Elliott was born in London, the son of Nina (née Mitchell) and Myles Laymen Farr Elliott. He attended Malvern College, trained at the Royal Academy of Dramatic Art in London and afterwards served as a radio operator and gunner in the Royal Air Force during World War II. In 1942, he was shot down over Denmark and spent the rest of the war in a P.O.W. camp in Silesia.

After the war, he made his film debut in Dear Mr. Prohack (1949). He went on to play a wide range of parts, often playing ineffectual and occasionally seedy characters, such as the journalist Bayliss in Defence of the Realm, the abortionist in Alfie, and the washed-up film director in The Apprenticeship of Duddy Kravitz.

He made many television appearances, notably in plays by Dennis Potter, including Follow The Yellow Brick Road (1972), Brimstone and Treacle (1976) and Blade on the Feather (1980). He took over for an ill Michael Aldridge for one season of The Man in Room 17 (1966) and also appeared in the series Thriller (1975).

In the 1980s he won three consecutive BAFTA awards as best supporting actor for Trading Places as Dan Aykroyd's kindly butler, A Private Function and Defence of the Realm, as well as an Academy Award nomination for A Room with a View. He also became familiar to a wider audience as the well meaning but ineffectual Dr. Marcus Brody in Raiders of the Lost Ark and Indiana Jones and the Last Crusade.

In 1988, Elliott was awarded the CBE for his services to acting. His career included many stage performances, including with the Royal Shakespeare Company.