Coltrane was born Anthony Robert McMillan in South Lanarkshire, the son of Jean McMillan Ross (née Howie), a teacher and pianist, and Ian Baxter McMillan, a general practitioner who also served as a forensic police surgeon.

He has an older sister, Annie, and a late younger sister, Jane. Coltrane is the great-grandson of Scottish businessman Thomas W. Howie. He was educated at the prestigious Glenalmond College in Perthshire, from which he was nearly expelled after hanging the prefects' gowns from the school clocktower.

Coltrane moved into acting in his early twenties, taking the stage name Coltrane (in tribute to jazz saxophonist John Coltrane) and working in theatre and stand-up comedy. Appearing in the 1981 BBC television comedy series A Kick Up the Eighties, his comic skills also brought him roles in the The Comic Strip Presents (1982) series and the comedy sketch shows Alfresco (1983-1984) and Laugh??? I Nearly Paid My Licence Fee (1984).

Coltrane soon moved into films, obtaining roles in a number of movies such as Flash Gordon (1980), Death Watch (1980), Scrubbers (1983), Krull (1983), Absolute Beginners (1986), Mona Lisa (1986) and appeared as "Annabelle" in The Fruit Machine (1988).

On television, he also appeared in The Young Ones, Tutti Frutti (1987), as Samuel Johnson in Blackadder (1987) (a role he later reprised in the more serious Boswell and Johnson's Tour of the Western Islands (1993)), and in a number of stand-up and sketch comedy shows.

He played the part of Falstaff in Kenneth Branagh's Henry V (1989) He co-starred with Eric Idle in Nuns on the Run (1990), and played the Pope in The Pope Must Die (1991). He also played a would-be private detective obsessed with Humphrey Bogart in the TV play The Bogie Man.

His roles went from strength to strength in the 1990s with the TV series Cracker (1993-1996, 2006 one-off special) and a BAFTA award as the stepping stone to parts in bigger films such as the James Bond films GoldenEye (1995) and The World Is Not Enough (1999) and a major supporting role in From Hell (2001) as well as half-giant Rubeus Hagrid in the Harry Potter films (2001-2011).

J.K. Rowling, author of the Harry Potter series, has stated that she created the character of Hagrid based on Coltrane's likeness.