Graves was born in North Somerset, to Mary Lousilla (née Roberts), a travel co-ordinator, and Richard Harding Graves, a music teacher and musician. He first entered the entertainment business by leaving Wyvern Secondary School at the age of fifteen, to become a circus clown.

Graves came to prominence in costume drama adaptations of E. M. Forster novels A Room with a View (1985) and Maurice (1987), before going to appear in films such as A Handful of Dust (1988), Different for Girls (1996) and Intimate Relations (1996).

His role in Intimate Relations won him the Best Actor award at the 1996 Montreal Film Festival. He was also acclaimed for his portrayal of Young Jolyon in the 2002 ITV production of The Forsyte Saga.

In addition to his screen work, Graves has also won acclaim for his stage acting, including Broadway roles in Closer (2000) and The Elephant Man (2002).

One of Graves' own personal favourites is that of Presley Stray in Philip Ridley's The Pitchfork Disney at the Bush Theatre in London in 1991, for which he won the Bass Charrington Best Actor Award.