Oliver was born in Cambridge, where his father, David Thomas Oliver, was a professor of law and Fellow of Trinity Hall, Cambridge. He was educated at The Leys School, Cambridge and Trinity Hall, Cambridge, graduating with a starred First in law in 1941. He later became an honorary fellow of Trinity Hall, and became University Commissary. From 1941 to 1945, he served in the 12th Battalion Royal Tank Regiment, and was mentioned in dispatches during his service in Italy. He was Honorary President of the Cambridge University Law Society.
Oliver was called to the Bar at Lincoln's Inn in 1948, and became a chancery barrister. He became a Queen's Counsel in 1965, a Bencher at Lincoln's Inn in 1973. He was knighted one year later, when he was appointed a Judge of the High Court in the Chancery Division. Between 1976 and 1980, he was a member of the Restrictive Practices Court. He was chairman of the Review Body on the Chancery Division from 1979 to 1981, recommending many changes to modernise its practice.
He became a Privy Counsellor in 1980, when he was promoted to become a Lord Justice of Appeal. He joined with Lord Denning in ruling that the Greater London Council's "Fares Fair" policy was illegal (a decision which was later upheld by the House of Lords). He was a leading contender to succeed Lord Denning as Master of the Rolls on his retirement in 1982, but the post went to Sir John Donaldson instead. In 1986, Oliver was appointed a Lord of Appeal in Ordinary, and was created a life peer with the title Baron Oliver of Aylmerton, of Aylmerton in the County of Norfolk. He joined the minority in the House of Lords judgment in the Spycatcher case in 1987, in favour of lifting the ban on its publication.
He retired as Lord of Appeal in 1992. In later life, he lost his sight due to macular degeneration.
Lord Oliver married twice, firstly Mary Chichester Rideal in 1945. They had a son and a daughter. After her death in 1985, he married secondly Wendy Anne Lloyd Jones (née Harrison) in 1987. He was survived by his second wife, and the two children from his first marriage.