Greville was the second son of Francis Greville, 1st Earl of Warwick and his wife, Elizabeth Hamilton, daughter of Lord Archibald Hamilton. George Greville, 2nd Earl of Warwick and Robert Fulke Greville were his brothers and he had four sisters. He was brought up in the family home, Warwick Castle.
His father had been created Earl Brooke three years before he was born and in 1759 had successfully petitioned to have the prestigious medieval title of a more senior extinct line of his family, Earl of Warwick, conferred on him as the senior male heir of the family and lieutenant of the county.
He was educated at the University of Edinburgh from 1764 to 1767.
Greville lived most of his adult life on a rigid income of ₤500 a year, generated from landowning and investments, with which managed to acquire antiquities from Gavin Hamilton in Rome. He also purchased through his uncle a genre piece by Annibale Carracci. Greville was the nephew of Sir William Hamilton, the British envoy at Naples who formed two collections of Greek vases, one of which is at the British Museum.
As a Fellow of the Royal Society, his special interest was in minerals and precious stones, which were catalogued by the émigré Jacques Louis, Comte de Bournon and were later purchased via Act of Parliament for the British Museum. He was good friends with James Smithson, whom he sponsored for membership in the Royal Society and with whom he exchanged minerals.