He was the second son of Edward Devereux, 11th Viscount Hereford and his wife Catherine Mytton. His maternal grandparents were Richard Mytton of Pontyscowryd and Garth, High Sheriff of Montgomeryshire and Dorothy Wynn.
On 15 December 1768, George married his 3rd cousin, Marianna Devereux. His namesake father-in-law was George Devereux of Tregoyd. They had five children:
His older brother Edward Devereux, 12th Viscount Hereford had succeeded their father but died childless on 1 August 1783. George succeeded him and further Viscount Hereford descent from him.