In 1736, he married Lady Mary Booth, the only daughter and heiress of George, 2nd Earl of Warrington. They had three children:
In 1738, he represented Leicestershire in the British House of Commons, but entered the House of Lords in 1739 upon inheriting the earldom. On 3 March 1744, he was appointed a Deputy Lieutenant of Lincolnshire, and on 8 March, of Staffordshire too.
He had inherited the Grey estates at Bradgate Park in Leicestershire and Enville in Staffordshire but decided to make Enville Hall the family seat. Bradgate House was therefore bricked up and the park there kept for hunting and game. The Enville grounds (750 acres) were significantly re-landscaped during the mid-18th century. Mary, Countess of Stamford also inherited large estates at Dunham Massey in Cheshire and Stalybridge in Lancashire on the death of her father in 1758.
He died at Enville Hall and was succeeded in the earldom by his son (George) Harry, 5th Earl of Stamford (later Earl of Stamford and Warrington).