He was the son of John Crichton-Stuart, 3rd Marquess of Bute and Gwendolen Fitzalan-Howard, a daughter of Edward Fitzalan-Howard, 1st Baron Howard of Glossop and granddaughter of Henry Howard, 13th Duke of Norfolk.
He was educated at Harrow School, and succeeded his father as Marquess of Bute in October 1900, nineteen years old. In early 1902 he was on a tour in the Far East. On reaching his majority in June 1902, he received the Honorary Freedom of the Burgh of Rothesay, and later the same month took the oath and his seat in the House of Lords.
The 4th Marquess, like his father, had a passion for architecture and was responsible for restoring Caerphilly Castle in South Wales. In 1936 he published a pamphlet entitled "A Plea for Scotland's Architectural Heritage", which argued for the preservation of Scotland's smaller burgh dwellings and advocated reconditioning traditional working class housing, rather than wholesale demolition. He became "the man who sold a city" when, in 1938, he disposed of the remaining Bute family estate in Cardiff.
On 6 July 1905, he married Augusta Bellingham, daughter of Sir (Alan) Henry Bellingham, 4th Baronet and Catherine Noel. The lavish wedding, at Castle Bellingham in Ireland, was followed by a party at Mount Stuart House in Scotland. A film company was employed to film the event, one of the earliest examples of the aristocratic classes making a private film.
They had seven children: