Normanby was born in London, the son of Constantine Phipps, 1st Marquess of Normanby, by his wife the Hon. Maria, daughter of Thomas Liddell, 1st Baron Ravensworth. He gained the courtesy title Viscount Normanby when his father succeeded as Earl of Mulgrave in 1831. When his father was made Marquess of Normanby in 1838, he became known by the courtesy title Earl of Mulgrave. Normanby entered the Coldstream Guards as an ensign, and became a lieutenant in 1838.
Normanby was returned to parliament for Scarborough in 1847, a seat he held until 1851 and again between 1852 and 1857. He was appointed Comptroller of the Household by Lord John Russell in 1851. When Lord Aberdeen became prime minister in early 1852, he became Treasurer of the Household, a post he held until 1858, the last three years under the premiership of Lord Palmerston. In the latter year he was appointed Governor of Nova Scotia, which he remained until 1863. In 1863 he also succeeded his father in the marquessate and took his seat in the House of Lords.
Normanby returned to the government in 1868 when he was appointed a Lord-in-Waiting by William Ewart Gladstone. The following year he was promoted to Captain of the Honourable Corps of Gentlemen-at-Arms. In 1871 he became Governor of Queensland. He continued in this post until 1874, and was then Governor of New Zealand from 1874 to 1879 and Governor of Victoria from 1879 to 1881.
Lord Normanby married Laura, daughter of Captain Robert Russell, R.N. in 1844. When he served as Lieutenant-Governor of Nova Scotia, he and his wife had the Prince of Wales (later King Edward VII) and his brother, Prince Alfred (Duke of Edinburgh) as their guests at Government House. The couple had several children. The Marchioness of Normanby died in London in January 1885, aged 69. Lord Normanby died at Brighton, Sussex, in April 1890, aged 70, and was succeeded by their eldest son, Constantine. One of their daughters, Lady Katherine Louisa Phipps, married the 3rd Earl of Ellesmere.