He attended Yale College and Yale Medical School. His first marriage to Susan A. Metcalfe September 28, 1838. He married his second wife, Maria Chapin, in 1842. It was after her death that he married Elizabeth Rutherford, granddaughter of the author Eliza Fenwick, in 1844. He was the father of five children, Elizabeth Fenwick Savage (b. 1846), Alexander Duncan Savage (b. 1848), Thomas Rutherford Savage (b. 1852), William Rutherford Savage (b. 1854), Jesse Duncan Savage (b. 1858). He was the grandfather of the American artist Thomas Casilear Cole (1888-1976).
In 1836 Savage was sent as a missionary to Liberia. During his time in Africa he acquired the skull and other bones from an unknown ape species, which he described in 1847 at the Boston Society of Natural History with American naturalist and anatomist Jeffries Wyman with the scientific name Troglodytes gorilla, now known as the western gorilla.