She was born in Boston, Massachusetts, the first daughter of Samuel Davenport Torrey (1789-1877) and his second wife, the former Susan Holman Waters (1803-1866). Her three sisters were Delia Chapin Torrey, Anna Davenport Torrey (who married geologist Edward Orton, Sr.), and Susan H. Torrey. She graduated from Mount Holyoke College (then Mount Holyoke Female Seminary) in 1845.
She married Alphonso Taft, widowed in 1852, on 26 December 1853 in Millbury, Massachusetts, becoming stepmother to his two living sons by his first wife, Fanny Phelps, Charles Phelps Taft, who became the publisher of the Cincinnati Times-Star and was a member of the U.S. House of Representatives from 1895 to 1897, and Peter Rawson "Rossy" Taft.
They had five children, four of whom lived to adulthood. The first, who died aged 14 months of pertussis, was Samuel Davenport Torrey Taft. The second was President William Howard Taft; next was Henry Waters Taft, who became a lawyer in New York City; fourth was Horace Dutton Taft, founder of the Taft School in Watertown, Connecticut, and the last was Frances Louis "Fanny" Taft, who married surgeon William A. Edwards.
The family lived in Cincinnati during her husband's tenure as judge of the Superior Court of Cincinnati, in Washington, D. C. when he was, successively, Secretary of War and Attorney General and in Austria-Hungary and Russia when he served as U.S. ambassador there.