Major Levin Handy purchased 357 acres (1.44 km2) of land outside of Salisbury in 1795 and began construction of this Federal-style building later that year. Due to lack of funds stemming from severe medical problems, Major Handy discontinued construction of the mansion and put it up for sale in 1803. In 1805, Dr. John D. Huston purchased the incomplete house and continued its construction. Sarah Huston, Dr. Huston's widow inherited the estate, which included eighteen enslaved people and $110. She sold some of the property for development in the late 1840s to early 1850s. In 1881, George Waller purchased the estate and his family lived there until 1945. In 1945, Fred A. Adkins purchased the property and renovated the house, modernizing it. In 1948, Mr. & Mrs. Ward A Garber purchased the estate. In 1970, Wicomico County purchased the estate and the mansion was placed in public trust in 1974 under the ownership of the City of Salisbury. In 1971, Poplar Hill Mansion was placed on the National Register of Historic Places.
According to legend, a slave girl named Sara perished in the house after her dress caught on fire in the second floor rear bedroom during the Huston period. Since that time, there have been reports of a "consoling" spirit.