In 1708 Antoine Lispenard bought from Jacob Leisler's son a half interest in the large peninsula, or "neck", jutting out from the mainland between New Rochelle Creek and Long Island Sound. Six years later he bought the other half. Across the inlet he built a dam and a tidal gristmill. Each incoming tide filled the millpond behind the dam, and then, as the tide ebbed, the water was released through a millrace to turn the mill wheel. Nearby the mill, on the neck itself, Lispenard built his home, a stone house of one-and-a-half stories, with the front eaves extending to form the roof of a wide porch.
In 1732 he sold his property to Joseph Rodman who later doubled the size of the house. By 1776 the house and Neck had passed to John R. Myers who owned it for the duration of the American Revolutionary War. During this brief period the house was used by the British as a hospital for their wounded soldiers.
In 1784 the property came into the possession of the Davenport family, and so is currently known as Davenport Neck. Generations of the Davenport family, and other owners after them made further changes to the house, so that the original structure built by Antoine Lispenard and Joseph Rodman is largely invisible. In the 1860s, the original roof was replaced by a modified mansard, topped by a cupola. Restoration by the late Louis Ferguson managed to reveal and preserve several elements of the original structure such as the hand-hewn beams of its frame and the lime mortar made from local oyster shells, used to cement its stone wall. Other features of the house include 20-inch-thick (510 mm) walls, 16-inch-wide (410 mm) pine floorboards and a 5 1⁄2-foot-wide (1.7 m) fireplace in the kitchen. Several Lispenard family grave-sites are also located on the property.
During the War of 1812, guards of the local militia were stationed in the neck to prevent a landing by the British fleet blockading the Sound. An incident erupted as the result of a relatively innocent encounter between an English landing party in search of fresh food and American sentries. Additional men were moved in and quartered in barns on the neck. The ensuing events, renowned locally as the Battle of Davenport's Neck, involved a false alarm that caused the entire militia to flee the area mistakenly believing they were under attack. In 1814, Newberry Davenport divided the neck between his two sons, Newberry Davenport, Sr. and Lawrence Davenport. Lawrence erected a house on his section in 1816, later known as Davenport Grange and the old Lispenard–Rodman House became the house of Newberry, Jr. This began a process whereby both branches of the family sold large plots of acreage to wealthy persons who established handsome homes and estates on the neck. About this time, Newberry Davenport, Jr. would have added the kitchen wing to the old house, continuing the stone fabric and gable roof of the main house. The grade level of the land around the house may have begun to be heightened at this time as access to the original basement kitchen was closed-off. There is no other evidence of the house changing significantly at this time. Newberry Dayenport, Jr. died in 1863, leaving his house and remaining estate to his two unmarried children, Lawrence Montgomery Davenport and Anna Davenport. Based on the financial successes of their predecessors, the later Davenports enjoyed a comfortable leisurely lifestyle, traveling to Europe, providing community services and participating in the sophisticated social circle that had enveloped around the prominent estates built in the neck. Lawrence M. Davenport had Alexander Jackson Davis design the neighboring San Souci. By 1865, he had sold it to Mrs. Anthony Walton White Evans, who hired Davis to design additions in 1871. The Davenports continued to sell property to the growing number of people searching for country seats on Long Island Sound.
Sometime after Newberry Davenport's demise the house was dramatically altered to conform to more of an estate taste with the addition of a gambrel-roofed second floor, porches and interior changes. The roof top ventilator was a functional part introducing the then modern idea of healthful living into the venerable structure. The principal (sound-side) rooms on the interior were embellished with new trim in a restrained manner. Windows were extended to floor level on the piazza and most sash were replaced with casements with an unusually configured muntin pattern. The rear and kitchen rooms remained essentially unaltered in the transformation. The property had been reduced in size to only a few acres as a result of the sale of waterfront lots immediately before the house to Adrian Iselin, a New York financier, and Clarksen N. Potter, former Congressman. The land level may have been further built up at this time and driveways rerouted.