It was built on the site of an Augustinian priory that was a daughter house of Notley Abbey in Long Crendon. Jasper Tudor, Duke of Bedford bought and largely remodelled the priory in the 16th century.
In late 1771 the manor was bequeathed to Sir Francis Bernard, 1st Baronet on the death of his cousin. He had served as governor of the provinces of New Jersey and Massachusetts Bay, where his uncompromising policies were instrumental in the events leading to the American Revolution. The house is still owned by the family.
The house and gardens are open to the public and the house is noted for its interior and is a grade I listed building.