According to the Domesday Book of 1086, immediately before the Norman Conquest of 1066 the manor of SIDREHA~ was held by the Saxon magnate Brictric, a great landholder in Devon and more widely in England. Following the Norman Conquest, it was one of the 107 Devonshire landholdings of Juhel de Totnes (died 1123/30), the first Anglo-Norman feudal baron of Totnes and feudal baron of Barnstaple, both in Devon. Juhel's tenant was a certain William. On his banishment by King William Rufus, Juhel's property was confiscated, and Sydenham later became the seat of the de Sydenham family, which as was usual took its name from its seat, whose overlords were the subsequent feudal barons of Totnes. In the 13th century Book of Fees Maurice de Sideham is listed as holding lands in Parva Sideham ("Little Sydenham") (which adjective was used to distinguish it from Sydenham Damerel, also in Lifton Hundred) from Reginald de Vautortes, feudal baron of Totnes, and is recorded by Pole (d.1635) as holding North Sidenham, this manor, in 1242. The de Sydenham family was succeeded by that of de Mauris, from which it passed by marriage to Trevage, and thence to the Wise family, which later was also seated at Mount Wise, Plymouth.
Part of the house dates from the fourteenth century, and is said to have originally formed a quadrangle or "H", but in the reign of Elizabeth it was built into the shape of an "E", and is a very perfect example of Tudor domestic architecture. Sir Thomas Wise (c.1576–1630), Knight of the Bath, of Sydenham, was Sheriff of Devon in 1612 and in 1621 served as a member of parliament for Bere Alston in Devon. He much beautified the house, and added such height and such a great amount of granite to it that his contemporary Risdon (d.1640) Risdon remarked: "as the very foundation is ready to reel under the burthen". A story is repeated in many histories of Devon, including Lysons (1822), that during the Civil War this Sydenham was captured in 1644 by Colonel Holborne. However it appears that the house which was in fact garrisoned and taken was Combe Sydenham, in the parish of Stogumber in Somerset. However, within the last forty years a sword and other weapons, also seventeenth century horseshoes, have been found may be taken as a proof that fighting of some sort did take place at the Wise seat. It is quite likely that King Charles II, when Prince of Wales, did come here, as he is known to have spent many weeks in the vicinity. His son was Thomas Wise (c.1605–1641), of Sydenham, Sheriff of Devon in 1638 and in 1625 a member of parliament for Callington in Cornwall and for Bere Alston in the parliaments of King Charles of 1625, 1626 and 1628 to 1629, and for Devon twice in 1640. The son of the latter was Edward Wise (1632–1675) of Sydenham, thrice MP for Okehampton, who was predeceased by both his childless sons, and whose sole heiress became his only daughter Arabella Wise, who married Edmund Tremayne (1649–1698) of Collacombe in the parish of Lamerton on Devon, to which family passed Sydenham.
Sydenham House has four separate entrances, each opening on to a court or garden. Access to the front-entrance, commonly called the Green Court, is through an iron gateway, and above the central door are sculpted the Wise arms. Most of the windows have eight rounded granite mullions and small leaded panes of glass, and in some the original glass survives. Two windows in the front date from the reign of Charles I (1625–1649) and have unusual fan-shaped lights. Over the large granite open fireplace in the great hall is engraved the date "1656", when the house underwent repair after damage, caused, it is said, in the Civil War. In making alterations to the kitchen chimney in about 1888, a little hiding-place, or priest's room, was found opening out of it, and in it was an oak table and the remains of a chair; and since then large and small unsuspected rooms have been discovered, and it has been said that in the largest a troop of cavalry could lie hidden. A secret passage leading from the house towards the river was found, bearing out the legend "that the Lady Wise of the day escaped with a large party by a secret passage near the river, and got into the woods undetected by the soldiers who were round the house." There is oak panelling in most of the rooms, and in the dining-room, the panelling is inlaid with an ivory-like substance. Secret passages exist to this day in the walls, which are of immense thickness, in some places being seven feet thick. There are three oak staircases, the main one being carved with figures standing at the angles, and another having newels. In the so-called "King's Room" there is an ancient bed, with old red silk curtains and the Prince of Wales's feathers over it, in which Kings Charles I and Charles II are reported to have slept.
Sydenham stands in a valley on the banks of the River Lyd. It is overshadowed by woods on a hill-side that is steep and fairly high. At the foot, there is a rushing stream which is crossed by a bridge exactly opposite the front of the house. A wide lawn slopes away from the house, and a very small straight rivulet runs through it just a foot or two from the path. At the foot of the slope, there is a tiny lake, which, though very narrow, divides the lawn from end to end, and beyond the water the ground rises gradually.