The manor of Fernham was in existence by the first half of the 13th century, when Juliana de Elsefeld quitclaimed six virgates of land at Fernham to William de Valence, 1st Earl of Pembroke. The Earl supported Henry III but the rebel Simon de Montfort, 6th Earl of Leicester defeated the King at the Battle of Lewes in 1264, and thereafter the manors of Shrivenham and Fernham was granted to his wife Joan de Valence, Countess of Pembroke for her maintenance. Shrivenham and Fernham descended with the same heirs until Richard Talbot, 2nd Baron Talbot died seised of the reversion of Fernham in 1356.
Fernham was part of the Church of England parish of Shrivenham until 1846, when it and neighbouring Longcot were formed into a separate ecclesiastical parish. The Church of England parish church of Saint John the Evangelist was designed in 13th-century style by the Gothic Revival architect J.W. Hugall and built in 1861 as a chapel of ease for Longcot. St. John's parish is now part of a single Church of England Benefice with the parishes of Ashbury, Bourton, Compton Beauchamp, Longcot, Shrivenham and Watchfield. In 2008 the parish controversially spent a £90,000 grant from the Big Lottery Fund to strip St. John's of its Victorian pews, lay a modern floor and reorder its interior for secular uses as a village hall.
Fernham had a Congregational chapel.