A castle has existed at the site at least since 1219. From then and for more than 400 years, it belonged to the Teutonic Order. As such, it was the residence of the territorial commander of the Bailiwick of Franconia, the wealthiest of the 13 provinces of the Teutonic Order. A number of both medieval and Renaissance buildings have occupied the site. In 1718-1721, the residence was transformed into the presently visible, Baroque palace by Karl Heinrich von Hornstein to designs by architect Franz Keller. In 1775, additional changes in Neoclassical style were made, to designs by architect Pierre Michel d’Ixnard.
In 1789, the seat of the residence of the Bailiwick of Franconia was moved to Bad Mergentheim, and a few years later the owner of the palace passed to the Kingdom of Bavaria. It was given as a gift in 1815 by the Bavarian king Maximilian I Joseph of Bavaria to field marshal Karl Philipp von Wrede. The field marshal made a comprehensive re-furnishing of the palace.
The palace was bought by the State of Bavaria in 1939. It is today administered by the Bavarian Administration of State-Owned Palaces, Gardens and Lakes and open to the public through guided tours. The administration also rents out parts of the residence for larger events. The western wing houses Kulturzentrum Ostpreußen, a cultural centre dedicated to the history and culture of East Prussia.
The residence is built in a mixture of Baroque and Neoclassical. It consists of, notably, a church, princely suites, and an adjacent park. The church is Baroque, as are several of the interior details, such as the stucco work by Franz Joseph Roth, frescos, wall panelling and original floors. The building also contains some of the finest Neoclassical interiors in Bavaria, owing both to the work of d’Ixnard and the re-furnishing carried out during the ownership of Karl Philipp von Wrede. A park lies adjacent to the residence. Originally a formal, Baroque garden (depicted for the first time in 1726), it was transformed into a landscape garden during the 19th century. A few remaining Baroque architectural elements were destroyed during World War II, and today only a single balustrade remains from the Baroque garden.