Among the principal holdings of the museum are the six The Lady and the Unicorn (La Dame à la licorne) tapestries.
The structure is perhaps the most outstanding example still extant of civic architecture in medieval Paris. It was formerly the town house (hôtel) of the abbots of Cluny, started in 1334. The structure was rebuilt by Jacques d'Amboise, abbot in commendam of Cluny 1485–1510; it combines Gothic and Renaissance elements. In 1843, it was made into a public museum, to hold relics of France's Gothic past preserved in the building by Alexandre du Sommerard.
Though it no longer possesses anything originally connected with the abbey of Cluny, the hôtel was at first part of a larger Cluniac complex that also included a building (no longer standing) for a religious college in the Place de la Sorbonne, just south of the present day Hôtel de Cluny along Boulevard Saint-Michel. Although originally intended for the use of the Cluny abbots, the residence was taken over by Jacques d'Amboise, Bishop of Clermont and Abbot of Jumièges, and rebuilt to its present form in the period of 1485-1500. Occupants of the house over the years have included Mary Tudor, the sister of Henry VIII of England. She resided here in 1515 after the death of her husband Louis XII, whose successor, Francis I, kept her under surveillance, particularly to see if she was pregnant. Seventeenth-century occupants included several papal nuncios, including Mazarin.
In the 18th century, the tower of the Hôtel de Cluny was used as an observatory by the astronomer Charles Messier who, in 1771, published his observations in the landmark Messier catalog. In 1789, the hôtel was confiscated by the state, and for the next three decades served several functions. At one point, it was owned by a physician who used the magnificent Flamboyant chapel on the first floor as a dissection room.
In 1833, Alexandre du Sommerard bought the Hôtel de Cluny and installed his large collection of medieval and Renaissance objects. Upon his death in 1842, the collection was purchased by the state; the building was opened as a museum in 1843, with Sommerard's son serving as its first curator. The present-day gardens, opened in 1971, include a "forêt de la licorne" inspired by the tapestries.