It is not known exactly when the construction of the castle started but it is believed that it took place around the late 1290s, by King Haakon V, replacing Tønsberg as one of the two most important Norwegian castles of the period (the other being Båhus). It was constructed in response to the Norwegian nobleman, Earl Alv Erlingsson of Sarpsborg’s earlier attack on Oslo that occurred in 1287. In the aftermath of the attack, it became clear that the city’s existing defences weren’t effective and therefore, a stronger defensive centre was needed.
The castle is mentioned in written sources for the first time in 1300 in a letter from King Haakon to a church in Oslo. However, the letter does not mention how far the construction of the castle had progressed by then.
The fortress was first used in battle in 1308, when it was besieged by the Swedish duke Eric of Södermanland, whose brother won the Swedish throne in 1309. The siege was eventually broken by a local Norwegian army in a battle. (This battle forms a major part of the plot of Sigrid Undset's historical novel In the Wilderness, the third volume of her tetralogy The Master of Hestviken.)
In 1449-1450 the castle was besieged again, this time by the Swedish king Karl Knutsson Bonde, but he had to lift the siege after a while. The castle was not besieged again until 1502 when Scottish soldiers in the service of the Danish king besieged the castle in order to regain it from the hands of the Norwegian nobleman Knut Alysson.