The present castle was constructed in the 1560s in the then Danish province of Scania by Danish riksråd Sten Rosensparre, though certain parts of the castle are presumed to be older.
The estate was originally called Skarholta or Skarolt. From at least the 14th century until 1624, it belonged to the Rosensparre family, who also called themselves Skarholt, and then to the Ruud family and the Trolle family.
In 1658, Scania became a Swedish province, and in 1661, it was bought by the Swedish count Pontus Fredrik De la Gardie. In reality, however, it was bought by the funds of his rich wife, Beata Elisabet von Königsmarck who, as a married woman and thereby a minor, could not formally be listed as its buyer and owner until after she was widowed in 1692. In practice, Beata Elisabet von Königsmarck managed the estate alone from 1661 until her death in 1723, when it was inherited by the grandson of her daughter, count Erik Brahe. His son, Magnus Fredrik Brahe, sold Skarholt to Charles XIV John of Sweden in 1826.
Skarholt was sold to baron Jules von Schwerin by Oscar I of Sweden and is still privately owned.
