The history of this place goes back to the 12th century when there was a fortified keep. The keep was incorporated in a bigger castle with round towers and curtain walls in the 14th century. The moat that was dug around the castle gave it an extra protection. The castle prospered for many centuries and during the renaissance period the military castle was transformed in a more luxurious residential castle. Early in the 18th century the castle lost its importance and was no longer permanently inhabited this resulted in decay. At the end of the 18th century important parts of the castle collapsed. At the beginning of the 19th century the remains were turned into a farm. The town of Millen bought the remains of the castle in 1972. The castle was restored in a peculiar way in the 1990's and hosts now a visitor center.