Sir John Charlton was licensed to crenellate his dwelling at Charlton in 1316. This defended manor house was known as Charlton Castle, and was apparently still used as a residence of the Lords of Powys in the earlier part of the 16th century. In 1588 it was sold to Francis Newport but then fell into disrepair.