After this time Dorstone Castle changed hands several times. Lady Fitzwalter died about 1422, and afterwards the castle belonged to Richard de la Mare of the de la Mare family and was then owned by the Lysters who sold it to Morgan Aubrey. It was then purchased by the Cornewall family of nearby Moccas Court in 1780.