Having inherited from his father both the title and the family's manor house, Clevedon Court in Somerset, he spent the rest of his life improving the town of Clevedon, setting up a lending library and allotments, and building and funding the local cottage hospital (which is still in existence). The west wing of the house was largely destroyed by a fire in 1882 and rebuilt. The house is now owned by the National Trust. He also built Hallum Hall on Dial Hill which was an orphanage. It was later renamed St Edith's children's home and remained in use as a children's home until 1974. The gothic Victorian building is listed and remains largely intact on the outside, but is now converted to luxury flats.