The 207 acres of land belonged to the Ewing family since the 1800s.[2] In the 1910s, Brentwood Hall was built for Rogers Caldwell, a prominent businessman across the Southern United States.[2] When he was convicted of fraud, his estate was seized by the state of Tennessee and turned into this center.[3] It was named after Buford Ellington, who served as Governor of Tennessee from 1959 to 1963 and again from 1967 to 1971.[2]