The Tom Thumb House is located south of the village center of Norfolk, on the south side of Windrow Road. It is set on a hillside above the road, from which it and an adjacent later house are screened by trees. It is a single-story structure, consisting of two roughly square sections joined at a corner. The walls of each section are made of large boulders, and are up to 2.5 feet (0.76 m) thick. The pyramidal roofs of each section, framed with hand-hewn timbers recycled from old barns, are covered in thick slate. One of the squares has the main living space, while the other houses a sleeping and kitchen area. Exposed framing and rough paneling, also recycled, are the principal exposed woodwork.[2]
The building was built in 1929 to a design by New York City architect Alfredo S.G. Taylor. Taylor spent many summers in Norfolk, and is credited with more than thirty commissions in the community. A number of them share the medieval features used in this building: the rough rustic stone and wood finishes, and heavy slate roof. Details of the building were published in American Architect and Building News after its construction. Taylor's work for the Childs family also included the Starling Childs Camp on Doolittle Lake, and the Childs Sports Building; the latter in particular, located north of Windrow Road, has similar style and construction methods to this building, but on a larger scale.[2][3] This house was referred to by the family as the "Teahouse"; its "Tom Thumb" name was applied in the American Architect and Building News article.[2]