In the early 19th century, the community was a hamlet known as "Tobacco Stick."[4] Around 1822, Harriet Tubman was born just outside the hamlet; in her youth, she lived in her father's cabin nearby, in what is now the Blackwater National Wildlife Refuge.[5]
According to the United States Census Bureau, the Madison CDP has a total area of 3.2 square miles (8.4 km2), of which 3.2 square miles (8.3 km2) is land and 0.04 square miles (0.1 km2), or 1.26%, is water.[3]