Joseph Madison was from Morgan County, Georgia. He started a business in Madison, Georgia, forming the firm, High & Berney. He later moved to Atlanta for greater opportunity, where he partnered with E. D. Herring to open up a dry goods store at 40 Whitehall Street (now Peachtree St. SE).[3] A new store opened at 50 Whitehall in 1882.[4] In 1884 High bought out Herring and renamed the firm J. M. High & Co.[3]
In 1887 High's department store opened a new location on the west side of Whitehall. By 1889, High was able to buy the southwest corner of Whitehall and Hunter (now King Blvd.) and erect a four story brick building with an ornate interior for his department store.[3]
In 1918 the store moved across the street to the former site of the Chamberlin-Johnson-DuBose department store, with 100,000 square feet (9,300 m2) of floor space. The new store was named the "daylight department store" as the large windows "did away with the need for artificial illumination".[3]
The old store on the west side of Whitehall was later occupied by the McClure Ten Cent Company until the 1930s, then the F & W Grand Company, until it was finally demolished to make way for the new building of the H. L. Green Company.[5]
From 1926-1927, High's was home to a broadcasting studio of radio station WDBE.[6]