The song is performed by Astaire on the lonely foggy deck of a ferry from New Jersey to Manhattan. It is sung to Ginger Rogers, who remains silent listening throughout. No dance sequence follows, which was unusual for the Astaire-Rogers numbers. Astaire and Rogers did dance to it later in their last movie The Barkleys of Broadway (1949) in which they played a married couple with marital issues. The song, in the context of Shall We Dance, notes some of the things that Peter (Astaire) will miss about Linda (Rogers). The lyrics include "the way you wear your hat, the way you sip your tea", and "the way you hold your knife, the way we danced till three". Each verse is followed by the line "no, no, they can't take that away from me". The basic meaning of the song is that even if the lovers part, though physically separated the nostalgic memories[5] cannot be forced from them. Thus, it is a song of mixed joy and sadness.
Our romance won't end on a sorrowful note, though by tomorrow you're gone. The song is ended, but as the songwriter wrote, 'the melody lingers on'. They may take you from me, I'll miss your fond caress, but though they take you from me I'll still possess....
George Gershwin died two months after the film's release, and he was posthumously nominated for the Academy Award for Best Original Song at the 1937 Oscars but lost out to "Sweet Leilani" which had been made tremendously popular by Bing Crosby.
^Horton, Matthew (2015). "Fred Astaire - "They Can't Take That Away From Me". In Dimery, Robert (ed.). 1001 Songs You Must Hear Before You Die. New York: Universe. p. 54.
