Armstrong married Nancy Irwin on July 1, 1823.[1] They had three sons, James Trooper, David Irwin and Francis Armstrong, and four daughters, Mary Elizabeth, Margaret, Susan Wells, and Nancy Irwin.[1] He died on June 12, 1847, at his plantation in the Choctaw Nation and was buried in Fort Coffee, Oklahoma.[1]