Qvist competed in the 1928 Summer Olympics where he earned a silver medal and in the 1936 Summer Olympics. During World War II he served as a Major in the Waffen-SS and was sentenced to 8 years in prison for his collaborationist crimes after the war.
In 1928 he and his horse Hidalgo won the silver medal as member of the Norwegian eventing team in the team eventing competition after finishing eighth in the individual eventing.
After the German occupation of Norway in World War II, Qvist joined a voluntary unit of the German Waffen-SS, the Norwegian Legion. As an officer he eventually came to command the unit at the rank of Major.