As an amateur, Dittmer was a three-sport star at Elkader High School, where he played basketball, baseball, and football. In Dittmer's senior year of high school, he was selected the best high-school baseball player in the state of Iowa, and he went on to play in a National All-star Game. He continued to play all three sports at the University of Iowa, where he was mainly known as a wide receiver. Dittmer was inducted into the University of Iowa Athletics Hall of Fame in 1993.