According to the official statistics, one gymnast, Adolf Spinnler, and one wrestler, Gustav Thiefenthaler, from Switzerland competed at the 1904 Summer Olympics in St. Louis, United States.
But there were more athletes with Swiss roots at the Olympics: Andreas Kempf competed in three gymnastics events, finishing 8th in the combined three events finals. He arrived in the United States in 1902 and represented the Kansas City Turnverein. Kempf applied for naturalization as a US citizen in 1908, but was denied citizenship. Emil Schwegler competed for the United States in three gymnastics events, representing the St. Louis Schweizer Turnverein. Born in Switzerland, he was naturalized as a US citizen with his parents and then was a college student in Kansas City, Missouri. And Oscar Schwab was born in Paris to a Swiss mother and was adopted by her American husband. He competed for the US in the quarter mile cycling race. After the games he raced mostly in Europe and was Swiss sprint champion in 1907.
In fact, eighteen-year old Gustav Tiefenthaler was also born in Switzerland, moved to the United States with his family when he was a child and was naturalized as a US citizen with his parents. So he was "less Swiss" than Andreas Kempf. Tiefenthaler represented the South Broadway Athletic Club of St. Louis. At the Olympics, Tiefenthaler had a single bout in the men's freestyle light flyweight event and lose, but still earned a bronze medal.
Adolf Spinnler on the other hand was clearly European, but he was born in Switzerland, later moved to Esslingen am Neckar in the German Empire and represented a German sports club - so his affiliation to a national team is also disputed.