Cavalaire-sur-Mer is probably derived from an ancient Phoenician colony of the name of Heraclea Caccabaria. There are also remains of a Gallo-Roman occupation in Pardigon.
It is located on the route of the old railway Saint Raphael–Toulon (sometimes called Train Pignes), now defunct. The old railway line can still be seen throughout the town, with the old station situated near the harbour and now hosting a museum.
During World War II, on August 16, 1944, it was one of the sites of a beach landing in Operation Dragoon, the Allied invasion of southern France. Every year, August 15 sees a parade of military vehicles and the reconstruction of a military camp.