Byzantine eparchial lists from the time of the 9th-10th century mention a bishop's centre by the name of Lyutitsa on the site of present-day Ivaylovgrad. According to John VI Kantakouzenos (1347–1354), his infantry reached the fortress in 1342–1343. The settlement was destroyed by the Ottoman Turks during their invasion of the Balkans in the 14th-15th century.
Information about the town during the Ottoman rule of Bulgaria is scarce, but it was the centre of a kaza under the name of Ortaköy in Edirne Province. Much of the Bulgarian population moved from the region due to organized robberies and internecine wars in the empire.
Landmarks in Ivaylovgrad and the surrounding area include the Ivaylovgrad Reservoir, the 2nd-3rd century RomanVilla Armira, the medieval Byzantine and Bulgarian fortress Lyutitsa, the 16th-century bridge Aterenski, as well as a number of ancient Thracian sites, and 19th-century churches.
