The family is descended from the watchman Johannes Pfeffel (1580–1634) of Neuburg an der Donau. His son Conrad Pfeffel (born 1605) was a tailor in Augsburg, and was the father of a pastor in Freiburg im Breisgau Johann Conrad Pfeffel (1636–1701). The latter's son Johann Conrad Pfeffel (1682–1738) moved from Freiburg to nearby Colmar in Alsace, where he served as Buergermeister (mayor) for half a year in 1727. He was the father of the historian, lawyer and diplomat Christian Friedrich Pfeffel von Kriegelstein (1726–1807) and the writer Gottlieb Konrad Pfeffel (1736–1809).
Among the children of Karl Maximilian was Freiherr Christian Hubert Theodor Marie Karl von Pfeffel (born 1843 in Munich, died 1922 in Paris), who married Hélène Arnous de Rivière (1862-1951) and had his name changed to de Pfeffel when he settled in France. They were the parents of Marie-Louise de Pfeffel (1882–1944), the great-grandmother of former British Prime Minister Boris Johnson, and of Yvonne de Pfeffel (born 1883 in Paris, died 1959 in Truro, Cornwall), the first female French Open doubles champion.
A mummified corpse of a woman buried in the Barfüsser Church in Basel, Switzerland, was identified in 2018, 43 years after the discovery, as Anna Catharina Bischoff (1719–1787), the mother-in-law of Christian Friedrich Pfeffel von Kriegelstein (1726–1807).