The third of Hilbert's list of mathematical problems, presented in 1900, was the first to be solved. The problem is related to the following question: given any two polyhedra of equal volume, is it always possible to cut the first into finitely many polyhedral pieces which can be reassembled to yield the second? Based on earlier writings by Carl Friedrich Gauss,^[1] David Hilbert conjectured that this is not always possible. This was confirmed within the year by his student Max Dehn, who proved that the answer in general is "no" by producing a counterexample.^[2]

The answer for the analogous question about polygons in 2 dimensions is "yes" and had been known for a long time; this is the Wallace–Bolyai–Gerwien theorem.

Unknown to Hilbert and Dehn, Hilbert's third problem was also proposed independently by Władysław Kretkowski for a math contest of 1882 by the Academy of Arts and Sciences of Kraków, and was solved by Ludwik Antoni Birkenmajer with a different method than Dehn's. Birkenmajer did not publish the result, and the original manuscript containing his solution was rediscovered years later.^[3]

The formula for the volume of a pyramid,

${\displaystyle {\frac {{\text{base area}}\times {\text{height}}}{3}},}$

had been known to Euclid, but all proofs of it involve some form of limiting process or calculus, notably the method of exhaustion or, in more modern form, Cavalieri's principle. Similar formulas in plane geometry can be proven with more elementary means. Gauss regretted this defect in two of his letters to Christian Ludwig Gerling, who proved that two symmetric tetrahedra are equidecomposable.^[3]

Gauss's letters were the motivation for Hilbert: is it possible to prove the equality of volume using elementary "cut-and-glue" methods? Because if not, then an elementary proof of Euclid's result is also impossible.

Dehn's proof is an instance in which abstract algebra is used to prove an impossibility result in geometry. Other examples are doubling the cube and trisecting the angle.

Two polyhedra are called scissors-congruent if the first can be cut into finitely many polyhedral pieces that can be reassembled to yield the second. Any two scissors-congruent polyhedra have the same volume. Hilbert asks about the converse.

For every polyhedron ${\displaystyle P}$, Dehn defines a value, now known as the Dehn invariant ${\displaystyle \operatorname {D} (P)}$, with the property that, if ${\displaystyle P}$ is cut into polyhedral pieces ${\displaystyle P_{1},P_{2},\dots P_{n}}$, then $${\displaystyle \operatorname {D} (P)=\operatorname {D} (P_{1})+\operatorname {D} (P_{2})+\cdots +\operatorname {D} (P_{n}).}$$ In particular, if two polyhedra are scissors-congruent, then they have the same Dehn invariant. He then shows that every cube has Dehn invariant zero while every regular tetrahedron has non-zero Dehn invariant. Therefore, these two shapes cannot be scissors-congruent.

A polyhedron's invariant is defined based on the lengths of its edges and the angles between its faces. If a polyhedron is cut into two, some edges are cut into two, and the corresponding contributions to the Dehn invariants should therefore be additive in the edge lengths. Similarly, if a polyhedron is cut along an edge, the corresponding angle is cut into two. Cutting a polyhedron typically also introduces new edges and angles; their contributions must cancel out. The angles introduced when a cut passes through a face add to ${\displaystyle \pi }$, and the angles introduced around an edge interior to the polyhedron add to ${\displaystyle 2\pi }$. Therefore, the Dehn invariant is defined in such a way that integer multiples of angles of ${\displaystyle \pi }$ give a net contribution of zero.

All of the above requirements can be met by defining ${\displaystyle \operatorname {D} (P)}$ as an element of the tensor product of the real numbers ${\displaystyle \mathbb {R} }$ (representing lengths of edges) and the quotient space ${\displaystyle \mathbb {R} /(\mathbb {Q} \pi )}$ (representing angles, with all rational multiples of ${\displaystyle \pi }$ replaced by zero). For some purposes, this definition can be made using the tensor product of modules over ${\displaystyle \mathbb {Z} }$ (or equivalently of abelian groups), while other aspects of this topic make use of a vector space structure on the invariants, obtained by considering the two factors ${\displaystyle \mathbb {R} }$ and ${\displaystyle \mathbb {R} /(\mathbb {Q} \pi )}$ to be vector spaces over ${\displaystyle \mathbb {Q} }$ and taking the tensor product of vector spaces over ${\displaystyle \mathbb {Q} }$. This choice of structure in the definition does not make a difference in whether two Dehn invariants, defined in either way, are equal or unequal.

For any edge ${\displaystyle e}$ of a polyhedron ${\displaystyle P}$, let ${\displaystyle \ell (e)}$ be its length and let ${\displaystyle \theta (e)}$ denote the dihedral angle of the two faces of ${\displaystyle P}$ that meet at ${\displaystyle e}$, measured in radians and considered modulo rational multiples of ${\displaystyle \pi }$. The Dehn invariant is then defined as $${\displaystyle \operatorname {D} (P)=\sum _{e}\ell (e)\otimes \theta (e)}$$ where the sum is taken over all edges ${\displaystyle e}$ of the polyhedron ${\displaystyle P}$. It is a valuation.

In light of Dehn's theorem above, one might ask "which polyhedra are scissors-congruent"? Sydler (1965) showed that two polyhedra are scissors-congruent if and only if they have the same volume and the same Dehn invariant.^[4] Børge Jessen later extended Sydler's results to four dimensions.^[5] In 1990, Dupont and Sah provided a simpler proof of Sydler's result by reinterpreting it as a theorem about the homology of certain classical groups.^[6]

Debrunner showed in 1980 that the Dehn invariant of any polyhedron with which all of three-dimensional space can be tiled periodically is zero.^[7]

Jessen also posed the question of whether the analogue of Jessen's results remained true for spherical geometry and hyperbolic geometry. In these geometries, Dehn's method continues to work, and shows that when two polyhedra are scissors-congruent, their Dehn invariants are equal. However, it remains an open problem whether pairs of polyhedra with the same volume and the same Dehn invariant, in these geometries, are always scissors-congruent.^[8]

Hilbert's original question was more complicated: given any two tetrahedra T₁ and T₂ with equal base area and equal height (and therefore equal volume), is it always possible to find a finite number of tetrahedra, so that when these tetrahedra are glued in some way to T₁ and also glued to T₂, the resulting polyhedra are scissors-congruent?

Dehn's invariant can be used to yield a negative answer also to this stronger question.

• Hill tetrahedron
• Onorato Nicoletti

• Benko, D. (2007). "A New Approach to Hilbert's Third Problem". The American Mathematical Monthly. 114 (8): 665–676. doi:10.1080/00029890.2007.11920458. S2CID 7213930.
• Schwartz, Rich (2010). "The Dehn–Sydler Theorem Explained" (PDF).
• Koji, Shiga; Toshikazu Sunada (2005). A Mathematical Gift, III: The Interplay Between Topology, Functions, Geometry, and Algebra. American Mathematical Society.

• Proof of Dehn's Theorem at Everything2
• Weisstein, Eric W. "Dehn Invariant". MathWorld.
• Dehn Invariant at Everything2
• Hazewinkel, M. (2001) [1994], "Dehn invariant", Encyclopedia of Mathematics, EMS Press