In mathematics, the orbit method (also known as the Kirillov theory, the method of coadjoint orbits and by a few similar names) establishes a correspondence between irreducible unitary representations of a Lie group and its coadjoint orbits: orbits of the action of the group on the dual space of its Lie algebra. The theory was introduced by Kirillov (1961, 1962) for nilpotent groups and later extended by Bertram Kostant, Louis Auslander, Lajos Pukánszky and others to the case of solvable groups. Roger Howe found a version of the orbit method that applies to p-adic Lie groups.^[1] David Vogan proposed that the orbit method should serve as a unifying principle in the description of the unitary duals of real reductive Lie groups.^[2]

One of the key observations of Kirillov was that coadjoint orbits of a Lie group G have natural structure of symplectic manifolds whose symplectic structure is invariant under G. If an orbit is the phase space of a G-invariant classical mechanical system then the corresponding quantum mechanical system ought to be described via an irreducible unitary representation of G. Geometric invariants of the orbit translate into algebraic invariants of the corresponding representation. In this way the orbit method may be viewed as a precise mathematical manifestation of a vague physical principle of quantization. In the case of a nilpotent group G the correspondence involves all orbits, but for a general G additional restrictions on the orbit are necessary (polarizability, integrality, Pukánszky condition). This point of view has been significantly advanced by Kostant in his theory of geometric quantization of coadjoint orbits.

For a Lie group ${\displaystyle G}$, the Kirillov orbit method gives a heuristic method in representation theory. It connects the Fourier transforms of coadjoint orbits, which lie in the dual space of the Lie algebra of G, to the infinitesimal characters of the irreducible representations. The method got its name after the Russian mathematician Alexandre Kirillov.

At its simplest, it states that a character of a Lie group may be given by the Fourier transform of the Dirac delta function supported on the coadjoint orbits, weighted by the square-root of the Jacobian of the exponential map, denoted by ${\displaystyle j}$. It does not apply to all Lie groups, but works for a number of classes of connected Lie groups, including nilpotent, some semisimple groups, and compact groups.

Let G be a connected, simply connected nilpotent Lie group. Kirillov proved that the equivalence classes of irreducible unitary representations of G are parametrized by the coadjoint orbits of G, that is the orbits of the action G on the dual space ${\displaystyle {\mathfrak {g}}^{*}}$ of its Lie algebra. The Kirillov character formula expresses the Harish-Chandra character of the representation as a certain integral over the corresponding orbit.

Complex irreducible representations of compact Lie groups have been completely classified. They are always finite-dimensional, unitarizable (i.e. admit an invariant positive definite Hermitian form) and are parametrized by their highest weights, which are precisely the dominant integral weights for the group. If G is a compact semisimple Lie group with a Cartan subalgebra h then its coadjoint orbits are closed and each of them intersects the positive Weyl chamber h^*₊ in a single point. An orbit is integral if this point belongs to the weight lattice of G. The highest weight theory can be restated in the form of a bijection between the set of integral coadjoint orbits and the set of equivalence classes of irreducible unitary representations of G: the highest weight representation L(λ) with highest weight λ∈h^*₊ corresponds to the integral coadjoint orbit G·λ. The Kirillov character formula amounts to the character formula earlier proved by Harish-Chandra.

• Dixmier mapping
• Pukánszky condition

• Dulfo; Pederson; Vergne (1990), The Orbit Method in Representation Theory: Proceedings of a Conference Held in Copenhagen, August to September 1988 (Progress in Mathematics), Birkhäuser
• Kirillov, A. A. (1961), "Unitary representations of nilpotent Lie groups", Doklady Akademii Nauk SSSR, 138: 283–284, ISSN 0002-3264, MR 0125908
• Kirillov, A. A. (1962), "Unitary representations of nilpotent Lie groups", Russian Mathematical Surveys, 17 (4): 53–104, Bibcode:1962RuMaS..17...53K, doi:10.1070/RM1962v017n04ABEH004118, ISSN 0042-1316, MR 0142001
• Kirillov, A. A. (1976) [1972], Elements of the theory of representations, Grundlehren der Mathematischen Wissenschaften, vol. 220, Berlin, New York: Springer-Verlag, ISBN 978-0-387-07476-4, MR 0412321
• Kirillov, A. A. (1999), "Merits and demerits of the orbit method", Bull. Amer. Math. Soc., 36 (4): 433–488, doi:10.1090/s0273-0979-99-00849-6.
• Kirillov, A. A. (2001) [1994], "Orbit method", Encyclopedia of Mathematics, EMS Press
• Kirillov, A. A. (2004), Lectures on the orbit method, Graduate Studies in Mathematics, vol. 64, Providence, RI: American Mathematical Society, ISBN 978-0-8218-3530-2.