Modular representation theory is a branch of mathematics, and is the part of representation theory that studies linear representations of finite groups over a field K of positive characteristic p, necessarily a prime number. As well as having applications to group theory, modular representations arise naturally in other branches of mathematics, such as algebraic geometry, coding theory^{[citation needed]}, combinatorics and number theory.

Within finite group theory, character-theoretic results proved by Richard Brauer using modular representation theory played an important role in early progress towards the classification of finite simple groups, especially for simple groups whose characterization was not amenable to purely group-theoretic methods because their Sylow 2-subgroups were too small in an appropriate sense. Also, a general result on embedding of elements of order 2 in finite groups called the Z* theorem, proved by George Glauberman using the theory developed by Brauer, was particularly useful in the classification program.

If the characteristic p of K does not divide the order |G|, then modular representations are completely reducible, as with ordinary (characteristic 0) representations, by virtue of Maschke's theorem. In the other case, when |G| ≡ 0 mod p, the process of averaging over the group needed to prove Maschke's theorem breaks down, and representations need not be completely reducible. Much of the discussion below implicitly assumes that the field K is sufficiently large (for example, K algebraically closed suffices), otherwise some statements need refinement.

The earliest work on representation theory over finite fields is by Dickson (1902) who showed that when p does not divide the order of the group, the representation theory is similar to that in characteristic 0. He also investigated modular invariants of some finite groups. The systematic study of modular representations, when the characteristic p divides the order of the group, was started by Brauer (1935) and was continued by him for the next few decades.

Finding a representation of the cyclic group of two elements over F₂ is equivalent to the problem of finding matrices whose square is the identity matrix. Over every field of characteristic other than 2, there is always a basis such that the matrix can be written as a diagonal matrix with only 1 or −1 occurring on the diagonal, such as

${\displaystyle {\begin{bmatrix}1&0\\0&-1\end{bmatrix}}.}$

Over F₂, there are many other possible matrices, such as

${\displaystyle {\begin{bmatrix}1&1\\0&1\end{bmatrix}}.}$

Over an algebraically closed field of positive characteristic, the representation theory of a finite cyclic group is fully explained by the theory of the Jordan normal form. Non-diagonal Jordan forms occur when the characteristic divides the order of the group.

Given a field K and a finite group G, the group algebra K[G] (which is the K-vector space with K-basis consisting of the elements of G, endowed with algebra multiplication by extending the multiplication of G by linearity) is an Artinian ring.

When the order of G is divisible by the characteristic of K, the group algebra is not semisimple, hence has non-zero Jacobson radical. In that case, there are finite-dimensional modules for the group algebra that are not projective modules. By contrast, in the characteristic 0 case every irreducible representation is a direct summand of the regular representation, hence is projective.

Modular representation theory was developed by Richard Brauer from about 1940 onwards to study in greater depth the relationships between the characteristic p representation theory, ordinary character theory and structure of G, especially as the latter relates to the embedding of, and relationships between, its p-subgroups. Such results can be applied in group theory to problems not directly phrased in terms of representations.

Brauer introduced the notion now known as the Brauer character. When K is algebraically closed of positive characteristic p, there is a bijection between roots of unity in K and complex roots of unity of order coprime to p. Once a choice of such a bijection is fixed, the Brauer character of a representation assigns to each group element of order coprime to p the sum of complex roots of unity corresponding to the eigenvalues (including multiplicities) of that element in the given representation.

The Brauer character of a representation determines its composition factors but not, in general, its equivalence type. The irreducible Brauer characters are those afforded by the simple modules. These are integral (though not necessarily non-negative) combinations of the restrictions to elements of order coprime to p of the ordinary irreducible characters. Conversely, the restriction to the elements of order coprime to p of each ordinary irreducible character is uniquely expressible as a non-negative integer combination of irreducible Brauer characters.

In the theory initially developed by Brauer, the link between ordinary representation theory and modular representation theory is best exemplified by considering the group algebra of the group G over a complete discrete valuation ring R with residue field K of positive characteristic p and field of fractions F of characteristic 0, such as the p-adic integers. The structure of R[G] is closely related both to the structure of the group algebra K[G] and to the structure of the semisimple group algebra F[G], and there is much interplay between the module theory of the three algebras.

Each R[G]-module naturally gives rise to an F[G]-module, and, by a process often known informally as reduction (mod p), to a K[G]-module. On the other hand, since R is a principal ideal domain, each finite-dimensional F[G]-module arises by extension of scalars from an R[G]-module.^{[citation needed]} In general, however, not all K[G]-modules arise as reductions (mod p) of R[G]-modules. Those that do are liftable.

In ordinary representation theory, the number of simple modules k(G) is equal to the number of conjugacy classes of G. In the modular case, the number l(G) of simple modules is equal to the number of conjugacy classes whose elements have order coprime to the relevant prime p, the so-called p-regular classes.

In modular representation theory, while Maschke's theorem does not hold when the characteristic divides the group order, the group algebra may be decomposed as the direct sum of a maximal collection of two-sided ideals known as blocks. When the field F has characteristic 0, or characteristic coprime to the group order, there is still such a decomposition of the group algebra F[G] as a sum of blocks (one for each isomorphism type of simple module), but the situation is relatively transparent when F is sufficiently large: each block is a full matrix algebra over F, the endomorphism ring of the vector space underlying the associated simple module.

To obtain the blocks, the identity element of the group G is decomposed as a sum of primitive idempotents in Z(R[G]), the center of the group algebra over the maximal order R of F. The block corresponding to the primitive idempotent e is the two-sided ideal e R[G]. For each indecomposable R[G]-module, there is only one such primitive idempotent that does not annihilate it, and the module is said to belong to (or to be in) the corresponding block (in which case, all its composition factors also belong to that block). In particular, each simple module belongs to a unique block. Each ordinary irreducible character may also be assigned to a unique block according to its decomposition as a sum of irreducible Brauer characters. The block containing the trivial module is known as the principal block.

In ordinary representation theory, every indecomposable module is irreducible, and so every module is projective. However, the simple modules with characteristic dividing the group order are rarely projective. Indeed, if a simple module is projective, then it is the only simple module in its block, which is then isomorphic to the endomorphism algebra of the underlying vector space, a full matrix algebra. In that case, the block is said to have 'defect 0'. Generally, the structure of projective modules is difficult to determine.

For the group algebra of a finite group, the (isomorphism types of) projective indecomposable modules are in a one-to-one correspondence with the (isomorphism types of) simple modules: the socle of each projective indecomposable is simple (and isomorphic to the top), and this affords the bijection, as non-isomorphic projective indecomposables have non-isomorphic socles. The multiplicity of a projective indecomposable module as a summand of the group algebra (viewed as the regular module) is the dimension of its socle (for large enough fields of characteristic zero, this recovers the fact that each simple module occurs with multiplicity equal to its dimension as a direct summand of the regular module).

Each projective indecomposable module (and hence each projective module) in positive characteristic p may be lifted to a module in characteristic 0. Using the ring R as above, with residue field K, the identity element of G may be decomposed as a sum of mutually orthogonal primitive idempotents (not necessarily central) of K[G]. Each projective indecomposable K[G]-module is isomorphic to e.K[G] for a primitive idempotent e that occurs in this decomposition. The idempotent e lifts to a primitive idempotent, say E, of R[G], and the left module E.R[G] has reduction (mod p) isomorphic to e.K[G].

When a projective module is lifted, the associated character vanishes on all elements of order divisible by p, and (with consistent choice of roots of unity), agrees with the Brauer character of the original characteristic p module on p-regular elements. The (usual character-ring) inner product of the Brauer character of a projective indecomposable with any other Brauer character can thus be defined: this is 0 if the second Brauer character is that of the socle of a non-isomorphic projective indecomposable, and 1 if the second Brauer character is that of its own socle. The multiplicity of an ordinary irreducible character in the character of the lift of a projective indecomposable is equal to the number of occurrences of the Brauer character of the socle of the projective indecomposable when the restriction of the ordinary character to p-regular elements is expressed as a sum of irreducible Brauer characters.

The composition factors of the projective indecomposable modules may be calculated as follows: Given the ordinary irreducible and irreducible Brauer characters of a particular finite group, the irreducible ordinary characters may be decomposed as non-negative integer combinations of the irreducible Brauer characters. The integers involved can be placed in a matrix, with the ordinary irreducible characters assigned rows and the irreducible Brauer characters assigned columns. This is referred to as the decomposition matrix, and is frequently labelled D. It is customary to place the trivial ordinary and Brauer characters in the first row and column respectively. The product of the transpose of D with D itself results in the Cartan matrix, usually denoted C; this is a symmetric matrix such that the entries in its j-th row are the multiplicities of the respective simple modules as composition factors of the j-th projective indecomposable module. The Cartan matrix is non-singular; in fact, its determinant is a power of the characteristic of K.

Since a projective indecomposable module in a given block has all its composition factors in that same block, each block has its own Cartan matrix.

To each block B of the group algebra K[G], Brauer associated a certain p-subgroup, known as its defect group (where p is the characteristic of K). Formally, it is the largest p-subgroup D of G for which there is a Brauer correspondent of B for the subgroup ${\displaystyle DC_{G}(D)}$, where ${\displaystyle C_{G}(D)}$ is the centralizer of D in G.

The defect group of a block is unique up to conjugacy and has a strong influence on the structure of the block. For example, if the defect group is trivial, then the block contains just one simple module, just one ordinary character, the ordinary and Brauer irreducible characters agree on elements of order prime to the relevant characteristic p, and the simple module is projective. At the other extreme, when K has characteristic p, the Sylow p-subgroup of the finite group G is a defect group for the principal block of K[G].

The order of the defect group of a block has many arithmetical characterizations related to representation theory. It is the largest invariant factor of the Cartan matrix of the block, and occurs with multiplicity one. Also, the power of p dividing the index of the defect group of a block is the greatest common divisor of the powers of p dividing the dimensions of the simple modules in that block, and this coincides with the greatest common divisor of the powers of p dividing the degrees of the ordinary irreducible characters in that block.

Other relationships between the defect group of a block and character theory include Brauer's result that if no conjugate of the p-part of a group element g is in the defect group of a given block, then each irreducible character in that block vanishes at g. This is one of many consequences of Brauer's second main theorem.

The defect group of a block also has several characterizations in the more module-theoretic approach to block theory, building on the work of J. A. Green, which associates a p-subgroup known as the vertex to an indecomposable module, defined in terms of relative projectivity of the module. For example, the vertex of each indecomposable module in a block is contained (up to conjugacy) in the defect group of the block, and no proper subgroup of the defect group has that property.

Brauer's first main theorem states that the number of blocks of a finite group that have a given p-subgroup as defect group is the same as the corresponding number for the normalizer in the group of that p-subgroup.

The easiest block structure to analyse with non-trivial defect group is when the latter is cyclic. Then there are only finitely many isomorphism types of indecomposable modules in the block, and the structure of the block is by now well understood, by virtue of work of Brauer, E.C. Dade, J.A. Green and J.G. Thompson, among others. In all other cases, there are infinitely many isomorphism types of indecomposable modules in the block.

Blocks whose defect groups are not cyclic can be divided into two types: tame and wild. The tame blocks (which only occur for the prime 2) have as a defect group a dihedral group, semidihedral group or (generalized) quaternion group, and their structure has been broadly determined in a series of papers by Karin Erdmann. The indecomposable modules in wild blocks are extremely difficult to classify, even in principle.

• Brauer, R. (1935), Über die Darstellung von Gruppen in Galoisschen Feldern, Actualités Scientifiques et Industrielles, vol. 195, Paris: Hermann et cie, pp. 1–15, review
• Dickson, Leonard Eugene (1902), "On the Group Defined for any Given Field by the Multiplication Table of Any Given Finite Group", Transactions of the American Mathematical Society, 3 (3), Providence, R.I.: American Mathematical Society: 285–301, doi:10.2307/1986379, ISSN 0002-9947, JSTOR 1986379
• Jean-Pierre Serre (1977). Linear Representations of Finite Groups. Springer-Verlag. ISBN 0-387-90190-6.
• Walter Feit (1982). The representation theory of finite groups. North-Holland Mathematical Library. Vol. 25. Amsterdam-New York: North-Holland Publishing. ISBN 0-444-86155-6.