Formally, we start with a categoryC with finite products (i.e. C has a terminal object 1 and any two objects of C have a product). A group object in C is an object G of C together with morphisms
m : G × G → G (thought of as the "group multiplication")
e : 1 → G (thought of as the "inclusion of the identity element")
inv : G → G (thought of as the "inversion operation")
m is associative, i.e. m (m × idG) = m (idG × m) as morphisms G × G × G → G, and where e.g. m × idG : G × G × G → G × G; here we identify G × (G × G) in a canonical manner with (G × G) × G.
e is a two-sided unit of m, i.e. m (idG × e) = p1, where p1 : G × 1 → G is the canonical projection, and m (e × idG) = p2, where p2 : 1 × G → G is the canonical projection
inv is a two-sided inverse for m, i.e. if d : G → G × G is the diagonal map, and eG : G → G is the composition of the unique morphism G → 1 (also called the counit) with e, then m (idG × inv) d = eG and m (inv × idG) d = eG.
Note that this is stated in terms of maps – product and inverse must be maps in the category – and without any reference to underlying "elements" of the group object – categories in general do not have elements of their objects.
Another way to state the above is to say G is a group object in a category C if for every object X in C, there is a group structure on the morphisms Hom(X, G) from X to G such that the association of X to Hom(X, G) is a (contravariant) functor from C to the category of groups.
Examples
Each set G for which a group structure (G, m, u, −1) can be defined can be considered a group object in the category of sets. The map m is the group operation, the map e (whose domain is a singleton) picks out the identity element u of G, and the map inv assigns to every group element its inverse. eG : G → G is the map that sends every element of G to the identity element.
A localic group is a group object in the category of locales.
The group objects in the category of groups (or monoids) are the abelian groups. The reason for this is that, if inv is assumed to be a homomorphism, then G must be abelian. More precisely: if A is an abelian group and we denote by m the group multiplication of A, by e the inclusion of the identity element, and by inv the inversion operation on A, then (A, m, e, inv) is a group object in the category of groups (or monoids). Conversely, if (A, m, e, inv) is a group object in one of those categories, then m necessarily coincides with the given operation on A, e is the inclusion of the given identity element on A, inv is the inversion operation and A with the given operation is an abelian group. See also Eckmann–Hilton argument.
Given a category C with finite coproducts, a cogroup object is an object G of C together with a "comultiplication" m: G → GG, a "coidentity" e: G → 0, and a "coinversion" inv: G → G that satisfy the dual versions of the axioms for group objects. Here 0 is the initial object of C. Cogroup objects occur naturally in algebraic topology.