In set theory and computability theory, Kleene's ${\displaystyle {\mathcal {O}}}$ is a canonical subset of the natural numbers when regarded as ordinal notations. It contains ordinal notations for every computable ordinal, that is, ordinals below Church–Kleene ordinal, ${\displaystyle \omega _{1}^{\text{CK}}}$. Since ${\displaystyle \omega _{1}^{\text{CK}}}$ is the first ordinal not representable in a computable system of ordinal notations the elements of ${\displaystyle {\mathcal {O}}}$ can be regarded as the canonical ordinal notations.

Kleene (1938) described a system of notation for all computable ordinals (those less than the Church–Kleene ordinal). It uses a subset of the natural numbers instead of finite strings of symbols. Unfortunately, there is in general no effective way to tell whether some natural number represents an ordinal, or whether two numbers represent the same ordinal. However, one can effectively find notations which represent the ordinal sum, product, and power (see ordinal arithmetic) of any two given notations in Kleene's ${\displaystyle {\mathcal {O}}}$; and given any notation for an ordinal, there is a computably enumerable set of notations which contains one element for each smaller ordinal and is effectively ordered.

The basic idea of Kleene's system of ordinal notations is to build up ordinals in an effective manner. For members ${\displaystyle p}$ of ${\displaystyle {\mathcal {O}}}$, the ordinal for which ${\displaystyle p}$ is a notation is ${\displaystyle |p|}$. ${\displaystyle {\mathcal {O}}}$ and ${\displaystyle <_{\mathcal {O}}}$ (a partial ordering of Kleene's ${\displaystyle {\mathcal {O}}}$) are the smallest sets such that the following holds.^{[citation needed]}

• ${\displaystyle 0\in {\mathcal {O}}\land |0|=0}$.
• ${\displaystyle i\in {\mathcal {O}}\land |i|=\alpha \rightarrow 2^{i}\in {\mathcal {O}}\land |2^{i}|=\alpha +1\land i<_{\mathcal {O}}2^{i}}$
• Suppose ${\displaystyle \{e\}}$ is the ${\displaystyle e}$-th partial computable function. If ${\displaystyle e}$ is total and ${\displaystyle {\textrm {range}}(\{e\})\subset {\mathcal {O}}\land \forall n(\{e\}(n)<_{\mathcal {O}}\{e\}(n+1))}$, then ${\displaystyle 3\cdot 5^{e}\in {\mathcal {O}}\land \forall n,\{e\}(n)<_{\mathcal {O}}3\cdot 5^{e}\land |3\cdot 5^{e}|=\lim _{k}|\{e\}(k)|}$
• ${\displaystyle p<_{\mathcal {O}}q\land q<_{\mathcal {O}}r\rightarrow p<_{\mathcal {O}}r}$

This definition has the advantages that one can computably enumerate the predecessors of a given ordinal (though not in the ${\displaystyle <_{\mathcal {O}}}$ ordering) and that the notations are downward closed, i.e., if there is a notation for ${\displaystyle \gamma }$ and ${\displaystyle \alpha <\gamma }$ then there is a notation for ${\displaystyle \alpha }$. There are alternate definitions, such as the set of indices of (partial) well-orderings of the natural numbers.^[1]

A member ${\displaystyle p}$ of Kleene's ${\displaystyle {\mathcal {O}}}$ is called a notation and is meant to give a definition of the ordinal ${\displaystyle |p|}$.

The successor notations are those such that ${\displaystyle |p|}$ is a successor ordinal ${\displaystyle \alpha +1}$. In Kleene's ${\displaystyle {\mathcal {O}}}$, a successor ordinal is defined in terms of a notation for the ordinal immediately preceding it. Specifically, a successor notation ${\displaystyle p}$ is of the form ${\displaystyle 2^{q}}$ for some other notation ${\displaystyle q}$, so that ${\displaystyle |p|=|q|+1}$.

The limit notations are those such that ${\displaystyle |p|}$ is a limit ordinal. In Kleene's ${\displaystyle {\mathcal {O}}}$, a notation for a limit ordinal ${\displaystyle \beta }$ amounts to a computable sequence of notations for smaller ordinals limiting to ${\displaystyle \beta }$. Any limit notation ${\displaystyle p}$ is of the form ${\displaystyle 3\cdot 5^{e}}$ where the ${\displaystyle e}$'th partial computable function ${\displaystyle \{e\}:\mathbb {N} \rightarrow \mathbb {N} }$ is a total function listing an infinite sequence ${\displaystyle \langle q_{0},q_{1}...\rangle }$ of notations, which are strictly increasing under the order ${\displaystyle <_{\mathcal {O}}}$. The limit of the sequence of ordinals ${\displaystyle \langle |q_{0}|,|q_{1}|...\rangle }$ is ${\displaystyle |p|}$.

Although ${\displaystyle q<_{\mathcal {O}}p}$ implies ${\displaystyle |q|<|p|}$, ${\displaystyle |q|<|p|}$ does not imply ${\displaystyle q<_{\mathcal {O}}p}$.

In order for ${\displaystyle q<_{\mathcal {O}}p}$, ${\displaystyle q}$ must be reachable from ${\displaystyle p}$ by repeatedly applying the operations ${\displaystyle 2^{n}\mapsto n}$ and ${\displaystyle 3\cdot 5^{e}\mapsto \{e\}(i)}$ for ${\displaystyle i\in \mathbb {N} }$. In other words, ${\displaystyle q<_{\mathcal {O}}p}$ when ${\displaystyle q}$ is eventually referenced in the definition of ${\displaystyle |p|}$ given by ${\displaystyle p}$.

For arbitrary ${\displaystyle p,q\in \mathbb {N} }$, say that ${\displaystyle q\prec p}$ when ${\displaystyle q}$ is reachable from ${\displaystyle p}$ by repeatedly applying the operations ${\displaystyle 2^{n}\mapsto n}$ and ${\displaystyle 3\cdot 5^{e}\mapsto \{e\}(i)}$ for ${\displaystyle i\in dom(\{e\})}$. The relation ${\displaystyle \prec }$ agrees with ${\displaystyle <_{\mathcal {O}}}$ on ${\displaystyle {\mathcal {O}}}$, but ${\displaystyle \prec }$ is computably enumerable: if ${\displaystyle q\prec p}$, then a computer program will eventually find a proof of this fact.

For any notation ${\displaystyle p\in {\mathcal {O}}}$, all ${\displaystyle q\prec p}$ are themselves notations in ${\displaystyle {\mathcal {O}}}$.

For ${\displaystyle p\in \mathbb {N} }$, ${\displaystyle p}$ is a notation of ${\displaystyle {\mathcal {O}}}$ only when all of the criteria below are met:

• For all ${\displaystyle q\preceq p}$, ${\displaystyle q}$ is either ${\displaystyle 0}$, a power of ${\displaystyle 2}$, or ${\displaystyle 3\cdot 5^{e}}$ for some ${\displaystyle e\in \mathbb {N} }$.
• For any ${\displaystyle q\preceq p}$, if ${\displaystyle q=3\cdot 5^{e}}$ then ${\displaystyle \{e\}}$ is total and strictly increasing under ${\displaystyle \prec }$; i.e. ${\displaystyle \{e\}(i)\prec \{e\}(i+1)}$ for any ${\displaystyle i\in \mathbb {N} }$.
• The set ${\displaystyle \{q\in \mathbb {N} :q\prec p\}}$ is well-founded, so that there are no infinite descending sequences ${\displaystyle ...\prec q_{1}\prec q_{0}\prec p}$.

• If ${\displaystyle |i|=\alpha }$ and ${\displaystyle |j|=\beta }$ and ${\displaystyle i<_{\mathcal {O}}j\,,}$ then ${\displaystyle \alpha <\beta }$; but the converse may fail to hold.
• ${\displaystyle <_{\mathcal {O}}}$ induces a tree structure on ${\displaystyle {\mathcal {O}}}$, so ${\displaystyle {\mathcal {O}}}$ is well-founded.
• ${\displaystyle {\mathcal {O}}}$ only branches at limit ordinals; and at each notation of a limit ordinal, ${\displaystyle {\mathcal {O}}}$ is infinitely branching.
• Since every computable function has countably many indices, each infinite ordinal receives countably many notations; the finite ordinals have unique notations, ${\displaystyle n}$ usually denoted ${\displaystyle n_{\mathcal {O}}}$.
• The first ordinal that doesn't receive a notation is called the Church–Kleene ordinal and is denoted by ${\displaystyle \omega _{1}^{\text{CK}}}$. Since there are only countably many computable functions, the ordinal ${\displaystyle \omega _{1}^{\text{CK}}}$ is evidently countable.
• The ordinals with a notation in Kleene's ${\displaystyle {\mathcal {O}}}$ are exactly the computable ordinals. (The fact that every computable ordinal has a notation follows from the closure of this system of ordinal notations under successor and effective limits.)
• ${\displaystyle <_{\mathcal {O}}}$ is not computably enumerable, but there is a computably enumerable relation which agrees with ${\displaystyle <_{\mathcal {O}}}$ precisely on members of ${\displaystyle {\mathcal {O}}}$.
• For any notation ${\displaystyle p}$, the set ${\displaystyle \lbrace q\mid q<_{\mathcal {O}}p\rbrace }$ of notations below ${\displaystyle p}$ is computably enumerable. However, Kleene's ${\displaystyle {\mathcal {O}}}$, when taken as a whole, is ${\displaystyle \Pi _{1}^{1}}$ (see analytical hierarchy) and not arithmetical because of the following:
• ${\displaystyle {\mathcal {O}}}$ is ${\displaystyle \Pi _{1}^{1}}$-complete (i.e. ${\displaystyle {\mathcal {O}}}$ is ${\displaystyle \Pi _{1}^{1}}$ and every ${\displaystyle \Pi _{1}^{1}}$ set is Turing reducible to it)^[2] and every ${\displaystyle \Sigma _{1}^{1}}$ subset of ${\displaystyle {\mathcal {O}}}$ is effectively bounded in ${\displaystyle {\mathcal {O}}}$ (a result of Spector).
• In fact, any ${\displaystyle \Pi _{1}^{1}}$ set is many-one reducible to ${\displaystyle {\mathcal {O}}}$.^[2]
• ${\displaystyle {\mathcal {O}}}$ is the universal system of ordinal notations in the sense that any specific set of ordinal notations can be mapped into it in a straightforward way. More precisely, there is a computable function ${\displaystyle f}$ such that if ${\displaystyle e}$ is an index for a computable well-ordering, then ${\displaystyle f(e)}$ is a member of ${\displaystyle {\mathcal {O}}}$ and ${\displaystyle <_{e}}$ is order-isomorphic to an initial segment of the set ${\displaystyle \{p\mid p<_{\mathcal {O}}f(e)\}}$.
• There is a computable function ${\displaystyle +_{\mathcal {O}}}$, which, for members of ${\displaystyle {\mathcal {O}}}$, mimics ordinal addition and has the property that ${\displaystyle \max\{p,q\}\leq p+_{\mathcal {O}}q}$. (Jockusch)

A path in ${\displaystyle {\mathcal {O}}}$ is a subset ${\displaystyle {\mathcal {P}}}$ of ${\displaystyle {\mathcal {O}}}$ which is totally ordered by ${\displaystyle <_{\mathcal {O}}}$ and is closed under predecessors, i.e. if ${\displaystyle p}$ is a member of a path ${\displaystyle {\mathcal {P}}}$ and ${\displaystyle q<_{\mathcal {O}}p}$ then ${\displaystyle q}$ is also a member of ${\displaystyle {\mathcal {P}}}$. A path ${\displaystyle {\mathcal {P}}}$ is maximal if there is no element of ${\displaystyle {\mathcal {O}}}$ which is above (in the sense of ${\displaystyle <_{\mathcal {O}}}$) every member of ${\displaystyle {\mathcal {P}}}$, otherwise ${\displaystyle {\mathcal {P}}}$ is non-maximal.

• A path ${\displaystyle {\mathcal {P}}}$ is non-maximal if and only if ${\displaystyle {\mathcal {P}}}$ is computably enumerable (c.e.). It follows by remarks above that every element ${\displaystyle p}$ of ${\displaystyle {\mathcal {O}}}$ determines a non-maximal path ${\displaystyle {\mathcal {P}}}$; and every non-maximal path is so determined.
• There are ${\displaystyle 2^{\aleph _{0}}}$ maximal paths through ${\displaystyle {\mathcal {O}}}$; since they are maximal, they are non-c.e.
• In fact, there are ${\displaystyle 2^{\aleph _{0}}}$ maximal paths within ${\displaystyle {\mathcal {O}}}$ of length ${\displaystyle \omega ^{2}}$. (Crossley, Schütte)
• For every non-zero ordinal ${\displaystyle \lambda <\omega _{1}^{CK}}$, there are ${\displaystyle 2^{\aleph _{0}}}$ maximal paths within ${\displaystyle {\mathcal {O}}}$ of length ${\displaystyle \omega ^{2}\cdot \lambda }$. (Aczel)
• Further, if ${\displaystyle {\mathcal {P}}}$ is a path whose length is not a multiple of ${\displaystyle \omega ^{2}}$ then ${\displaystyle {\mathcal {P}}}$ is not maximal. (Aczel)
• For each c.e. degree ${\displaystyle d}$, there is a member ${\displaystyle e_{d}}$ of ${\displaystyle {\mathcal {O}}}$ such that the path ${\displaystyle {\mathcal {P}}=\lbrace p\mid p<_{\mathcal {O}}e_{d}\rbrace }$ has many-one degree ${\displaystyle d}$. In fact, for each computable ordinal ${\displaystyle \alpha \geq \omega ^{2}}$, a notation ${\displaystyle e_{d}}$ exists with ${\displaystyle |e_{d}|=\alpha }$. (Jockusch)
• There exist ${\displaystyle \aleph _{0}}$ paths through ${\displaystyle {\mathcal {O}}}$ which are ${\displaystyle \Pi _{1}^{1}}$. Given a progression of computably enumerable theories based on iterating Uniform Reflection, each such path is incomplete with respect to the set of true ${\displaystyle \Pi _{1}^{0}}$ sentences. (Feferman & Spector)
• There exist ${\displaystyle \Pi _{1}^{1}}$ paths through ${\displaystyle {\mathcal {O}}}$ each initial segment of which is not merely c.e., but computable. (Jockusch)
• Various other paths in ${\displaystyle {\mathcal {O}}}$ have been shown to exist, each with specific kinds of reducibility properties. (See references below)

• Computable ordinal
• Large countable ordinal
• Ordinal notation