In mathematics, and in particular the study of dynamical systems, the idea of stable and unstable sets or stable and unstable manifolds give a formal mathematical definition to the general notions embodied in the idea of an attractor or repellor. In the case of hyperbolic dynamics, the corresponding notion is that of the hyperbolic set.

The gravitational tidal forces acting on the rings of Saturn provide an easy-to-visualize physical example. The tidal forces flatten the ring into the equatorial plane, even as they stretch it out in the radial direction. Imagining the rings to be sand or gravel particles ("dust") in orbit around Saturn, the tidal forces are such that any perturbations that push particles above or below the equatorial plane results in that particle feeling a restoring force, pushing it back into the plane. Particles effectively oscillate in a harmonic well, damped by collisions. The stable direction is perpendicular to the ring. The unstable direction is along any radius, where forces stretch and pull particles apart. Two particles that start very near each other in phase space will experience radial forces causing them to diverge, radially. These forces have a positive Lyapunov exponent; the trajectories lie on a hyperbolic manifold, and the movement of particles is essentially chaotic, wandering through the rings. The center manifold is tangential to the rings, with particles experiencing neither compression nor stretching. This allows second-order gravitational forces to dominate, and so particles can be entrained by moons or moonlets in the rings, phase locking to them. The gravitational forces of the moons effectively provide a regularly repeating small kick, each time around the orbit, akin to a kicked rotor, such as found in a phase-locked loop.

The discrete-time motion of particles in the ring can be approximated by the Poincaré map. The map effectively provides the transfer matrix of the system. The eigenvector associated with the largest eigenvalue of the matrix is the Frobenius–Perron eigenvector, which is also the invariant measure, i.e the actual density of the particles in the ring. All other eigenvectors of the transfer matrix have smaller eigenvalues, and correspond to decaying modes.

The following provides a definition for the case of a system that is either an iterated function or has discrete-time dynamics. Similar notions apply for systems whose time evolution is given by a flow.

Let ${\displaystyle X}$ be a topological space, and ${\displaystyle f\colon X\to X}$ a homeomorphism. If ${\displaystyle p}$ is a fixed point for ${\displaystyle f}$, the stable set of ${\displaystyle p}$ is defined by

${\displaystyle W^{s}(f,p)=\{q\in X:f^{n}(q)\to p{\mbox{ as }}n\to \infty \}}$

and the unstable set of ${\displaystyle p}$ is defined by

${\displaystyle W^{u}(f,p)=\{q\in X:f^{-n}(q)\to p{\mbox{ as }}n\to \infty \}.}$

Here, ${\displaystyle f^{-1}}$ denotes the inverse of the function ${\displaystyle f}$, i.e. ${\displaystyle f\circ f^{-1}=f^{-1}\circ f=id_{X}}$, where ${\displaystyle id_{X}}$ is the identity map on ${\displaystyle X}$.

If ${\displaystyle p}$ is a periodic point of least period ${\displaystyle k}$, then it is a fixed point of ${\displaystyle f^{k}}$, and the stable and unstable sets of ${\displaystyle p}$ are defined by

${\displaystyle W^{s}(f,p)=W^{s}(f^{k},p)}$

and

${\displaystyle W^{u}(f,p)=W^{u}(f^{k},p).}$

Given a neighborhood ${\displaystyle U}$ of ${\displaystyle p}$, the local stable and unstable sets of ${\displaystyle p}$ are defined by

${\displaystyle W_{\mathrm {loc} }^{s}(f,p,U)=\{q\in U:f^{n}(q)\in U{\mbox{ for each }}n\geq 0\}}$

and

${\displaystyle W_{\mathrm {loc} }^{u}(f,p,U)=W_{\mathrm {loc} }^{s}(f^{-1},p,U).}$

If ${\displaystyle X}$ is metrizable, we can define the stable and unstable sets for any point by

${\displaystyle W^{s}(f,p)=\{q\in X:d(f^{n}(q),f^{n}(p))\to 0{\mbox{ for }}n\to \infty \}}$

and

${\displaystyle W^{u}(f,p)=W^{s}(f^{-1},p),}$

where ${\displaystyle d}$ is a metric for ${\displaystyle X}$. This definition clearly coincides with the previous one when ${\displaystyle p}$ is a periodic point.

Suppose now that ${\displaystyle X}$ is a compact smooth manifold, and ${\displaystyle f}$ is a ${\displaystyle {\mathcal {C}}^{k}}$ diffeomorphism, ${\displaystyle k\geq 1}$. If ${\displaystyle p}$ is a hyperbolic periodic point, the stable manifold theorem assures that for some neighborhood ${\displaystyle U}$ of ${\displaystyle p}$, the local stable and unstable sets are ${\displaystyle {\mathcal {C}}^{k}}$ embedded disks, whose tangent spaces at ${\displaystyle p}$ are ${\displaystyle E^{s}}$ and ${\displaystyle E^{u}}$ (the stable and unstable spaces of ${\displaystyle Df(p)}$), respectively; moreover, they vary continuously (in a certain sense) in a neighborhood of ${\displaystyle f}$ in the ${\displaystyle {\mathcal {C}}^{k}}$ topology of ${\displaystyle \mathrm {Diff} ^{k}(X)}$ (the space of all ${\displaystyle {\mathcal {C}}^{k}}$ diffeomorphisms from ${\displaystyle X}$ to itself). Finally, the stable and unstable sets are ${\displaystyle {\mathcal {C}}^{k}}$ injectively immersed disks. This is why they are commonly called stable and unstable manifolds. This result is also valid for nonperiodic points, as long as they lie in some hyperbolic set (stable manifold theorem for hyperbolic sets).

If ${\displaystyle X}$ is a (finite-dimensional) vector space and ${\displaystyle f}$ an isomorphism, its stable and unstable sets are called stable space and unstable space, respectively.

• Invariant manifold
• Center manifold
• Limit set
• Julia set
• Slow manifold
• Inertial manifold
• Normally hyperbolic invariant manifold
• Lagrangian coherent structure

• Abraham, Ralph; Marsden, Jerrold E. (1978). Foundations of Mechanics. Reading Mass.: Benjamin/Cummings. ISBN 0-8053-0102-X.
• Irwin, Michael C. (2001). "Stable Manifolds". Smooth Dynamical Systems. World Scientific. pp. 143–160. ISBN 981-02-4599-8.
• Sritharan, S. S. (1990). Invariant Manifold Theory for Hydrodynamic Transition. New York: John Wiley & Sons. ISBN 0-582-06781-2.

This article incorporates material from Stable manifold on PlanetMath, which is licensed under the Creative Commons Attribution/Share-Alike License.