This article includes a list of references, related reading, or external links, but its sources remain unclear because it lacks inline citations. Please help improve this article by introducing more precise citations. (June 2013) (Learn how and when to remove this message)

Outer billiards is a dynamical system based on a convex shape in the plane. Classically, this system is defined for the Euclidean plane^[1] but one can also consider the system in the hyperbolic plane^[2] or in other spaces that suitably generalize the plane. Outer billiards differs from a usual dynamical billiard in that it deals with a discrete sequence of moves outside the shape rather than inside of it.

Let P be a convex shape in the plane. Given a point x0 outside P, there is typically a unique point x1 (also outside P) so that the line segment connecting x0 to x1 is tangent to P at its midpoint and a person walking from x0 to x1 would see P on the right. (See Figure.) The map F: x0 -> x1 is called the outer billiards map.

The inverse (or backwards) outer billiards map is also defined, as the map x1 -> x0. One gets the inverse map simply by replacing the word right by the word left in the definition given above. The figure shows the situation in the Euclidean plane, but the definition in the hyperbolic plane is essentially the same.

An outer billiards orbit is the set of all iterations of the point, namely ... x0 ↔ x1 ↔ x2 ↔ x3 ... That is, start at x0 and iteratively apply both the outer billiards map and the backwards outer billiards map. When P is a strictly convex shape, such as an ellipse, every point in the exterior of P has a well defined orbit. When P is a polygon, some points might not have well-defined orbits, on account of the potential ambiguity of choosing the midpoint of the relevant tangent line. Nevertheless, in the polygonal case, almost every point has a well-defined orbit.

• An orbit is called periodic if it eventually repeats.
• An orbit is called aperiodic (or non-periodic) if it is not periodic.
• An orbit is called bounded (or stable) if some bounded region in the plane contains the whole orbit.
• An orbit is called unbounded (or unstable) if it is not bounded.

Defining an outer billiards system in a higher-dimensional space is beyond the scope of this article. Unlike the case of ordinary billiards, the definition is not straightforward. One natural setting for the map is a complex vector space. In this case, there is a natural choice of line tangent to a convex body at each point. One obtains these tangents by starting with the normals and using the complex structure to rotate 90 degrees. These distinguished tangent lines can be used to define the outer billiards map roughly as above.^[1]

Most people attribute the introduction of outer billiards to Bernhard Neumann in the late 1950s,^[3] though it seems that a few people cite an earlier construction in 1945, due to M. Day. Jürgen Moser popularized the system in the 1970s as a toy model for celestial mechanics.^[4][5] This system has been studied classically in the Euclidean plane, and more recently in the hyperbolic plane. One can also consider higher-dimensional spaces, though no serious study has yet been made. Bernhard Neumann informally posed the question as to whether or not one can have unbounded orbits in an outer billiards system, and Moser put it in writing in 1973.^[4] Sometimes this basic question has been called the Moser-Neumann question. This question, originally posed for shapes in the Euclidean plane and solved only recently, has been a guiding problem in the field.

In the 70's, Jürgen Moser sketched a proof, based on K.A.M. theory, that outer billiards relative to a 6-times-differentiable shape of positive curvature has all orbits bounded. In 1982, Raphael Douady gave the full proof of this result.^[6] A big advance in the polygonal case came over a period of several years when three teams of authors, Vivaldi-Shaidenko,^[7] Kolodziej,^[8] and Gutkin-Simanyi,^[9] each using different methods, showed that outer billiards relative to a quasirational polygon has all orbits bounded. The notion of quasirational is technical (see references) but it includes the class of regular polygons and convex rational polygons, namely those convex polygons whose vertices have rational coordinates. In the case of rational polygons, all the orbits are periodic. In 1995, Sergei Tabachnikov showed that outer billiards for the regular pentagon has some aperiodic orbits, thus clarifying the distinction between the dynamics in the rational and regular cases.^[1] In 1996, Philip Boyland showed that outer billiards relative to some shapes can have orbits which accumulate on the shape.^[10] In 2005, Daniel Genin showed that all orbits are bounded when the shape is a trapezoid, thus showing that quasirationality is not a necessary condition for the system to have all orbits bounded.^[11] (Not all trapezoids are quasirational.)

In 2007, Richard Schwartz showed that outer billiards has some unbounded orbits when defined relative to the Penrose Kite, thus answering the original Moser-Neumann question in the affirmative.^[12] The Penrose kite is the convex quadrilateral from the kites-and-darts Penrose tilings. Subsequently, Schwartz showed that outer billiards has unbounded orbits when defined relative to any irrational kite.^[13] An irrational kite is a quadrilateral with the following property: One of the diagonals of the quadrilateral divides the region into two triangles of equal area and the other diagonal divides the region into two triangles whose areas are not rational multiples of each other. In 2008, Dmitry Dolgopyat and Bassam Fayad showed that outer billiards defined relative to the semidisk has unbounded orbits.^[14] The semidisk is the region one gets by cutting a disk in half. The proof of Dolgopyat-Fayad is robust, and also works for regions obtained by cutting a disk nearly in half, when the word nearly is suitably interpreted.

In 2003, Filiz Doǧru and Sergei Tabachnikov showed that all orbits are unbounded for a certain class of convex polygons in the hyperbolic plane.^[15] The authors call such polygons large. (See the reference for the definition.) Filiz Doǧru and Samuel Otten then extended this work in 2011 by specifying the conditions under which a regular polygonal table in the hyperbolic plane have all orbits unbounded, that is, are large.^[16]

In ordinary polygonal billiards, the existence of periodic orbits is a major unsolved problem. For instance, it is unknown if every triangular shaped table has a periodic billiard path. More progress has been made for outer billiards, though the situation is far from well understood. As mentioned above, all the orbits are periodic when the system is defined relative to a convex rational polygon in the Euclidean plane. Moreover, it is a recent theorem of Chris Culter (written up by Sergei Tabachnikov) that outer billiards relative to any convex polygon has periodic orbits—in fact a periodic orbit outside of any given bounded region.^[17]

Outer billiards is a subject still in its beginning phase. Most problems are still unsolved. Here are some open problems in the area.

• Show that outer billiards relative to almost every convex polygon has unbounded orbits.
• Show that outer billiards relative to a regular polygon has almost every orbit periodic. The cases of the equilateral triangle and the square are trivial, and Tabachnikov answered this for the regular pentagon. These are the only cases known.
• more broadly, characterize the structure of the set of periodic orbits relative to the typical convex polygon.
• understand the structure of periodic orbits relative to simple shapes in the hyperbolic plane, such as small equilateral triangles.

• Illumination problem