This article may be too technical for most readers to understand. Please help improve it to make it understandable to non-experts, without removing the technical details. (June 2024) (Learn how and when to remove this message)

Hyperelliptic curve cryptography is similar to elliptic curve cryptography (ECC) insofar as the Jacobian of a hyperelliptic curve is an abelian group in which to do arithmetic, just as we use the group of points on an elliptic curve in ECC.

An (imaginary) hyperelliptic curve of genus ${\displaystyle g}$ over a field ${\displaystyle K}$ is given by the equation ${\displaystyle C:y^{2}+h(x)y=f(x)\in K[x,y]}$ where ${\displaystyle h(x)\in K[x]}$ is a polynomial of degree not larger than ${\displaystyle g}$ and ${\displaystyle f(x)\in K[x]}$ is a monic polynomial of degree ${\displaystyle 2g+1}$. From this definition it follows that elliptic curves are hyperelliptic curves of genus 1. In hyperelliptic curve cryptography ${\displaystyle K}$ is often a finite field. The Jacobian of ${\displaystyle C}$, denoted ${\displaystyle J(C)}$, is a quotient group, thus the elements of the Jacobian are not points, they are equivalence classes of divisors of degree 0 under the relation of linear equivalence. This agrees with the elliptic curve case, because it can be shown that the Jacobian of an elliptic curve is isomorphic with the group of points on the elliptic curve.^[1] The use of hyperelliptic curves in cryptography came about in 1989 from Neal Koblitz. Although introduced only 3 years after ECC, not many cryptosystems implement hyperelliptic curves because the implementation of the arithmetic isn't as efficient as with cryptosystems based on elliptic curves or factoring (RSA). The efficiency of implementing the arithmetic depends on the underlying finite field ${\displaystyle K}$, in practice it turns out that finite fields of characteristic 2 are a good choice for hardware implementations while software is usually faster in odd characteristic.^[2]

The Jacobian on a hyperelliptic curve is an Abelian group and as such it can serve as group for the discrete logarithm problem (DLP). In short, suppose we have an Abelian group ${\displaystyle G}$ and ${\displaystyle g}$ an element of ${\displaystyle G}$, the DLP on ${\displaystyle G}$ entails finding the integer ${\displaystyle a}$ given two elements of ${\displaystyle G}$, namely ${\displaystyle g}$ and ${\displaystyle g^{a}}$. The first type of group used was the multiplicative group of a finite field, later also Jacobians of (hyper)elliptic curves were used. If the hyperelliptic curve is chosen with care, then Pollard's rho method is the most efficient way to solve DLP. This means that, if the Jacobian has ${\displaystyle n}$ elements, that the running time is exponential in ${\displaystyle \log(n)}$. This makes it possible to use Jacobians of a fairly small order, thus making the system more efficient. But if the hyperelliptic curve is chosen poorly, the DLP will become quite easy to solve. In this case there are known attacks which are more efficient than generic discrete logarithm solvers^[3] or even subexponential.^[4] Hence these hyperelliptic curves must be avoided. Considering various attacks on DLP, it is possible to list the features of hyperelliptic curves that should be avoided.

All generic attacks on the discrete logarithm problem in finite abelian groups such as the Pohlig–Hellman algorithm and Pollard's rho method can be used to attack the DLP in the Jacobian of hyperelliptic curves. The Pohlig-Hellman attack reduces the difficulty of the DLP by looking at the order of the group we are working with. Suppose the group ${\displaystyle G}$ that is used has ${\displaystyle n=p_{1}^{r_{1}}\cdots p_{k}^{r_{k}}}$ elements, where ${\displaystyle p_{1}^{r_{1}}\cdots p_{k}^{r_{k}}}$ is the prime factorization of ${\displaystyle n}$. Pohlig-Hellman reduces the DLP in ${\displaystyle G}$ to DLPs in subgroups of order ${\displaystyle p_{i}}$ for ${\displaystyle i=1,...,k}$. So for ${\displaystyle p}$ the largest prime divisor of ${\displaystyle n}$, the DLP in ${\displaystyle G}$ is just as hard to solve as the DLP in the subgroup of order ${\displaystyle p}$. Therefore, we would like to choose ${\displaystyle G}$ such that the largest prime divisor ${\displaystyle p}$ of ${\displaystyle \#G=n}$ is almost equal to ${\displaystyle n}$ itself. Requiring ${\textstyle {\frac {n}{p}}\leq 4}$ usually suffices.

The index calculus algorithm is another algorithm that can be used to solve DLP under some circumstances. For Jacobians of (hyper)elliptic curves there exists an index calculus attack on DLP. If the genus of the curve becomes too high, the attack will be more efficient than Pollard's rho. Today it is known that even a genus of ${\displaystyle g=3}$ cannot assure security.^[5] Hence we are left with elliptic curves and hyperelliptic curves of genus 2.

Another restriction on the hyperelliptic curves we can use comes from the Menezes-Okamoto-Vanstone-attack / Frey-Rück-attack. The first, often called MOV for short, was developed in 1993, the second came about in 1994. Consider a (hyper)elliptic curve ${\displaystyle C}$ over a finite field ${\displaystyle \mathbb {F} _{q}}$ where ${\displaystyle q}$ is the power of a prime number. Suppose the Jacobian of the curve has ${\displaystyle n}$ elements and ${\displaystyle p}$ is the largest prime divisor of ${\displaystyle n}$. For ${\displaystyle k}$ the smallest positive integer such that ${\displaystyle p|q^{k}-1}$ there exists a computable injective group homomorphism from the subgroup of ${\displaystyle J(C)}$ of order ${\displaystyle p}$ to ${\displaystyle \mathbb {F} _{q^{k}}^{*}}$. If ${\displaystyle k}$ is small, we can solve DLP in ${\displaystyle J(C)}$ by using the index calculus attack in ${\textstyle \mathbb {F} _{q^{k}}^{*}}$. For arbitrary curves ${\displaystyle k}$ is very large (around the size of ${\displaystyle q^{g}}$); so even though the index calculus attack is quite fast for multiplicative groups of finite fields this attack is not a threat for most curves. The injective function used in this attack is a pairing and there are some applications in cryptography that make use of them. In such applications it is important to balance the hardness of the DLP in ${\displaystyle J(C)}$ and ${\textstyle \mathbb {F} _{q^{k}}^{*}}$; depending on the security level values of ${\displaystyle k}$ between 6 and 12 are useful. The subgroup of ${\textstyle \mathbb {F} _{q^{k}}^{*}}$ is a torus. There exists some independent usage in torus based cryptography.

We also have a problem, if ${\displaystyle p}$, the largest prime divisor of the order of the Jacobian, is equal to the characteristic of ${\displaystyle \mathbb {F} _{q}.}$ By a different injective map we could then consider the DLP in the additive group ${\displaystyle \mathbb {F} _{q}}$ instead of DLP on the Jacobian. However, DLP in this additive group is trivial to solve, as can easily be seen. So also these curves, called anomalous curves, are not to be used in DLP.

Hence, in order to choose a good curve and a good underlying finite field, it is important to know the order of the Jacobian. Consider a hyperelliptic curve ${\textstyle C}$ of genus ${\textstyle g}$ over the field ${\textstyle \mathbb {F} _{q}}$ where ${\textstyle q}$ is the power of a prime number and define ${\textstyle C_{k}}$ as ${\textstyle C}$ but now over the field ${\textstyle \mathbb {F} _{q^{k}}}$. It can be shown that the order of the Jacobian of ${\textstyle C_{k}}$ lies in the interval ${\textstyle [({\sqrt {q}}^{k}-1)^{2g},({\sqrt {q}}^{k}+1)^{2g}]}$, called the Hasse-Weil interval.^[6]

But there is more, we can compute the order using the zeta-function on hyperelliptic curves. Let ${\textstyle A_{k}}$ be the number of points on ${\textstyle C_{k}}$. Then we define the zeta-function of ${\textstyle C=C_{1}}$ as ${\textstyle Z_{C}(t)=\exp(\sum _{i=1}^{\infty }{A_{i}{\frac {t^{i}}{i}}})}$. For this zeta-function it can be shown that ${\textstyle Z_{C}(t)={\frac {P(t)}{(1-t)(1-qt)}}}$ where ${\textstyle P(t)}$ is a polynomial of degree ${\textstyle 2g}$ with coefficients in ${\textstyle \mathbb {Z} }$.^[7] Furthermore ${\textstyle P(t)}$ factors as ${\textstyle P(t)=\prod _{i=1}^{g}{(1-a_{i}t)(1-{\bar {a_{i}}}t)}}$ where ${\textstyle a_{i}\in \mathbb {C} }$ for all ${\textstyle i=1,...,g}$. Here ${\textstyle {\bar {a}}}$ denotes the complex conjugate of ${\displaystyle a}$. Finally we have that the order of ${\textstyle J(C_{k})}$ equals ${\textstyle \prod _{i=1}^{g}{|1-a_{i}^{k}|^{2}}}$. Hence orders of Jacobians can be found by computing the roots of ${\textstyle P(t)}$.

• Colm Ó hÉigeartaigh Implementation of some hyperelliptic curves algorithms using MIRACL
• DJ Bernstein Surface1271: high-speed genus-2-hyperelliptic-curve cryptography – incomplete work from 2006 intending to produce a Diffie–Hellman variant, but stalled due to difficulties in choosing surfaces (in turn because point-counting for large surfaces is unavailable). Contains software for the Pentium M scalar multiplication on a Kummer surface.