The Merkle–Hellman knapsack cryptosystem was one of the earliest public key cryptosystems. It was published by Ralph Merkle and Martin Hellman in 1978. A polynomial time attack was published by Adi Shamir in 1984. As a result, the cryptosystem is now considered insecure.^{[1]: 465  [2]: 190}

The concept of public key cryptography was introduced by Whitfield Diffie and Martin Hellman in 1976.^[3] At that time they proposed the general concept of a "trap-door one-way function", a function whose inverse is computationally infeasible to calculate without some secret "trap-door information"; but they had not yet found a practical example of such a function. Several specific public-key cryptosystems were then proposed by other researchers over the next few years, such as RSA in 1977 and Merkle-Hellman in 1978.^[4]

Merkle–Hellman is a public key cryptosystem, meaning that two keys are used, a public key for encryption and a private key for decryption. It is based on the subset sum problem (a special case of the knapsack problem).^[5] The problem is as follows: given a set of integers ${\displaystyle A}$ and an integer ${\displaystyle c}$, find a subset of ${\displaystyle A}$ which sums to ${\displaystyle c}$. In general, this problem is known to be NP-complete. However, if ${\displaystyle A}$ is superincreasing, meaning that each element of the set is greater than the sum of all the numbers in the set lesser than it, the problem is "easy" and solvable in polynomial time with a simple greedy algorithm.

In Merkle–Hellman, decrypting a message requires solving an apparently "hard" knapsack problem. The private key contains a superincreasing list of numbers ${\displaystyle W}$, and the public key contains a non-superincreasing list of numbers ${\displaystyle B}$, which is actually a "disguised" version of ${\displaystyle W}$. The private key also contains some "trapdoor" information that can be used to transform a hard knapsack problem using ${\displaystyle B}$ into an easy knapsack problem using ${\displaystyle W}$.

Unlike some other public key cryptosystems such as RSA, the two keys in Merkle-Hellman are not interchangeable; the private key cannot be used for encryption. Thus Merkle-Hellman is not directly usable for authentication by cryptographic signing, although Shamir published a variant that can be used for signing.^[6]

1. Choose a block size ${\displaystyle n}$. Integers up to ${\displaystyle n}$ bits in length can be encrypted with this key.

2. Choose a random superincreasing sequence of ${\displaystyle n}$ positive integers

${\displaystyle W=(w_{1},w_{2},\dots ,w_{n})}$

The superincreasing requirement means that ${\displaystyle w_{k}>\sum _{i=1}^{k-1}w_{i}}$, for ${\displaystyle 1<k\leq n}$.

3. Choose a random integer ${\displaystyle q}$ such that

${\displaystyle q>\sum _{i=1}^{n}w_{i}}$

4. Choose a random integer ${\displaystyle r}$ such that ${\displaystyle \gcd(r,q)=1}$ (that is, ${\displaystyle r}$ and ${\displaystyle q}$ are coprime).

5. Calculate the sequence

${\displaystyle B=(b_{1},b_{2},\dots ,b_{n})}$

where ${\displaystyle b_{i}=rw_{i}{\bmod {q}}}$.

The public key is ${\displaystyle B}$ and the private key is ${\displaystyle (W,q,r)}$.

Let ${\displaystyle m}$ be an ${\displaystyle n}$-bit message consisting of bits ${\displaystyle m_{1}m_{2}\dots m_{n}}$, with ${\displaystyle m_{1}}$ the highest order bit. Select each ${\displaystyle b_{i}}$ for which ${\displaystyle m_{i}}$ is nonzero, and add them together. Equivalently, calculate

${\displaystyle c=\sum _{i=1}^{n}m_{i}b_{i}}$.

The ciphertext is ${\displaystyle c}$.

To decrypt a ciphertext ${\displaystyle c}$, we must find the subset of ${\displaystyle B}$ which sums to ${\displaystyle c}$. We do this by transforming the problem into one of finding a subset of ${\displaystyle W}$. That problem can be solved in polynomial time since ${\displaystyle W}$ is superincreasing.

1. Calculate the modular inverse of ${\displaystyle r}$ modulo ${\displaystyle q}$ using the Extended Euclidean algorithm. The inverse will exist since ${\displaystyle r}$ is coprime to ${\displaystyle q}$.

${\displaystyle r':=r^{-1}{\pmod {q}}}$

The computation of ${\displaystyle r'}$ is independent of the message, and can be done just once when the private key is generated.

2. Calculate

${\displaystyle c':=cr'{\bmod {q}}}$

3. Solve the subset sum problem for ${\displaystyle c'}$ using the superincreasing sequence ${\displaystyle W}$, by the simple greedy algorithm described below. Let ${\displaystyle X=(x_{1},x_{2},\dots ,x_{k})}$ be the resulting list of indexes of the elements of ${\displaystyle W}$ which sum to ${\displaystyle c'}$. (That is, ${\displaystyle c'=\sum _{i=1}^{k}w_{x_{i}}}$.)

4. Construct the message ${\displaystyle m}$ with a 1 in each ${\displaystyle x_{i}}$ bit position and a 0 in all other bit positions:

${\displaystyle m=\sum _{i=1}^{k}2^{n-x_{i}}}$

This simple greedy algorithm finds the subset of a superincreasing sequence ${\displaystyle W}$ which sums to ${\displaystyle c'}$, in polynomial time:

1. Initialize ${\displaystyle X}$ to an empty list.

2. Find the largest element in ${\displaystyle W}$ which is less than or equal to ${\displaystyle c'}$, say ${\displaystyle w_{j}}$.

3. Subtract: ${\displaystyle c':=c'-w_{j}}$.

4. Append ${\displaystyle j}$ to the list ${\displaystyle X}$.

5. If ${\displaystyle c'}$ is greater than zero, return to step 2.

Create a key to encrypt 8-bit numbers by creating a random superincreasing sequence of 8 values:

${\displaystyle W=(2,7,11,21,42,89,180,354)}$

The sum of these is 706, so select a larger value for ${\displaystyle q}$:

${\displaystyle q=881}$.

Choose ${\displaystyle r}$ to be coprime to ${\displaystyle q}$:

${\displaystyle r=588}$.

Construct the public key ${\displaystyle B}$ by multiplying each element in ${\displaystyle W}$ by ${\displaystyle r}$ modulo ${\displaystyle q}$:

${\displaystyle {\begin{aligned}&(2*588){\bmod {8}}81=295\\&(7*588){\bmod {8}}81=592\\&(11*588){\bmod {8}}81=301\\&(21*588){\bmod {8}}81=14\\&(42*588){\bmod {8}}81=28\\&(89*588){\bmod {8}}81=353\\&(180*588){\bmod {8}}81=120\\&(354*588){\bmod {8}}81=236\end{aligned}}}$

Hence ${\displaystyle B=(295,592,301,14,28,353,120,236)}$.

Let the 8-bit message be ${\displaystyle m=97=01100001_{2}}$. We multiply each bit by the corresponding number in ${\displaystyle B}$ and add the results:

  0 * 295
+ 1 * 592
+ 1 * 301
+ 0 * 14
+ 0 * 28
+ 0 * 353
+ 0 * 120
+ 1 * 236
    = 1129

The ciphertext ${\displaystyle c}$ is 1129.

To decrypt 1129, first use the Extended Euclidean Algorithm to find the modular inverse of ${\displaystyle r}$ mod ${\displaystyle q}$:

${\displaystyle r'=r^{-1}{\bmod {q}}=588^{-1}{\bmod {8}}81=442}$.

Compute ${\displaystyle c'=cr'{\bmod {q}}=1129*442{\bmod {8}}81=372}$.

Use the greedy algorithm to decompose 372 into a sum of ${\displaystyle w_{i}}$ values:

${\displaystyle {\begin{aligned}c'&=372\\&w_{8}=354\leq 372\\c'&=372-354=18\\&w_{3}=11\leq 18\\c'&=18-11=7\\&w_{2}=7\leq 7\\c'&=7-7=0\end{aligned}}}$

Thus ${\displaystyle 372=354+11+7=w_{8}+w_{3}+w_{2}}$, and the list of indexes is ${\displaystyle X=(8,3,2)}$. The message can now be computed as

${\displaystyle m=\sum _{i=1}^{3}2^{n-x_{i}}=2^{8-8}+2^{8-3}+2^{8-2}=1+32+64=97}$.

This section needs expansion. You can help by adding to it. (September 2020)

In 1984 Adi Shamir published an attack on the Merkle-Hellman cryptosystem which can decrypt encrypted messages in polynomial time without using the private key. ^[7] The attack analyzes the public key ${\displaystyle B=(b_{1},b_{2},\dots ,b_{n})}$ and searches for a pair of numbers ${\displaystyle u}$ and ${\displaystyle m}$ such that ${\displaystyle (ub_{i}{\bmod {m}})}$ is a superincreasing sequence. The ${\displaystyle (u,m)}$ pair found by the attack may not be equal to ${\displaystyle (r',q)}$ in the private key, but like that pair it can be used to transform a hard knapsack problem using ${\displaystyle B}$ into an easy problem using a superincreasing sequence. The attack operates solely on the public key; no access to encrypted messages is necessary.

Shamir's attack on the Merkle-Hellman cryptosystem works in polynomial time even if the numbers in the public key are randomly shuffled, a step which is usually not included in the description of the cryptosystem, but can be helpful against some more primitive attacks.