Zstd

".zst" redirects here. For the game console emulator also using this file extension, see ZSNES.

Zstandard
Original author(s)Yann Collet
Developer(s)Yann Collet, Nick Terrell, Przemysław Skibiński[1]
Initial release23 January 2015 (2015-01-23)
Stable release
1.5.6[2] Edit this on Wikidata / 27 March 2024; 7 months ago (27 March 2024)
Repository
Written inC
Operating systemCross-platform
PlatformPortable
TypeData compression
LicenseBSD-3-Clause or GPL-2.0-or-later (dual-licensed)
Websitefacebook.github.io/zstd/ Edit this on Wikidata

Zstandard is a lossless data compression algorithm developed by Yann Collet at Facebook. Zstd is the corresponding reference implementation in C, released as open-source software on 31 August 2016.[3][4]

Features

Zstandard was designed to give a compression ratio comparable to that of the DEFLATE algorithm (developed in 1991 and used in the original ZIP and gzip programs), but faster, especially for decompression. It is tunable with compression levels ranging from negative 7 (fastest)[5] to 22 (slowest in compression speed, but best compression ratio).

Starting from version 1.3.2 (October 2017), zstd optionally implements very long range search and deduplication (--long, 128 MiB window) similar to rzip or lrzip.[6]

Compression speed can vary by a factor of 20 or more between the fastest and slowest levels, while decompression is uniformly fast, varying by less than 20% between the fastest and slowest levels.[7] The Zstandard command-line has an "adaptive" (--adapt) mode that varies compression level depending on I/O conditions, mainly how fast it can write the output.

Zstd at its maximum compression level gives a compression ratio close to lzma, lzham, and ppmx, and performs better[vague] than lza or bzip2.[improper synthesis?][8][9] Zstandard reaches the current Pareto frontier, as it decompresses faster than any other currently available algorithm with similar or better compression ratio.[10][11]

Dictionaries can have a large impact on the compression ratio of small files, so Zstandard can use a user-provided compression dictionary. It also offers a training mode, able to generate a dictionary from a set of samples.[12][13] In particular, one dictionary can be loaded to process large sets of files with redundancy between files, but not necessarily within each file, e.g., log files.

Design

Zstandard combines a dictionary-matching stage (LZ77) with a large search window and a fast entropy-coding stage. It uses both Huffman coding (used for entries in the Literals section)[14] and finite-state entropy (FSE) - a fast tabled version of ANS, tANS, used for entries in the Sequences section. Because of the way that FSE carries over state between symbols, decompression involves processing symbols within the Sequences section of each block in reverse order (from last to first).

Usage

Zstandard
Filename extension
.zst[15]
Internet media type
application/zstd[15]
Magic number28 b5 2f fd[15]
Type of formatData compression
StandardRFC 8878
Websitegithub.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md
Zstandard Dictionary
Internet media typeapplication/zstd
Magic number37 a4 30 ec[15]
StandardRFC 8878
Websitegithub.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#dictionary-format

The Linux kernel has included Zstandard since November 2017 (version 4.14) as a compression method for the btrfs and squashfs filesystems.[16][17][18]

In 2017, Allan Jude integrated Zstandard into the FreeBSD kernel,[19] and it was subsequently integrated as a compressor option for core dumps (both user programs and kernel panics). It was also used to create a proof-of-concept OpenZFS compression method[7] which was integrated in 2020.[20]

The AWS Redshift and RocksDB databases include support for field compression using Zstandard.[21]

In March 2018, Canonical tested[22] the use of zstd as a deb package compression method by default for the Ubuntu Linux distribution. Compared with xz compression of deb packages, zstd at level 19 decompresses significantly faster, but at the cost of 6% larger package files. Support was added to Debian (and subsequently, Ubuntu) in April 2018 (in version 1.6~rc1).[23][22][24]

In 2018 the algorithm was published as RFC 8478, which also defines an associated media type "application/zstd", filename extension "zst", and HTTP content encoding "zstd".[25]

Arch Linux added support for zstd as a package compression method in October 2019 with the release of the pacman 5.2 package manager[26] and in January 2020 switched from xz to zstd for the packages in the official repository. Arch uses zstd -c -T0 --ultra -20 -, the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 14 times faster, decompression memory increased by 50 MiB when using multiple threads, compression memory increases but scales with the number of threads used.[27][28][29] Arch Linux later also switched to zstd as default compression algorithm for mkinitcpio initial ramdisk generator.[30]

Fedora added ZStandard support to RPM in May 2018 (Fedora release 28) and used it for packaging the release in October 2019 (Fedora 31).[31] In Fedora 33, the filesystem is compressed by default with zstd.[32][33]

Full implementation of the algorithm with an option to choose the compression level is used in the .NSZ/.XCZ[34] file formats developed by the homebrew community for the Nintendo Switch hybrid game console.[35] Similarly, it is also one of many supported compression algorithms in the .RVZ Wii and GameCube disc image file format.

On 15 June 2020, Zstandard was implemented in version 6.3.8 of the zip file format with codec number 93, deprecating the previous codec number of 20 as it was implemented in version 6.3.7, released on 1 June.[36][37]

In March 2024, Google Chrome version 123 (and Chromium-based browsers such as Brave or Microsoft Edge) added zstd support in the HTTP header Content-Encoding.[38] In May 2024, Firefox release 126.0 added zstd support in the HTTP header Content-Encoding.[39]

License

The reference implementation is licensed under the BSD license, published at GitHub.[40] Since version 1.0, it had an additional Grant of Patent Rights.[41]

From version 1.3.1,[42] this patent grant was dropped and the license was changed to a BSD + GPLv2 dual license.[43]

See also

References

  1. ^ "Contributors to facebook/zstd". github.com. Archived from the original on 27 January 2021. Retrieved 26 January 2021.
  2. ^ "Release Zstandard v1.5.6 - Chrome Edition · facebook/zstd". Retrieved 27 March 2024.
  3. ^ Sergio De Simone (2 September 2016). "Facebook Open-Sources New Compression Algorithm Outperforming Zlib". InfoQ. Retrieved 20 April 2019.
  4. ^ "Life imitates satire: Facebook touts zlib killer just like Silicon Valley's Pied Piper". The Register. 31 August 2016. Retrieved 6 September 2016.
  5. ^ "Release Zstandard v1.3.4 - faster everything · facebook/zstd". GitHub. Retrieved 27 March 2024.
  6. ^ "Command Line Interface for Zstandard library". GitHub. 28 October 2021.
  7. ^ a b "ZStandard in ZFS" (PDF). open-zfs.org. 2017. Retrieved 20 April 2019.
  8. ^ Matt Mahoney. "Silesia Open Source Compression Benchmark". Retrieved 10 May 2019.
  9. ^ Matt Mahoney (29 August 2016). "Large Text Compression Benchmark, .2157 zstd". Retrieved 1 September 2016.
  10. ^ TurboBench: Static/Dynamic web content compression benchmark, PowTurbo
  11. ^ Matt Mahoney, Silesia Open Source Compression Benchmark
  12. ^ "Facebook developers report massive speedups and compression ratio improvements when using dictionaries" (PDF). Fermilab. 11 October 2017. Retrieved 27 March 2024.
  13. ^ "Smaller and faster data compression with Zstandard". Facebook. 31 August 2016.
  14. ^ "facebook/zstd". GitHub. 28 October 2021.
  15. ^ a b c d Collet, Yann (February 2021). Kucherawy, Murray S. (ed.). Zstandard Compression and the application/zstd Media Type. Internet Engineering Task Force Request for Comments. doi:10.17487/RFC8878. RFC 8878. Retrieved 26 February 2023.
  16. ^ Corbet, Jonathan (17 September 2017). "The rest of the 4.14 merge window [LWN.net]". lwn.net. Retrieved 27 March 2024.
  17. ^ "Linux_4.14 - Linux Kernel Newbies". Kernelnewbies.org. 30 December 2017. Retrieved 16 August 2018.
  18. ^ Larabel, Michael (8 September 2017). "Zstd Compression For Btrfs & Squashfs Set For Linux 4.14, Already Used Within Facebook - Phoronix". www.phoronix.com.
  19. ^ "Integrate ZSTD into the kernel · freebsd/Freebsd-SRC@28ef165". GitHub.
  20. ^ "Add ZSTD support to ZFS · openzfs/ZFS@10b3c7f". GitHub.
  21. ^ "Zstandard Encoding - Amazon Redshift". 20 April 2019.
  22. ^ a b Larabel, Michael (12 March 2018). "Canonical Working On Zstd-Compressed Debian Packages For Ubuntu". phoronix.com. Phoronix Media. Retrieved 29 October 2019. The developers at Canonical are considering a feature freeze exception to get this newly-developed Zstd Apt/Dpkg support in Ubuntu 18.04 LTS. In doing so, they mention they would be looking at enabling Zstd compression for packages by default in Ubuntu 18.10.
  23. ^ "New Ubuntu Installs Could Be Speed Up by 10% with the Zstd Compression Algorithm". Softpedia. 12 March 2018. Retrieved 13 August 2018.
  24. ^ "Debian Changelog for apt". Debian. 19 April 2021. Retrieved 7 November 2022.
  25. ^ Collet, Yann (October 2018). Kucherawy, Murray S. (ed.). Zstandard Compression and the application/zstd Media Type. Internet Engineering Task Force Request for Comments. doi:10.17487/RFC8478. RFC 8478. Retrieved 7 October 2020.
  26. ^ Larabel, Michael (16 October 2019). "Arch Linux Nears Roll-Out of ZSTD Compressed Packages for Faster Pacman Installs". Phoronix.
  27. ^ Broda, Mara (4 January 2020). "Now using Zstandard instead of xz for package compression". Arch Linux. Retrieved 5 January 2020.
  28. ^ Broda, Mara (25 March 2019). "RFC: (devtools) Changing default compression method to zstd". arch-dev-public (Mailing list).
  29. ^ Broda, Mara; Polyak, Levente (27 December 2019). "makepkg.conf: change default compression method to zstd". GitHub.
  30. ^ Razzolini, Giancarlo (19 February 2021). "News: Moving to Zstandard images by default on mkinitcpio". Arch Linux. Retrieved 28 December 2021.
  31. ^ "Changes/Switch RPMS to ZSTD compression". Fedora Project Wiki.
  32. ^ "Fedora Workstation 34 feature focus: Btrfs transparent compression". Fedora Magazine. 14 April 2021. Retrieved 12 May 2022.
  33. ^ "Changes/BtrfsTransparentCompression". Fedora Project Wiki. Retrieved 12 May 2022.
  34. ^ "RELEASE - nsZip - NSP compressor/decompressor to reduce storage". GBAtemp.net - The Independent Video Game Community. 20 October 2019. Retrieved 3 November 2019.
  35. ^ Bosshard, Nico (31 October 2019), nsZip is a tool to compress/decompress Nintendo Switch games using the here specified NSZ file format: nicoboss/nsZip, retrieved 3 November 2019
  36. ^ APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.8, 15 June 2020, retrieved 7 July 2020
  37. ^ APPNOTE.TXT - .ZIP File Format Specification Version: 6.3.7, 1 June 2020, retrieved 6 June 2020
  38. ^ "New in Chrome 123 | Chrome Blog". Chrome for Developers. 19 March 2024. Retrieved 16 April 2024.
  39. ^ "Firefox 126.0, See All New Features, Updates and Fixes". Retrieved 15 May 2024.
  40. ^ "Facebook open sources Zstandard data compression algorithm, aims to replace technology behind Zip". ZDnet. 31 August 2016. Retrieved 1 September 2016.
  41. ^ "zstd/PATENTS at v1.3.0 · facebook/zstd". GitHub. Retrieved 27 March 2024.
  42. ^ "Release Zstandard v1.3.1 · facebook/zstd". GitHub. Retrieved 27 March 2024.
  43. ^ "New license by Cyan4973 · Pull Request #801 · facebook/zstd". GitHub. Retrieved 27 March 2024.

Read other articles:

Dangun

DangunLukisan Dangun karya pelukis Chae Yong-sin (1850-1941 M)Nama KoreaHangul단군왕검 Hanja檀君王儉 Alih AksaraDangun WanggeomMcCune–ReischauerTan'gun Wanggŏm Dangun Wanggeom adalah seorang pendiri Gojoseon, kerajaan pertama bangsa Korea yang meliputi wilayah Liaoning di Manchuria dan Semenanjung Korea saat ini.[1][2] Berdasarkan beberapa sumber sejarah Tiongkok kuno yang menuliskan tentang sejarah Korea, Dangun adalah pendiri kerajaan Gojoseon, yang diperkirakan t...

 

Juliarti Rahayu Gunawan

Juliarti Rahayu GunawanLahirJuliarti Rahayu Gunawan20 November 1960 (umur 63)Jakarta, IndonesiaNama lainJuliarti RahayuAlmamaterUniversitas TarumanegaraTinggi1,78 m (5 ft 10 in)Pemenang kontes kecantikanGelarMiss Universe Indonesia 1976Warna rambutHitamWarna mataCokelatKompetisiutamaMiss Universe Indonesia 1976(Pemenang)Miss Universe 1976(Tidak ditempatkan) Juliarti Rahayu Gunawan (lahir 20 November 1960), yang lebih dikenal sebagai Juliarti Rahayu adalah seorang akt...

 

Keuskupan Clogher (Katolik)

Untuk keuskupan bernama sama dalam Gereja Irlandia, lihat Keuskupan Clogher (Gereja Irlandia). Keuskupan ClogherDioecesis ClogheriensisDeoise ChlochairKatolik Katedral St Macartan, MonaghanLokasiNegara Irlandia Utara Republik IrlandiaProvinsi gerejawiProvinsi ArmaghStatistikLuas3.456 sq mi (8.950 km2)Populasi- Total- Katolik(per 2016)11370088,400 (77.7%)InformasiDenominasiKatolik RomaRitusRitus LatinKatedralKatedral St Macartan, MonaghanPelindungSt...

ماريا إسابيل غارثيا سواريث

هذه المقالة يتيمة إذ تصل إليها مقالات أخرى قليلة جدًا. فضلًا، ساعد بإضافة وصلة إليها في مقالات متعلقة بها. (مايو 2021) ماريا إسابيل غارثيا سواريث معلومات شخصية الميلاد 26 مارس 1978 (العمر 46 سنة)أفيليس مواطنة إسبانيا  الطول 169 سنتيمتر  الوزن 58 كيلوغرام  الحياة العملية المهن...

 

Organisasi Kerja Sama dan Pembangunan Ekonomi

Organisasi Kerja Sama dan Pembangunan EkonomiOrganisation de coopération et de développement économiques     Negara anggota pendiri (1961)     Negara anggota lainSingkatanOECDOCDETanggal pendirian16 April 1948; 75 tahun lalu (1948-04-16) (bernama OEEC)aDireformasi tahun September 1961 (1961-09) (bernama OECD)TipeOrganisasi antarpemerintahKantor pusatParis, PrancisJumlah anggota 38 negara[1]  Australia  Austria &#...

 

Hamblen County, Tennessee

County in Tennessee, United States County in TennesseeHamblen CountyCountyHamblen County Courthouse in Morristown LogoMotto(s): Service, Community, IndustryLocation within the U.S. state of TennesseeTennessee's location within the U.S.Coordinates: 36°13′N 83°16′W / 36.22°N 83.27°W / 36.22; -83.27Country United StatesState TennesseeFounded1870Named forHezekiah Hamblen[1]SeatMorristownLargest cityMorristownGovernment • MayorBill...

Internazionali Femminili di Palermo 2013 - Doppio

Internazionali Femminili di Palermo 2013Doppio Sport Tennis Vincitori Kristina Mladenovic Katarzyna Piter Finalisti Karolína Plíšková Kristýna Plíšková Punteggio 6-1, 5-7, [10-8] Tornei Singolare Singolare (q)   Doppio Doppio 2012 2014 Voce principale: Internazionali Femminili di Palermo 2013. Il doppio dell'Internazionali Femminili di Palermo 2013 è stato un torneo di tennis facente parte del WTA Tour 2013. Renata Voráčová e Barbora Záhlavová-Strýcová erano le detentric...

 

Mujin wakusei Survive

Mujin wakusei Survive無人惑星サヴァイヴ(Mujin wakusei Survive)Scena di un episodio Generefantascienza, avventura Serie TV animeAutoreYuichiro Yano StudioMadhouse ReteAnimax, NHK 1ª TV16 ottobre 2003 – 28 ottobre 2004 Episodi52 (completa) Mujin wakusei Survive (無人惑星サヴァイヴ?, Mujin wakusei Savaivu, lett. Sopravvivere sul pianeta disabitato o Il pianeta disabitato Survive), è una serie animata prodotta in Giappone su ...

 

Baret

Baret merah Baret adalah sebuah topi berbentuk bulat, pipih, dan lembut, biasanya terbuat dari tenunan, wol rajutan tangan, katun rajutan, flanel wol,[1] atau serat akrilik. Produksi massal baret dimulai pada abad ke-19 di Prancis dan Spanyol, negara-negara yang tetap terkait. Baret biasanya dipakai sebagai bagian dari banyak seragam unit militer dan polisi di seluruh dunia, serta oleh organisasi lainnya.[2] Penggunaan Sebagai penutup kepala seragam Kepraktisan baret telah lam...

Elek Gyula Aréna

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these template messages) This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed.Find sources: Elek Gyula Aréna – news · newspapers · books · scholar · JSTOR (June 2020) (Learn how and when to remove t...

 

Championnat du Guatemala de football 1967-1968

Liga Nacional de Guatemala1967-1968 Généralités Sport Football Organisateur(s) FNFG Édition 16e Lieu(x) Guatemala Date du Jour inconnu 1967au Jour inconnu 1968 Participants 10 équipes Matchs joués 135 Site web officiel Site officiel Hiérarchie Hiérarchie 1er échelon Niveau inférieur Primera División Palmarès Tenant du titre Aurora FC Vainqueur Aurora FC Deuxième CSD Municipal Troisième Xelaju MC Relégué(s) Deportivo Botrán Navigation Saison précédente Saison suivante ...

 

4th Secretariat of the Workers' Party of Korea

This article relies largely or entirely on a single source. Relevant discussion may be found on the talk page. Please help improve this article by introducing citations to additional sources.Find sources: 4th Secretariat of the Workers' Party of Korea – news · newspapers · books · scholar · JSTOR (February 2021) The 4th Secretariat of the Workers' Party of Korea (WPK)(4차 조선로동당 비서국), officially the Secretariat of the 4th Congress of th...

Hovnanadzor

Hovnanadzor (hy) Հովնանաձոր Administration Pays Arménie Région Lorri Maire Mandat Karen Abovyan[1] 2012-2016 Démographie Population 80 hab. (2008) Densité 9,2 hab./km2 Géographie Coordonnées 41° 01′ 54″ nord, 44° 31′ 37″ est Superficie 870 ha = 8,7 km2 Fuseau horaire UTC+4 Localisation Géolocalisation sur la carte : Arménie Hovnanadzor Géolocalisation sur la carte : Arménie Hovnanadzor Géoloc...

 

Cigaritis nilus

Species of butterfly Cigaritis nilus Figures 11 and 12 Scientific classification Domain: Eukaryota Kingdom: Animalia Phylum: Arthropoda Class: Insecta Order: Lepidoptera Family: Lycaenidae Genus: Cigaritis Species: C. nilus Binomial name Cigaritis nilus(Hewitson, 1865)[1] Synonyms Aphnaeus nilus Hewitson, 1865 Apharitis nilus Aphnaeus subaureus Grose-Smith, 1898 Spindasis kaduglii Bethune-Baker, 1916 Spindasis subaurea f. sabulosa Hawker-Smith, 1929 Cigaritis Nilus, the Saharan S...

 

Trams in Kingston upon Hull

Historic tram system in Kingston upon Hull See also: Trolleybuses in Kingston upon Hull Plan of the Hull tram system The Kingston upon Hull tramway network was a network of 4 ft 8+1⁄2 in (1,435 mm) standard gauge tram lines following the five main roads radially out of the city centre of Kingston upon Hull, East Riding of Yorkshire, England. Two of these lines went west, and two east. The fifth went to the north, and branched to include extra lines serving suburban ...

Bobcat

Medium-sized North American wild cat For other uses, see Bobcat (disambiguation). Red Lynx redirects here. For the wrestler, see Candy Okutsu. For the video game developer, see RedLynx. Bobcat Bobcat in Columbus Zoo Conservation status Least Concern  (IUCN 3.1)[1] CITES Appendix II (CITES)[1] Scientific classification Domain: Eukaryota Kingdom: Animalia Phylum: Chordata Class: Mammalia Order: Carnivora Suborder: Feliformia Family: Felidae Subfamily: Felinae Genus: Ly...

 

Constantine II of Kakheti

King of Kakheti Constantine IIKing of Kakheti (more...) Reign1722–1732PredecessorDavid II of KakhetiSuccessorTeimuraz II of KakhetiBornIsfahan, Safavid IranDiedDecember 28, 1732DynastyBagrationi dynastyFatherHeraclius I of KakhetiReligionIslam Constantine II (Georgian: კონსტანტინე II) (died December 28, 1732), also known as Mahmād Qulī Khān (მაჰმად ყული-ხანი) in Iran, was a king (mepe) of Kakheti in eastern Georgia of the Bagrationi d...

 

National Republican Party

Historical American political party This article is about the American political party. For other uses, see National Republican Party (disambiguation). National Republican redirects here. For the group founded in 2019, see Never Trump movement § Developments following the 2016 election. For the former newspaper, see National Republican (newspaper). Not to be confused with Republican Party (United States), National Union Party (United States), or Democratic-Republican Party. This article...

Virginia Central Railroad

Virginia Central Railroad Création 1836 Disparition 1er juillet 1878 Fondateur(s) Agence des travaux publics de Virginie Successeur Chesapeake and Ohio Railway Siège social Richmond, Virginie États-Unis Localisation Virginie modifier  La Virginia Central Railroad est une ancienne compagnie de chemin de fer qui opérait dans l'État de Virginie aux États-Unis. Elle fut fondée en 1836 sous le nom de Louisa Railroad par l'agence des travaux publics de Virginie (« Virginia B...

 

Lake Louise State Park

Not to be confused with Lake Louisa State Park. This article is about the park in Minnesota. For the state recreation area in Alaska, see Lake Louise State Recreation Area. Reservoir in Mower County, MinnesotaLake LouiseLake LouiseShow map of MinnesotaLake LouiseShow map of the United StatesLocationMower County, MinnesotaCoordinates43°31′24″N 092°30′55″W / 43.52333°N 92.51528°W / 43.52333; -92.51528 (Lake Louise)TypereservoirPrimary inflowsLittle Io...