DeepSpeed

DeepSpeed

Original author(s)	Microsoft Research
Developer(s)	Microsoft
Initial release	May 18, 2020; 4 years ago (2020-05-18)

Stable release	v0.14.4 / June 21, 2024; 5 months ago (2024-06-21)

Repository	github.com/microsoft/DeepSpeed
Written in	Python, CUDA, C++
Type	Software library
License	Apache License 2.0
Website	deepspeed.ai

DeepSpeed is an open source deep learning optimization library for PyTorch.^[1]

Library

The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware.^[2]^[3] DeepSpeed is optimized for low latency, high throughput training. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 1 trillion or more parameters.^[4] Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on GitHub.^[5]

The team claimed to achieve up to a 6.2x throughput improvement, 2.8x faster convergence, and 4.6x less communication.^[6]

References

^ "Microsoft Updates Windows, Azure Tools with an Eye on The Future". PCMag UK. May 22, 2020.
^ Yegulalp, Serdar (February 10, 2020). "Microsoft speeds up PyTorch with DeepSpeed". InfoWorld.
^ "Microsoft unveils "fifth most powerful" supercomputer in the world". Neowin. 18 June 2023.
^ "Microsoft trains world's largest Transformer language model". February 10, 2020.
^ "microsoft/DeepSpeed". July 10, 2020 – via GitHub.
^ "DeepSpeed: Accelerating large-scale model inference and training via system optimizations and compression". Microsoft Research. 2021-05-24. Retrieved 2021-06-19.

External links

v t e Deep learning software
Comparison
Open source	Apache MXNet Apache SINGA Caffe Deeplearning4j DeepSpeed Dlib Keras Microsoft Cognitive Toolkit ML.NET OpenNN PyTorch TensorFlow Theano Torch ONNX OpenVINO MindSpore
Proprietary	Apple Core ML IBM Watson Neural Designer Wolfram Mathematica MATLAB Deep Learning Toolbox
Category

Microsoft free and open-source software (FOSS)

Overview

Software

Applications	3D Movie Maker Atom Conference XP Family.Show File Manager Open Live Writer Microsoft PowerToys Terminal Windows Calculator Windows Console Windows Package Manager WorldWide Telescope XML Notepad
Video games	Allegiance
Programming languages	Bosque C# Dafny F# F* GW-BASIC IronPython IronRuby Lean P Power Fx PowerShell Project Verona Q# Small Basic Online TypeScript Visual Basic
Frameworks, development tools	.NET .NET Framework .NET Gadgeteer .NET MAUI .NET Micro Framework AirSim ASP.NET ASP.NET AJAX ASP.NET Core ASP.NET MVC ASP.NET Razor ASP.NET Web Forms Avalonia Babylon.js BitFunnel Blazor C++/WinRT CCF ChakraCore CLR Profiler Dapr DeepSpeed DiskSpd Dryad Dynamic Language Runtime eBPF on Windows Electron Entity Framework Fluent Design System Fluid Framework Infer.NET LightGBM Managed Extensibility Framework Microsoft Automatic Graph Layout Microsoft C++ Standard Library Microsoft Cognitive Toolkit Microsoft Design Language Microsoft Detours Microsoft Enterprise Library Microsoft SEAL mimalloc Mixed Reality Toolkit ML.NET mod_mono Mono MonoDevelop MSBuild MsQuic Neural Network Intelligence npm NuGet OneFuzz Open Management Infrastructure Open Neural Network Exchange Open Service Mesh Open XML SDK Orleans Playwright ProcDump ProcMon Python Tools for Visual Studio R Tools for Visual Studio RecursiveExtractor Roslyn Sandcastle SignalR StyleCop SVNBridge T2 Temporal Prover Text Template Transformation Toolkit TLA+ Toolbox U-Prove vcpkg Virtual File System for Git Voldemort VoTT Vowpal Wabbit Windows App SDK Windows Communication Foundation Windows Driver Frameworks KMDF UMDF Windows Forms Windows Presentation Foundation Windows Template Library Windows UI Library WinJS WinObjC WiX XDP for Windows XSP xUnit.net Z3 Theorem Prover
Operating systems	MS-DOS (v1.25, v2.0 & v4.0) Barrelfish SONiC Azure Linux
Other	ChronoZoom Extensible Storage Engine FlexWiki FourQ Gollum Project Mu ReactiveX SILK TLAPS TPM 2.0 Reference Implementation WikiBhasha

Licenses

Forges

Read other articles:

Terong, Dlingo, Bantul

TerongKalurahanNegara IndonesiaProvinsiDaerah Istimewa YogyakartaKabupatenBantulKecamatanDlingoKode Kemendagri34.02.11.2005 Luas788.671,5 Ha[1]Jumlah penduduk-Kepadatan- Terong (Jawa: Térong) adalah desa di Kecamatan Dlingo, Bantul, Daerah Istimewa Yogyakarta, Indonesia yang terletak paling Utara dari Kecamatan Dlingo. Desa ini merupakan desa yang secara politik paling dinamis dibanding desa-desa lainnya di Kecamatan Dlingo, hal ini bisa ditunjukkan dengan partisipasi politi...

Daftar Ketua Kwartir Nasional Gerakan Pramuka

Ketua Kwartir Nasional Gerakan PramukaLambang Gerakan PramukaPetahanaBudi Wasesosejak 27 Desember 2018Masa jabatan5 tahunDibentuk1961Pejabat pertamaSri Sultan Hamengkubuwono IXSitus webpramuka.or.id Ketua Kwartir Nasional Gerakan Pramuka atau yang biasa disebut dengan Ka Kwarnas adalah Ketua Pengurus satuan organisasi yang mengelola Gerakan Pramuka secara Nasional. Ketua Kwarnas ditetapkan oleh Musyawarah Nasional (Munas) untuk masa bakti berikutnya, dan dilantik oleh Ketua Presidium Pim...

Strada statale 544 di Trinitapoli

Strada statale 544di TrinitapoliLocalizzazioneStato Italia Regioni Puglia DatiClassificazioneStrada statale InizioFoggia FineNSA 113 presso Ponte su fiume Ofanto Lunghezza57,000[1] km Provvedimento di istituzioneD.M. 17/04/1967 - G.U. 151 del 19/06/1967[2] GestoreTratte ANAS: nessuna (dal 2001 la gestione è passata alla Provincia di Foggia; dal 2010 è passato alla Provincia di Barletta-Andria-Trani il tratto competente) Manuale La ex strada statale 544 di Trinitapo...

مالفرين (نيويورك)

مالفرين الإحداثيات 40°40′31″N 73°40′23″W / 40.675277777778°N 73.673055555556°W / 40.675277777778; -73.673055555556 [1] تاريخ التأسيس 20 أبريل 1921 تقسيم إداري البلد الولايات المتحدة[2][3] التقسيم الأعلى مقاطعة ناسو خصائص جغرافية المساحة 2.740048 كيلومتر مربع2.739702 �...

Laika

Soviet dog, first animal to orbit Earth (1957) This article is about the Soviet dog. For the Greek genre of music, see Laïko. For other uses, see Laika (disambiguation). LaikaLaika in a flight harnessOther name(s)Kudryavka (Кудрявка, Curly)SpeciesCanis familiarisBreedMongrel, possibly part-husky (or part-Samoyed) and part-terrierSexFemaleBornLaika (Лайка)c. 1954Moscow, Russian SFSR, Soviet UnionDied3 November 1957(1957-11-03) (aged 2–3)Sputnik 2, in low Earth o...

James Dwight Dana House

Historic house in Connecticut, United States United States historic placeJames Dwight Dana HouseU.S. National Register of Historic PlacesU.S. National Historic LandmarkU.S. Historic districtContributing property James Dwight Dana house in 2009Show map of ConnecticutShow map of the United StatesLocation24 Hillhouse Avenue, New Haven, ConnecticutCoordinates41°18′48.24″N 72°55′26.76″W / 41.3134000°N 72.9241000°W / 41.3134000; -72.9241000Built1849ArchitectHenry...

Promozione Trentino-Alto Adige 1981-1982

Voce principale: Promozione 1981-1982. Promozione1981-1982 Competizione Promozione Sport Calcio Edizione 8ª Organizzatore FIGC - LNDComitato Regionale Trentino-Alto Adige Luogo ItaliaTrentino-Alto Adige Partecipanti 16 Cronologia della competizione 1980-1981 1982-1983 Manuale Nella stagione 1981-1982 la Promozione era sesto livello del calcio italiano (il massimo livello regionale). Qui vi sono le statistiche relative al campionato in Trentino-Alto Adige. Il campionato è strutturato ...

Karim Aga Khan IV

Karim Aga Khan IVAga Khan IV nel 2014Imam degli Ismailiti NizaritiIn carica11 luglio 1957 –in carica PredecessoreAga Khan III Nome completoKarīm al-Husaynī TrattamentoPrincipe NascitaGinevra, 13 dicembre 1936 (87 anni) DinastiaQajar PadrePrincipe Ali Aga Khan MadreTajuddawlah Aga Khan (Joan Barbara Yarde-Buller) ConiugiSalimah Aga Khan (Sarah Frances Croker-Poole)Inaara Aga Khan (Gabriele Thyssen) FigliZahraRahim HusaynAly ReligioneIslam (Nizariti) Karīm al-Husaynī, Āg�...

Inertial electrostatic confinement

Fusion power research concept A fusor, exhibiting nuclear fusion in star mode Inertial electrostatic confinement, or IEC, is a class of fusion power devices that use electric fields to confine the plasma rather than the more common approach using magnetic fields found in magnetic confinement fusion (MCF) designs. Most IEC devices directly accelerate their fuel to fusion conditions, thereby avoiding energy losses seen during the longer heating stages of MCF devices. In theory, this makes them ...

Kais Saied

His ExcellencyKais Saiedقيس سعيدSaied pada tahun 2019 Presiden Tunisia ke-7PetahanaMulai menjabat 23 Oktober 2019Perdana MenteriYoussef ChahedElyes Fakhfakh[1]Hichem Mechichi (ditunjuk)PendahuluBeji Caid EssebsiMohamed Ennaceur (penjabat)PenggantiPetahana Informasi pribadiLahir22 Februari 1958 (umur 66)Tunis, Tunisia[2]KebangsaanTunisiaPartai politikIndependenSuami/istriIchraf ChebilAnak3Alma materUniversitas Tunis Institut Hukum Humaniter InternasionalProfesiY...

East London line

London Overground line Not to be confused with East London Transit. East London lineClass 378 train at Hoxton in 2010, with the City of London skyline in backgroundOverviewStatusOperational[1]OwnerTransport for London (TfL)Network RailLocaleGreater LondonTerminiHighbury & IslingtonDalston JunctionNew CrossCrystal PalaceWest CroydonStations23ServiceTypeSuburban rail, Rapid TransitSystemNational RailServices3Operator(s)London OvergroundDepot(s)New Cross GateRolling stockClass 378 Ca...

1984 World Rally Championship

12th season of the FIA World Rally Championship 1984 World Rally Championship World Drivers' Champion:Stig BlomqvistWorld Manufacturers' Champion:Audi Previous 1983 Next 1985 World RallyChampionship Current season 2024 World Rally Championship 2024 WRC2 2024 WRC3 2024 Junior WRC Support categories Current: WRC2 WRC3 Junior WRC Former: PWRC SWRC FIA 2-Litre Current car classes Rally1 Rally2 Rally3 Rally4 Rally5 Related lists Drivers Champions Female Numbers Co-drivers Champions Manufact...

Parc floral de Paris

EXPO 1969 ParisOverviewBIE-classHorticultural expositionLocationCountryFranceCityParisHorticultural expositionsPreviousVienna 1964 in ViennaNextFloriade 1972 in AmsterdamSpecialized expositionsPreviousHemisFair '68 in San AntonioNextExpo 71 in BudapestUniversal expositionsPreviousExpo 67 in MontrealNextExpo '70 in Osaka The Parc floral de Paris is a public park and botanical garden located within the Bois de Vincennes in the 12th arrondissement of Paris. Created in 1969, the park remains the ...

Euroleague Basketball

Questa voce o sezione sull'argomento competizioni cestistiche non cita le fonti necessarie o quelle presenti sono insufficienti. Puoi migliorare questa voce aggiungendo citazioni da fonti attendibili secondo le linee guida sull'uso delle fonti. Euroleague BasketballSport Pallacanestro Tiposquadre di club FederazioneFIBA Europe ContinenteEuropa OrganizzatoreEuroleague Basketball MottoI feel devotion TitoloVincitore dell'Euroleague Basketball Cadenzaannuale Aperturaottobre Partecipanti18 ...

ABC News Radio

Radio service of ABC News in the United States For the Australian radio service, see ABC NewsRadio. See also: ABC News Now (radio network) This article may need to be rewritten to comply with Wikipedia's quality standards. You can help. The talk page may contain suggestions. (February 2024) ABC News RadioTypeRadio networkCountryUnited StatesHeadquartersABC News World HeadquartersNew York CityUnited StatesBrandingABC NewsOwnershipOwnerABC Audio(ABC News)distributor: Skyview NetworksHistoryLaun...

Сосна смолистая

Запрос «Сосна красная» перенаправляется сюда; о другом виде см. Сосна красная китайская. Сосна смолистая Научная классификация Домен:ЭукариотыЦарство:РастенияПодцарство:Зелёные растенияКлада:Высшие растенияКлада:Сосудистые растенияКлада:Семенные растенияНадотде...

Kuwana

Kuwana-shi 桑名市 Kuwana et la baie d'Ise. Drapeau Administration Pays Japon Région Kansai Préfecture Mie Maire Narutaka Itō Code postal 〒511-8601 Démographie Population 141 458 hab. (septembre 2020) Densité 1 035 hab./km2 Géographie Coordonnées 35° 04′ 00″ nord, 136° 40′ 00″ est Superficie 13 668 ha = 136,68 km2 Localisation Géolocalisation sur la carte : Japon Kuwana-shi Géolocalisation ...

كاديز (كنتاكي)

لمعانٍ أخرى، طالع كاديز (توضيح). كاديز الإحداثيات 36°52′04″N 87°49′03″W / 36.8678°N 87.8175°W / 36.8678; -87.8175 [1] تاريخ التأسيس 1820 تقسيم إداري البلد الولايات المتحدة[2][3] التقسيم الأعلى مقاطعة تريغ عاصمة لـ مقاطعة تريغ خصائص جغرافية ...

後七日御修法

後七日御修法の出仕のため灌頂院へ向かう高僧後七日御修法（ごしちにちみしほ/ごしちにちみしゅほう）とは、毎年正月8日から14日まで東寺で行われる、玉体安穏・鎮護国家・五穀豊穣・万民豊楽を祈る法会[1][2]。元来は宮中真言院で行われた宮中行事で、空海の奏上によって承和2年（835年）に始められた[3][1]。以降、幾たびかの中断などを挟...

De agri cultura

Book by Marcus Porcius Cato Censorius maior De agri cultura (XV sec., Biblioteca Medicea Laurenziana, pluteo 51.2) De agri cultura[a] ([deː ˈaɡriː kʊlˈtuːraː]), also known as On Farming or On Agriculture, is a treatise on Roman agriculture by Cato the Elder. It is the oldest surviving work of Latin prose. Alexander Hugh McDonald, in his article for the Oxford Classical Dictionary, dated this essay's composition to about 160 BC and noted that for all of its lack of form...

PROFILPELAJAR.COM