梯度消失问题

梯度消失问题(Vanishing gradient problem)是一种机器学习中的难题,出現在以梯度下降法反向传播训练人工神經網路的時候。在每次訓練的迭代中,神经网路权重的更新值与误差函数偏導數成比例,然而在某些情况下,梯度值会几乎消失,使得权重无法得到有效更新,甚至神經網路可能完全无法继续训练。舉個例子來說明問題起因,一個传统的激勵函数双曲正切函数,其梯度值在 (-1, 1)范围内,反向传播以链式法则来计算梯度。

這樣做的效果,相当于在n層網路中,将n个這些小数字相乘來計算“前端”層的梯度,这就使梯度(误差信号)随着n呈指數遞減,导致前端層的訓練非常緩慢。

反向傳播使研究人員從頭開始訓練監督式深度人工神經網路,最初收效甚微。 1991年賽普·霍克賴特(Hochreiter)的畢業論文[1][2]正式確認了“梯度消失問題”失敗的原因。梯度消失問題不僅影響多層前饋網絡[3]還影響循環網路[4]循環網路是通過將前饋網路深度展開來訓練,在網路處理的輸入序列的每個時間步驟中,都會產生一個新的層。

當所使用的激勵函數之導數可以取較大值時,則可能會遇到相關的梯度爆炸問題(exploding gradient problem)

解決方案

多級層次結構

長短期記憶

更快的硬體

殘差網路(Residual Networks,ResNets)

其他的激活函數

其他

參考文獻

  1. ^ S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut f. Informatik, Technische Univ. Munich, 1991.
  2. ^ S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. In S. C. Kremer and J. F. Kolen, editors, A Field Guide to Dynamical Recurrent Neural Networks. IEEE Press, 2001.
  3. ^ Goh, Garrett B.; Hodas, Nathan O.; Vishnu, Abhinav. Deep learning for computational chemistry. Journal of Computational Chemistry. 2017-06-15, 38 (16): 1291–1307. PMID 28272810. arXiv:1701.04503可免费查阅. doi:10.1002/jcc.24764 (英语). 
  4. ^ Pascanu, Razvan; Mikolov, Tomas; Bengio, Yoshua. On the difficulty of training Recurrent Neural Networks. 2012-11-21. arXiv:1211.5063可免费查阅 [cs.LG]. 

Read other articles:

Bahadur Shah I قطب الدین محمد بہادر شاهBadhishah-I-HindustaniKaisar Mughal ke-7 dari IndiaBerkuasa19 Juni 1707 – 27 Februari 1712Penobatan19 Juni 1707 di DelhiPendahuluMuhammad Azam ShahPenerusJahandar ShahInformasi pribadiKelahiran14 Oktober 1643Burhanpur, Kemaharajaan MughalKematian27 Februari 1712(1712-02-27) (umur 68)Lahore, Kemaharajaan MughalPemakamanMasjid Moti, DelhiNama lengkapAbul-nasr Sayyid Qutb-ud-din Muhammad Shah Alam Bahadur Shah BadshahNama anu...

 

Kanker dan tumor diakibatkan oleh serangkaian mutasi. Setiap mutasi mengubah perilaku sel. Karsinogenesis, juga disebut onkogenesis atau tumorigenesis, adalah proses pembentukan kanker. Proses ini terjadi ketika sel normal berubah menjadi sel kanker. Karsinogenesis memiliki ciri berupa perubahan di tingkatan seluler, genetik, dan epigenetik, serta pembelahan sel yang tidak normal. Pembelahan sel merupakan proses fisiologis yang berlangsung di dalam semua jaringan. Dalam keadaan normal, terdap...

 

Pour la nouvelle de G. Simenon, voir Peine de mort (nouvelle). Si ce bandeau n'est plus pertinent, retirez-le. Cliquez ici pour en savoir plus. Cet article ne cite pas suffisamment ses sources (septembre 2023). Si vous disposez d'ouvrages ou d'articles de référence ou si vous connaissez des sites web de qualité traitant du thème abordé ici, merci de compléter l'article en donnant les références utiles à sa vérifiabilité et en les liant à la section « Notes et références&#...

Microsoft SwiftKeyTangkapan layar Tipeperusahaan bisnis dan input method (en) Versi pertamaJuli 2010; 13 tahun lalu (2010-07)GenrePapan ketik virtualLisensiPerangkat lunak milik peroranganKlasifikasi Alexa139.856 Karakteristik teknisSistem operasiiOS, Android, Windows 10 (versi 1809 sampai 1909)Ukuran24 MB (Android) 139,7 MB (iOS)Informasi pengembangPembuatJon Reynolds Ben MedlockPengembangMicrosoftPenerbitGoogle Play dan App Store Sumber kode Google Playcom.touchtype.swiftkey iTunes Sto...

 

Pour les articles homonymes, voir Ministère du Tourisme. Cet article est une ébauche concernant la politique et le Venezuela. Vous pouvez partager vos connaissances en l’améliorant (comment ?) selon les recommandations des projets correspondants. Consultez la liste des tâches à accomplir en page de discussion. Ministère du Tourisme Situation Type Ministère vénézuélien Organisation Ministre du Tourisme Alí Padrón Site web [1] modifier  Le ministère du Tourisme ou MINT...

 

Language family See Northeastern Neo-Aramaic for the other languages of the larger group. Central Neo-AramaicWestern Neo-SyriacGeographicdistributionMardin and Diyarbakır provinces in Turkey, Qamishli and al Hasakah in SyriaLinguistic classificationAfro-AsiaticSemiticCentral SemiticNorthwest SemiticAramaicEastern AramaicCentral Neo-AramaicSubdivisions Turoyo Mlahsô † Glottologturo1240 Central Neo-Aramaic languages represent a specific group of Neo-Aramaic languages, that is designated as ...

French wine region in the Loire valley Not to be confused with Pouilly-Fuissé from Burgundy. Pouilly FuméWine regionTypeAOCYear established1937Country FrancePart ofNivernaisClimate regionmoderate temperate oceanicSoil conditionskimmeridgian marl and clay with flintsTotal area1,200 hectaresVarietals producedSauvignon blanc[1]Wine producedFrom 65 to 75 hl/ha[2] Pouilly-Fumé is an appellation d'origine contrôlée (AOC) for the dry sauvignon blanc white wine produced arou...

 

Sacra Famiglia di Francesco IAutoreRaffaello Sanzio e aiuti Data1518 Tecnicaolio su tavola trasportata su tela Dimensioni207×140 cm UbicazioneMusée du Louvre, Parigi La Sacra Famiglia di Francesco I è un dipinto a olio su tavola trasportato su tela (207x140 cm) di Raffaello e aiuti, datato 1518 e conservato nel Museo del Louvre di Parigi. L'opera è firmata e datata sull'orlo della veste della Vergine: RAPHAEL VRBINAS S[anti] PINGEBAT MDXVIII. Indice 1 Storia 2 Descrizione e stile 3 B...

 

Public company registered under the corporate law of the European Union This article is part of a series onCorporate law By jurisdiction Anguilla Australia BVI Canada Cayman Islands India South Africa UK United States Vietnam European Union France Germany General corporate forms Company Conglomerate Cooperative Corporation Holding company Joint-stock Partnership General Limited Limited liability Private limited Shell corporation Shelf corporation Sole proprietorship Corporate formsby jurisdic...

森川智之配音演员本名同上原文名森川 智之(もりかわ としゆき)罗马拼音Morikawa Toshiyuki昵称モリモリ[1]、帝王[1]国籍 日本出生 (1967-01-26) 1967年1月26日(57歲) 日本東京都品川區[1](神奈川縣川崎市[2]、橫濱市[3]成長)职业配音員、旁白、歌手、藝人音乐类型J-POP出道作品外國人取向的日語教材代表作品但丁(Devil May Cry)D-boy(宇宙騎...

 

Indian bank Not to be confused with Federal Reserve Bank. Federal Bank LimitedHeadquarters in AluvaCompany typePublicTraded asNSE: FEDERALBNKBSE: 500469LSE: FEDSIndustryBankingFinancial servicesFounded23 April 1931; 93 years ago (1931-04-23)(as Travancore Federal Bank) at Nedumpuram, Thiruvalla2 December 1949; 74 years ago (1949-12-02)(as Federal Bank)FounderK. P. Hormis[1]HeadquartersAluva, Kochi, Kerala, IndiaNumber of locations1,50...

 

Санкт-Петербурзький національний дослідницький університет інформаційних технологій, механіки та оптикиИТМО 59°57′26″ пн. ш. 30°18′28″ сх. д. / 59.95745000002777658° пн. ш. 30.30795277780577734° сх. д. / 59.95745000002777658; 30.30795277780577734Координати: 59°57′26″ пн. ш. 30°18′...

جزء من سلسلة مقالات حولالمطبخ العربي حسب البلد الجزيرة العربية البحرين الإمارات الكويت سلطنة عمان قطر السعودية اليمن المغرب الكبير الجزائر ليبيا موريتانيا المغرب تونس الشام العراق الأردن لبنان فلسطين سوريا نهر النيل مصر السودان القرن الأفريقي جيبوتي الصومال المحيط اله�...

 

Код валюты — относительно короткая последовательность цифр и/или букв, используемая для представления наименования денежной единицы и её идентификации в различных системах передачи информации. Коды валют сгруппированы в классификаторы валют, которые, как правило, �...

 

Peta wilayah Komune Besnate (merah) di Provinsi Varese (emas), Lombardia, Italia. Besnate komune di Italia Tempat Negara berdaulatItaliaDaerah di ItaliaLombardyProvinsi di ItaliaProvinsi Varese NegaraItalia Ibu kotaBesnate PendudukTotal5.475  (2023 )GeografiLuas wilayah7,48 km² [convert: unit tak dikenal]Ketinggian296 m Berbatasan denganCavaria con Premezzo Albizzate Gallarate Jerago con Orago Mornago Arsago Seprio Sumirago SejarahSanto pelindungMartinus dari Tours Informasi tambah...

Distribution of advertising by direct mail or letterbox drop Typical advertising mail Part of a series onPollutionSolid waste after being shredded to a uniform size Air Air quality index Atmospheric dispersion modeling Chlorofluorocarbon Combustion Exhaust gas Haze Global dimming Global distillation Indoor air quality Ozone depletion Particulates Persistent organic pollutant Smog Soot Volatile organic compound Waste Biological Biological hazard Genetic Introduced species Invasive species Digi...

 

English archaeologist Arthur Cruttenden MaceMace working on items from Tutankhamun's tomb, 1923Born(1874-07-17)17 July 1874Glenorchy, TasmaniaDied6 April 1928(1928-04-06) (aged 53)Haywards Heath, Sussex, EnglandNationalityBritishEducationKeble College, OxfordOccupationArchaeologistKnown forWorked in Egypt for the Metropolitan Museum of Art and on Tutankhamun's tombRelativesFlinders Petrie, cousin Mace and Alfred Lucas conserve a chariot from Tutankhamun's tomb, December 1923 Arthur ...

 

This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed.Find sources: Stromboli 1950 film – news · newspapers · books · scholar · JSTOR (November 2022) (Learn how and when to remove this message) 1950 filmStromboliItalian theatrical release posterDirected byRoberto RosselliniScreenplay bySergio AmideiGian Paolo CallegariArt ...

توماس ك. بلات (بالإنجليزية: Thomas C. Platt)‏    معلومات شخصية الميلاد 15 يوليو 1833 [1]  قرية أويغو  الوفاة 6 مارس 1910 (76 سنة) [1]  نيويورك  مواطنة الولايات المتحدة  مناصب عضو مجلس الشيوخ الأمريكي[2]   عضو خلال الفترة4 مارس 1881  – 16 مايو 1881  الدائرة الإنتخ...

 

جزء من سلسلة مقالات حولحقوق الإنسان النظرية حقوق الادعاء وحقوق الحرية حقوق فردية حقوق طبيعية وحقوق قانونية حقوق سلبية وإيجابية حقوق الإنسان المدنية والسياسية الاقتصادية والاجتماعية والثقافية الأجيال الثلاثة الحقوق حسب الفئة الطفل المستهلك الدائنون المتهمون الصم المسن...