词法分析

词法分析（英語：lexical analysis）是计算机科学中将字符序列转换为标记（token）序列的过程。进行词法分析的程序或者函数叫作词法分析器（lexical analyzer，简称lexer），也叫扫描器（scanner）。词法分析器一般以函数的形式存在，供语法分析器调用。

标记

这里的标记是一个字串，是构成源代码的最小单位。从输入字符流中生成标记的过程叫作标记化（tokenization），在这个过程中，词法分析器还会对标记进行分类。

词法分析器通常不会关心标记之间的关系（属于语法分析的范畴），举例来说：词法分析器能够将括号识别为标记，但并不保证括号是否匹配。

针对如下C语言表达式： 

sum=3+2;

将其标记化后可以得到下表内容：

语素 标记类型
sum 标识符
= 赋值操作符
3 数字
+ 加法操作符
2 数字
; 语句结束

标记经常使用正则表达式进行定义，像lex一类的词法分析器生成器就支持使用正则表达式。语法分析器读取输入字符流、从中识别出语素、最后生成不同类型的标记。其间一旦发现无效标记，便会报错。

扫描器

词法分析的第一阶段即扫描器，通常基于有限状态自动机。扫描器能够识别其所能处理的标记中可能包含的所有字符序列（单个这样的字符序列即前面所说的“语素”）。例如“整数”标记可以包含所有数字字符序列。很多情况下，根据第一个非空白字符便可以推导出该标记的类型，于是便可逐个处理之后的字符，直到出现不属于该类型标记字符集中的字符（即最长一致原则）。

标记生成器

标记化（tokenization）即将输入字符串分割为标记、进而将标记进行分类的过程。生成的标记随后便被用来进行语法分析。

例如对于如下字符串： The quick brown fox jumps over the lazy dog

计算机并不知道这是以空格分隔的九个英语单词，只知道这是普通的43个字符构成的字符串。可以通过一定的方法（这里即使用空格作为分隔符）将语素（这里即英语单词）从输入字符串中分割出来。分割后的结果用XML可以表示如下： 

<sentence>
  <word>The</word>
  <word>quick</word>
  <word>brown</word>
  <word>fox</word>
  <word>jumps</word>
  <word>over</word>
  <word>the</word>
  <word>lazy</word>
  <word>dog</word>
</sentence>

然而，语素只是一类字符构成的字符串（字符序列），要构建标记，语法分析器需要第二阶段的评估器（Evaluator）。评估器根据语素中的字符序列生成一个“值”，这个“值”和语素的类型便构成了可以送入语法分析器的标记。一些诸如括号的语素并没有“值”，评估器函数便可以什么都不返回。整数、标识符、字符串的评估器则要复杂的多。评估器有时会抑制语素，被抑制的语素（例如空白语素和注释语素）随后不会被送入语法分析器。

例如对于某程序设计语言的源程序片段：

net_worth_future = (assets - liabilities);

在进行语法分析后可能生成以下单词流（空格被抑制）： 

NAME "net_worth_future" 
EQUALS 
OPEN_PARENTHESIS 
NAME "assets" 
MINUS 
NAME "liabilities" 
CLOSE_PARENTHESIS 
SEMICOLON

尽管在某些情况下需要手工编写词法分析器，一般情况下词法分析器都用自动化工具生成。

关联项目

参考文献

Read other articles:

Perubahan budaya

Artikel ini memiliki beberapa masalah. Tolong bantu memperbaikinya atau diskusikan masalah-masalah ini di halaman pembicaraannya. (Pelajari bagaimana dan kapan saat yang tepat untuk menghapus templat pesan ini) artikel ini perlu dirapikan agar memenuhi standar Wikipedia. Tidak ada alasan yang diberikan. Silakan kembangkan artikel ini semampu Anda. Merapikan artikel dapat dilakukan dengan wikifikasi atau membagi artikel ke paragraf-paragraf. Jika sudah dirapikan, silakan hapus templat ini. (Pe...

 

كأس اليونان 1980–81

كأس اليونان 1980–81 تفاصيل الموسم كأس اليونان  النسخة 39  البلد اليونان  المنظم الاتحاد الإغريقي لكرة القدم  البطل نادي أولمبياكوس  عدد المشاركين 58   كأس اليونان 1979–80  كأس اليونان 1981–82  تعديل مصدري - تعديل   كأس اليونان 1980–81 (باليونانية: Κύπελλο Ελλάδος...

 

Cloud Strife

Protagonist in Final Fantasy VII Fictional character Cloud StrifeFinal Fantasy characterCloud Strife artwork by Tetsuya Nomura for Final Fantasy VII.First appearanceFinal Fantasy VIIJanuary 31, 1997Created byYoshinori Kitase, Kazushige Nojima, Tetsuya Nomura, Hironobu SakaguchiDesigned byTetsuya NomuraVoiced by English Steve Burton (2002–18)Cody Christian (2020–present)Major Dodson (2020, young) Japanese Kenyu Horiuchi (Ehrgeiz, arcade version)Nozomu Sasaki (Ehrgeiz, home console version)...

Rooks County, Kansas

County in Kansas, United States County in KansasRooks CountyCountyRooks County Courthouse in Stockton (2014)Location within the U.S. state of KansasKansas's location within the U.S.Coordinates: 39°27′05″N 99°06′07″W / 39.4514°N 99.1019°W / 39.4514; -99.1019Country United StatesState KansasFoundedFebruary 26, 1867Named forJohn C. RooksSeatStocktonLargest cityPlainvilleArea • Total895 sq mi (2,320 km2) • Land89...

 

Франц (герцог Саксен-Кобург-Заальфельдский)

Франц Саксен-Кобург-Заальфельдскийнем. Franz von Sachsen-Coburg-Saalfeld герцог Саксен-Кобург-Заальфельдский 8 сентября 1800 — 9 декабря 1806 Предшественник Эрнст Фридрих Саксен-Кобург-Заальфельдский Преемник Эрнст I Саксен-Кобург-Заальфельдский Рождение 15 июля 1750(1750-07-15)Кобург, Сакс...

 

Sungai Anadyr

Peta Sungai Anadyr Sungai Anadyr (Rusia: Ана́дырь) merupakan sebuah sungai yang terletak di Siberia bagian timur laut Rusia. Sungai ini bermuara dari Pegunungan Stanovoi hingga mulut Teluk Anadyr di Laut Bering. Sungai ini memiliki panjang 1.150 km. Sungai ini mengalami pembekuan terjadi pada bulan Oktober hingga April. Lihat pula Operasi Anadyr Pranala luar Tourist and environmental information Diarsipkan 2010-09-01 di Wayback Machine. Chukotka Autonomous Okrug website, in Engli...

Petrus Canisius Mandagi

Ini adalah nama patrilineal Minahasa, marganya adalah Mandagi (rumpun dari Sulawesi Utara) Yang MuliaPetrus Canisius MandagiM.S.C.Uskup Agung MeraukeGerejaGereja Katolik RomaKeuskupan agungMeraukePenunjukan11 November 2020(3 tahun, 183 hari)PendahuluNicolaus Adi Seputra, M.S.C.ImamatTahbisan imam18 Desember 1975[1] (48 tahun, 146 hari)oleh Theodorus Hubertus Moors, M.S.C.Tahbisan uskup18 September 1994(29 tahun, 237 hari)oleh Andreas Peter Corne...

 

Aleksander Tammert

Questa voce sull'argomento atleti estoni è solo un abbozzo. Contribuisci a migliorarla secondo le convenzioni di Wikipedia. Segui i suggerimenti del progetto di riferimento. Aleksander TammertNazionalità Estonia Altezza197 cm Peso120 kg Atletica leggera SpecialitàLancio del disco SocietàSK Audentes Termine carriera2016 Record Peso 18,05 m (1999) Disco 70,82 m (2006) CarrieraNazionale 1995-2012 Estonia- Palmarès Competizione Ori Argenti Bronzi Giochi olimpici 0 0 1 Europei 0 0 ...

 

Authoring Instructional Materials

Management system AIM was developed for and is primarily used by the United States Navy Authoring Instructional Materials (AIM) is a management system consisting of a set of commercial and government software used by the United States Navy for the development and design of training curricula and instructional content.[1] First proposed in the 1970s,[2] AIM was designed to maximize the efficiency of the curriculum development process through the use of computer-based automation...

Ibn Adlan

Arab cryptologist, linguist and poet ʻAfīf al-DīnʻAlī ibn ʻAdlānal-Mawsili al-Nahwi al-MutarjimBorn1187Mosul, Ayyubid SultanateDied1268 (aged 80–81)Cairo, Mamluk SultanateScientific careerFieldsCryptologyLiteratureArabic languageInstitutionsAl-Salihiyya Mosque, Cairo ʻAfīf al-Dīn ʻAlī ibn ʻAdlān al-Mawsilī (Arabic: عفيف لدين علي بن عدلان الموصلي; 1187–1268 CE), born in Mosul, was an Arab cryptologist, linguist and poet who is known for his ...

 

Death's-head hawkmoth

Species of moth Death's-head hawkmoth Acherontia lachesis Scientific classification Domain: Eukaryota Kingdom: Animalia Phylum: Arthropoda Class: Insecta Order: Lepidoptera Family: Sphingidae Tribe: Acherontiini Genus: AcherontiaLaspeyres, 1809 Species A. atropos — African death's head hawkmoth A. styx — lesser death's head hawkmoth A. lachesis — greater death's head hawkmoth Larva Thorn-like horn on the back The name death's-head hawkmoth refers to any of three moth species of the genu...

 

ミュウツー

この記事は英語版の対応するページを翻訳することにより充実させることができます。（2021年10月）翻訳前に重要な指示を読むには右にある[表示]をクリックしてください。 英語版記事を日本語へ機械翻訳したバージョン（Google翻訳）。 万が一翻訳の手がかりとして機械翻訳を用いた場合、翻訳者は必ず翻訳元原文を参照して機械翻訳の誤りを訂正し、正確な翻訳にし�...

徐光启

此條目需要补充更多来源。 (2023年5月21日)请协助補充多方面可靠来源以改善这篇条目，无法查证的内容可能會因為异议提出而被移除。致使用者：请搜索一下条目的标题（来源搜索：徐光启 — 网页、新闻、书籍、学术、图像），以检查网络上是否存在该主题的更多可靠来源（判定指引）。 此條目的引用需要清理，使其符合格式。 (2023年5月21日)参考文献应符合正确的引�...

 

Athlétisme aux Jeux olympiques d'été de 1948

Pour des articles plus généraux, voir Jeux olympiques d'été de 1948 et Athlétisme aux Jeux olympiques. Athlétisme aux Jeux olympiques d'été de 1948 Arrivée de la finale du 100 m masculinGénéralités Sport Athlétisme Éditions 11e Lieu(x) Londres Date 30 juillet au 7 août 1948 Nations 53 Participants 745 Épreuves 33 Site(s) Stade de Wembley Navigation 1896 • 1900 • 1904 • 1908 • 1912 • 1920 • 1924 • 1928 • 1932 • 1936 • 1948 • 1952 • 1956 • 1960 •...

 

Zecharias Frankel

Bohemian-German rabbi and historian (1801–1875) Zecharias FrankelPersonalBorn(1801-09-30)30 September 1801Prague, BohemiaDied13 February 1875(1875-02-13) (aged 73)Breslau, Silesia, Kingdom of Prussia, Germany(now Wrocław, Poland)ReligionJudaismSpouseRachel Maier Zecharias Frankel, also known as Zacharias Frankel (30 September 1801 – 13 February 1875) was a Bohemian-German rabbi and a historian who studied the historical development of Judaism. He was born in Prague and died in Bresl...

Brescia Calcio 1990-1991

Voce principale: Brescia Calcio. Brescia CalcioStagione 1990-1991 Sport calcio Squadra Brescia Allenatore Bruno Mazzia (1ª-3ª) Bruno Bolchi (4ª-38ª) Presidente Claudio Cremonesi Serie B9º posto Coppa ItaliaSecondo turno Maggiori presenzeCampionato: Giunta, Carnasciali (35) Miglior marcatoreCampionato: Ganz (11)Totale: Ganz (13) StadioStadio Mario Rigamonti Maggior numero di spettatori12 533 vs Verona (7 aprile 1991) Minor numero di spettatori3 397 vs Pescara (12 maggio 19...

 

Max Hirsch (economist)

Australian economist and businessman (1852–1909) For the German labor economist, see Max Hirsch (labor economist). Max HirschBorn(1852-09-21)21 September 1852?CologneDied(1909-03-04)4 March 1909NationalityGerman Australian.Academic careerSchool ortraditionGeorgist, Austrian, ClassicalInfluencesDavid Ricardo, John Stuart Mill, Henry George, Eugen von Böhm-Bawerk.ContributionsIntroduced Austrian theory of capital and interest into the Georgist school; proposed a correction to the Law of...

 

Peureulak Sultanate

Sultanate in SumatraPart of a series on the History of Indonesia Timeline Prehistory Paleolithic Java Man 1,000,000 BP Flores Man 94,000–12,000 BP Neolithic Toba catastrophe 75,000 BP Buni culture 400 BCE Hindu and Buddhist kingdoms Kutai Kingdom 350–1605 Tarumanagara Kingdom 400s–500s Kantoli 400s-500s Kalingga Kingdom 500s–600s Melayu Kingdom 600s–1347 Srivijaya Empire 600s–1025 Shailendra Dynasty 600s–900s Mataram Kingdom 716–1016 Bali Kingdom 914–1908 Sunda Kingd...

South Dakota Coyotes football

College football team of University of South Dakota This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed.Find sources: South Dakota Coyotes football – news · newspapers · books · scholar · JSTOR (January 2017) (Learn how and when to remove this message) South Dakota Coyotes football2024 South Dakota Coyotes football team First...

 

Enki

For other uses, see Enki (disambiguation). God in Sumerian mythology Enki𒀭𒂗𒆠God of creation, intelligence, crafts, fertility, semen, magic, mischiefDetail of Enki from the Adda Seal, an ancient Akkadian cylinder seal dating to circa 2,300 BC[1]SymbolGoat, fish, goat-fish, chimeraGenealogyParentsAn and Nammu[2]SiblingsEnlilConsortNinhursag, DamkinaChildrenMarduk, Dumuzid, Ninsar, Ninkurra, Uttu, NintiEquivalentsGreek equivalentPoseidon,[3] Prometheus[4]...