What Does A Parser Do
Parsing – Wikipedia
Parsing, syntax analysis, or syntactic analysis is the process of analyzing a string of symbols, either in natural language, computer languages or data structures, conforming to the rules of a formal grammar. The term parsing comes from Latin pars (orationis), meaning part (of speech). [1]
The term has slightly different meanings in different branches of linguistics and computer science. Traditional sentence parsing is often performed as a method of understanding the exact meaning of a sentence or word, sometimes with the aid of devices such as sentence diagrams. It usually emphasizes the importance of grammatical divisions such as subject and predicate.
Within computational linguistics the term is used to refer to the formal analysis by a computer of a sentence or other string of words into its constituents, resulting in a parse tree showing their syntactic relation to each other, which may also contain semantic and other information (p-values). [citation needed] Some parsing algorithms may generate a parse forest or list of parse trees for a syntactically ambiguous input. [2]
The term is also used in psycholinguistics when describing language comprehension. In this context, parsing refers to the way that human beings analyze a sentence or phrase (in spoken language or text) “in terms of grammatical constituents, identifying the parts of speech, syntactic relations, etc. “[1] This term is especially common when discussing what linguistic cues help speakers to interpret garden-path sentences.
Within computer science, the term is used in the analysis of computer languages, referring to the syntactic analysis of the input code into its component parts in order to facilitate the writing of compilers and interpreters. The term may also be used to describe a split or separation.
Human languages[edit]
Traditional methods[edit]
The traditional grammatical exercise of parsing, sometimes known as clause analysis, involves breaking down a text into its component parts of speech with an explanation of the form, function, and syntactic relationship of each part. [3] This is determined in large part from study of the language’s conjugations and declensions, which can be quite intricate for heavily inflected languages. To parse a phrase such as ‘man bites dog’ involves noting that the singular noun ‘man’ is the subject of the sentence, the verb ‘bites’ is the third person singular of the present tense of the verb ‘to bite’, and the singular noun ‘dog’ is the object of the sentence. Techniques such as sentence diagrams are sometimes used to indicate relation between elements in the sentence.
Parsing was formerly central to the teaching of grammar throughout the English-speaking world, and widely regarded as basic to the use and understanding of written language. However, the general teaching of such techniques is no longer current. [citation needed]
Computational methods[edit]
In some machine translation and natural language processing systems, written texts in human languages are parsed by computer programs. [4] Human sentences are not easily parsed by programs, as there is substantial ambiguity in the structure of human language, whose usage is to convey meaning (or semantics) amongst a potentially unlimited range of possibilities but only some of which are germane to the particular case. [5] So an utterance “Man bites dog” versus “Dog bites man” is definite on one detail but in another language might appear as “Man dog bites” with a reliance on the larger context to distinguish between those two possibilities, if indeed that difference was of concern. It is difficult to prepare formal rules to describe informal behaviour even though it is clear that some rules are being followed. [citation needed]
In order to parse natural language data, researchers must first agree on the grammar to be used. The choice of syntax is affected by both linguistic and computational concerns; for instance some parsing systems use lexical functional grammar, but in general, parsing for grammars of this type is known to be NP-complete. Head-driven phrase structure grammar is another linguistic formalism which has been popular in the parsing community, but other research efforts have focused on less complex formalisms such as the one used in the Penn Treebank. Shallow parsing aims to find only the boundaries of major constituents such as noun phrases. Another popular strategy for avoiding linguistic controversy is dependency grammar parsing.
Most modern parsers are at least partly statistical; that is, they rely on a corpus of training data which has already been annotated (parsed by hand). This approach allows the system to gather information about the frequency with which various constructions occur in specific contexts. (See machine learning. ) Approaches which have been used include straightforward PCFGs (probabilistic context-free grammars), [6] maximum entropy, [7] and neural nets. [8] Most of the more successful systems use lexical statistics (that is, they consider the identities of the words involved, as well as their part of speech). However such systems are vulnerable to overfitting and require some kind of smoothing to be effective. [citation needed]
Parsing algorithms for natural language cannot rely on the grammar having ‘nice’ properties as with manually designed grammars for programming languages. As mentioned earlier some grammar formalisms are very difficult to parse computationally; in general, even if the desired structure is not context-free, some kind of context-free approximation to the grammar is used to perform a first pass. Algorithms which use context-free grammars often rely on some variant of the CYK algorithm, usually with some heuristic to prune away unlikely analyses to save time. (See chart parsing. ) However some systems trade speed for accuracy using, e. g., linear-time versions of the shift-reduce algorithm. A somewhat recent development has been parse reranking in which the parser proposes some large number of analyses, and a more complex system selects the best option. [citation needed] Semantic parsers convert texts into representations of their meanings. [9]
Psycholinguistics[edit]
In psycholinguistics, parsing involves not just the assignment of words to categories (formation of ontological insights), but the evaluation of the meaning of a sentence according to the rules of syntax drawn by inferences made from each word in the sentence (known as connotation). This normally occurs as words are being heard or read. Consequently, psycholinguistic models of parsing are of necessity incremental, meaning that they build up an interpretation as the sentence is being processed, which is normally expressed in terms of a partial syntactic structure. Creation of initially wrong structures occurs when interpreting garden-path sentences.
Discourse analysis[edit]
Discourse analysis examines ways to analyze language use and semiotic events. Persuasive language may be called rhetoric.
Computer languages[edit]
Parser[edit]
A parser is a software component that takes input data (frequently text) and builds a data structure – often some kind of parse tree, abstract syntax tree or other hierarchical structure, giving a structural representation of the input while checking for correct syntax. The parsing may be preceded or followed by other steps, or these may be combined into a single step. The parser is often preceded by a separate lexical analyser, which creates tokens from the sequence of input characters; alternatively, these can be combined in scannerless parsing. Parsers may be programmed by hand or may be automatically or semi-automatically generated by a parser generator. Parsing is complementary to templating, which produces formatted output. These may be applied to different domains, but often appear together, such as the scanf/printf pair, or the input (front end parsing) and output (back end code generation) stages of a compiler.
The input to a parser is often text in some computer language, but may also be text in a natural language or less structured textual data, in which case generally only certain parts of the text are extracted, rather than a parse tree being constructed. Parsers range from very simple functions such as scanf, to complex programs such as the frontend of a C++ compiler or the HTML parser of a web browser. An important class of simple parsing is done using regular expressions, in which a group of regular expressions defines a regular language and a regular expression engine automatically generating a parser for that language, allowing pattern matching and extraction of text. In other contexts regular expressions are instead used prior to parsing, as the lexing step whose output is then used by the parser.
The use of parsers varies by input. In the case of data languages, a parser is often found as the file reading facility of a program, such as reading in HTML or XML text; these examples are markup languages. In the case of programming languages, a parser is a component of a compiler or interpreter, which parses the source code of a computer programming language to create some form of internal representation; the parser is a key step in the compiler frontend. Programming languages tend to be specified in terms of a deterministic context-free grammar because fast and efficient parsers can be written for them. For compilers, the parsing itself can be done in one pass or multiple passes – see one-pass compiler and multi-pass compiler.
The implied disadvantages of a one-pass compiler can largely be overcome by adding fix-ups, where provision is made for code relocation during the forward pass, and the fix-ups are applied backwards when the current program segment has been recognized as having been completed. An example where such a fix-up mechanism would be useful would be a forward GOTO statement, where the target of the GOTO is unknown until the program segment is completed. In this case, the application of the fix-up would be delayed until the target of the GOTO was recognized. Conversely, a backward GOTO does not require a fix-up, as the location will already be known.
Context-free grammars are limited in the extent to which they can express all of the requirements of a language. Informally, the reason is that the memory of such a language is limited. The grammar cannot remember the presence of a construct over an arbitrarily long input; this is necessary for a language in which, for example, a name must be declared before it may be referenced. More powerful grammars that can express this constraint, however, cannot be parsed efficiently. Thus, it is a common strategy to create a relaxed parser for a context-free grammar which accepts a superset of the desired language constructs (that is, it accepts some invalid constructs); later, the unwanted constructs can be filtered out at the semantic analysis (contextual analysis) step.
For example, in Python the following is syntactically valid code:
The following code, however, is syntactically valid in terms of the context-free grammar, yielding a syntax tree with the same structure as the previous, but is syntactically invalid in terms of the context-sensitive grammar, which requires that variables be initialized before use:
Rather than being analyzed at the parsing stage, this is caught by checking the values in the syntax tree, hence as part of semantic analysis: context-sensitive syntax is in practice often more easily analyzed as semantics.
Overview of process[edit]
The following example demonstrates the common case of parsing a computer language with two levels of grammar: lexical and syntactic.
The first stage is the token generation, or lexical analysis, by which the input character stream is split into meaningful symbols defined by a grammar of regular expressions. For example, a calculator program would look at an input such as “12 * (3 + 4)^2” and split it into the tokens 12, *, (, 3, +, 4, ), ^, 2, each of which is a meaningful symbol in the context of an arithmetic expression. The lexer would contain rules to tell it that the characters *, +, ^, ( and) mark the start of a new token, so meaningless tokens like “12*” or “(3” will not be generated.
The next stage is parsing or syntactic analysis, which is checking that the tokens form an allowable expression. This is usually done with reference to a context-free grammar which recursively defines components that can make up an expression and the order in which they must appear. However, not all rules defining programming languages can be expressed by context-free grammars alone, for example type validity and proper declaration of identifiers. These rules can be formally expressed with attribute grammars.
The final phase is semantic parsing or analysis, which is working out the implications of the expression just validated and taking the appropriate action. [10] In the case of a calculator or interpreter, the action is to evaluate the expression or program; a compiler, on the other hand, would generate some kind of code. Attribute grammars can also be used to define these actions.
Types of parsers[edit]
The task of the parser is essentially to determine if and how the input can be derived from the start symbol of the grammar. This can be done in essentially two ways:
Top-down parsing – Top-down parsing can be viewed as an attempt to find left-most derivations of an input-stream by searching for parse trees using a top-down expansion of the given formal grammar rules. Tokens are consumed from left to right. Inclusive choice is used to accommodate ambiguity by expanding all alternative right-hand-sides of grammar rules. [11] This is known as the primordial soup approach. Very similar to sentence diagramming, primordial soup breaks down the constituencies of sentences. [12]
Bottom-up parsing – A parser can start with the input and attempt to rewrite it to the start symbol. Intuitively, the parser attempts to locate the most basic elements, then the elements containing these, and so on. LR parsers are examples of bottom-up parsers. Another term used for this type of parser is Shift-Reduce parsing.
LL parsers and recursive-descent parser are examples of top-down parsers which cannot accommodate left recursive production rules. Although it has been believed that simple implementations of top-down parsing cannot accommodate direct and indirect left-recursion and may require exponential time and space complexity while parsing ambiguous context-free grammars, more sophisticated algorithms for top-down parsing have been created by Frost, Hafiz, and Callaghan[13][14] which accommodate ambiguity and left recursion in polynomial time and which generate polynomial-size representations of the potentially exponential number of parse trees. Their algorithm is able to produce both left-most and right-most derivations of an input with regard to a given context-free grammar.
An important distinction with regard to parsers is whether a parser generates a leftmost derivation or a rightmost derivation (see context-free grammar). LL parsers will generate a leftmost derivation and LR parsers will generate a rightmost derivation (although usually in reverse). [11]
Some graphical parsing algorithms have been designed for visual programming languages. [15][16] Parsers for visual languages are sometimes based on graph grammars. [17]
Adaptive parsing algorithms have been used to construct “self-extending” natural language user interfaces. [18]
Parser development software[edit]
Some of the well known parser development tools include the following:
ANTLR
Bison
Coco/R
Definite clause grammar
GOLD
JavaCC
Lemon
Lex
LuZc
Parboiled
Parsec
Ragel
Spirit Parser Framework
Syntax Definition Formalism
SYNTAX
XPL
Yacc
PackCC
Lookahead[edit]
C program that cannot be parsed with less than 2 token lookahead. Top: C grammar excerpt. [19] Bottom: a parser has digested the tokens “int v;main(){” and is about choose a rule to derive Stmt. Looking only at the first lookahead token “v”, it cannot decide which of both alternatives for Stmt to choose; the latter requires peeking at the second token.
Lookahead establishes the maximum incoming tokens that a parser can use to decide which rule it should use. Lookahead is especially relevant to LL, LR, and LALR parsers, where it is often explicitly indicated by affixing the lookahead to the algorithm name in parentheses, such as LALR(1).
Most programming languages, the primary target of parsers, are carefully defined in such a way that a parser with limited lookahead, typically one, can parse them, because parsers with limited lookahead are often more efficient. One important change[citation needed] to this trend came in 1990 when Terence Parr created ANTLR for his Ph. D. thesis, a parser generator for efficient LL(k) parsers, where k is any fixed value.
LR parsers typically have only a few actions after seeing each token. They are shift (add this token to the stack for later reduction), reduce (pop tokens from the stack and form a syntactic construct), end, error (no known rule applies) or conflict (does not know whether to shift or reduce).
Lookahead has two advantages. [clarification needed]
It helps the parser take the correct action in case of conflicts. For example, parsing the if statement in the case of an else clause.
It eliminates many duplicate states and eases the burden of an extra stack. A C language non-lookahead parser will have around 10, 000 states. A lookahead parser will have around 300 states.
Example: Parsing the Expression 1 + 2 * 3[dubious – discuss]
Set of expression parsing rules (called grammar) is as follows,
Rule1:
E → E + E
Expression is the sum of two expressions.
Rule2:
E → E * E
Expression is the product of two expressions.
Rule3:
E → number
Expression is a simple number
Rule4:
+ has less precedence than *
Most programming languages (except for a few such as APL and Smalltalk) and algebraic formulas give higher precedence to multiplication than addition, in which case the correct interpretation of the example above is 1 + (2 * 3).
Note that Rule4 above is a semantic rule. It is possible to rewrite the grammar to incorporate this into the syntax. However, not all such rules can be translated into syntax.
Simple non-lookahead parser actions
Initially Input = [1, +, 2, *, 3]
Shift “1” onto stack from input (in anticipation of rule3). Input = [+, 2, *, 3] Stack = [1]
Reduces “1” to expression “E” based on rule3. Stack = [E]
Shift “+” onto stack from input (in anticipation of rule1). Input = [2, *, 3] Stack = [E, +]
Shift “2” onto stack from input (in anticipation of rule3). Input = [*, 3] Stack = [E, +, 2]
Reduce stack element “2” to Expression “E” based on rule3. Stack = [E, +, E]
Reduce stack items [E, +, E] and new input “E” to “E” based on rule1. Stack = [E]
Shift “*” onto stack from input (in anticipation of rule2). Input = [3] Stack = [E, *]
Shift “3” onto stack from input (in anticipation of rule3). Input = [] (empty) Stack = [E, *, 3]
Reduce stack element “3” to expression “E” based on rule3. Stack = [E, *, E]
Reduce stack items [E, *, E] and new input “E” to “E” based on rule2. Stack = [E]
The parse tree and resulting code from it is not correct according to language semantics.
To correctly parse without lookahead, there are three solutions:
The user has to enclose expressions within parentheses. This often is not a viable solution.
The parser needs to have more logic to backtrack and retry whenever a rule is violated or not complete. The similar method is followed in LL parsers.
Alternatively, the parser or grammar needs to have extra logic to delay reduction and reduce only when it is absolutely sure which rule to reduce first. This method is used in LR parsers. This correctly parses the expression but with many more states and increased stack depth.
Lookahead parser actions[clarification needed]
Shift 1 onto stack on input 1 in anticipation of rule3. It does not reduce immediately.
Reduce stack item 1 to simple Expression on input + based on rule3. The lookahead is +, so we are on path to E +, so we can reduce the stack to E.
Shift + onto stack on input + in anticipation of rule1.
Shift 2 onto stack on input 2 in anticipation of rule3.
Reduce stack item 2 to Expression on input * based on rule3. The lookahead * expects only E before it.
Now stack has E + E and still the input is *. It has two choices now, either to shift based on rule2 or reduction based on rule1. Since * has higher precedence than + based on rule4, we shift * onto stack in anticipation of rule2.
Shift 3 onto stack on input 3 in anticipation of rule3.
Reduce stack item 3 to Expression after seeing end of input based on rule3.
Reduce stack items E * E to E based on rule2.
Reduce stack items E + E to E based on rule1.
The parse tree generated is correct and simply more efficient[clarify][citation needed] than non-lookahead parsers. This is the strategy followed in LALR parsers.
See also[edit]
Backtracking
Chart parser
Compiler-compiler
Deterministic parsing
Generating strings
Grammar checker
LALR parser
Lexical analysis
Pratt parser
Shallow parsing
Left corner parser
Parsing expression grammar
DMS Software Reengineering Toolkit
Program transformation
Source code generation
References[edit]
^ a b “Parse”. Retrieved 27 November 2010.
^ Masaru Tomita (6 December 2012). Generalized LR Parsing. Springer Science & Business Media. ISBN 978-1-4615-4034-2.
^ “Grammar and Composition”.
^ Christopher D.. Manning; Christopher D. Manning; Hinrich Schütze (1999). Foundations of Statistical Natural Language Processing. MIT Press. ISBN 978-0-262-13360-9.
^ Jurafsky, Daniel (1996). “A Probabilistic Model of Lexical and Syntactic Access and Disambiguation”. Cognitive Science. 20 (2): 137–194. CiteSeerX 10. 1. 150. 5711. doi:10. 1207/s15516709cog2002_1.
^ Klein, Dan, and Christopher D. Manning. “Accurate unlexicalized parsing. ” Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1. Association for Computational Linguistics, 2003.
^ Charniak, Eugene. “A maximum-entropy-inspired parser. ” Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference. Association for Computational Linguistics, 2000.
^ Chen, Danqi, and Christopher Manning. “A fast and accurate dependency parser using neural networks. ” Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 2014.
^ Jia, Robin; Liang, Percy (2016-06-11). “Data Recombination for Neural Semantic Parsing”. arXiv:1606. 03622 [].
^ Berant, Jonathan, and Percy Liang. “Semantic parsing via paraphrasing. ” Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2014.
^ a b Aho, A. V., Sethi, R. and Ullman, J. (1986) ” Compilers: principles, techniques, and tools. ” Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA.
^ Sikkel, Klaas, 1954- (1997). Parsing schemata: a framework for specification and analysis of parsing algorithms. Berlin: Springer. ISBN 9783642605413. OCLC 606012644. CS1 maint: multiple names: authors list (link)
^ Frost, R., Hafiz, R. and Callaghan, P. (2007) ” Modular and Efficient Top-Down Parsing for Ambiguous Left-Recursive Grammars. ” 10th International Workshop on Parsing Technologies (IWPT), ACL-SIGPARSE, Pages: 109 – 120, June 2007, Prague.
^ Frost, R., Hafiz, R. (2008) ” Parser Combinators for Ambiguous Left-Recursive Grammars. ” 10th International Symposium on Practical Aspects of Declarative Languages (PADL), ACM-SIGPLAN, Volume 4902/2008, Pages: 167 – 181, January 2008, San Francisco.
^ Rekers, Jan, and Andy Schürr. “Defining and parsing visual languages with layered graph grammars. ” Journal of Visual Languages & Computing 8. 1 (1997): 27-55.
^ Rekers, Jan, and A. Schurr. “A graph grammar approach to graphical parsing. ” Visual Languages, Proceedings., 11th IEEE International Symposium on. IEEE, 1995.
^ Zhang, Da-Qian, Kang Zhang, and Jiannong Cao. “A context-sensitive graph grammar formalism for the specification of visual languages. ” The Computer Journal 44. 3 (2001): 186-200.
^ Jill Fain Lehman (6 December 2012). Adaptive Parsing: Self-Extending Natural Language Interfaces. ISBN 978-1-4615-3622-2.
^ taken from Brian W. Kernighan and Dennis M. Ritchie (Apr 1988). The C Programming Language. Prentice Hall Software Series (2nd ed. ). Englewood Cliffs/NJ: Prentice Hall. ISBN 0131103628. (Appendix A. 13 “Grammar”, p. 193 ff)
21. Free Parse HTML Codes [1]
Further reading[edit]
Chapman, Nigel P., LR Parsing: Theory and Practice, Cambridge University Press, 1987. ISBN 0-521-30413-X
Grune, Dick; Jacobs, Ceriel J. H., Parsing Techniques – A Practical Guide, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands. Originally published by Ellis Horwood, Chichester, England, 1990; ISBN 0-13-651431-6
External links[edit]
Look up parse or parsing in Wiktionary, the free dictionary.
The Lemon LALR Parser Generator
Stanford Parser The Stanford Parser
Turin University Parser Natural language parser for the Italian, open source, developed in Common Lisp by Leonardo Lesmo, University of Torino, Italy.
Short history of parser construction
Parsing | Set 1 (Introduction, Ambiguity and Parsers)
In this article we will study about various types of parses. It is one of the most important topic in Compiler from GATE point of view. The working of various parsers will be explained from GATE question solving point of view. Prerequisite – basic knowledge of grammars, parse trees, ambiguity. Role of the parser:In the syntax analysis phase, a compiler verifies whether or not the tokens generated by the lexical analyzer are grouped according to the syntactic rules of the language. This is done by a parser. The parser obtains a string of tokens from the lexical analyzer and verifies that the string can be the grammar for the source language. It detects and reports any syntax errors and produces a parse tree from which intermediate code can be generated. Attention reader! Don’t stop learning now. Get hold of all the important CS Theory concepts for SDE interviews with the CS Theory Course at a student-friendly price and become industry going to types of parsers we will discuss on some ideas about the some important things required for understanding parsing. Context Free Grammars: The syntax of a programming language is described by a context free grammar (CFG). CFG consists of set of terminals, set of non terminals, a start symbol and set of productions. Notation –??? where? is a is a single variable [V]?? (V+T)* Ambiguity A grammar that produces more than one parse tree for some sentence is said to be ambiguous. Eg- consider a grammar S -> aS | Sa | a Now for string aaa we will have 4 parse trees, hence ambiguous For more information refer Removing Left Recursion: A grammar is left recursive if it has a non terminal (variable) S such that their is a derivation S -> Sα | β where α? (V+T)* and β? (V+T)* (sequence of terminals and non terminals that do not start with S) Due to the presence of left recursion some top down parsers enter into infinite loop so we have to eliminate left recursion. Let the productions is of the form A -> Aα1 | Aα2 | Aα3 | ….. | Aαm | β1 | β2 | …. | βn Where no βi begins with an A. then we replace the A-productions by A -> β1 A’ | β2 A’ | ….. | βn A’ A’ -> α1A’ | α2A’ | α3A’| ….. | αmA’ | ε The nonterminal A generates the same strings as before but is no longer left recursive. Let’s look at some example to understand better Removing Left Factoring: A grammar is said to be left factored when it is of the form – A -> αβ1 | αβ2 | αβ3 | …… | αβn | γ i. e the productions start with the same terminal (or set of terminals). On seeing the input α we cannot immediately tell which production to choose to expand A. Left factoring is a grammar transformation that is useful for producing a grammar suitable for predictive or top down parsing. When the choice between two alternative A-productions is not clear, we may be able to rewrite the productions to defer the decision until enough of the input has been seen to make the right choice. For the grammar A -> αβ1 | αβ2 | αβ3 | …… | αβn | γ The equivalent left factored grammar will be – A -> αA’ | γ A’ -> β1 | β2 | β3 | …… | βn The process of deriving the string from the given grammar is known as derivation (parsing). Depending upon how derivation is done we have two kinds of parsers:- Top Down ParserBottom Up ParserWe will be studying the parsers from GATE point of view. Top Down Parser Top down parsing attempts to build the parse tree from root to leaf. Top down parser will start from start symbol and proceeds to string. It follows leftmost derivation. In leftmost derivation, the leftmost non-terminal in each sentential is always chosen. Recursive Descent Parsing S()
{ Choose any S production, S ->X1X2…. ;
for (i = 1 to k)
{
If ( Xi is a non-terminal)
Call procedure Xi();
else if ( Xi equals the current input, increment input)
Else /* error has occurred, backtrack and try another possibility */}}Lets understand it better with an example A recursive descent parsing program consist of a set of procedures, one for each nonterminal. Execution begins with the procedure for the start symbol which halts if its procedure body scans the entire input string. Non Recursive Predictive Parsing: This type if parsing does not require backtracking. Predictive parsers can be constructed for LL(1) grammar, the first ‘L’ stands for scanning the input from left to right, the second ‘L’ stands for leftmost derivation and ‘1’ for using one input symbol lookahead at each step to make parsing action decisions. Before moving on to LL(1) parsers please go through FIRST and FOLLOW Construction of LL(1)predictive parsing table For each production A -> α repeat following steps – Add A -> α under M[A, b] for all b in FIRST(α) If FIRST(α) contains ε then add A -> α under M[A, c] for all c in FOLLOW(A). Size of parsing table = (No. of terminals + 1) * #variables Eg – consider the grammar S -> (L) | a L -> SL’ L’ -> ε | SL’ For any grammar if M have multiple entries than it is not LL(1) grammar Eg – S -> iEtSS’/a S’ ->eS/ε E -> b Important Notes 1. If a grammar contain left factoring then it can not be LL(1)
Eg – S -> aS | a —- both productions go in a
2. If a grammar contain left recursion it can not be LL(1)
Eg – S -> Sa | b
S -> Sa goes to FIRST(S) = b
S -> b goes to b, thus b has 2 entries hence not LL(1)
3. If a grammar is ambiguous then it can not be LL(1)
4. Every regular grammar need not be LL(1) because
regular grammar may contain left factoring, left recursion or ambiguity. We will discuss Bottom Up parser in next article (Set 2). This article is contributed by Parul Sharma
A Guide to Parsing: Algorithms and Technology (Part 1) – DZone AI
We already introduced a few parsing terms when we listed the major tools and libraries used for parsing in Java, C#, Python, and JavaScript. In this article, we make a more in-depth presentation of the concepts and algorithms used in parsing so that you can get a better understanding of this fascinating world.
We have tried to be practical in this article. Our goal is to help practitioners, not to explain the full theory. We just explain what you need to know to understand and build parsers.
Definition of Parsing
Parsing is defined as “the analysis of an input to organize the data according to the rule of a grammar. ”
There are a few ways to define parsing. However, the gist remains the same: parsing means to find the underlying structure of the data we are given.
In a way, parsing can be considered the inverse of templating: identifying the structure and extracting the data. In templating, instead, we have a structure and we fill it with data. In the case of parsing, you have to determine the model from the raw representation, while for templating, you have to combine the data with the model to create the raw representation. Raw representation is usually text, but it can also be binary data.
Fundamentally, parsing is necessary because different entities need the data to be in different forms. Parsing allows transforming data in a way that can be understood by a specific software. The obvious example is programs — they are written by humans, but they must be executed by computers. So, humans write them in a form that they can understand, then a software transforms them in a way that can be used by a computer.
However, parsing might be necessary even when passing data between two software that have different needs. For instance, it is needed when you have to serialize or deserialize a class.
The Big Picture
In this section, we are going to describe the fundamental components of a parser. We are not trying to give you formal explanations, but practical ones.
Regular Expressions
Regular expressions are a sequence of characters that can be defined by a pattern.
Regular expressions are often touted as the thing that you should not use for parsing. This is not strictly correct because you can use regular expressions for parsing simple input. The problem is that some programmers only know regular expressions, so they use them to try to parse everything — even the things that they should not. The result is usually a series of regular expressions hacked together that are very fragile.
You can use regular expressions to parse some simpler languages, but this excludes most programming languages — even the ones that look simple enough, like HTML. In fact, languages that can be parsed with just regular expressions are called regular languages. There is a formal mathematical definition, but that is beyond the scope of this article.
One important consequence of the theory is that regular languages can also be parsed or expressed by a finite state machine. That is to say, regular expressions and finite state machines are equally powerful. This is the reason they are used to implement lexers, as we are going to see later.
A regular language can be defined by a series of regular expressions, while more complex languages need something more. A simple rule of thumb is that if a grammar of a language has recursive, or nested, elements, it is not a regular language. For instance, HTML can contain an arbitrary number of tags inside any tag; therefore, it is not a regular language and it cannot be parsed using solely regular expressions, no matter how clever.
Regular Expressions in Grammar
The familiarity of a typical programmer with regular expressions lends them to be often used to define the grammar of a language. More precisely, their syntax is used to define the rules of a lexer or a parser. For example, the Kleene star (*) is used in a rule to indicate that a particular element can be present zero or an infinite amount of times.
The definition of the rule should not be confused with how the actual lexer or parser is implemented. You can implement a lexer using the regular expression engine provided by your language. However, usually, the regular expressions defined in the grammar are converted are actually converted to a finite-state machine to gain better performance.
Structure of a Parser
Having clarified the role of regular expressions, we can look at the general structure of a parser. A complete parser is usually composed of two parts: a lexer, also known as scanner or tokenizer, and the proper parser. The parser needs the lexer because it does not work directly on the text but on the output produced by the lexer. Not all parsers adopt this two-step schema; some parsers do not depend on a separate lexer and they combine the two steps. They are called scannerless parsers.
A lexer and a parser work in sequence: the lexer scans the input and produces the matching tokens; the parser then scans the tokens and produces the parsing result.
Let’s look at the following example and imagine that we are trying to parse addition.
437 + 734
The lexer scans the text and finds 4, 3, and 7, and then a space (). The job of the lexer is to recognize that the characters 437 constitute one token of type NUM. Then the lexer finds a + symbol, which corresponds to the second token of type PLUS, and lastly, it finds another token of type NUM.
The parser will typically combine the tokens produced by the lexer and group them.
The definitions used by lexers and parsers are called rules or productions. In our example, a lexer rule will specify that a sequence of digits correspond to a token of type NUM, while a parser rule will specify that a sequence of tokens of type NUM, PLUS, NUM corresponds to a sum expression.
It is now typical to find suites that can generate both a lexer and parser. In the past, it was instead more common to combine two different tools: one to produce the lexer and one to produce the parser. For example, this was the case of the venerable lex and yacc couple: using lex, it was possible to generate a lexer, while using yacc, it was possible to generate a parser.
Scannerless Parsers
Scannerless parsers operate differently because they process directly the original text instead of processing a list of tokens produced by a lexer. That is to say, a scannerless parser works as a lexer and a parser combined.
While it is certainly important to know for debugging purposes if a particular parsing tool is a scannerless parser, in many cases, it is not relevant to define a grammar. That is because you usually still define the pattern that group sequences of characters to create (virtual) tokens; then, you combine them to obtain the final result. This is simply because it is more convenient to do so. In other words, the grammar of a scannerless parser looks very similar to one for a tool with separate steps. Again, you should not confuse how things are defined for your convenience and how things work behind the scenes.
You can check out the rest of the series below:
Part 2
Part 3
Part 4
Part 5
Part 6
Part 7
Part 8
Part 9
Topics:
ai,
tutorial,
algorithms,
parsing,
lexer
Frequently Asked Questions about what does a parser do
What is the role of parser?
Role of the parser : … The parser obtains a string of tokens from the lexical analyzer and verifies that the string can be the grammar for the source language. It detects and reports any syntax errors and produces a parse tree from which intermediate code can be generated.Jun 28, 2021
What is parsing and how does it work?
Parsing is defined as “the analysis of an input to organize the data according to the rule of a grammar.” There are a few ways to define parsing. However, the gist remains the same: parsing means to find the underlying structure of the data we are given.Dec 16, 2017
What does parsing do in programming?
To parse is to break up a sentence or group of words into separate components, including the definition of each part’s function or form. … Parsing is used in all high-level programming languages. Languages like C++ and Java are parsed by their respective compilers before being transformed into executable machine code.Mar 23, 2017
