COMPILER DESIGN

Detailed explanation-1: -When expression sum=3+2 is tokenized then what is the token category of 3? Sum “Identifier” = “Assignment operator” 3 “Integer literal” + “Addition operator” 2 “Integer literal”; “End of statement". 3.

Detailed explanation-2: -The lexical analyzer is responsible for removing the white spaces and comments from the source program. It corresponds to the error messages with the source program. It helps to identify the tokens. The input characters are read by the lexical analyzer from the source code.

Detailed explanation-3: -For example, A Z and a z. Identifiers are a combination of alphabetic, numeric, underscore, and $ characters. They can be up to 1024 characters long. Identifiers must begin with an alphabetic character or the underscore character (a-z A-Z ).

Detailed explanation-4: -The process of forming tokens from an input stream of characters is called tokenization, and the lexer categorizes them according to a symbol type.

Detailed explanation-5: -In programming language, keywords, constants, identifiers, strings, numbers, operators and punctuations symbols can be considered as tokens. int value = 100; contains the tokens: int (keyword), value (identifier), = (operator), 100 (constant) and; (symbol).