COMPILER DESIGN

Detailed explanation-1: -The main task of lexical analysis is to read input characters in the code and produce tokens. “Get next token” is a command which is sent from the parser to the lexical analyzer. On receiving this command, the lexical analyzer scans the input until it finds the next token. It returns the token to Parser.

Detailed explanation-2: -Tokens are defined often by regular expressions, which are understood by a lexical analyzer generator such as lex. The lexical analyzer (generated automatically by a tool like lex, or hand-crafted) reads in a stream of characters, identifies the lexemes in the stream, and categorizes them into tokens.

Detailed explanation-3: -In programming language, keywords, constants, identifiers, strings, numbers, operators and punctuations symbols can be considered as tokens.

Detailed explanation-4: -A lexical token may consist of one or more characters, and every single character is in exactly one token. The tokens can be keywords, comments, numbers, white space, or strings.