MCQ IN COMPUTER SCIENCE & ENGINEERING

Detailed explanation-1: -A pattern explains what can be a token, and these patterns are defined by means of regular expressions. In programming language, keywords, constants, identifiers, strings, numbers, operators and punctuations symbols can be considered as tokens. For example, in C language, the variable declaration line.

Detailed explanation-2: -REPRESENTATION OF TOKENS USING REGULAR EXPRESSION: 0, 1 is the binary alphabet. ASCII and EBCDIC are two examples of computer alphabets. A string over some alphabet is a finite sequence of symbols drawn from that alphabet. The length of a string s is written as |s|, is the number of occurrences of symbols in s.

Detailed explanation-3: -A regular expression (shortened as regex or regexp; sometimes referred to as rational expression) is a sequence of characters that specifies a search pattern in text. Usually such patterns are used by string-searching algorithms for “find” or “find and replace” operations on strings, or for input validation.

Detailed explanation-4: -Regular expressions have the capability to express finite languages by defining a pattern for finite strings of symbols. So regular expressions are used in token specification. Regular expressions are the algebraic expressions that are used to describe tokens of a programming language.