HCT::Language::Lexer - Base class of lexer used by HCT.
Creates a new HCT::Language::Lexer object.
Returns input object created by HCT::Std::IO.
If FH set,
creates a new input object.
Virtual method that provides skip from the comments. Returns true if current position should be skipped.
Skips whitespaces. Just removes leading white spaces.
Virtual method that provides skip the current positions. Returns true or false.
If EXPR passed successfully,
returns matched value and sets new line position.
Else,
goes back to the previous position and returns undef.
If POS set, changes current line position. Returns current line position.
Important: works not directly with current line position (pos),
but with variable that store previous position value.
Returns current line length.
Returns current line number.
Returns true if end of file (input), or false.
Returns true if end of line, or false.
Moves to the next line.
If line is ended (pos equal to len),
gets a new line.
If line is empty,
moves once again.
If file is finished returns EOF.
Returns true if we can move forward or false.
The scanner has encoded within it information on the possible sequences of characters that can be contained within any of the tokens it handles. Returns token.
Returns EMPTY_TOKEN to stop the parser.
Could be rewritten if needed,
see CDL lexer.
Returns parser handler.
Will be updated as soon,
as scan called.
Returns object of trace created by HCT::Language::Trace.
Tries to find corresponding token. Return true if token was found, or false as finish.
Virtual method.
Tries to get around main lexer method next_token.
Will be called at the end of next_token,
if will not found correspond token.
Returns true if token was found, or false.
Virtual method to provide reserve of tokens.
Makes reserve of tokens by calling reserve_tokens and fills lexemelist by sorting lexemetable.
Creates an objects of new tokens by using HCT::Language::Token and stores them into the objtable.
Each token name will be register in the table of names - nametable,
and their lexemes in lexemetable.
Each of them will be linked with position in objtable.
Returns token object,
where KEY can be name or lexeme.
Example:
$token_obj = $lexer->lookup_token (lexeme => 'if');
$token_obj = $lexer->lookup_token (name => 'TOK_IF');
Gets token id and returns token object from objlist. Commonly this method using by lookup_token.
Shifts and returns token from token stack.
Psuh new token to the stack and returns true.
The lexer state solution involves coordination between the lexer and the parser. In particular, the parser must tell the lexer whether in "this context" a keyword is expected or an identifier is expected. There are several problems with coordinating the lexer's state with the parser's context. One of the most frequently noted ones is that it makes (multiple token) lookahead more difficult. A related problem occurs if the context is miscommunicated between the parser and the lexer, the lexer may return a keyword when only identifers are expected or return the keyword as an identifier when the keyword was supposed to be treated as a keyword.
This document is part of HCT, the Hardware Complexity Tool.
Generated: Fri Jul 17 10:38:47 2009