HCT::Language::Lexer - Base class of lexer used by HCT.
Creates a new
Returns input object created by
If FH set,
creates a new input object.
Virtual method that provides skip from the comments. Returns true if current position should be skipped.
Skips whitespaces. Just removes leading white spaces.
Virtual method that provides skip the current positions. Returns true or false.
If EXPR passed successfully,
returns matched value and sets new line position.
goes back to the previous position and returns
If POS set, changes current line position. Returns current line position.
Important: works not directly with current line position (
but with variable that store previous position value.
Returns current line length.
Returns current line number.
Returns true if end of file (input), or false.
Returns true if end of line, or false.
Moves to the next line.
If line is ended (
pos equal to
gets a new line.
If line is empty,
moves once again.
If file is finished returns
Returns true if we can move forward or false.
The scanner has encoded within it information on the possible sequences of characters that can be contained within any of the tokens it handles. Returns token.
EMPTY_TOKEN to stop the parser.
Could be rewritten if needed,
see CDL lexer.
Returns parser handler.
Will be updated as soon,
Returns object of trace created by
Tries to find corresponding token. Return true if token was found, or false as finish.
Tries to get around main lexer method
Will be called at the end of
if will not found correspond token.
Returns true if token was found, or false.
Virtual method to provide reserve of tokens.
Makes reserve of tokens by calling
reserve_tokens and fills
lexemelist by sorting
Creates an objects of new tokens by using
HCT::Language::Token and stores them into the
Each token name will be register in the table of names -
and their lexemes in
Each of them will be linked with position in
Returns token object,
where KEY can be
$token_obj = $lexer->lookup_token (lexeme => 'if'); $token_obj = $lexer->lookup_token (name => 'TOK_IF');
Gets token id and returns token object from
objlist. Commonly this method using by
Shifts and returns token from token stack.
Psuh new token to the stack and returns true.
The lexer state solution involves coordination between the lexer and the parser. In particular, the parser must tell the lexer whether in "this context" a keyword is expected or an identifier is expected. There are several problems with coordinating the lexer's state with the parser's context. One of the most frequently noted ones is that it makes (multiple token) lookahead more difficult. A related problem occurs if the context is miscommunicated between the parser and the lexer, the lexer may return a keyword when only identifers are expected or return the keyword as an identifier when the keyword was supposed to be treated as a keyword.
This document is part of HCT, the Hardware Complexity Tool.
Generated: Fri Jul 17 10:38:47 2009