PUMA Library Reference Manual
Puma::Token Class Reference

#include <Puma/Token.h>

+ Inheritance diagram for Puma::Token:

Description

Abstraction of a lexical token.

A token is created by a scanner (see Puma::CScanner). It is part a token chain (see Puma::Unit) and encapsulates a lexical unit of the scanned text.

The information provided about a lexical unit is:

  • token type
  • location in the scanned text
  • portion of the scanned text represented by this token
  • language of the token

Public Types

enum  {
  ID_END_OF_FILE = -1,
  ID_UNKNOWN = -2,
  ID_ERROR = -3,
  ID_WARNING = -4
}
 Special token types. More...
 

Public Member Functions

 Token (int type=0, LanguageID lang=LanguageID(0), const char *text="")
 Constructor. More...
 
 Token (const Token &copy)
 Copy-constructor. More...
 
virtual ~Token ()
 Destructor. More...
 
void print (std::ostream &os) const
 Print the token text on the given stream. More...
 
virtual ListElementduplicate ()
 Duplicate this token. More...
 
Unitunit () const
 Get the unit this token belongs to. More...
 
void reset (int type=0, const char *text=0, LanguageID lang=LanguageID(0))
 Reset the token object optionally providing new content. More...
 
void location (const Location &loc)
 Set the location of the token (usually the line and column in a file). More...
 
void cont_lines (Array< int > *lines)
 Set the continuous lines appeared in this token. More...
 
int line_breaks () const
 Count the number of line breaks in the text of this token. More...
 
void macro_call (bool is_call=true)
 Set the token to be a macro call. More...
 
const char * text () const
 Get the token text. More...
 
DStringdtext () const
 Get the token text from the global dictionary. More...
 
char * get_static_text () const
 Get the static token text. More...
 
int type () const
 Get the type of the token (see Puma::CTokens). More...
 
const Locationlocation () const
 Get the location of the token (usually the line and column in a file). More...
 
bool is_macro_generated () const
 Check if the token is macro generated. More...
 
bool is_macro_call () const
 Check if the token is a macro call. More...
 
bool is_core () const
 Check if this is a core language token. More...
 
bool is_identifier () const
 Check if the token is an identifier. More...
 
bool is_keyword () const
 Check if the token is a keyword. More...
 
bool is_comma () const
 Check if this is a comma. More...
 
bool is_open () const
 Check if this is a left parenthesis. More...
 
bool is_close () const
 Check if this is a right parenthesis. More...
 
bool is_whitespace () const
 Check if this is a white-space token. More...
 
bool is_comment () const
 Check if this is a comment. More...
 
bool is_preprocessor () const
 Check if this is a preprocessor token. More...
 
bool is_macro_op () const
 Check if this is function-like macro operand. More...
 
bool is_directive () const
 Check if this is a compiler directive token. More...
 
bool is_wildcard () const
 Check if this is a wildcard token. More...
 
void * operator new (size_t size)
 Overwrite operator new for better performance. More...
 
void operator delete (void *ptr)
 Free the memory allowed with the Token's own operator new. More...
 
- Public Member Functions inherited from Puma::ListElement
 ListElement ()
 Constructor. More...
 
virtual ~ListElement ()
 Destructor. More...
 
Listbelonging_to () const
 Get the list this element belongs to. More...
 

Static Public Member Functions

static char * get_static_text (int token_type)
 Get the static token text for the given token type. More...
 

Static Public Attributes

static LanguageID comment_id
 Comment tokens. More...
 
static LanguageID pre_id
 Preprocessor tokens. More...
 
static LanguageID dir_id
 Compiler directives. More...
 
static LanguageID cpp_id
 C/C++ core language tokens. More...
 
static LanguageID white_id
 White-spaces. More...
 
static LanguageID wildcard_id
 Wildcards. More...
 
static LanguageID keyword_id
 Keywords. More...
 
static LanguageID open_id
 Left parenthesis. More...
 
static LanguageID macro_op_id
 Function-like macro operands. More...
 
static LanguageID identifier_id
 Identifiers. More...
 
static LanguageID comma_id
 Comma. More...
 
static LanguageID close_id
 Right parenthesis. More...
 

Member Enumeration Documentation

anonymous enum

Special token types.

Enumerator
ID_END_OF_FILE 

End-of-file token.

ID_UNKNOWN 

Unknown token type.

ID_ERROR 

Error token type.

ID_WARNING 

Warning token type.

Constructor & Destructor Documentation

Puma::Token::Token ( int  type = 0,
LanguageID  lang = LanguageID(0),
const char *  text = "" 
)

Constructor.

Parameters
typeThe token type.
langThe language of the token.
textThe optional token text (defaults to the empty string).
Puma::Token::Token ( const Token copy)

Copy-constructor.

Parameters
copyThe token to copy.
virtual Puma::Token::~Token ( )
virtual

Destructor.

Frees the token text.

Member Function Documentation

void Puma::Token::cont_lines ( Array< int > *  lines)
inline

Set the continuous lines appeared in this token.

Parameters
linesThe continuous line array.
DString& Puma::Token::dtext ( ) const

Get the token text from the global dictionary.

Returns
A reference to the global dictionary managed version of the token text.
virtual ListElement* Puma::Token::duplicate ( )
inlinevirtual

Duplicate this token.

Returns
A copy of this token. Has to be destroyed by the caller.

Implements Puma::ListElement.

char* Puma::Token::get_static_text ( ) const

Get the static token text.

Only for keywords, operators, and other tokens with a constant token text.

Returns
The static text, or NULL for tokens with dynamic text (like identifiers).
static char* Puma::Token::get_static_text ( int  token_type)
static

Get the static token text for the given token type.

Only for keywords, operators, and other tokens with a constant token text.

Parameters
token_typeThe token type.
Returns
The static text, or NULL for tokens with dynamic text (like identifiers).
bool Puma::Token::is_close ( ) const
inline

Check if this is a right parenthesis.

Subset of core tokens.

Returns
True if this is a right parenthesis.
bool Puma::Token::is_comma ( ) const
inline

Check if this is a comma.

Subset of core tokens.

Returns
True if this is a comma.
bool Puma::Token::is_comment ( ) const
inline

Check if this is a comment.

Returns
True if this is a comment token.
bool Puma::Token::is_core ( ) const
inline

Check if this is a core language token.

Returns
True if this is core language token.
bool Puma::Token::is_directive ( ) const
inline

Check if this is a compiler directive token.

Returns
True if this is compiler directive token.
bool Puma::Token::is_identifier ( ) const
inline

Check if the token is an identifier.

Subset of core tokens.

Returns
True if this is an identifier.
bool Puma::Token::is_keyword ( ) const
inline

Check if the token is a keyword.

Subset of identifiers.

Returns
True if this is a keyword.
bool Puma::Token::is_macro_call ( ) const
inline

Check if the token is a macro call.

Returns
True if it is a macro call.
bool Puma::Token::is_macro_generated ( ) const

Check if the token is macro generated.

Returns
True if macro generated.
bool Puma::Token::is_macro_op ( ) const
inline

Check if this is function-like macro operand.

Returns
True if this is an operand of a function-like macro.
bool Puma::Token::is_open ( ) const
inline

Check if this is a left parenthesis.

Subset of core tokens.

Returns
True if this is a left parenthesis.
bool Puma::Token::is_preprocessor ( ) const
inline

Check if this is a preprocessor token.

Returns
True if this is a preprocessor token.
bool Puma::Token::is_whitespace ( ) const
inline

Check if this is a white-space token.

Returns
True if this is a white space.
bool Puma::Token::is_wildcard ( ) const
inline

Check if this is a wildcard token.

Subset of core tokens.

Returns
True if this is a wildcard token.
int Puma::Token::line_breaks ( ) const

Count the number of line breaks in the text of this token.

Returns
The number of line breaks found.
void Puma::Token::location ( const Location loc)
inline

Set the location of the token (usually the line and column in a file).

Parameters
locThe location.
const Location& Puma::Token::location ( ) const
inline

Get the location of the token (usually the line and column in a file).

Returns
A reference to the token location.
void Puma::Token::macro_call ( bool  is_call = true)
inline

Set the token to be a macro call.

Parameters
is_callIs a macro call or not.
void Puma::Token::operator delete ( void *  ptr)

Free the memory allowed with the Token's own operator new.

The memory is not freed but kept back for reuse.

Parameters
ptrA pointer to the memory to free.
void* Puma::Token::operator new ( size_t  size)

Overwrite operator new for better performance.

Tokens are allocated very often. To reduce the overhead of allocating a lot of small portions of memory, this implementation reuses already allocated memory by using a constant size memory pool (see Puma::ConstSizeMemPool).

Parameters
sizeThe size of the memory to allocate in byte.
Returns
A pointer to the allocated memory.
void Puma::Token::print ( std::ostream &  os) const

Print the token text on the given stream.

Parameters
osThe output stream.
void Puma::Token::reset ( int  type = 0,
const char *  text = 0,
LanguageID  lang = LanguageID(0) 
)

Reset the token object optionally providing new content.

Parameters
typeThe new token type.
textThe new token text.
langThe new token language.
const char* Puma::Token::text ( ) const

Get the token text.

Returns
The constant token text.
int Puma::Token::type ( ) const
inline

Get the type of the token (see Puma::CTokens).

Returns
The numerical value of the token type.
Unit* Puma::Token::unit ( ) const

Get the unit this token belongs to.

Returns
A pointer to the unit.

Member Data Documentation

LanguageID Puma::Token::close_id
static

Right parenthesis.

LanguageID Puma::Token::comma_id
static

Comma.

LanguageID Puma::Token::comment_id
static

Comment tokens.

LanguageID Puma::Token::cpp_id
static

C/C++ core language tokens.

LanguageID Puma::Token::dir_id
static

Compiler directives.

LanguageID Puma::Token::identifier_id
static

Identifiers.

LanguageID Puma::Token::keyword_id
static

Keywords.

LanguageID Puma::Token::macro_op_id
static

Function-like macro operands.

LanguageID Puma::Token::open_id
static

Left parenthesis.

LanguageID Puma::Token::pre_id
static

Preprocessor tokens.

LanguageID Puma::Token::white_id
static

White-spaces.

LanguageID Puma::Token::wildcard_id
static

Wildcards.