#include <Puma/Token.h>

Inheritance diagram for Puma::Token:

Description

Abstraction of a lexical token.

A token is created by a scanner (see Puma::CScanner). It is part a token chain (see Puma::Unit) and encapsulates a lexical unit of the scanned text.

The information provided about a lexical unit is:

token type
location in the scanned text
portion of the scanned text represented by this token
language of the token

Public Types
enum	{ ID_END_OF_FILE = -1 , ID_UNKNOWN = -2 , ID_ERROR = -3 , ID_WARNING = -4 }
	Special token types. More...

Public Member Functions
	Token (int type=0, LanguageID lang=LanguageID(0), const char *text="")
	Constructor.

	Token (const Token &copy)
	Copy-constructor.

virtual	~Token ()
	Destructor.

void	print (std::ostream &os) const
	Print the token text on the given stream.

virtual ListElement *	duplicate ()
	Duplicate this token.

Unit *	unit () const
	Get the unit this token belongs to.

void	reset (int type=0, const char *text=0, LanguageID lang=LanguageID(0))
	Reset the token object optionally providing new content.

void	location (const Location &loc)
	Set the location of the token (usually the line and column in a file).

void	cont_lines (Array< int > *lines)
	Set the continuous lines appeared in this token.

int	line_breaks () const
	Count the number of line breaks in the text of this token.

void	macro_call (bool is_call=true)
	Set the token to be a macro call.

const char *	text () const
	Get the token text.

DString &	dtext () const
	Get the token text from the global dictionary.

char *	get_static_text () const
	Get the static token text.

int	type () const
	Get the type of the token (see Puma::CTokens).

const Location &	location () const
	Get the location of the token (usually the line and column in a file).

bool	is_macro_generated () const
	Check if the token is macro generated.

bool	is_macro_call () const
	Check if the token is a macro call.

bool	is_core () const
	Check if this is a core language token.

bool	is_identifier () const
	Check if the token is an identifier.

bool	is_keyword () const
	Check if the token is a keyword.

bool	is_comma () const
	Check if this is a comma.

bool	is_open () const
	Check if this is a left parenthesis.

bool	is_close () const
	Check if this is a right parenthesis.

bool	is_whitespace () const
	Check if this is a white-space token.

bool	is_comment () const
	Check if this is a comment.

bool	is_preprocessor () const
	Check if this is a preprocessor token.

bool	is_macro_op () const
	Check if this is function-like macro operand.

bool	is_directive () const
	Check if this is a compiler directive token.

bool	is_wildcard () const
	Check if this is a wildcard token.

void *	operator new (size_t size)
	Overwrite operator new for better performance.

void	operator delete (void *ptr)
	Free the memory allowed with the Token's own operator new.

Public Member Functions inherited from Puma::ListElement
	ListElement ()
	Constructor.

virtual	~ListElement ()
	Destructor.

List *	belonging_to () const
	Get the list this element belongs to.

Static Public Member Functions
static char *	get_static_text (int token_type)
	Get the static token text for the given token type.

Static Public Attributes
static LanguageID	comment_id
	Comment tokens.

static LanguageID	pre_id
	Preprocessor tokens.

static LanguageID	dir_id
	Compiler directives.

static LanguageID	cpp_id
	C/C++ core language tokens.

static LanguageID	white_id
	White-spaces.

static LanguageID	wildcard_id
	Wildcards.

static LanguageID	keyword_id
	Keywords.

static LanguageID	open_id
	Left parenthesis.

static LanguageID	macro_op_id
	Function-like macro operands.

static LanguageID	identifier_id
	Identifiers.

static LanguageID	comma_id
	Comma.

static LanguageID	close_id
	Right parenthesis.

Member Enumeration Documentation

◆ anonymous enum

anonymous enum

Special token types.

Enumerator
ID_END_OF_FILE	End-of-file token.
ID_UNKNOWN	Unknown token type.
ID_ERROR	Error token type.
ID_WARNING	Warning token type.

Constructor & Destructor Documentation

◆ Token() [1/2]

Puma::Token::Token	(	int	type = 0,
		LanguageID	lang = LanguageID(0),
		const char *	text = "" )

Constructor.

Parameters

type	The token type.
lang	The language of the token.
text	The optional token text (defaults to the empty string).

◆ Token() [2/2]

Puma::Token::Token ( const Token & copy )

Copy-constructor.

Parameters

copy	The token to copy.

◆ ~Token()

virtual Puma::Token::~Token ( )

virtual

Destructor.

Frees the token text.

Member Function Documentation

◆ cont_lines()

void Puma::Token::cont_lines ( Array< int > * lines )

inline

Set the continuous lines appeared in this token.

Parameters

lines The continuous line array.

◆ dtext()

DString & Puma::Token::dtext ( ) const

Get the token text from the global dictionary.

Returns: A reference to the global dictionary managed version of the token text.

◆ duplicate()

virtual ListElement * Puma::Token::duplicate ( )

inlinevirtual

Duplicate this token.

Returns: A copy of this token. Has to be destroyed by the caller.

Implements Puma::ListElement.

◆ get_static_text() [1/2]

char * Puma::Token::get_static_text ( ) const

Get the static token text.

Only for keywords, operators, and other tokens with a constant token text.

Returns: The static text, or NULL for tokens with dynamic text (like identifiers).

◆ get_static_text() [2/2]

static char * Puma::Token::get_static_text ( int token_type )

static

Get the static token text for the given token type.

Only for keywords, operators, and other tokens with a constant token text.

Parameters

token_type The token type.

Returns: The static text, or NULL for tokens with dynamic text (like identifiers).

◆ is_close()

bool Puma::Token::is_close ( ) const

inline

Check if this is a right parenthesis.

Subset of core tokens.

Returns: True if this is a right parenthesis.

◆ is_comma()

bool Puma::Token::is_comma ( ) const

inline

Check if this is a comma.

Subset of core tokens.

Returns: True if this is a comma.

◆ is_comment()

bool Puma::Token::is_comment ( ) const

inline

Check if this is a comment.

Returns: True if this is a comment token.

◆ is_core()

bool Puma::Token::is_core ( ) const

inline

Check if this is a core language token.

Returns: True if this is core language token.

◆ is_directive()

bool Puma::Token::is_directive ( ) const

inline

Check if this is a compiler directive token.

Returns: True if this is compiler directive token.

◆ is_identifier()

bool Puma::Token::is_identifier ( ) const

inline

Check if the token is an identifier.

Subset of core tokens.

Returns: True if this is an identifier.

◆ is_keyword()

bool Puma::Token::is_keyword ( ) const

inline

Check if the token is a keyword.

Subset of identifiers.

Returns: True if this is a keyword.

◆ is_macro_call()

bool Puma::Token::is_macro_call ( ) const

inline

Check if the token is a macro call.

Returns: True if it is a macro call.

◆ is_macro_generated()

bool Puma::Token::is_macro_generated ( ) const

Check if the token is macro generated.

Returns: True if macro generated.

◆ is_macro_op()

bool Puma::Token::is_macro_op ( ) const

inline

Check if this is function-like macro operand.

Returns: True if this is an operand of a function-like macro.

◆ is_open()

bool Puma::Token::is_open ( ) const

inline

Check if this is a left parenthesis.

Subset of core tokens.

Returns: True if this is a left parenthesis.

◆ is_preprocessor()

bool Puma::Token::is_preprocessor ( ) const

inline

Check if this is a preprocessor token.

Returns: True if this is a preprocessor token.

◆ is_whitespace()

bool Puma::Token::is_whitespace ( ) const

inline

Check if this is a white-space token.

Returns: True if this is a white space.

◆ is_wildcard()

bool Puma::Token::is_wildcard ( ) const

inline

Check if this is a wildcard token.

Subset of core tokens.

Returns: True if this is a wildcard token.

◆ line_breaks()

int Puma::Token::line_breaks ( ) const

Count the number of line breaks in the text of this token.

Returns: The number of line breaks found.

◆ location() [1/2]

const Location & Puma::Token::location ( ) const

inline

Get the location of the token (usually the line and column in a file).

Returns: A reference to the token location.

◆ location() [2/2]

void Puma::Token::location ( const Location & loc )

inline

Set the location of the token (usually the line and column in a file).

Parameters

loc	The location.

◆ macro_call()

void Puma::Token::macro_call ( bool is_call = true )

inline

Set the token to be a macro call.

Parameters

is_call Is a macro call or not.

◆ operator delete()

void Puma::Token::operator delete ( void * ptr )

Free the memory allowed with the Token's own operator new.

The memory is not freed but kept back for reuse.

Parameters

ptr	A pointer to the memory to free.

◆ operator new()

void * Puma::Token::operator new ( size_t size )

Overwrite operator new for better performance.

Tokens are allocated very often. To reduce the overhead of allocating a lot of small portions of memory, this implementation reuses already allocated memory by using a constant size memory pool (see Puma::ConstSizeMemPool).

Parameters

size	The size of the memory to allocate in byte.

Returns: A pointer to the allocated memory.

◆ print()

void Puma::Token::print ( std::ostream & os ) const

Print the token text on the given stream.

Parameters

os	The output stream.

◆ reset()

void Puma::Token::reset	(	int	type = 0,
		const char *	text = 0,
		LanguageID	lang = LanguageID(0) )

Reset the token object optionally providing new content.

Parameters

type	The new token type.
text	The new token text.
lang	The new token language.

◆ text()

const char * Puma::Token::text ( ) const

Get the token text.

Returns: The constant token text.

◆ type()

int Puma::Token::type ( ) const

inline

Get the type of the token (see Puma::CTokens).

Returns: The numerical value of the token type.

◆ unit()

Unit * Puma::Token::unit ( ) const

Get the unit this token belongs to.

Returns: A pointer to the unit.

Member Data Documentation

◆ close_id

LanguageID Puma::Token::close_id

static

Right parenthesis.

◆ comma_id

LanguageID Puma::Token::comma_id

static

Comma.

◆ comment_id

LanguageID Puma::Token::comment_id

static

Comment tokens.

◆ cpp_id

LanguageID Puma::Token::cpp_id

static

C/C++ core language tokens.

◆ dir_id

LanguageID Puma::Token::dir_id

static

Compiler directives.

◆ identifier_id

LanguageID Puma::Token::identifier_id

static

Identifiers.

◆ keyword_id

LanguageID Puma::Token::keyword_id

static

Keywords.

◆ macro_op_id

LanguageID Puma::Token::macro_op_id

static

Function-like macro operands.

◆ open_id

LanguageID Puma::Token::open_id

static

Left parenthesis.

◆ pre_id

LanguageID Puma::Token::pre_id

static

Preprocessor tokens.

◆ white_id

LanguageID Puma::Token::white_id

static

White-spaces.

◆ wildcard_id

LanguageID Puma::Token::wildcard_id

static

Wildcards.

Description

Public Types

Public Member Functions

Static Public Member Functions

Static Public Attributes

Member Enumeration Documentation

◆ anonymous enum

Constructor & Destructor Documentation

◆ Token() [1/2]

◆ Token() [2/2]

◆ ~Token()

Member Function Documentation

◆ cont_lines()

◆ dtext()

◆ duplicate()

◆ get_static_text() [1/2]

◆ get_static_text() [2/2]

◆ is_close()

◆ is_comma()

◆ is_comment()

◆ is_core()

◆ is_directive()

◆ is_identifier()

◆ is_keyword()

◆ is_macro_call()

◆ is_macro_generated()

◆ is_macro_op()

◆ is_open()

◆ is_preprocessor()

◆ is_whitespace()

◆ is_wildcard()

◆ line_breaks()

◆ location() [1/2]

◆ location() [2/2]

◆ macro_call()

◆ operator delete()

◆ operator new()

◆ print()

◆ reset()

◆ text()

◆ type()

◆ unit()

Member Data Documentation

◆ close_id

◆ comma_id

◆ comment_id

◆ cpp_id

◆ dir_id

◆ identifier_id

◆ keyword_id

◆ macro_op_id

◆ open_id

◆ pre_id

◆ white_id

◆ wildcard_id