libconf.Tokenizer Class Reference

Public Member Functions
	__init__ (self, filename)
	tokenize (self, string)

Public Attributes
	filename = filename
int	row = 1
int	column = 1

Static Public Attributes
	token_map

Detailed Description

Tokenize an input string

Typical usage:

    tokens = list(Tokenizer("<memory>").tokenize("""a = 7; b = ();"""))

The filename argument to the constructor is used only in error messages, no
data is loaded from the file. The input data is received as argument to the
tokenize function, which yields tokens or throws a ConfigParseError on
invalid input.

Include directives are not supported, they must be handled at a higher
level (cf. the TokenStream class).

Constructor & Destructor Documentation

__init__()

libconf.Tokenizer.__init__	(		self,
			filename )

Member Function Documentation

tokenize()

libconf.Tokenizer.tokenize	(		self,
			string )

Yield tokens from the input string or throw ConfigParseError

Member Data Documentation

column

libconf.Tokenizer.column = 1

filename

libconf.Tokenizer.filename = filename

row

libconf.Tokenizer.row = 1

token_map

libconf.Tokenizer.token_map

static

Initial value:

=  compile_regexes([
        (FltToken,  'float',     r'([-+]?(\d+)?\.\d*([eE][-+]?\d+)?)|'
                                 r'([-+]?(\d+)(\.\d*)?[eE][-+]?\d+)'),
        (IntToken,  'hex64',     r'0[Xx][0-9A-Fa-f]+(L(L)?)'),
        (IntToken,  'hex',       r'0[Xx][0-9A-Fa-f]+'),
        (IntToken,  'integer64', r'[-+]?[0-9]+L(L)?'),
        (IntToken,  'integer',   r'[-+]?[0-9]+'),
        (BoolToken, 'boolean',   r'(?i)(true|false)\b'),
        (StrToken,  'string',    r'"([^"\\]|\\.)*"'),
        (Token,     'name',      r'[A-Za-z\*][-A-Za-z0-9_\*]*'),
        (Token,     '}',         r'\}'),
        (Token,     '{',         r'\{'),
        (Token,     ')',         r'\‍)'),
        (Token,     '(',         r'\‍('),
        (Token,     ']',         r'\]'),
        (Token,     '[',         r'\['),
        (Token,     ',',         r','),
        (Token,     ';',         r';'),
        (Token,     '=',         r'='),
        (Token,     ':',         r':'),
    ])

---

The documentation for this class was generated from the following file:

• python/shared/libconf.py