Module tokenizer


Classes

Token
Tokenizer
UngetBufferFull

Variables

COMMENT
DELIMITER
EOF
EOL
IDENTIFIER
QUOTED_STRING
WHITESPACE
_DELIMITERS
_QUOTING_DELIMITERS
__package__

[hide private]