Lexer (xonsh.lexer)

Lexer for xonsh code.

Written using a hybrid of tokenize and PLY.

class xonsh.lexer.Lexer

Implements a lexer for the xonsh language.

Attributes

fname	(str) Filename
last	(token) The last token seen.
lineno	(int) The last line number seen.

build(**kwargs)

Part of the PLY lexer API.

input(s)

Calls the lexer on the string s.

reset()

token()

Retrieves the next token.

tokens = ('POWEQUAL', 'MINUS', 'NEWLINE', 'INDENT', 'DEDENT', 'RSHIFT', 'PERIOD', 'PLUS', 'ATDOLLAR', 'PIPE', 'NUMBER', 'STRING', 'RSHIFTEQUAL', 'LT', 'AMPERSANDEQUAL', 'DOUBLEDIVEQUAL', 'PLUSEQUAL', 'RARROW', 'SEMI', 'XOREQUAL', 'GT', 'DOUBLEDIV', 'POW', 'TIMESEQUAL', 'TILDE', 'AWAIT', 'ASYNC', 'GE', 'SEARCHPATH', 'XOR', 'DOLLAR_NAME', 'DIVIDE', 'MINUSEQUAL', 'COLON', 'DOUBLE_QUESTION', 'AMPERSAND', 'DIVEQUAL', 'PIPEEQUAL', 'ATEQUAL', 'MOD', 'EQ', 'EQUALS', 'IOREDIRECT', 'ELLIPSIS', 'LSHIFT', 'TIMES', 'AT', 'MODEQUAL', 'COMMA', 'NE', 'QUESTION', 'LE', 'LSHIFTEQUAL', 'NAME', 'WS', 'LPAREN', 'RPAREN', 'LBRACKET', 'RBRACKET', 'LBRACE', 'RBRACE', 'AT_LPAREN', 'BANG_LPAREN', 'BANG_LBRACKET', 'DOLLAR_LPAREN', 'DOLLAR_LBRACE', 'DOLLAR_LBRACKET', 'ATDOLLAR_LPAREN', 'FALSE', 'NONE', 'TRUE', 'AND', 'AS', 'ASSERT', 'BREAK', 'CLASS', 'CONTINUE', 'DEF', 'DEL', 'ELIF', 'ELSE', 'EXCEPT', 'FINALLY', 'FOR', 'FROM', 'GLOBAL', 'IF', 'IMPORT', 'IN', 'IS', 'LAMBDA', 'NONLOCAL', 'NOT', 'OR', 'PASS', 'RAISE', 'RETURN', 'TRY', 'WHILE', 'WITH', 'YIELD')

xonsh.lexer.get_tokens(s)

Given a string containing xonsh code, generates a stream of relevant PLY tokens using handle_token.

xonsh.lexer.handle_double_amps(state, token)

xonsh.lexer.handle_double_pipe(state, token)

xonsh.lexer.handle_error_space(state, token)

Function for handling special whitespace characters in subprocess mode

xonsh.lexer.handle_error_token(state, token)

Function for handling error tokens

xonsh.lexer.handle_ignore(state, token)

Function for handling tokens that should be ignored

xonsh.lexer.handle_name(state, token)

Function for handling name tokens

xonsh.lexer.handle_rbrace(state, token)

Function for handling }

xonsh.lexer.handle_rbracket(state, token)

Function for handling ]

xonsh.lexer.handle_rparen(state, token)

Function for handling )

xonsh.lexer.handle_token(state, token)

General-purpose token handler. Makes use of token_map or special_map to yield one or more PLY tokens from the given input.

Parameters:

state :

The current state of the lexer, including information about whether we are in Python mode or subprocess mode, which changes the lexer’s behavior. Also includes the stream of tokens yet to be considered.

token :

The token (from tokenize) currently under consideration

xonsh.lexer.special_handlers = {0: <function handle_ignore>, 1: <function handle_name>, (53, '&&'): <function handle_double_amps>, (53, '@('): <function _make_matcher_handler.<locals>._inner_handler>, (53, '}'): <function handle_rbrace>, (53, '||'): <function handle_double_pipe>, (53, '@$('): <function _make_matcher_handler.<locals>._inner_handler>, (53, '{'): <function _make_matcher_handler.<locals>._inner_handler>, 57: <function handle_ignore>, (53, ')'): <function handle_rparen>, (53, '$('): <function _make_matcher_handler.<locals>._inner_handler>, (53, '$['): <function _make_matcher_handler.<locals>._inner_handler>, (53, '['): <function _make_matcher_handler.<locals>._inner_handler>, (53, '!['): <function _make_matcher_handler.<locals>._inner_handler>, (53, '('): <function _make_matcher_handler.<locals>._inner_handler>, (56, ' '): <function handle_error_space>, (53, ']'): <function handle_rbracket>, 56: <function handle_error_token>, (53, '${'): <function _make_matcher_handler.<locals>._inner_handler>, 58: <function handle_ignore>, 59: <function handle_ignore>, (53, '!('): <function _make_matcher_handler.<locals>._inner_handler>}

Mapping from tokenize tokens (or token types) to the proper function for generating PLY tokens from them. In addition to yielding PLY tokens, these functions may manipulate the Lexer’s state.

xonsh.lexer.token_map = {(53, '**='): 'POWEQUAL', (53, '-'): 'MINUS', 4: 'NEWLINE', 5: 'INDENT', 6: 'DEDENT', (53, '>>'): 'RSHIFT', (53, '.'): 'PERIOD', (53, '+'): 'PLUS', (53, '@$'): 'ATDOLLAR', (53, '|'): 'PIPE', 2: 'NUMBER', 3: 'STRING', (53, '>>='): 'RSHIFTEQUAL', (53, '<'): 'LT', (53, '&='): 'AMPERSANDEQUAL', (53, '//='): 'DOUBLEDIVEQUAL', (53, '+='): 'PLUSEQUAL', (53, '->'): 'RARROW', (53, ';'): 'SEMI', (53, '^='): 'XOREQUAL', (53, '>'): 'GT', (53, '//'): 'DOUBLEDIV', (53, '**'): 'POW', (53, '*='): 'TIMESEQUAL', (53, '~'): 'TILDE', 54: 'AWAIT', 55: 'ASYNC', (53, '>='): 'GE', 60: 'SEARCHPATH', (53, '^'): 'XOR', 62: 'DOLLAR_NAME', (53, '/'): 'DIVIDE', (53, '-='): 'MINUSEQUAL', (53, ':'): 'COLON', (53, '??'): 'DOUBLE_QUESTION', (53, '&'): 'AMPERSAND', (53, '/='): 'DIVEQUAL', (53, '|='): 'PIPEEQUAL', (53, '@='): 'ATEQUAL', (53, '%'): 'MOD', (53, '=='): 'EQ', (53, '='): 'EQUALS', 61: 'IOREDIRECT', (53, '...'): 'ELLIPSIS', (53, '<<'): 'LSHIFT', (53, '*'): 'TIMES', (53, '@'): 'AT', (53, '%='): 'MODEQUAL', (53, ','): 'COMMA', (53, '!='): 'NE', (53, '?'): 'QUESTION', (53, '<='): 'LE', (53, '<<='): 'LSHIFTEQUAL'}

Mapping from tokenize tokens (or token types) to PLY token types. If a simple one-to-one mapping from tokenize to PLY exists, the lexer will look it up here and generate a single PLY token of the given type. Otherwise, it will fall back to handling that token using one of the handlers in special_handlers.