Token (RSyntaxTextArea API)

java.lang.Object
- org.fife.ui.rsyntaxtextarea.Token

All Implemented Interfaces:

TokenTypes
```
public class Token
extends Object
implements TokenTypes
```
A generic token that functions as a node in a linked list of syntax highlighted tokens for some language.
A Token is a piece of text representing some logical token in source code for a programming language. For example, the line of C code:
```
 int i = 0;
 
```
would be broken into 8 Tokens: the first representing int, the second whitespace, the third i, the fourth whitespace, the fifth =, etc.
Version:

0.3

Author:

Robert Futrell

Field Summary

Fields
Modifier and Type	Field and Description
`int`	`offset` The offset into the document at which this token resides.
`char[]`	`text` The text this token represents.
`int`	`textCount`
`int`	`textOffset`
`int`	`type` The type of token this is; for example, `TokenTypes.FUNCTION`.

Constructor Summary

Constructors
Constructor and Description
`Token()` Creates a "null token."
`Token(char[] line, int beg, int end, int startOffset, int type)` Constructor.
`Token(Segment line, int beg, int end, int startOffset, int type)` Constructor.
`Token(Token t2)` Creates this token as a copy of the passed-in token.

Method Summary

Methods
Modifier and Type	Method and Description
`StringBuffer`	`appendHTMLRepresentation(StringBuffer sb, RSyntaxTextArea textArea, boolean fontFamily)` Appends HTML code for painting this token, using the given text area's color scheme.
`StringBuffer`	`appendHTMLRepresentation(StringBuffer sb, RSyntaxTextArea textArea, boolean fontFamily, boolean tabsToSpaces)` Appends HTML code for painting this token, using the given text area's color scheme.
`boolean`	`containsPosition(int pos)` Returns whether the token straddles the specified position in the document.
`void`	`copyFrom(Token t2)` Makes one token point to the same text segment, and have the same value as another token.
`int`	`documentToToken(int pos)` Returns the position in the token's internal char array corresponding to the specified document position.
`boolean`	`endsWith(char[] ch)` Returns whether this token's lexeme ends with the specified characters.
`String`	`getHTMLRepresentation(RSyntaxTextArea textArea)` Returns a `String` containing HTML code for painting this token, using the given text area's color scheme.
`int`	`getLanguageIndex()` Returns the language index of this token.
`Token`	`getLastNonCommentNonWhitespaceToken()` Returns the last token in this list that is not whitespace or a comment.
`Token`	`getLastPaintableToken()` Returns the last paintable token in this token list, or `null` if there is no paintable token.
`String`	`getLexeme()` Returns the text of this token, as a string.
`int`	`getListOffset(RSyntaxTextArea textArea, TabExpander e, float x0, float x)` Determines the offset into this token list (i.e., into the document) that covers pixel location `x` if the token list starts at pixel location `x0`.
`Token`	`getNextToken()` Returns the token after this one in the linked list.
`int`	`getOffsetBeforeX(RSyntaxTextArea textArea, TabExpander e, float startX, float endBeforeX)` Returns the position in the document that represents the last character in the token that will fit into `endBeforeX-startX` pixels.
`float`	`getWidth(RSyntaxTextArea textArea, TabExpander e, float x0)` Returns the width of this token given the specified parameters.
`float`	`getWidthUpTo(int numChars, RSyntaxTextArea textArea, TabExpander e, float x0)` Returns the width of a specified number of characters in this token.
`boolean`	`is(char[] lexeme)` Returns whether this token's lexeme matches a specific character array.
`boolean`	`is(int type, char[] lexeme)` Returns whether this token is of the specified type, with the specified lexeme.
`boolean`	`is(int type, String lexeme)` Returns whether this token is of the specified type, with the specified lexeme.
`boolean`	`isComment()` Returns whether this token is a comment.
`boolean`	`isCommentOrWhitespace()` Returns whether this token is a comment or whitespace.
`boolean`	`isHyperlink()` Returns whether this token is a hyperlink.
`boolean`	`isIdentifier()` Returns whether this token is an identifier.
`boolean`	`isLeftCurly()` Returns whether this token is a `TokenTypes.SEPARATOR` representing a single left curly brace.
`boolean`	`isPaintable()` Returns whether or not this token is "paintable;" i.e., whether or not the type of this token is one such that it has an associated syntax style.
`boolean`	`isRightCurly()` Returns whether this token is a `TokenTypes.SEPARATOR` representing a single right curly brace.
`boolean`	`isSingleChar(char ch)` Returns whether this token is the specified single character.
`boolean`	`isSingleChar(int type, char ch)` Returns whether this token is the specified single character, and of a specific type.
`boolean`	`isWhitespace()` Returns whether or not this token is whitespace.
`Rectangle`	`listOffsetToView(RSyntaxTextArea textArea, TabExpander e, int pos, int x0, Rectangle rect)` Returns the bounding box for the specified document location.
`void`	`makeStartAt(int pos)` Makes this token start at the specified offset into the document.
`void`	`moveOffset(int amt)` Moves the starting offset of this token.
`void`	`set(char[] line, int beg, int end, int offset, int type)` Sets the value of this token to a particular segment of a document.
`void`	`setHyperlink(boolean hyperlink)` Sets whether this token is a hyperlink.
`void`	`setLanguageIndex(int languageIndex)` Sets the language index for this token.
`void`	`setNextToken(Token nextToken)` Sets the "next token" pointer of this token to point to the specified token.
`boolean`	`startsWith(char[] chars)` Returns whether this token starts with the specified characters.
`int`	`tokenToDocument(int pos)` Returns the position in the document corresponding to the specified position in this token's internal char array (`textOffset` - `textOffset+textCount-1`).
`String`	`toString()` Returns this token as a `String`, which is useful for debugging.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - text
```
public char[] text
```
    The text this token represents. This is implemented as a segment so we can point directly to the text in the document without having to make a copy of it.
  - textOffset
```
public int textOffset
```
  - textCount
```
public int textCount
```
  - offset
```
public int offset
```
    The offset into the document at which this token resides.
  - type
```
public int type
```
    The type of token this is; for example, TokenTypes.FUNCTION.
- Constructor Detail
  - Token
```
public Token()
```
    Creates a "null token." The token itself is not null; rather, it signifies that it is the last token in a linked list of tokens and that it is not part of a "multi-line token."
  - Token
```
public Token(Segment line,
     int beg,
     int end,
     int startOffset,
     int type)
```
    Constructor.
    
    Parameters:
    line - The segment from which to get the token.
    beg - The first character's position in line.
    end - The last character's position in line.
    startOffset - The offset into the document at which this token begins.
    type - A token type listed as "generic" above.
  - Token
```
public Token(char[] line,
     int beg,
     int end,
     int startOffset,
     int type)
```
    Constructor.
    
    Parameters:
    line - The segment from which to get the token.
    beg - The first character's position in line.
    end - The last character's position in line.
    startOffset - The offset into the document at which this token begins.
    type - A token type listed as "generic" above.
  - Token
```
public Token(Token t2)
```
    Creates this token as a copy of the passed-in token.
    
    Parameters:
    t2 - The token from which to make a copy.
- Method Detail
  - appendHTMLRepresentation
```
public StringBuffer appendHTMLRepresentation(StringBuffer sb,
                                    RSyntaxTextArea textArea,
                                    boolean fontFamily)
```
    Appends HTML code for painting this token, using the given text area's color scheme.
    
    Parameters:
    sb - The buffer to append to.
    textArea - The text area whose color scheme to use.
    fontFamily - Whether to include the font family in the HTML for this token. You can pass false for this parameter if, for example, you are making all your HTML be monospaced, and don't want any crazy fonts being used in the editor to be reflected in your HTML.
    
    Returns:
    The buffer appended to.
    See Also:
    getHTMLRepresentation(RSyntaxTextArea)
  - appendHTMLRepresentation
```
public StringBuffer appendHTMLRepresentation(StringBuffer sb,
                                    RSyntaxTextArea textArea,
                                    boolean fontFamily,
                                    boolean tabsToSpaces)
```
    Appends HTML code for painting this token, using the given text area's color scheme.
    
    Parameters:
    sb - The buffer to append to.
    textArea - The text area whose color scheme to use.
    fontFamily - Whether to include the font family in the HTML for this token. You can pass false for this parameter if, for example, you are making all your HTML be monospaced, and don't want any crazy fonts being used in the editor to be reflected in your HTML.
    tabsToSpaces - Whether to convert tabs into spaces.
    
    Returns:
    The buffer appended to.
    See Also:
    getHTMLRepresentation(RSyntaxTextArea)
  - containsPosition
```
public boolean containsPosition(int pos)
```
    Returns whether the token straddles the specified position in the document.
    
    Parameters:
    pos - The position in the document to check.
    
    Returns:
    Whether the specified position is straddled by this token.
  - copyFrom
```
public void copyFrom(Token t2)
```
    Makes one token point to the same text segment, and have the same value as another token.
    
    Parameters:
    t2 - The token from which to copy.
  - documentToToken
```
public int documentToToken(int pos)
```
    Returns the position in the token's internal char array corresponding to the specified document position.
    Note that this method does NOT do any bounds checking; you can pass in a document position that does not correspond to a position in this token, and you will not receive an Exception or any other notification; it is up to the caller to ensure valid input.
    
    Parameters:
    pos - A position in the document that is represented by this token.
    
    Returns:
    The corresponding token position >= textOffset and < textOffset+textCount.
    See Also:
    tokenToDocument(int)
  - endsWith
```
public boolean endsWith(char[] ch)
```
    Returns whether this token's lexeme ends with the specified characters.
    
    Parameters:
    ch - The characters.
    
    Returns:
    Whether this token's lexeme ends with the specified characters.
    See Also:
    startsWith(char[])
  - getHTMLRepresentation
```
public String getHTMLRepresentation(RSyntaxTextArea textArea)
```
    Returns a String containing HTML code for painting this token, using the given text area's color scheme.
    
    Parameters:
    textArea - The text area whose color scheme to use.
    
    Returns:
    The HTML representation of the token.
    See Also:
    appendHTMLRepresentation(StringBuffer, RSyntaxTextArea, boolean)
  - getLanguageIndex
```
public int getLanguageIndex()
```
    Returns the language index of this token.
    
    Returns:
    The language index. A value of 0 denotes the "main" language, any positive value denotes a specific secondary language.
    See Also:
    setLanguageIndex(int)
  - getLastNonCommentNonWhitespaceToken
```
public Token getLastNonCommentNonWhitespaceToken()
```
    Returns the last token in this list that is not whitespace or a comment.
    
    Returns:
    The last non-comment, non-whitespace token, or null if there isn't one.
  - getLastPaintableToken
```
public Token getLastPaintableToken()
```
    Returns the last paintable token in this token list, or null if there is no paintable token.
    
    Returns:
    The last paintable token in this token list.
  - getLexeme
```
public String getLexeme()
```
    Returns the text of this token, as a string.
    Note that this method isn't used much by the ryntaxtextarea package internally, as it tries to limit memory allocation.
    
    Returns:
    The text of this token.
  - getListOffset
```
public int getListOffset(RSyntaxTextArea textArea,
                TabExpander e,
                float x0,
                float x)
```
    Determines the offset into this token list (i.e., into the document) that covers pixel location x if the token list starts at pixel location x0
    . This method will return the document position "closest" to the x-coordinate (i.e., if they click on the "right-half" of the w in awe, the caret will be placed in between the w and e; similarly, clicking on the left-half places the caret between the a and w). This makes it useful for methods such as viewToModel found in javax.swing.text.View subclasses.
    
    Parameters:
    textArea - The text area from which the token list was derived.
    e - How to expand tabs.
    x0 - The pixel x-location that is the beginning of tokenList.
    x - The pixel-position for which you want to get the corresponding offset.
    
    Returns:
    The position (in the document, NOT into the token list!) that covers the pixel location. If tokenList is null or has type Token.NULL, then -1
  - getNextToken
```
public Token getNextToken()
```
    Returns the token after this one in the linked list.
    
    Returns:
    The next token.
    See Also:
    setNextToken(org.fife.ui.rsyntaxtextarea.Token)
  - getOffsetBeforeX
```
public int getOffsetBeforeX(RSyntaxTextArea textArea,
                   TabExpander e,
                   float startX,
                   float endBeforeX)
```
    Returns the position in the document that represents the last character in the token that will fit into endBeforeX-startX pixels. For example, if you're using a monospaced 8-pixel-per-character font, have the token "while" and startX is 0 and endBeforeX is 30, this method will return the document position of the "i" in "while", because the "i" ends at pixel 24, while the "l" ends at 32. If not even the first character fits in endBeforeX-startX, the first character's position is still returned so calling methods don't go into infinite loops.
    
    Parameters:
    textArea - The text area in which this token is being painted.
    e - How to expand tabs.
    startX - The x-coordinate at which the token will be painted. This is needed because of tabs.
    endBeforeX - The x-coordinate for which you want to find the last character of t which comes before it.
    
    Returns:
    The last document position that will fit in the specified amount of pixels.
  - getWidth
```
public float getWidth(RSyntaxTextArea textArea,
             TabExpander e,
             float x0)
```
    Returns the width of this token given the specified parameters.
    
    Parameters:
    textArea - The text area in which the token is being painted.
    e - Describes how to expand tabs. This parameter cannot be null.
    x0 - The pixel-location at which the token begins. This is needed because of tabs.
    
    Returns:
    The width of the token, in pixels.
    See Also:
    getWidthUpTo(int, org.fife.ui.rsyntaxtextarea.RSyntaxTextArea, javax.swing.text.TabExpander, float)
  - getWidthUpTo
```
public float getWidthUpTo(int numChars,
                 RSyntaxTextArea textArea,
                 TabExpander e,
                 float x0)
```
    Returns the width of a specified number of characters in this token. For example, for the token "while", specifying a value of 3 here returns the width of the "whi" portion of the token.
    
    Parameters:
    numChars - The number of characters for which to get the width.
    textArea - The text area in which the token is being painted.
    e - How to expand tabs. This value cannot be null.
    x0 - The pixel-location at which this token begins. This is needed because of tabs.
    
    Returns:
    The width of the specified number of characters in this token.
    See Also:
    getWidth(org.fife.ui.rsyntaxtextarea.RSyntaxTextArea, javax.swing.text.TabExpander, float)
  - is
```
public boolean is(char[] lexeme)
```
    Returns whether this token's lexeme matches a specific character array.
    
    Parameters:
    lexeme - The lexeme to check for.
    
    Returns:
    Whether this token has that lexeme.
    See Also:
    is(int, char[]), is(int, String), isSingleChar(int, char), startsWith(char[])
  - is
```
public boolean is(int type,
         char[] lexeme)
```
    Returns whether this token is of the specified type, with the specified lexeme.
    This method is preferred over the other overload in performance-critical code where this operation may be called frequently, since it does not involve any String allocations.
    
    Parameters:
    type - The type to check for.
    lexeme - The lexeme to check for.
    
    Returns:
    Whether this token has that type and lexeme.
    See Also:
    is(int, String), is(char[]), isSingleChar(int, char), startsWith(char[])
  - is
```
public boolean is(int type,
         String lexeme)
```
    Returns whether this token is of the specified type, with the specified lexeme.
    The other overload of this method is preferred over this one in performance-critical code, as this one involves a String allocation while the other does not.
    
    Parameters:
    type - The type to check for.
    lexeme - The lexeme to check for.
    
    Returns:
    Whether this token has that type and lexeme.
    See Also:
    is(int, char[]), isSingleChar(int, char), startsWith(char[])
  - isComment
```
public boolean isComment()
```
    Returns whether this token is a comment.
    
    Returns:
    Whether this token is a comment.
    See Also:
    isWhitespace(), isCommentOrWhitespace()
  - isCommentOrWhitespace
```
public boolean isCommentOrWhitespace()
```
    Returns whether this token is a comment or whitespace.
    
    Returns:
    Whether this token is a comment or whitespace.
    See Also:
    isComment(), isWhitespace()
  - isHyperlink
```
public boolean isHyperlink()
```
    Returns whether this token is a hyperlink.
    
    Returns:
    Whether this token is a hyperlink.
    See Also:
    setHyperlink(boolean)
  - isIdentifier
```
public boolean isIdentifier()
```
    Returns whether this token is an identifier.
    
    Returns:
    Whether this token is an identifier.
  - isLeftCurly
```
public boolean isLeftCurly()
```
    Returns whether this token is a TokenTypes.SEPARATOR representing a single left curly brace.
    
    Returns:
    Whether this token is a left curly brace.
    See Also:
    isRightCurly()
  - isRightCurly
```
public boolean isRightCurly()
```
    Returns whether this token is a TokenTypes.SEPARATOR representing a single right curly brace.
    
    Returns:
    Whether this token is a right curly brace.
    See Also:
    isLeftCurly()
  - isPaintable
```
public boolean isPaintable()
```
    Returns whether or not this token is "paintable;" i.e., whether or not the type of this token is one such that it has an associated syntax style. What this boils down to is whether the token type is greater than Token.NULL.
    
    Returns:
    Whether or not this token is paintable.
  - isSingleChar
```
public boolean isSingleChar(char ch)
```
    Returns whether this token is the specified single character.
    
    Parameters:
    ch - The character to check for.
    
    Returns:
    Whether this token's lexeme is the single character specified.
    See Also:
    isSingleChar(int, char)
  - isSingleChar
```
public boolean isSingleChar(int type,
                   char ch)
```
    Returns whether this token is the specified single character, and of a specific type.
    
    Parameters:
    type - The token type.
    ch - The character to check for.
    
    Returns:
    Whether this token is of the specified type, and with a lexeme Equaling the single character specified.
    See Also:
    isSingleChar(char)
  - isWhitespace
```
public boolean isWhitespace()
```
    Returns whether or not this token is whitespace.
    
    Returns:
    true iff this token is whitespace.
    See Also:
    isComment(), isCommentOrWhitespace()
  - listOffsetToView
```
public Rectangle listOffsetToView(RSyntaxTextArea textArea,
                         TabExpander e,
                         int pos,
                         int x0,
                         Rectangle rect)
```
    Returns the bounding box for the specified document location. The location must be in the specified token list; if it isn't, null is returned.
    
    Parameters:
    textArea - The text area from which the token list was derived.
    e - How to expand tabs.
    pos - The position in the document for which to get the bounding box in the view.
    x0 - The pixel x-location that is the beginning of tokenList.
    rect - The rectangle in which we'll be returning the results. This object is reused to keep from frequent memory allocations.
    
    Returns:
    The bounding box for the specified position in the model.
  - makeStartAt
```
public void makeStartAt(int pos)
```
    Makes this token start at the specified offset into the document.
    
    Parameters:
    pos - The offset into the document this token should start at. Note that this token must already contain this position; if it doesn't, an exception is thrown.
    
    Throws:
    
    IllegalArgumentException - If pos is not already contained by this token.
    See Also:
    moveOffset(int)
  - moveOffset
```
public void moveOffset(int amt)
```
    Moves the starting offset of this token.
    
    Parameters:
    amt - The amount to move the starting offset. This should be between 0 and textCount, inclusive.
    
    Throws:
    
    IllegalArgumentException - If amt is an invalid value.
    See Also:
    makeStartAt(int)
  - set
```
public void set(char[] line,
       int beg,
       int end,
       int offset,
       int type)
```
    Sets the value of this token to a particular segment of a document. The "next token" value is cleared.
    
    Parameters:
    line - The segment from which to get the token.
    beg - The first character's position in line.
    end - The last character's position in line.
    offset - The offset into the document at which this token begins.
    type - A token type listed as "generic" above.
  - setHyperlink
```
public void setHyperlink(boolean hyperlink)
```
    Sets whether this token is a hyperlink.
    
    Parameters:
    hyperlink - Whether this token is a hyperlink.
    See Also:
    isHyperlink()
  - setLanguageIndex
```
public void setLanguageIndex(int languageIndex)
```
    Sets the language index for this token. If this value is positive, it denotes a specific "secondary" language this token represents (such as JavaScript code or CSS embedded in an HTML file). If this value is 0, this token is in the "main" language being edited. Negative values are invalid and treated as 0.
    
    Parameters:
    languageIndex - The new language index. A value of 0 denotes the "main" language, any positive value denotes a specific secondary language. Negative values will be treated as 0.
    See Also:
    getLanguageIndex()
  - setNextToken
```
public void setNextToken(Token nextToken)
```
    Sets the "next token" pointer of this token to point to the specified token.
    
    Parameters:
    nextToken - The new next token.
    See Also:
    getNextToken()
  - startsWith
```
public boolean startsWith(char[] chars)
```
    Returns whether this token starts with the specified characters.
    
    Parameters:
    chars - The characters.
    
    Returns:
    Whether this token starts with those characters.
    See Also:
    endsWith(char[]), is(int, char[])
  - tokenToDocument
```
public int tokenToDocument(int pos)
```
    Returns the position in the document corresponding to the specified position in this token's internal char array (textOffset - textOffset+textCount-1).
    Note that this method does NOT do any bounds checking; you can pass in an invalid token position, and you will not receive an Exception or any other indication that the returned document position is invalid. It is up to the user to ensure valid input.
    
    Parameters:
    pos - A position in the token's internal char array (textOffset - textOffset+textCount).
    
    Returns:
    The corresponding position in the document.
    See Also:
    documentToToken(int)
  - toString
```
public String toString()
```
    Returns this token as a String, which is useful for debugging.
    
    Overrides:
    
    toString in class Object
    
    Returns:
    A string describing this token.

Class Token

Field Summary

Fields inherited from interface org.fife.ui.rsyntaxtextarea.TokenTypes

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

text

textOffset

textCount

offset

type

Constructor Detail

Token

Token

Token

Token

Method Detail

appendHTMLRepresentation

appendHTMLRepresentation

containsPosition

copyFrom

documentToToken

endsWith

getHTMLRepresentation

getLanguageIndex

getLastNonCommentNonWhitespaceToken

getLastPaintableToken

getLexeme

getListOffset

getNextToken

getOffsetBeforeX

getWidth

getWidthUpTo

is

is

is

isComment

isCommentOrWhitespace

isHyperlink

isIdentifier

isLeftCurly

isRightCurly

isPaintable

isSingleChar

isSingleChar

isWhitespace

listOffsetToView

makeStartAt

moveOffset

set

setHyperlink

setLanguageIndex

setNextToken

startsWith

tokenToDocument

toString