TSourceToken (gudusoft.gsqlparser 2.0.1.0 API)

java.lang.Object
- gudusoft.gsqlparser.TSourceToken

```
public class TSourceToken
extends Object
```
Represents a source token which is the basic syntactic unit of SQL. A token can be a key word, an identifier, a quoted identifier, a literal (or constant), or a special character symbol. Tokens are normally separated by whitespace (space, tab, newline), but need not be if there is no ambiguity (which is generally only the case if a special character is adjacent to some other token type).
The parse tree node consists of source tokens.

A list of source token will be available after parse or tokenize the input SQL.
```
 

   TGSqlParser sqlparser = new TGSqlParser(EDbVendor.dbvoracle);
   sqlparser.sqltext = "select col from t";
   int ret = sqlparser.parse();
   if (ret == 0){
    for(int i=0;i<sqlparser.sourcetokenlist.size();i++){
      TSourceToken st =  sqlparser.sourcetokenlist.get(i);
      System.out.println(st.tokentype.toString()+" "+st.toString());
    }
   }else{
     System.out.println(sqlparser.getErrormessage());
   }

 
 
```
Get a list of source tokens after call the method TGSqlParser.parse() or just call TGSqlParser.tokenizeSqltext() if you only need to access tokens of input SQL without generating the full parse tree.

tokencode is the unique id represents the type of token, some typical tokens are: whitespace, return, keyword, identifier. This value is mainly used by the parser internally.
tokentype uniquely identify the token type in a more meaningful way. It's more easier to use this field in your program than tokencode.

Field Summary

Fields
Modifier and Type	Field and Description
`TSourceToken`	`alternativeToken` Deprecated. since v1.8.8.0, use scriptWriter technology to modify the node
`String`	`astext` Text representation for this token.
`long`	`columnNo` the column number of the first character in this token
`TSourceTokenList`	`container` Container for this token which is a list of source token, this is the reference to `TGSqlParser.sourcetokenlist`
`String`	`dolqstart` Start part of Dollar-quoted String Constants of PostgreSQL.
`long`	`lineNo` the line number of the first character in this token
`long`	`offset` Token's offset from the beginning of the input query.
`int`	`posinlist` Index of this token in the `container`, start from 0
`TCustomSqlStatement`	`stmt` SQL statement that owns this token.
`int`	`tag` Space to save a value for temporary use
`int`	`tokencode` Unique id of this token used by parser internally.
`ETokenStatus`	`tokenstatus` Maintenance the status of this token during lex and parsing.
`ETokenType`	`tokentype` Uniquely identify the token type in a more meaningful way.

Constructor Summary

Constructors
Constructor and Description

TSourceToken()
Class constructor

TSourceToken(String s)
Class constructor, set a string value.

Constructors
Constructor and Description
`TSourceToken()` Class constructor
`TSourceToken(String s)` Class constructor, set a string value.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`boolean`	`firstTokenOfLine()` Check to see if this token is the first token in a line of the input SQL
`TSourceToken`	`getAlternativeToken()` Deprecated. since v1.8.8.0, use scriptWriter technology to modify the node If you find a start token of parse tree node has an alternativetoken, then text of this node should be modified by using TParseTreeNode.setString(String sqlSegment) 1. new string will be tokenized into a list of source tokens: stlist 2. link alternativetoken of start token of this node to the first token in stlist generated in step 1. 3. link back alternativetoken of last token in stlist to the last token of this node.
`EDbObjectType`	`getDbObjectType()` Token in a `TObjectName` has the same database object type as the objectName.
`int`	`getDbObjType()` Token in a `TObjectName` has the same database object type as the objectName.
`EDbVendor`	`getDbvendor()` The database vendor which the SQL script includes this token will run against
`TGSqlParser`	`getGsqlparser()` The parser that create this token.
`TSourceToken`	`getLinkToken()` Gets the linked token.
`TParseTreeNodeList`	`getNodesEndWithThisToken()` Deprecated. since v1.8.8.0, use scriptWriter technology to modify the node A list of nodes whose end token is this token.
`TParseTreeNodeList`	`getNodesStartFromThisToken()` Deprecated. since v1.8.8.0, use scriptWriter technology to modify the node A list of node whose start token is this token
`TSourceToken`	`getReplaceToken()` Used in sql formatter package only
`String`	`getTextWithoutQuoted()` String literal in SQL usually inside ``.
`TSourceTokenList`	`getTokensAfter()` Used in sql formatter package only.
`TSourceTokenList`	`getTokensBefore()` Used in sql formatter package only
`boolean`	`isnonsolidtoken()` Is this token a solid token or not.
`static boolean`	`isnonsolidtoken(ETokenType tokentype)` Space, return, comments are treated as non-solid token by default
`boolean`	`issolidtoken()` Is this token a non-solid token or not.
`boolean`	`lastTokenOfLine()` Check to see if this token is the last token of in a line in the input SQL
`TSourceToken`	`nextSolidToken()` The next token whose `tokentype` is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
`TSourceToken`	`nextSolidToken(boolean treatCommentAsSolidToken)` The next token whose `tokentype` is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
`TSourceToken`	`nextToken()`
`TSourceToken`	`prevSolidToken()` The previous token whose `tokentype` is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
`TSourceToken`	`prevSolidToken(boolean treatCommentAsSolidToken)` The previous token whose `tokentype` is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment
`int`	`removeMeFromTokenList()` This method is not used.
`TSourceToken`	`searchToken(int targetTokenCode, int range)` Search a token before or after this token in the same source token list.
`TSourceToken`	`searchToken(String targetTokenText, int range)` Search a token before or after this token in the same source token list.
`TSourceToken`	`searchTokenAfterObjectName()` Search the first non-solid token after the next objectName.
`void`	`setAlternativeToken(TSourceToken alternativeToken)` Deprecated. since v1.8.8.0, use scriptWriter technology to modify the node
`void`	`setDbObjectType(EDbObjectType dbObjectType)` Set the database object type of this token
`void`	`setDbObjType(int dbObjType)` Deprecated. use `setDbObjectType(gudusoft.gsqlparser.EDbObjectType)` instead.
`void`	`setDbvendor(EDbVendor dbvendor)` The database vendor which the SQL script includes this token will run against
`void`	`setGsqlparser(TGSqlParser gsqlparser)`
`void`	`setLinkToken(TSourceToken linkToken)` Create a link between two tokens.
`void`	`setReplaceToken(TSourceToken replaceToken)` Used in sql formatter package only
`void`	`setString(String str)` set new string of this token
`String`	`toScript()` The string text of this token
`String`	`toString()` The original string text for this token.
`String`	`toStringDebug()` String text with the debug information such as coordinate, token code, token type
`String`	`toUnQuotedString()` Remove double quote `""`, bracket quote `[]` , left/right brace `{}` from a delimited identifier and return string text of this identifier.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - tokencode
```
public int tokencode
```
    Unique id of this token used by parser internally. check available value start from TBaseType.cmtslashstar
  - lineNo
```
public long lineNo
```
    the line number of the first character in this token
  - columnNo
```
public long columnNo
```
    the column number of the first character in this token
  - offset
```
public long offset
```
    Token's offset from the beginning of the input query.
```
     
    public void testOffset(){
        TGSqlParser sqlparser = new TGSqlParser(EDbVendor.dbvoracle);
        sqlparser.sqltext = "select f from t\n" +
                "where f>1\n";
        assertTrue(sqlparser.parse() == 0);
        for (int i=0;i<sqlparser.sourcetokenlist.size();i++){
            TSourceToken st = sqlparser.sourcetokenlist.get(i);
            String textFromOffset = sqlparser.sqltext.toString().substring((int)st.offset,(int)st.offset+st.toString().length());
            assertTrue(st.toString().equalsIgnoreCase(textFromOffset));
        }
    }
     
 
```
  - tokentype
```
public ETokenType tokentype
```
    Uniquely identify the token type in a more meaningful way. It's more easier to use this field in your program than tokencode. check available value in ETokenType
  - container
```
public TSourceTokenList container
```
    Container for this token which is a list of source token, this is the reference to TGSqlParser.sourcetokenlist
  - posinlist
```
public int posinlist
```
    Index of this token in the container, start from 0
```
     
    public void testPosinList(){
        TGSqlParser sqlparser = new TGSqlParser(EDbVendor.dbvoracle);
        sqlparser.sqltext = "select f from t\n" +
                "where f>1\n";
        assertTrue(sqlparser.parse() == 0);
        for (int i=0;i<sqlparser.sourcetokenlist.size();i++){
            assertTrue(i == sqlparser.sourcetokenlist.get(i).posinlist);
        }
    }
     
 
```
  - astext
```
public String astext
```
    Text representation for this token.
  - tokenstatus
```
public ETokenStatus tokenstatus
```
    Maintenance the status of this token during lex and parsing. Used by the parser internally.
  - dolqstart
```
public String dolqstart
```
    Start part of Dollar-quoted String Constants of PostgreSQL. While the standard syntax for specifying string constants is usually convenient, it can be difficult to understand when the desired string contains many single quotes or backslashes, since each of those must be doubled. To allow more readable queries in such situations, PostgreSQL provides another way, called "dollar quoting", to write string constants. A dollar-quoted string constant consists of a dollar sign ($), an optional "tag" of zero or more characters, another dollar sign, an arbitrary sequence of characters that makes up the string content, a dollar sign, the same tag that began this dollar quote, and a dollar sign.
    For example, here are two different ways to specify the string "Dianne's horse" using dollar quoting:
```
     $$Dianne's horse$$
     $SomeTag$Dianne's horse$SomeTag$
 
```
    This field will return $$ and $SomeTag$ accordingly.
  - stmt
```
public TCustomSqlStatement stmt
```
    SQL statement that owns this token.
  - tag
```
public int tag
```
    Space to save a value for temporary use
  - alternativeToken
```
public TSourceToken alternativeToken
```
    Deprecated. since v1.8.8.0, use scriptWriter technology to modify the node
- Constructor Detail
  - TSourceToken
```
public TSourceToken()
```
    Class constructor
  - TSourceToken
```
public TSourceToken(String s)
```
    Class constructor, set a string value.
    
    Parameters:
    
    s - the string value this toke represent for.
- Method Detail
  - getTextWithoutQuoted
```
public String getTextWithoutQuoted()
```
    String literal in SQL usually inside ``. Quoted identifier in SQL usually surrounded by [ and ], This method only returns the text inside those surroundings.
    
    Returns:
    
    text inside `` and []
  - toScript
```
public String toScript()
```
    The string text of this token
    
    Returns:
    
    the string text of this token
  - getGsqlparser
```
public TGSqlParser getGsqlparser()
```
    The parser that create this token.
    
    Returns:
    
    the parser that created this token, return null if this token created manually.
  - setGsqlparser
```
public void setGsqlparser(TGSqlParser gsqlparser)
```
  - getNodesEndWithThisToken
```
public TParseTreeNodeList getNodesEndWithThisToken()
```
    Deprecated. since v1.8.8.0, use scriptWriter technology to modify the node A list of nodes whose end token is this token.
    
    Returns:
    
    A list of nodes whose end token is this token.
  - getNodesStartFromThisToken
```
public TParseTreeNodeList getNodesStartFromThisToken()
```
    Deprecated. since v1.8.8.0, use scriptWriter technology to modify the node A list of node whose start token is this token
    
    Returns:
    
    A list of node whose start token is this token
  - setDbObjType
```
public void setDbObjType(int dbObjType)
```
    Deprecated. use setDbObjectType(gudusoft.gsqlparser.EDbObjectType) instead.
    
    Parameters:
    
    dbObjType - the database object type
  - getDbObjType
```
public int getDbObjType()
```
    Token in a TObjectName has the same database object type as the objectName. Please use TObjectName.getDbObjectType() instead of this method if possible.
    
    Returns:
    
    the type of the database object
  - setDbObjectType
```
public void setDbObjectType(EDbObjectType dbObjectType)
```
    Set the database object type of this token
    
    Parameters:
    
    dbObjectType - database object type
  - getDbObjectType
```
public EDbObjectType getDbObjectType()
```
    Token in a TObjectName has the same database object type as the objectName. Please use TObjectName.getDbObjectType() instead of this method if possible.
    
    Returns:
    
    the type of the database object
  - setAlternativeToken
```
public void setAlternativeToken(TSourceToken alternativeToken)
```
    Deprecated. since v1.8.8.0, use scriptWriter technology to modify the node
  - getAlternativeToken
```
public TSourceToken getAlternativeToken()
```
    Deprecated. since v1.8.8.0, use scriptWriter technology to modify the node
    If you find a start token of parse tree node has an alternativetoken,
    then text of this node should be modified by using TParseTreeNode.setString(String sqlSegment)
    
    1. new string will be tokenized into a list of source tokens: stlist
    2. link alternativetoken of start token of this node to the first token in stlist generated in step 1.
    3. link back alternativetoken of last token in stlist to the last token of this node.
  - setDbvendor
```
public void setDbvendor(EDbVendor dbvendor)
```
    The database vendor which the SQL script includes this token will run against
    
    Parameters:
    
    dbvendor - the database vendor such as Oracle, DB2 and so on.
  - getDbvendor
```
public EDbVendor getDbvendor()
```
    The database vendor which the SQL script includes this token will run against
    
    Returns:
    
    dbvendor the database vendor such as Oracle, DB2 and so on.
  - setString
```
public void setString(String str)
```
    set new string of this token
    
    Parameters:
    
    str - the new string text
  - toString
```
public String toString()
```
    The original string text for this token.
    
    Overrides:
    
    toString in class Object
    
    Returns:
    
    the string text
  - toStringDebug
```
public String toStringDebug()
```
    String text with the debug information such as coordinate, token code, token type
    
    Returns:
    
    the string value with full debug info
  - isnonsolidtoken
```
public static boolean isnonsolidtoken(ETokenType tokentype)
```
    Space, return, comments are treated as non-solid token by default
    
    Parameters:
    
    tokentype - token type
    
    Returns:
    
    true if token type is not one of ttwhitespace,ttreturn,ttsimplecomment,ttbracketedcomment
  - isnonsolidtoken
```
public boolean isnonsolidtoken()
```
    Is this token a solid token or not.
    
    Returns:
    
    true if it's a non-solid token.
  - issolidtoken
```
public boolean issolidtoken()
```
    Is this token a non-solid token or not.
    
    Returns:
    
    true if it's a solid token.
  - getTokensAfter
```
public TSourceTokenList getTokensAfter()
```
    Used in sql formatter package only.
    
    Returns:
    
    source token list
  - getTokensBefore
```
public TSourceTokenList getTokensBefore()
```
    Used in sql formatter package only
    
    Returns:
    
    source token list
  - setReplaceToken
```
public void setReplaceToken(TSourceToken replaceToken)
```
    Used in sql formatter package only
    
    Parameters:
    
    replaceToken - replaced token
  - getReplaceToken
```
public TSourceToken getReplaceToken()
```
    Used in sql formatter package only
    
    Returns:
    
    replaced token
  - nextToken
```
public TSourceToken nextToken()
```
  - searchToken
```
public TSourceToken searchToken(int targetTokenCode,
                                int range)
```
    Search a token before or after this token in the same source token list. The result token should has the tokencode equals to the targetTokenCode in a specified range.
    
    Parameters:
    
    targetTokenCode, - the token code need to be searched
    
    range, - > 0, search token start from the next token and forward, = 0, just compare with this token, < 0, search from the previous token and backword.
    
    Returns:
    
    the token with the same token code, otherwise, return null.
  - searchToken
```
public TSourceToken searchToken(String targetTokenText,
                                int range)
```
    Search a token before or after this token in the same source token list. The result token should has the string text equals to the targetTokenText in a specified range.
    
    Parameters:
    
    targetTokenText, - the target string text
    
    range, - > 0, search token start from the next token and forward, = 0, just compare with this token, < 0, search from the previous token and backword.
    
    Returns:
    
    the token with the same token code, otherwise, return null.
  - searchTokenAfterObjectName
```
public TSourceToken searchTokenAfterObjectName()
```
    Search the first non-solid token after the next objectName.
    Take this SQL for example:
```
 return new scott.func(x1);
 
```
    If this token is new, then call searchTokenAfterObjectName will return ( token.
    Returns:
    
    solid token after the next objectName
  - nextSolidToken
```
public TSourceToken nextSolidToken()
```
    The next token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
    
    Returns:
    
    the next solid token, returns null if not found.
  - nextSolidToken
```
public TSourceToken nextSolidToken(boolean treatCommentAsSolidToken)
```
    The next token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
    
    Parameters:
    
    treatCommentAsSolidToken, - set to true will treat comment token as a solid token
    
    Returns:
    
    the solid token if found, otherwise, return null
  - prevSolidToken
```
public TSourceToken prevSolidToken()
```
    The previous token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment.
    
    Returns:
    
    the solid token if found, otherwise, return null
  - prevSolidToken
```
public TSourceToken prevSolidToken(boolean treatCommentAsSolidToken)
```
    The previous token whose tokentype is not ttreturn,ttwhitespace,ttsimplecomment and ttbracketedcomment
    
    Parameters:
    
    treatCommentAsSolidToken - set to true will treat comment token as a solid token
    
    Returns:
    
    the solid token if found, otherwise, return null
  - firstTokenOfLine
```
public boolean firstTokenOfLine()
```
    Check to see if this token is the first token in a line of the input SQL
    
    Returns:
    
    true if this is the first token of a line, otherwise, return false
  - lastTokenOfLine
```
public boolean lastTokenOfLine()
```
    Check to see if this token is the last token of in a line in the input SQL
    
    Returns:
    
    true if this is the last token of a line, otherwise, return false
  - removeMeFromTokenList
```
public int removeMeFromTokenList()
```
    This method is not used.
    
    Returns:
    
    not used.
  - setLinkToken
```
public void setLinkToken(TSourceToken linkToken)
```
    Create a link between two tokens. Make it easy to access another linked token. Usually, those are two tokens like the parenthesis in this SQL: (select * from t)
    
    Parameters:
    
    linkToken - the token need to be linked
  - getLinkToken
```
public TSourceToken getLinkToken()
```
    Gets the linked token. Take this SQL for example: (select * from t), if this token is '(', then you call this method will return ')' token.
    
    Returns:
    
    the paired parenthesis in select, expression
  - toUnQuotedString
```
public String toUnQuotedString()
```
    Remove double quote "", bracket quote [] , left/right brace {} from a delimited identifier and return string text of this identifier.
    
    Returns:
    
    string text of this token

Class TSourceToken

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

tokencode

lineNo

columnNo

offset

tokentype

container

posinlist

astext

tokenstatus

dolqstart

stmt

tag

alternativeToken

Constructor Detail

TSourceToken

TSourceToken

Method Detail

getTextWithoutQuoted

toScript

getGsqlparser

setGsqlparser

getNodesEndWithThisToken

getNodesStartFromThisToken

setDbObjType

getDbObjType

setDbObjectType

getDbObjectType

setAlternativeToken

getAlternativeToken

setDbvendor

getDbvendor

setString

toString

toStringDebug

isnonsolidtoken

isnonsolidtoken

issolidtoken

getTokensAfter

getTokensBefore

setReplaceToken

getReplaceToken

nextToken

searchToken

searchToken

searchTokenAfterObjectName

nextSolidToken

nextSolidToken

prevSolidToken

prevSolidToken

firstTokenOfLine

lastTokenOfLine

removeMeFromTokenList

setLinkToken

getLinkToken

toUnQuotedString