Function TokenInit() | Xbase++

Prepares a string for the incremental tokenizer.

TokenInit( [<@cString>]  , ;
           [<cDelimiter>], ;
           [<nSkipStep>]   ;  // Default: 0
         ) --> lSuccess

Returns .T. if the string could be initialized for the tokenizer otherwise .F.

<cDelimiter> is a string containing token delimiters. It defaults to a string with the following characters:

CHR() - 0, 9, 10, 13, 26, 32, 138, 141 
plus  - ,.;:!?/\<>()#&%+-*

All parts of a string which are separated by a character of <cDelimiter> are recognized as token. If two delimiters are adjacent to each other the token is empty (a null string).

<nSkipStep> defines after how many characters TokenNext() returns even if a token is empty.

NOTES:

The parameter <cString> must be passed by reference. It is prepared for the tokenizer by exchanging all possible delimiting characters with the first character of <cDelimiter>. This enhances performance considerably. If <cString> is needed after tokenizing is complete a copy of it should be passed to TokenInit(). The function alters <cString>.

Calling TokenInit() without a parameter resets the tokenizer to the beginning of <cString>.

If you see anything in the documentation that is not correct, does not match your experience with the particular feature or requires further clarification, please use this form to report a documentation issue.

Function TokenInit() Foundation