File: |
tokenizer.php |
Role: |
Class source |
Content type: |
text/plain |
Description: |
source code |
Class: |
Tokenizer
Tokenizer split strings into tokens |
Author: |
By Domenico Pontari |
Last change: |
New algorithm for the generation of tokens: using "explode" function instead of "preg_split". So that it is easier to manage special chars. Performances improvement expected. |
Date: |
14 years ago |
Size: |
13,263 bytes |
|
|
|