token_get_all()

(PHP 4 >= 4.2.0, PHP 5, PHP 7)

将提供的源码按 PHP 标记进行分割

说明

token_get_all()解析提供的$source源码字符，然后使用 Zend 引擎的语法分析器获取源码中的 PHP 语言的解析器代号

解析器代号列表见解析器代号列表,或者使用token_name()翻译获取这个代号的字符串表示.

参数

需要解析的 PHP 源码.

返回值

An array of token identifiers. Each individual token identifier is either a single character (i.e.:;,., or a three element array containing the token index in element 0, the string content of the original token in element 1 and the line number in element 2.

范例

Example #1token_get_all()examples

 tags in a normal file. */
$tokens = token_get_all('/* comment */'); // => array(array(T_INLINE_HTML, '/* comment */'));
?>

更新日志

Yes, some problems (On WAMP, PHP 5.3.0 ) with get_token_all() 
1 : bug line numbers
 Since PHP 5.2.2 token_get_all() should return Line numbers in element 2..
.. but for instance (5.3.0 on WAMP), it work perfectly only with PHP code (not HMTL miwed), but if you have some T_INLINE_HTML detected by token_get_all() , sometimes you find wrongs line numbers (return next line)... :(
2: bug warning message can impact loops
Warning with php code uncompleted (ex : php code line by line) :
for example if a comment tag is not closed token_get_all() can block loops on this warning :
Warning: Unterminated comment starting line
This problem seem not occur in CLI mod (php command line), but only in web mod.
Waiting more stability, used token_get_all() only on PHP code (not HMTL miwed) :
First extract entirely PHP code (with open et close php tag), 
Second use token_get_all() on the pure PHP code.
3 : Why there not function to extract PHP code (to extract HTML, we have Tidy..)?
Waiting, I used a function :
The code at end this post :
http://www.developpez.net/forums/d786381/php/langage/
fonctions/analyser-fichier-php-token_get_all/
This function not support :
- Old notation : "" and ""
- heredoc syntax 
- nowdoc syntax (since PHP 5.3.0)

I wanted to use the tokenizer functions to count source lines of code, including counting comments. Attempting to do this with regular expressions does not work well because of situations where /* appears in a string, or other situations. The token_get_all() function makes this task easy by detecting all the comments properly. However, it does not tokenize newline characters. I wrote the below set of functions to also tokenize newline characters as T_NEW_LINE.

Example usage:

鹏仔微信 15129739599 鹏仔QQ344225443 鹏仔前端 pjxi.com 共享博客 sharedbk.com