{
    "mode": "perldoc",
    "parameter": "JSON::Tokenize",
    "section": "",
    "url": "https://www.chedong.com/phpMan.php/perldoc/JSON%3A%3ATokenize/json",
    "generated": "2026-06-11T10:52:38Z",
    "synopsis": "use JSON::Tokenize ':all';\nmy $input = '{\"tuttie\":[\"fruity\", true, 100]}';\nmy $token = tokenizejson ($input);\nprinttokens ($token, 0);\nsub printtokens\n{\nmy ($token, $depth) = @;\nwhile ($token) {\nmy $start = tokenizestart ($token);\nmy $end = tokenizeend ($token);\nmy $type = tokenizetype ($token);\nprint \"   \" x $depth;\nmy $value = substr ($input, $start, $end - $start);\nprint \"'$value' has type '$type'.\\n\";\nmy $child = tokenizechild ($token);\nif ($child) {\nprinttokens ($child, $depth+1);\n}\nmy $next = tokenizenext ($token);\n$token = $next;\n}\n}\nThis outputs\n'{\"tuttie\":[\"fruity\", true, 100]}' has type 'object'.\n'\"tuttie\"' has type 'string'.\n':' has type 'colon'.\n'[\"fruity\", true, 100]' has type 'array'.\n'\"fruity\"' has type 'string'.\n',' has type 'comma'.\n'true' has type 'literal'.\n',' has type 'comma'.\n'100' has type 'number'.",
    "sections": {
        "NAME": {
            "content": "JSON::Tokenize - Tokenize JSON\n",
            "subsections": []
        },
        "SYNOPSIS": {
            "content": "use JSON::Tokenize ':all';\n\nmy $input = '{\"tuttie\":[\"fruity\", true, 100]}';\nmy $token = tokenizejson ($input);\nprinttokens ($token, 0);\n\nsub printtokens\n{\nmy ($token, $depth) = @;\nwhile ($token) {\nmy $start = tokenizestart ($token);\nmy $end = tokenizeend ($token);\nmy $type = tokenizetype ($token);\nprint \"   \" x $depth;\nmy $value = substr ($input, $start, $end - $start);\nprint \"'$value' has type '$type'.\\n\";\nmy $child = tokenizechild ($token);\nif ($child) {\nprinttokens ($child, $depth+1);\n}\nmy $next = tokenizenext ($token);\n$token = $next;\n}\n}\n\nThis outputs\n\n'{\"tuttie\":[\"fruity\", true, 100]}' has type 'object'.\n'\"tuttie\"' has type 'string'.\n':' has type 'colon'.\n'[\"fruity\", true, 100]' has type 'array'.\n'\"fruity\"' has type 'string'.\n',' has type 'comma'.\n'true' has type 'literal'.\n',' has type 'comma'.\n'100' has type 'number'.\n",
            "subsections": []
        },
        "VERSION": {
            "content": "This documents version 0.61 of JSON::Tokenize corresponding to git commit\n033269fa8972fdce8626aa65cd11a5394ab50492\n<https://github.com/benkasminbullock/JSON-Parse/commit/033269fa8972fdce8626aa65cd11a5394ab50492>\nreleased on Thu Feb 11 09:14:04 2021 +0900.\n",
            "subsections": []
        },
        "DESCRIPTION": {
            "content": "This is a module for tokenizing a JSON string. \"Tokenizing\" means breaking the string into\nindividual tokens, without creating any Perl structures. It uses the same underlying code as\nJSON::Parse. Tokenizing can be used for tasks such as picking out or searching through parts of\na large JSON structure without storing each part of the entire structure in memory.\n\nThis module is an experimental part of JSON::Parse and its interface is likely to change. The\ntokenizing functions are currently written in a very primitive way.\n",
            "subsections": []
        },
        "FUNCTIONS": {
            "content": "tokenizechild\nmy $child = tokenizechild ($child);\n\nWalk the tree of tokens.\n\ntokenizeend\nmy $end = tokenizeend ($token);\n\nGet the end of the token as a byte offset from the start of the string. Note this is a byte\noffset not a character offset.\n\ntokenizejson\nmy $token = tokenizejson ($json);\n\ntokenizenext\nmy $next = tokenizenext ($token);\n\nWalk the tree of tokens.\n\ntokenizestart\nmy $start = tokenizestart ($token);\n\nGet the start of the token as a byte offset from the start of the string. Note this is a byte\noffset not a character offset.\n\ntokenizetext\nmy $text = tokenizetext ($json, $token);\n\nGiven a token $token from this parsing and the JSON in $json, return the text which corresponds\nto the token. This is a convenience function written in Perl which uses \"tokenizestart\" and\n\"tokenizeend\" and \"substr\" to get the string from $json.\n\ntokenizetype\nmy $type = tokenizetype ($token);\n\nGet the type of the token as a string. The possible return values are\n\n\"array\",\n\"initial state\",\n\"invalid\",\n\"literal\",\n\"number\",\n\"object\",\n\"string\",\n\"unicode escape\"\n",
            "subsections": []
        },
        "AUTHOR": {
            "content": "Ben Bullock, <bkb@cpan.org>\n\nCOPYRIGHT & LICENCE\nThis package and associated files are copyright (C) 2016-2021 Ben Bullock.\n\nYou can use, copy, modify and redistribute this package and associated files under the Perl\nArtistic Licence or the GNU General Public Licence.\n",
            "subsections": []
        }
    },
    "summary": "JSON::Tokenize - Tokenize JSON",
    "flags": [],
    "examples": [],
    "see_also": []
}