# pydoc > tokenize

---
type: CommandReference
command: tokenize
mode: pydoc
section: ""
source: pydoc3
---

## Quick Reference

- `tokenize(readline)` — tokenize a Python source file (bytes), yields 5‑tuples
- `generate_tokens(readline)` — tokenize source as unicode strings, same API
- `untokenize(iterable)` — transform tokens back into Python source code (bytes)
- `detect_encoding(readline)` — detect encoding from a source file, returns (encoding, lines)

## Name

tokenize — Tokenization help for Python programs.

## Synopsis

The `tokenize` module provides a tokenizer for Python source code. It decodes bytes according to PEP‑263 and yields 5‑tuples: `(type, string, (srow, scol), (erow, ecol), line)`. The first token is always an `ENCODING` token. It can also detect the encoding of a source file.

## Functions

- `tokenize(readline)` — generator that produces 5‑tuples from a callable returning bytes (e.g., `file.readline`). The first token is `ENCODING`.
- `generate_tokens(readline)` — same as `tokenize()` but expects `readline` to return `str` objects.
- `untokenize(iterable)` — transforms token sequences back into Python source code as bytes. Each element must have at least two elements (token number, token value). Round‑trip invariant: full input yields identical source.
- `detect_encoding(readline)` — detects encoding from a source file (reads up to two lines). Returns `(encoding: str, lines: list of bytes)`. Handles BOM and PEP‑263 cookies. Raises `SyntaxError` on conflict or invalid charset.

## Classes

- `TokenInfo(type, string, start, end, line)` — a named tuple (subclass of `tuple`). Fields: `type`, `string`, `start`, `end`, `line`. Readonly property `exact_type`. Inherits tuple methods: `__add__`, `__contains__`, `count`, `index`, etc. Also has `_asdict()`, `_replace()`, `_make(iterable)`.

## Data

- `tok_name` — dict mapping token numbers to their symbolic names.
- `__all__` — list of exported names: `['tok_name', 'ISTERMINAL', 'ISNONTERMINAL', 'ISEOF', 'ENDMARKER', ...]`.
- Constants: `AMPER`, `AMPEQUAL`, `ASYNC`, `AT`, `ATEQUAL`, `AWAIT`, `CIRCUMFLEX`, `CIRCUMFLEXEQUAL`, `COLON`, `COLONEQUAL`, `COMMA`, `COMMENT`, `DEDENT`, `DOT`, `DOUBLESLASH`, `DOUBLESLASHEQUAL`, `DOUBLESTAR`, `DOUBLESTAREQUAL`, `ELLIPSIS`, `ENCODING`, `ENDMARKER`, `EQEQUAL`, `EQUAL`, `ERRORTOKEN`, `GREATER`, `GREATEREQUAL`, `INDENT`, `LBRACE`, `LEFTSHIFT`, `LEFTSHIFTEQUAL`, `LESS`, `LESSEQUAL`, `LPAR`, `LSQB`, `MINEQUAL`, `MINUS`, `NAME`, `NEWLINE`, `NL`, `NOTEQUAL`, `NT_OFFSET`, `NUMBER`, `N_TOKENS`, `OP`, `PERCENT`, `PERCENTEQUAL`, `PLUS`, `PLUSEQUAL`, `RARROW`, `RBRACE`, `RIGHTSHIFT`, `RIGHTSHIFTEQUAL`, `RPAR`, `RSQB`, `SEMI`, `SLASH`, `SLASHEQUAL`, `SOFT_KEYWORD`, `STAR`, `STAREQUAL`, `STRING`, `TILDE`, `TYPE_COMMENT`, `TYPE_IGNORE`, `VBAR`, `VBAREQUAL`.

## Examples

python
import tokenize
import io

# Tokenize a file
with open('example.py', 'rb') as f:
    tokens = tokenize.tokenize(f.readline)
    for tok in tokens:
        print(tok)
python
# Tokenize a string (unicode)
import tokenize
import io

source = "print('hello')"
tokens = tokenize.generate_tokens(io.StringIO(source).readline)
for tok in tokens:
    print(tok)
python
# Untokenize
import tokenize
import io
from io import BytesIO

with open('example.py', 'rb') as f:
    t1 = [tok[:2] for tok in tokenize.tokenize(f.readline)]
newcode = tokenize.untokenize(t1)
readline = BytesIO(newcode).readline
t2 = [tok[:2] for tok in tokenize.tokenize(readline)]
assert t1 == t2
## See Also

- [Python tokenize module documentation](https://docs.python.org/3/library/tokenize.html)
- [`token` module](https://docs.python.org/3/library/token.html) — constants for token types
- [PEP 263](https://www.python.org/dev/peps/pep-0263/) — defining source code encodings