perldoc > HTML::TokeParser

man perldoc info pydoc3 ri search

📘 NAME

HTML::TokeParser — Alternative HTML::Parser interface

🚀 Quick Reference

Use Case	Command	Description
📄 Create parser from file	HTML::TokeParser->new("index.html")	Opens and parses a file; returns undef on failure
📄 Create parser from filehandle	HTML::TokeParser->new($fh)	Reads from an open filehandle (not closed)
📄 Create parser from string	HTML::TokeParser->new(\$document)	Parses a scalar reference (literal HTML)
🔍 Get next token	$p->get_token	Returns arrayref: type (S/E/T/C/D/PI) + data
🏷️ Get next tag (skip others)	$p->get_tag	Returns start/end tag info, or undef at end
🏷️ Get specific tag	$p->get_tag("a", "/a")	Skips until one of the listed tags found
📝 Get text until tag	$p->get_text("p")	Returns all text before first specified tag
✂️ Get trimmed text	$p->get_trimmed_text	Collapses whitespace, removes leading/trailing
💬 Get phrase text	$p->get_phrase	Ignores phrasal-level tags, collapses whitespace
↩️ Push back tokens	$p->unget_token(@tokens)	Returns tokens to be read next
🔗 Extract all links	$p->get_tag("a") + get_trimmed_text("/a")	Iterates <a> tags, prints URL and text
📌 Extract title	$p->get_tag("title") + get_trimmed_text	Gets the text between <title>…</title>

📝 SYNOPSIS

use HTML::TokeParser;
$p = HTML::TokeParser->new("index.html") ||
     die "Can't open: $!";
$p->empty_element_tags(1);  # configure its behaviour

while (my $token = $p->get_token) {
    #...
}

📖 DESCRIPTION

The HTML::TokeParser is an alternative interface to the HTML::Parser class. It is an HTML::PullParser subclass with a predeclared set of token types. If you wish the tokens to be reported differently you probably want to use the HTML::PullParser directly.

The following methods are available:

🛠️ $p = HTML::TokeParser->new( $filename, %opt )

Constructs a new parser. The argument is either a file name, a file handle object, or a reference to the complete document. Extra options are key/value pairs processed by base classes.

• 🔹 Plain scalar → treated as filename to open. Returns undef on failure, $! explains why.
• 🔹 Reference to scalar → the scalar is the literal document to parse. Do not change it before extracting all tokens.
• 🔹 Other object → read from (like filehandle). Read until EOF, not closed.

A newly constructed parser has unbroken_text enabled by default. See HTML::Parser for other attributes. It is often a good idea to enable empty_element_tags.

🔴 Important: Parsing raw undecoded UTF-8 may produce invalid results. When parsing UTF-8 encoded files, turn on decoding:

open(my $fh, "<:utf8", "index.html") || die "Can't open 'index.html': $!";
my $p = HTML::TokeParser->new( $fh );
# ...

If a filename is passed, the file is opened in raw mode and the result is only valid for Latin-1 or pure ASCII.

When parsing from a UTF-8 encoded string buffer, decode it first:

utf8::decode($document);
my $p = HTML::TokeParser->new( \$document );
# ...

🔍 $p->get_token

Returns the next token as an array reference, or undef at end of document. The first element is a type string. Token structure depends on type:

• ["S", $tag, $attr, $attrseq, $text] — start tag
• ["E", $tag, $text] — end tag
• ["T", $text, $is_data] — text
• ["C", $text] — comment
• ["D", $text] — declaration
• ["PI", $token0, $text] — process instruction

🔍 $attr is a hash reference, $attrseq is an array reference. See HTML::Parser for details.

↩️ $p->unget_token( @tokens )

Pushes back tokens so they are returned by the next get_token call.

🏷️ $p->get_tag / $p->get_tag( @tags )

Returns the next start or end tag (skipping other tokens), or undef if none remain. If one or more arguments are given, skip until one of the specified tag types is found. For example:

$p->get_tag("font", "/font");

Returns an array reference similar to get_token but without the type code. Start tag: [$tag, $attr, $attrseq, $text]. End tag tagname is prefixed with /: ["/$tag", $text].

📝 $p->get_text / $p->get_text( @endtags )

Returns all text found at the current position. Returns zero-length string if next token is not text. Entities are converted to characters. If one or more arguments are given, return text occurring before the first of the specified tags. For example:

$p->get_text("p", "br");

Text might span tags that should be textified. Controlled by the $p->{textify} attribute, which is a hash. If a start tag name matches a key, that tag is converted to text. The hash value specifies which attribute to use for the text (e.g., alt). If missing, the uppercase tag name in brackets is returned (e.g., [IMG]). The value can also be a subroutine reference. Default textify:

{img => "alt", applet => "alt"}

This means <IMG> and <APPLET> are treated as text, using the ALT attribute.

✂️ $p->get_trimmed_text / $p->get_trimmed_text( @endtags )

Same as get_text, but collapses sequences of whitespace to a single space, and removes leading/trailing whitespace.

💬 $p->get_phrase

Returns all text found at the current position, ignoring any phrasal-level tags. Text is extracted until the first non-phrasal-level tag. Textification works as in get_text. Whitespace is collapsed as in get_trimmed_text. The definition of phrasal-level tags comes from HTML::Tagset.

📚 EXAMPLES

🔗 Extract all links

use HTML::TokeParser;
$p = HTML::TokeParser->new(shift||"index.html");

while (my $token = $p->get_tag("a")) {
    my $url = $token->[1]{href} || "-";
    my $text = $p->get_trimmed_text("/a");
    print "$url\t$text\n";
}

📌 Extract the document title

use HTML::TokeParser;
$p = HTML::TokeParser->new(shift||"index.html");
if ($p->get_tag("title")) {
    my $title = $p->get_trimmed_text;
    print "Title: $title\n";
}

🔗 SEE ALSO

• HTML::PullParser
• HTML::Parser

©️ COPYRIGHT

Copyright 1998–2005 Gisle Aas.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

📘 NAME 🚀 Quick Reference 📝 SYNOPSIS 📖 DESCRIPTION 📚 EXAMPLES 🔗 SEE ALSO ©️ COPYRIGHT