perldoc > Plucene::Analysis::Tokenizer(3pm)

NAME
    Plucene::Analysis::Tokenizer - base class for tokenizers

SYNOPSIS
            my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
                    ->new({ reader => $reader });

DESCRIPTION
    This is an abstract base class for tokenizers.

    A Tokenizer is a TokenStream whose input is a Reader.

METHODS
  new
            my $tokenizer = Plucene::Analysis::Tokenizer::Subclass
                    ->new({ reader => $reader });

    This will create a new tokenizer.

  next
    This must be defined in a subclass

  close
    Close the input reader.