# Plucene::Analysis::Tokenizer - phpMan

## NAME
    [Plucene::Analysis::Tokenizer] - base class for tokenizers

## SYNOPSIS
            my $tokenizer = [Plucene::Analysis::Tokenizer::Subclass]
                    ->new({ reader => $reader });

## DESCRIPTION
    This is an abstract base class for tokenizers.

    A Tokenizer is a TokenStream whose input is a Reader.

## METHODS
  new
            my $tokenizer = [Plucene::Analysis::Tokenizer::Subclass]
                    ->new({ reader => $reader });

    This will create a new tokenizer.

  next
    This must be defined in a subclass

  close
    Close the input reader.