# Web::Scraper - Web Scraping Toolkit using HTML and CSS Selectors or X... - perldoc - [phpMan]

## NAME
    [Web::Scraper](https://www.chedong.com/phpMan.php/perldoc/Web%3A%3AScraper/markdown) - Web Scraping Toolkit using HTML and CSS Selectors or XPath expressions

## SYNOPSIS
      use URI;
      use [Web::Scraper](https://www.chedong.com/phpMan.php/perldoc/Web%3A%3AScraper/markdown);
      use Encode;

      # First, create your scraper block
      my $authors = scraper {
          # Parse all TDs inside 'table[width="100%]"', store them into
          # an array 'authors'.  We embed other scrapers for each TD.
          process 'table[width="100%"] td', "authors[]" => scraper {
            # And, in each TD,
            # get the URI of "a" element
            process "a", uri => '@href';
            # get text inside "small" element
            process "small", fullname => 'TEXT';
          };
      };

      my $res = $authors->scrape( URI->new("<http://search.cpan.org/author/?A>") );

      # iterate the array 'authors'
      for my $author (@{$res->{authors}}) {
          # output is like:
          # Andy Adler      <http://search.cpan.org/~aadler/>
          # Aaron K Dancygier       <http://search.cpan.org/~aakd/>
          # Aamer Akhter    <http://search.cpan.org/~aakhter/>
          print [Encode::encode](https://www.chedong.com/phpMan.php/perldoc/Encode%3A%3Aencode/markdown)("utf8", "$author->{fullname}\t$author->{uri}\n");
      }

    The structure would resemble this (visually) { authors => [ { fullname => $fullname, link =>
    $uri }, { fullname => $fullname, link => $uri }, ] }

## DESCRIPTION
    [Web::Scraper](https://www.chedong.com/phpMan.php/perldoc/Web%3A%3AScraper/markdown) is a web scraper toolkit, inspired by Ruby's equivalent Scrapi. It provides a
    DSL-ish interface for traversing HTML documents and returning a neatly arranged Perl data
    structure.

    The *scraper* and *process* blocks provide a method to define what segments of a document to
    extract. It understands HTML and CSS Selectors as well as XPath expressions.

## METHODS
  scraper
      $scraper = scraper { ... };

    Creates a new [Web::Scraper](https://www.chedong.com/phpMan.php/perldoc/Web%3A%3AScraper/markdown) object by wrapping the DSL code that will be fired when *scrape*
    method is called.

  scrape
      $res = $scraper->scrape(URI->new($uri));
      $res = $scraper->scrape($html_content);
      $res = $scraper->scrape(\$html_content);
      $res = $scraper->scrape($http_response);
      $res = $scraper->scrape($html_element);

    Retrieves the HTML from URI, [HTTP::Response](https://www.chedong.com/phpMan.php/perldoc/HTTP%3A%3AResponse/markdown), [HTML::Tree](https://www.chedong.com/phpMan.php/perldoc/HTML%3A%3ATree/markdown) or text strings and creates a DOM
    object, then fires the callback scraper code to retrieve the data structure.

    If you pass URI or [HTTP::Response](https://www.chedong.com/phpMan.php/perldoc/HTTP%3A%3AResponse/markdown) object, [Web::Scraper](https://www.chedong.com/phpMan.php/perldoc/Web%3A%3AScraper/markdown) will automatically guesses the encoding
    of the content by looking at Content-Type headers and META tags. Otherwise you need to decode
    the HTML to Unicode before passing it to *scrape* method.

    You can optionally pass the base URL when you pass the HTML content as a string instead of URI
    or [HTTP::Response](https://www.chedong.com/phpMan.php/perldoc/HTTP%3A%3AResponse/markdown).

      $res = $scraper->scrape($html_content, "<http://example.com/foo>");

    This way [Web::Scraper](https://www.chedong.com/phpMan.php/perldoc/Web%3A%3AScraper/markdown) can resolve the relative links found in the document.

  process
      scraper {
          process "tag.class", key => 'TEXT';
          process '//tag[contains(@foo, "bar")]', key2 => '@attr';
          process '//comment()', 'comments[]' => 'TEXT';
      };

    *process* is the method to find matching elements from HTML with CSS selector or XPath
    expression, then extract text or attributes into the result stash.

    If the first argument begins with "//" or "id(" it's treated as an XPath expression and
    otherwise CSS selector.

      # <span class="date">2008/12/21</span>
      # date => "2008/12/21"
      process ".date", date => 'TEXT';

      # <div class="body"><a href="<http://example.com/>">foo</a></div>
      # link => URI->new("<http://example.com/>")
      process ".body > a", link => '@href';

      # <div class="body"><!-- HTML Comment here --><a href="<http://example.com/>">foo</a></div>
      # comment => " HTML Comment here "
      #
      # NOTES: A comment nodes are accessed when installed
      # the [HTML::TreeBuilder::XPath](https://www.chedong.com/phpMan.php/perldoc/HTML%3A%3ATreeBuilder%3A%3AXPath/markdown) (version >= 0.14) and/or
      # the [HTML::TreeBuilder::LibXML](https://www.chedong.com/phpMan.php/perldoc/HTML%3A%3ATreeBuilder%3A%3ALibXML/markdown) (version >= 0.13)
      process "//div[contains(@class, 'body')]/comment()", comment => 'TEXT';

      # <div class="body"><a href="<http://example.com/>">foo</a></div>
      # link => URI->new("<http://example.com/>"), text => "foo"
      process ".body > a", link => '@href', text => 'TEXT';

      # <ul><li>foo</li><li>bar</li></ul>
      # list => [ "foo", "bar" ]
      process "li", "list[]" => "TEXT";

      # <ul><li id="1">foo</li><li id="2">bar</li></ul>
      # list => [ { id => "1", text => "foo" }, { id => "2", text => "bar" } ];
      process "li", "list[]" => { id => '@id', text => "TEXT" };

  process_first
    "process_first" is the same as "process" but stops when the first matching result is found.

      # <span class="date">2008/12/21</span>
      # <span class="date">2008/12/22</span>
      # date => "2008/12/21"
      process_first ".date", date => 'TEXT';

  result
    "result" allows one to return not the default value after processing but a single value
    specified by a key or a hash reference built from several keys.

      process 'a', 'want[]' => 'TEXT';
      result 'want';

## EXAMPLES
    There are many examples in the "eg/" dir packaged in this distribution. It is recommended to
    look through these.

## NESTED SCRAPERS
    Scrapers can be nested thus allowing to scrape already captured data.

      # <ul>
      # <li class="foo"><a href="foo1">bar1</a></li>
      # <li class="bar"><a href="foo2">bar2</a></li>
      # <li class="foo"><a href="foo3">bar3</a></li>
      # </ul>
      # friends => [ {href => 'foo1'}, {href => 'foo2'} ];
      process 'li', 'friends[]' => scraper {
        process 'a', href => '@href',
      };

## FILTERS
    Filters are applied to the result after processing. They can be declared as anonymous
    subroutines or as class names.

      process $exp, $key => [ 'TEXT', sub { s/foo/bar/ } ];
      process $exp, $key => [ 'TEXT', 'Something' ];
      process $exp, $key => [ 'TEXT', '+[MyApp::Filter::Foo](https://www.chedong.com/phpMan.php/perldoc/MyApp%3A%3AFilter%3A%3AFoo/markdown)' ];

    Filters can be stacked

      process $exp, $key => [ '@href', 'Foo', '+[MyApp::Filter::Bar](https://www.chedong.com/phpMan.php/perldoc/MyApp%3A%3AFilter%3A%3ABar/markdown)', \&baz ];

    More about filters you can find in [Web::Scraper::Filter](https://www.chedong.com/phpMan.php/perldoc/Web%3A%3AScraper%3A%3AFilter/markdown) documentation.

XML backends
    By default [HTML::TreeBuilder::XPath](https://www.chedong.com/phpMan.php/perldoc/HTML%3A%3ATreeBuilder%3A%3AXPath/markdown) is used, this can be replaces by a [XML::LibXML](https://www.chedong.com/phpMan.php/perldoc/XML%3A%3ALibXML/markdown) backend using
    [Web::Scraper::LibXML](https://www.chedong.com/phpMan.php/perldoc/Web%3A%3AScraper%3A%3ALibXML/markdown) module.

      use [Web::Scraper::LibXML](https://www.chedong.com/phpMan.php/perldoc/Web%3A%3AScraper%3A%3ALibXML/markdown);

      # same as [Web::Scraper](https://www.chedong.com/phpMan.php/perldoc/Web%3A%3AScraper/markdown)
      my $scraper = scraper { ... };

## AUTHOR
    Tatsuhiko Miyagawa <<miyagawa@bulknews.net>>

## LICENSE
    This library is free software; you can redistribute it and/or modify it under the same terms as
    Perl itself.

## SEE ALSO
    <<http://blog.labnotes.org/category/scrapi/>>

    [HTML::TreeBuilder::XPath](https://www.chedong.com/phpMan.php/perldoc/HTML%3A%3ATreeBuilder%3A%3AXPath/markdown)