ConGlomCo to Open New Corporate Office in Ougadougou
\nBAKERSFIELD, CA, 2000-04-24 -- ConGlomCo's vice president in charge\nof world conquest, Rock Feldspar, announced today the opening of a\nnew office in Ougadougou, the capital city of Burkino Faso, gateway\nto the bustling \"Silicon Sahara\" of Africa...\n[...etc...]\n\n...and what you've got to do is, for each document, copy whatever text is in the \"h1\" element,\nso that you can, for example, make a table of contents of it. Now, there are three ways to do\nthis:\n\n* You can just use a regexp to scan the file for a text pattern.\n\nFor many very simple tasks, this will do fine. Many HTML documents are, in practice, very\nconsistently formatted as far as placement of linebreaks and whitespace, so you could just\nget away with scanning the file like so:\n\nsub getheading {\nmy $filename = $[0];\nlocal *HTML;\nopen(HTML, $filename)\nor die \"Couldn't open $filename);\nmy $heading;\nLine:\nwhile() {\nif( m{(.*?)
}i ) { # match it!\n$heading = $1;\nlast Line;\n}\n}\nclose(HTML);\nwarn \"No heading in $filename?\"\nunless defined $heading;\nreturn $heading;\n}\n\nThis is quick and fast, but awfully fragile -- if there's a newline in the middle of a\nheading's text, it won't match the above regexp, and you'll get an error. The regexp will\nalso fail if the \"h1\" element's start-tag has any attributes. If you have to adapt your code\nto fit more kinds of start-tags, you'll end up basically reinventing part of HTML::Parser,\nat which point you should probably just stop, and use HTML::Parser itself:\n\n* You can use HTML::Parser to scan the file for an \"h1\" start-tag token, then capture all the\ntext tokens until the \"h1\" close-tag. This approach is extensively covered in the Ken\nMacFarlane's TPJ17 article \"Parsing HTML with HTML::Parser\". (A variant of this approach is\nto use HTML::TokeParser, which presents a different and rather handier interface to the\ntokens that HTML::Parser picks out.)\n\nUsing HTML::Parser is less fragile than our first approach, since it's not sensitive to the\nexact internal formatting of the start-tag (much less whether it's split across two lines).\nHowever, when you need more information about the context of the \"h1\" element, or if you're\nhaving to deal with any of the tricky bits of HTML, such as parsing of tables, you'll find\nout the flat list of tokens that HTML::Parser returns isn't immediately useful. To get\nsomething useful out of those tokens, you'll need to write code that knows some things about\nwhat elements take no content (as with \"hr\" elements), and that a \"\" end-tags are\nomissible, so a \"\" will end any currently open paragraph -- and you're well on your way\nto pointlessly reinventing much of the code in HTML::TreeBuilder\n\nFootnote: And, as the person who last rewrote that module, I can attest that it wasn't\nterribly easy to get right! Never underestimate the perversity of people coding HTML.\n\n, at which point you should probably just stop, and use HTML::TreeBuilder itself:\n\n* You can use HTML::Treebuilder, and scan the tree of element objects that you get back.\n\nThe last approach, using HTML::TreeBuilder, is the diametric opposite of first approach: The\nfirst approach involves just elementary Perl and one regexp, whereas the TreeBuilder approach\ninvolves being at home with the concept of tree-shaped data structures and modules with\nobject-oriented interfaces, as well as with the particular interfaces that HTML::TreeBuilder and\nHTML::Element provide.\n\nHowever, what the TreeBuilder approach has going for it is that it's the most robust, because it\ninvolves dealing with HTML in its \"native\" format -- it deals with the tree structure that HTML\ncode represents, without any consideration of how the source is coded and with what tags\nomitted.\n\nSo, to extract the text from the \"h1\" elements of an HTML document:\n\nsub getheading {\nmy $tree = HTML::TreeBuilder->new;\n$tree->parsefile($[0]); # !\nmy $heading;\nmy $h1 = $tree->lookdown('tag', 'h1'); # !\nif($h1) {\n$heading = $h1->astext; # !\n} else {\nwarn \"No heading in $[0]?\";\n}\n$tree->delete; # clear memory!\nreturn $heading;\n}\n\nThis uses some unfamiliar methods that need explaining. The \"parsefile\" method that we've seen\nbefore, builds a tree based on source from the file given. The \"delete\" method is for marking a\ntree's contents as available for garbage collection, when you're done with the tree. The\n\"astext\" method returns a string that contains all the text bits that are children (or\notherwise descendants) of the given node -- to get the text content of the $h1 object, we could\njust say:\n\n$heading = join '', $h1->contentlist;\n\nbut that will work only if we're sure that the \"h1\" element's children will be only text bits --\nif the document contained:\n\n
Local Man Sees Blade Again
\n\nthen the sub-tree would be:\n\n. h1\n. \"Local Man Sees \"\n. cite\n. \"Blade\"\n. \" Again'\n\nso \"join '', $h1->contentlist\" will be something like:\n\nLocal Man Sees HTML::Element=HASH(0x15424040) Again\n\nwhereas \"$h1->astext\" would yield:\n\nLocal Man Sees Blade Again\n\nand depending on what you're doing with the heading text, you might want the \"asHTML\" method\ninstead. It returns the (sub)tree represented as HTML source. \"$h1->asHTML\" would yield:\n\nLocal Man Sees Blade Again
\n\nHowever, if you wanted the contents of $h1 as HTML, but not the $h1 itself, you could say:\n\njoin '',\nmap(\nref($) ? $->asHTML : $,\n$h1->contentlist\n)\n\nThis \"map\" iterates over the nodes in $h1's list of children; and for each node that's just a\ntext bit (as \"Local Man Sees \" is), it just passes through that string value, and for each node\nthat's an actual object (causing \"ref\" to be true), \"asHTML\" will used instead of the string\nvalue of the object itself (which would be something quite useless, as most object values are).\nSo that \"asHTML\" for the \"cite\" element will be the string \"Blade\". And then,\nfinally, \"join\" just puts into one string all the strings that the \"map\" returns.\n\nLast but not least, the most important method in our \"getheading\" sub is the \"lookdown\"\nmethod. This method looks down at the subtree starting at the given object ($h1), looking for\nelements that meet criteria you provide.\n\nThe criteria are specified in the method's argument list. Each criterion can consist of two\nscalars, a key and a value, which express that you want elements that have that attribute (like\n\"tag\", or \"src\") with the given value (\"h1\"); or the criterion can be a reference to a\nsubroutine that, when called on the given element, returns true if that is a node you're looking\nfor. If you specify several criteria, then that's taken to mean that you want all the elements\nthat each satisfy *all* the criteria. (In other words, there's an \"implicit AND\".)\n\nAnd finally, there's a bit of an optimization -- if you call the \"lookdown\" method in a scalar\ncontext, you get just the *first* node (or undef if none) -- and, in fact, once \"lookdown\"\nfinds that first matching element, it doesn't bother looking any further.\n\nSo the example:\n\n$h1 = $tree->lookdown('tag', 'h1');\n\nreturns the first element at-or-under $tree whose \"tag\" attribute has the value \"h1\".\n" }, { "name": "Complex Criteria in Tree Scanning", "content": "Now, the above \"lookdown\" code looks like a lot of bother, with barely more benefit than just\ngrepping the file! But consider if your criteria were more complicated -- suppose you found that\nsome of the press releases that you were scanning had several \"h1\" elements, possibly before or\nafter the one you actually want. For example:\n\nVisit Our Corporate Partner\n
![](\"/dyna/vendad\")
\n
\nConGlomCo President Schreck to Visit Regional HQ\n
![](\"/photos/Schreckvisit.jpg\")
\n
\n\nHere, you want to ignore the first \"h1\" element because it contains an ad, and you want the text\nfrom the second \"h1\". The problem is in formalizing the way you know that it's an ad. Since ad\nbanners are always entreating you to \"visit\" the sponsoring site, you could exclude \"h1\"\nelements that contain the word \"visit\" under them:\n\nmy $realh1 = $tree->lookdown(\n'tag', 'h1',\nsub {\n$[0]->astext !~ m/\\bvisit/i\n}\n);\n\nThe first criterion looks for \"h1\" elements, and the second criterion limits those to only the\nones whose text content doesn't match \"m/\\bvisit/\". But unfortunately, that won't work for our\nexample, since the second \"h1\" mentions \"ConGlomCo President Schreck to *Visit* Regional HQ\".\n\nInstead you could try looking for the first \"h1\" element that doesn't contain an image:\n\nmy $realh1 = $tree->lookdown(\n'tag', 'h1',\nsub {\nnot $[0]->lookdown('tag', 'img')\n}\n);\n\nThis criterion sub might seem a bit odd, since it calls \"lookdown\" as part of a larger\n\"lookdown\" operation, but that's fine. Note that when considered as a boolean value, a\n\"lookdown\" in a scalar context value returns false (specifically, undef) if there's no matching\nelement at or under the given element; and it returns the first matching element (which, being a\nreference and object, is always a true value), if any matches. So, here,\n\nsub {\nnot $[0]->lookdown('tag', 'img')\n}\n\nmeans \"return true only if this element has no 'img' element as descendants (and isn't an 'img'\nelement itself).\"\n\nThis correctly filters out the first \"h1\" that contains the ad, but it also incorrectly filters\nout the second \"h1\" that contains a non-advertisement photo besides the headline text you want.\n\nThere clearly are detectable differences between the first and second \"h1\" elements -- the only\nsecond one contains the string \"Schreck\", and we could just test for that:\n\nmy $realh1 = $tree->lookdown(\n'tag', 'h1',\nsub {\n$[0]->astext =~ m{Schreck}\n}\n);\n\nAnd that works fine for this one example, but unless all thousand of your press releases have\n\"Schreck\" in the headline, that's just not a general solution. However, if all the ads-in-\"h1\"s\nthat you want to exclude involve a link whose URL involves \"/dyna/\", then you can use that:\n\nmy $realh1 = $tree->lookdown(\n'tag', 'h1',\nsub {\nmy $link = $[0]->lookdown('tag','a');\nreturn 1 unless $link;\n# no link means it's fine\nreturn 0 if $link->attr('href') =~ m{/dyna/};\n# a link to there is bad\nreturn 1; # otherwise okay\n}\n);\n\nOr you can look at it another way and say that you want the first \"h1\" element that either\ncontains no images, or else whose image has a \"src\" attribute whose value contains \"/photos/\":\n\nmy $realh1 = $tree->lookdown(\n'tag', 'h1',\nsub {\nmy $img = $[0]->lookdown('tag','img');\nreturn 1 unless $img;\n# no image means it's fine\nreturn 1 if $img->attr('src') =~ m{/photos/};\n# good if a photo\nreturn 0; # otherwise bad\n}\n);\n\nRecall that this use of \"lookdown\" in a scalar context means to return the first element at or\nunder $tree that matches all the criteria. But if you notice that you can formulate criteria\nthat'll match several possible \"h1\" elements, some of which may be bogus but the *last* one of\nwhich is always the one you want, then you can use \"lookdown\" in a list context, and just use\nthe last element of that list:\n\nmy @h1s = $tree->lookdown(\n'tag', 'h1',\n...maybe more criteria...\n);\ndie \"What, no h1s here?\" unless @h1s;\nmy $realh1 = $h1s[-1]; # last or only\n\nA Case Study: Scanning Yahoo News's HTML\nThe above (somewhat contrived) case involves extracting data from a bunch of pre-existing HTML\nfiles. In that sort of situation, if your code works for all the files, then you know that the\ncode *works* -- since the data it's meant to handle won't go changing or growing; and,\ntypically, once you've used the program, you'll never need to use it again.\n\nThe other kind of situation faced in many data extraction tasks is where the program is used\nrecurringly to handle new data -- such as from ever-changing Web pages. As a real-world example\nof this, consider a program that you could use (suppose it's crontabbed) to extract\nheadline-links from subsections of Yahoo News (\"http://dailynews.yahoo.com/\").\n\nYahoo News has several subsections:\n\nhttp://dailynews.yahoo.com/h/tc/ for technology news\nhttp://dailynews.yahoo.com/h/sc/ for science news\nhttp://dailynews.yahoo.com/h/hl/ for health news\nhttp://dailynews.yahoo.com/h/wl/ for world news\nhttp://dailynews.yahoo.com/h/en/ for entertainment news\n\nand others. All of them are built on the same basic HTML template -- and a scarily complicated\ntemplate it is, especially when you look at it with an eye toward making up rules that will\nselect where the real headline-links are, while screening out all the links to other parts of\nYahoo, other news services, etc. You will need to puzzle over the HTML source, and scrutinize\nthe output of \"$tree->dump\" on the parse tree of that HTML.\n\nSometimes the only way to pin down what you're after is by position in the tree. For example,\nheadlines of interest may be in the third column of the second row of the second table element\nin a page:\n\nmy $table = ( $tree->lookdown('tag','table') )[1];\nmy $row2 = ( $table->lookdown('tag', 'tr' ) )[1];\nmy $col3 = ( $row2->look-down('tag', 'td') )[2];\n...then do things with $col3...\n\nOr they may be all the links in a \"p\" element that has at least three \"br\" elements as children:\n\nmy $p = $tree->lookdown(\n'tag', 'p',\nsub {\n2 < grep { ref($) and $->tag eq 'br' }\n$[0]->contentlist\n}\n);\n@links = $p->lookdown('tag', 'a');\n\nBut almost always, you can get away with looking for properties of the of the thing itself,\nrather than just looking for contexts. Now, if you're lucky, the document you're looking through\nhas clear semantic tagging, such is as useful in CSS -- note the class=\"headlinelink\" bit here:\n\nElvis\nseen in tortilla\n\nIf you find anything like that, you could leap right in and select links with:\n\n@links = $tree->lookdown('class','headlinelink');\n\nRegrettably, your chances of seeing any sort of semantic markup principles really being followed\nwith actual HTML are pretty thin.\n\nFootnote: In fact, your chances of finding a page that is simply free of HTML errors are\neven thinner. And surprisingly, sites like Amazon or Yahoo are typically worse as far as\nquality of code than personal sites whose entire production cycle involves simply being\nsaved and uploaded from Netscape Composer.\n\nThe code may be sort of \"accidentally semantic\", however -- for example, in a set of pages I was\nscanning recently, I found that looking for \"td\" elements with a \"width\" attribute value of\n\"375\" got me exactly what I wanted. No-one designing that page ever conceived of \"width=375\" as\n*meaning* \"this is a headline\", but if you impute it to mean that, it works.\n\nAn approach like this happens to work for the Yahoo News code, because the headline-links are\ndistinguished by the fact that they (and they alone) contain a \"b\" element:\n\nElvis seen in tortilla\n\nor, diagrammed as a part of the parse tree:\n\n. a [href=\"...longnewsurl...\"]\n. b\n. \"Elvis seen in tortilla\"\n\nA rule that matches these can be formalized as \"look for any 'a' element that has only one\ndaughter node, which must be a 'b' element\". And this is what it looks like when cooked up as a\n\"lookdown\" expression and prefaced with a bit of code that retrieves the text of the given\nYahoo News page and feeds it to TreeBuilder:\n\nuse strict;\nuse HTML::TreeBuilder 2.97;\nuse LWP::UserAgent;\nsub getheadlines {\nmy $url = $[0] || die \"What URL?\";\n\nmy $response = LWP::UserAgent->new->request(\nHTTP::Request->new( GET => $url )\n);\nunless($response->issuccess) {\nwarn \"Couldn't get $url: \", $response->statusline, \"\\n\";\nreturn;\n}\n\nmy $tree = HTML::TreeBuilder->new();\n$tree->parse($response->content);\n$tree->eof;\n\nmy @out;\nforeach my $link (\n$tree->lookdown( # !\n'tag', 'a',\nsub {\nreturn unless $[0]->attr('href');\nmy @c = $[0]->contentlist;\n@c == 1 and ref $c[0] and $c[0]->tag eq 'b';\n}\n)\n) {\npush @out, [ $link->attr('href'), $link->astext ];\n}\n\nwarn \"Odd, fewer than 6 stories in $url!\" if @out < 6;\n$tree->delete;\nreturn @out;\n}\n\n...and add a bit of code to actually call that routine and display the results...\n\nforeach my $section (qw[tc sc hl wl en]) {\nmy @links = getheadlines(\n\"http://dailynews.yahoo.com/h/$section/\"\n);\nprint\n$section, \": \", scalar(@links), \" stories\\n\",\nmap((\" \", $->[0], \" : \", $->[1], \"\\n\"), @links),\n\"\\n\";\n}\n\nAnd we've got our own headline-extractor service! This in and of itself isn't no amazingly\nuseful (since if you want to see the headlines, you *can* just look at the Yahoo News pages),\nbut it could easily be the basis for quite useful features like filtering the headlines for\nmatching certain keywords of interest to you.\n\nNow, one of these days, Yahoo News will decide to change its HTML template. When this happens,\nthis will appear to the above program as there being no links that meet the given criteria; or,\nless likely, dozens of erroneous links will meet the criteria. In either case, the criteria will\nhave to be changed for the new template; they may just need adjustment, or you may need to scrap\nthem and start over.\n\n*Regardez, duvet!*\nIt's often quite a challenge to write criteria to match the desired parts of an HTML parse tree.\nVery often you *can* pull it off with a simple \"$tree->lookdown('tag', 'h1')\", but sometimes\nyou do have to keep adding and refining criteria, until you might end up with complex filters\nlike what I've shown in this article. The benefit to learning how to deal with HTML parse trees\nis that one main search tool, the \"lookdown\" method, can do most of the work, making simple\nthings easy, while still making hard things possible.\n\n[end body of article]\n\n[Author Credit]\nSean M. Burke (\"sburke@cpan.org\") is the current maintainer of \"HTML::TreeBuilder\" and\n\"HTML::Element\", both originally by Gisle Aas.\n\nSean adds: \"I'd like to thank the folks who listened to me ramble incessantly about\nHTML::TreeBuilder and HTML::Element at this year's Yet Another Perl Conference and O'Reilly Open\nSource Software Convention.\"\n"
}
]
},
"BACK": {
"content": "Return to the HTML::Tree docs.\n",
"subsections": []
}
},
"summary": "HTML::Tree::Scanning -- article: \"Scanning HTML\"",
"flags": [],
"examples": [],
"see_also": []
}![](\"/photos/Schreckvisitlarge.jpg\"\n)
\n