{ "content": [ { "type": "text", "text": "# WWW::Mechanize::FAQ (info)\n\n## NAME\n\nWWW::Mechanize::FAQ - Frequently Asked Questions about WWW::Mechanize\n\n## Sections\n\n- **NAME**\n- **VERSION**\n- **How to get help with WWW::Mechanize**\n- **JavaScript**\n- **Why doesn't this work: Debugging your Mechanize program**\n- **AUTHOR**\n- **COPYRIGHT AND LICENSE**\n\nUse structuredContent.sections for detailed options, examples, and full documentation.\n" } ], "structuredContent": { "command": "WWW::Mechanize::FAQ", "section": "", "mode": "info", "summary": "WWW::Mechanize::FAQ - Frequently Asked Questions about WWW::Mechanize", "synopsis": null, "tldr_summary": null, "tldr_examples": [], "tldr_source": null, "flags": [], "examples": [], "see_also": [], "section_outline": [ { "name": "NAME", "lines": 2, "subsections": [] }, { "name": "VERSION", "lines": 2, "subsections": [] }, { "name": "How to get help with WWW::Mechanize", "lines": 12, "subsections": [] }, { "name": "JavaScript", "lines": 210, "subsections": [] }, { "name": "Why doesn't this work: Debugging your Mechanize program", "lines": 174, "subsections": [] }, { "name": "AUTHOR", "lines": 2, "subsections": [] }, { "name": "COPYRIGHT AND LICENSE", "lines": 6, "subsections": [] } ], "sections": { "NAME": { "content": "WWW::Mechanize::FAQ - Frequently Asked Questions about WWW::Mechanize\n", "subsections": [] }, "VERSION": { "content": "version 2.06\n", "subsections": [] }, "How to get help with WWW::Mechanize": { "content": "If your question isn't answered here in the FAQ, please turn to the\ncommunities at:\n\no StackOverflow\n\n\no #lwp on irc.perl.org\n\no \n\no The libwww-perl mailing list at \n", "subsections": [] }, "JavaScript": { "content": "I have this web page that has JavaScript on it, and my Mech program doesn't\nwork.\nThat's because WWW::Mechanize doesn't operate on the JavaScript. It\nonly understands the HTML parts of the page.\n\nI thought Mech was supposed to work like a web browser.\nIt does pretty much, but it doesn't support JavaScript.\n\nI added some basic attempts at picking up URLs in \"window.open()\" calls\nand return them in \"$mech->links\". They work sometimes.\n\nSince Javascript is completely visible to the client, it cannot be used\nto prevent a scraper from following links. But it can make life\ndifficult. If you want to scrape specific pages, then a solution is\nalways possible.\n\nOne typical use of Javascript is to perform argument checking before\nposting to the server. The URL you want is probably just buried in the\nJavascript function. Do a regular expression match on\n\"$mech->content()\" to find the link that you want and \"$mech->get\" it\ndirectly (this assumes that you know what you are looking for in\nadvance).\n\nIn more difficult cases, the Javascript is used for URL mangling to\nsatisfy the needs of some middleware. In this case you need to figure\nout what the Javascript is doing (why are these URLs always really\nlong?). There is probably some function with one or more arguments\nwhich calculates the new URL. Step one: using your favorite browser,\nget the before and after URLs and save them to files. Edit each file,\nconverting the argument separators ('?', '&' or ';') into newlines. Now\nit is easy to use diff or comm to find out what Javascript did to the\nURL. Step 2 - find the function call which created the URL - you will\nneed to parse and interpret its argument list. The Javascript Debugger\nin the Firebug extension for Firefox helps with the analysis. At this\npoint, it is fairly trivial to write your own function which emulates\nthe Javascript for the pages you want to process.\n\nHere's another approach that answers the question, \"It works in\nFirefox, but why not Mech?\" Everything the web server knows about the\nclient is present in the HTTP request. If two requests are identical,\nthe results should be identical. So the real question is \"What is\ndifferent between the mech request and the Firefox request?\"\n\nThe Firefox extension \"Tamper Data\" is an effective tool for examining\nthe headers of the requests to the server. Compare that with what LWP\nis sending. Once the two are identical, the action of the server should\nbe the same as well.\n\nI say \"should\", because this is an oversimplification - some values are\nnaturally unique, e.g. a SessionID, but if a SessionID is present, that\nis probably sufficient, even though the value will be different between\nthe LWP request and the Firefox request. The server could use the\nsession to store information which is troublesome, but that's not the\nfirst place to look (and highly unlikely to be relevant when you are\nrequesting the login page of your site).\n\nGenerally the problem is to be found in missing or incorrect POSTDATA\narguments, Cookies, User-Agents, Accepts, etc. If you are using mech,\nthen redirects and cookies should not be a problem, but are listed here\nfor completeness. If you are missing headers, \"$mech->addheader\" can\nbe used to add the headers that you need.\n\nWhich modules work like Mechanize and have JavaScript support?\nIn no particular order: Gtk2::WebKit::Mechanize, Win32::IE::Mechanize,\nWWW::Mechanize::Firefox, WWW::Scripter, WWW::Selenium\n\nHow do I do X?\nCan I do [such-and-such] with WWW::Mechanize?\nIf it's possible with LWP::UserAgent, then yes. WWW::Mechanize is a\nsubclass of LWP::UserAgent, so all the wondrous magic of that class is\ninherited.\n\nHow do I use WWW::Mechanize through a proxy server?\nSee the docs in LWP::UserAgent on how to use the proxy. Short version:\n\n$mech->proxy(['http', 'ftp'], 'http://proxy.example.com:8000/');\n\nor get the specs from the environment:\n\n$mech->envproxy();\n\n# Environment set like so:\ngopherproxy=http://proxy.my.place/\nwaisproxy=http://proxy.my.place/\nnoproxy=\"localhost,my.domain\"\nexport gopherproxy waisproxy noproxy\n\nHow can I see what fields are on the forms?\nUse the mech-dump utility, optionally installed with Mechanize.\n\n$ mech-dump --forms http://search.cpan.org\nDumping forms\nGET http://search.cpan.org/search\nquery=\nmode=all (option) [*all|module|dist|author]\n=CPAN Search (submit)\n\nHow do I get Mech to handle authentication?\nuse MIME::Base64;\n\nmy $agent = WWW::Mechanize->new();\nmy @args = (\nAuthorization => \"Basic \" .\nMIME::Base64::encode( USER . ':' . PASS )\n);\n\n$agent->credentials( ADDRESS, REALM, USER, PASS );\n$agent->get( URL, @args );\n\nIf you want to use the credentials for all future requests, you can\nalso use the LWP::UserAgent \"defaultheader()\" method instead of the\nextra arguments to \"get()\"\n\n$mech->defaultheader(\nAuthorization => 'Basic ' . encodebase64( USER . ':' . PASSWORD ) );\n\nHow can I get WWW::Mechanize to execute this JavaScript?\nYou can't. JavaScript is entirely client-based, and WWW::Mechanize is\na client that doesn't understand JavaScript. See the top part of this\nFAQ.\n\nHow do I check a checkbox that doesn't have a value defined?\nSet it to the value of \"on\".\n\n$mech->field( mycheckbox => 'on' );\n\nHow do I handle frames?\nYou don't deal with them as frames, per se, but as links. Extract them\nwith\n\nmy @framelinks = $mech->findlink( tag => \"frame\" );\n\nHow do I get a list of HTTP headers and their values?\nAll HTTP::Headers methods work on a HTTP::Response object which is\nreturned by the get(), reload(), response()/res(), click(),\nsubmitform(), and request() methods.\n\nmy $mech = WWW::Mechanize->new( autocheck => 1 );\n$mech->get( 'http://my.site.com' );\nmy $response = $mech->response();\nfor my $key ( $response->headerfieldnames() ) {\nprint $key, \" : \", $response->header( $key ), \"\\n\";\n}\n\nHow do I enable keep-alive?\nSince WWW::Mechanize is a subclass of LWP::UserAgent, you can use the\nsame mechanism to enable keep-alive:\n\nuse LWP::ConnCache;\n...\n$mech->conncache(LWP::ConnCache->new);\n\nHow can I change/specify the action parameter of an HTML form?\nYou can access the action of the form by utilizing the HTML::Form\nobject returned from one of the specifying form methods.\n\nUsing \"$mech->formnumber($number)\":\n\nmy $mech = WWW::mechanize->new;\n$mech->get('http://someurlhere.com');\n# Access the form using its Zero-Based Index by DOM order\n$mech->formnumber(0)->action('http://newAction'); #ABS URL\n\nUsing \"$mech->formname($number)\":\n\nmy $mech = WWW::mechanize->new;\n$mech->get('http://someurlhere.com');\n#Access the form using its Zero-Based Index by DOM order\n$mech->formname('trgForm')->action('http://newAction'); #ABS URL\n\nHow do I save an image? How do I save a large tarball?\nAn image is just content. You get the image and save it.\n\n$mech->get( 'photo.jpg' );\n$mech->savecontent( '/path/to/my/directory/photo.jpg' );\n\nYou can also save any content directly to disk using the\n\":contentfile\" flag to \"get()\", which is part of LWP::UserAgent.\n\n$mech->get( 'http://www.cpan.org/src/stable.tar.gz',\n':contentfile' => 'stable.tar.gz' );\n\nHow do I pick a specific value from a \"