{
    "mode": "perldoc",
    "parameter": "lwptut",
    "section": "",
    "url": "https://www.chedong.com/phpMan.php/perldoc/lwptut/json",
    "generated": "2026-10-07T21:05:46Z",
    "sections": {
        "NAME": {
            "content": "lwptut -- An LWP Tutorial\n",
            "subsections": []
        },
        "DESCRIPTION": {
            "content": "LWP (short for \"Library for WWW in Perl\") is a very popular group of Perl modules for accessing\ndata on the Web. Like most Perl module-distributions, each of LWP's component modules comes with\ndocumentation that is a complete reference to its interface. However, there are so many modules\nin LWP that it's hard to know where to start looking for information on how to do even the\nsimplest most common things.\n\nReally introducing you to using LWP would require a whole book -- a book that just happens to\nexist, called *Perl & LWP*. But this article should give you a taste of how you can go about\nsome common tasks with LWP.\n",
            "subsections": [
                {
                    "name": "Getting documents with LWP::Simple",
                    "content": "If you just want to get what's at a particular URL, the simplest way to do it is LWP::Simple's\nfunctions.\n\nIn a Perl program, you can call its get($url) function. It will try getting that URL's content.\nIf it works, then it'll return the content; but if there's some error, it'll return undef.\n\nmy $url = 'http://www.npr.org/programs/fa/?todayDate=current';\n# Just an example: the URL for the most recent /Fresh Air/ show\n\nuse LWP::Simple;\nmy $content = get $url;\ndie \"Couldn't get $url\" unless defined $content;\n\n# Then go do things with $content, like this:\n\nif($content =~ m/jazz/i) {\nprint \"They're talking about jazz today on Fresh Air!\\n\";\n}\nelse {\nprint \"Fresh Air is apparently jazzless today.\\n\";\n}\n\nThe handiest variant on \"get\" is \"getprint\", which is useful in Perl one-liners. If it can get\nthe page whose URL you provide, it sends it to STDOUT; otherwise it complains to STDERR.\n\n% perl -MLWP::Simple -e \"getprint 'http://www.cpan.org/RECENT'\"\n\nThat is the URL of a plain text file that lists new files in CPAN in the past two weeks. You can\neasily make it part of a tidy little shell command, like this one that mails you the list of new\n\"Acme::\" modules:\n\n% perl -MLWP::Simple -e \"getprint 'http://www.cpan.org/RECENT'\"  \\\n| grep \"/by-module/Acme\" | mail -s \"New Acme modules! Joy!\" $USER\n\nThere are other useful functions in LWP::Simple, including one function for running a HEAD\nrequest on a URL (useful for checking links, or getting the last-revised time of a URL), and two\nfunctions for saving/mirroring a URL to a local file. See the LWP::Simple documentation for the\nfull details, or chapter 2 of *Perl & LWP* for more examples.\n"
                },
                {
                    "name": "The Basics of the LWP Class Model",
                    "content": "LWP::Simple's functions are handy for simple cases, but its functions don't support cookies or\nauthorization, don't support setting header lines in the HTTP request, generally don't support\nreading header lines in the HTTP response (notably the full HTTP error message, in case of an\nerror). To get at all those features, you'll have to use the full LWP class model.\n\nWhile LWP consists of dozens of classes, the main two that you have to understand are\nLWP::UserAgent and HTTP::Response. LWP::UserAgent is a class for \"virtual browsers\" which you\nuse for performing requests, and HTTP::Response is a class for the responses (or error messages)\nthat you get back from those requests.\n\nThe basic idiom is \"$response = $browser->get($url)\", or more fully illustrated:\n\n# Early in your program:\n\nuse LWP 5.64; # Loads all important LWP classes, and makes\n#  sure your version is reasonably recent.\n\nmy $browser = LWP::UserAgent->new;\n\n...\n\n# Then later, whenever you need to make a get request:\nmy $url = 'http://www.npr.org/programs/fa/?todayDate=current';\n\nmy $response = $browser->get( $url );\ndie \"Can't get $url -- \", $response->statusline\nunless $response->issuccess;\n\ndie \"Hey, I was expecting HTML, not \", $response->contenttype\nunless $response->contenttype eq 'text/html';\n# or whatever content-type you're equipped to deal with\n\n# Otherwise, process the content somehow:\n\nif($response->decodedcontent =~ m/jazz/i) {\nprint \"They're talking about jazz today on Fresh Air!\\n\";\n}\nelse {\nprint \"Fresh Air is apparently jazzless today.\\n\";\n}\n\nThere are two objects involved: $browser, which holds an object of class LWP::UserAgent, and\nthen the $response object, which is of class HTTP::Response. You really need only one browser\nobject per program; but every time you make a request, you get back a new HTTP::Response object,\nwhich will have some interesting attributes:\n\n*   A status code indicating success or failure (which you can test with\n\"$response->issuccess\").\n\n*   An HTTP status line that is hopefully informative if there's failure (which you can see with\n\"$response->statusline\", returning something like \"404 Not Found\").\n\n*   A MIME content-type like \"text/html\", \"image/gif\", \"application/xml\", etc., which you can\nsee with \"$response->contenttype\"\n\n*   The actual content of the response, in \"$response->decodedcontent\". If the response is\nHTML, that's where the HTML source will be; if it's a GIF, then \"$response->decodedcontent\"\nwill be the binary GIF data.\n\n*   And dozens of other convenient and more specific methods that are documented in the docs for\nHTTP::Response, and its superclasses HTTP::Message and HTTP::Headers.\n"
                },
                {
                    "name": "Adding Other HTTP Request Headers",
                    "content": "The most commonly used syntax for requests is \"$response = $browser->get($url)\", but in truth,\nyou can add extra HTTP header lines to the request by adding a list of key-value pairs after the\nURL, like so:\n\n$response = $browser->get( $url, $key1, $value1, $key2, $value2, ... );\n\nFor example, here's how to send some commonly used headers, in case you're dealing with a site\nthat would otherwise reject your request:\n\nmy @nsheaders = (\n'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',\n'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*',\n'Accept-Charset' => 'iso-8859-1,*,utf-8',\n'Accept-Language' => 'en-US',\n);\n\n...\n\n$response = $browser->get($url, @nsheaders);\n\nIf you weren't reusing that array, you could just go ahead and do this:\n\n$response = $browser->get($url,\n'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',\n'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*',\n'Accept-Charset' => 'iso-8859-1,*,utf-8',\n'Accept-Language' => 'en-US',\n);\n\nIf you were only ever changing the 'User-Agent' line, you could just change the $browser\nobject's default line from \"libwww-perl/5.65\" (or the like) to whatever you like, using the\nLWP::UserAgent \"agent\" method:\n\n$browser->agent('Mozilla/4.76 [en] (Win98; U)');\n"
                },
                {
                    "name": "Enabling Cookies",
                    "content": "A default LWP::UserAgent object acts like a browser with its cookies support turned off. There\nare various ways of turning it on, by setting its \"cookiejar\" attribute. A \"cookie jar\" is an\nobject representing a little database of all the HTTP cookies that a browser knows about. It can\ncorrespond to a file on disk or an in-memory object that starts out empty, and whose collection\nof cookies will disappear once the program is finished running.\n\nTo give a browser an in-memory empty cookie jar, you set its \"cookiejar\" attribute like so:\n\nuse HTTP::CookieJar::LWP;\n$browser->cookiejar( HTTP::CookieJar::LWP->new );\n\nTo save a cookie jar to disk, see \"dumpcookies\" in HTTP::CookieJar. To load cookies from disk\ninto a jar, see \"loadcookies\" in HTTP::CookieJar.\n"
                },
                {
                    "name": "Posting Form Data",
                    "content": "Many HTML forms send data to their server using an HTTP POST request, which you can send with\nthis syntax:\n\n$response = $browser->post( $url,\n[\nformkey1 => value1,\nformkey2 => value2,\n...\n],\n);\n\nOr if you need to send HTTP headers:\n\n$response = $browser->post( $url,\n[\nformkey1 => value1,\nformkey2 => value2,\n...\n],\nheaderkey1 => value1,\nheaderkey2 => value2,\n);\n\nFor example, the following program makes a search request to AltaVista (by sending some form\ndata via an HTTP POST request), and extracts from the HTML the report of the number of matches:\n\nuse strict;\nuse warnings;\nuse LWP 5.64;\nmy $browser = LWP::UserAgent->new;\n\nmy $word = 'tarragon';\n\nmy $url = 'http://search.yahoo.com/yhs/search';\nmy $response = $browser->post( $url,\n[ 'q' => $word,  # the Altavista query string\n'fr' => 'altavista', 'pg' => 'q', 'avkw' => 'tgz', 'kl' => 'XX',\n]\n);\ndie \"$url error: \", $response->statusline\nunless $response->issuccess;\ndie \"Weird content type at $url -- \", $response->contenttype\nunless $response->contentishtml;\n\nif( $response->decodedcontent =~ m{([0-9,]+)(?:<.*?>)? results for} ) {\n# The substring will be like \"996,000</strong> results for\"\nprint \"$word: $1\\n\";\n}\nelse {\nprint \"Couldn't find the match-string in the response\\n\";\n}\n"
                },
                {
                    "name": "Sending GET Form Data",
                    "content": "Some HTML forms convey their form data not by sending the data in an HTTP POST request, but by\nmaking a normal GET request with the data stuck on the end of the URL. For example, if you went\nto \"www.imdb.com\" and ran a search on \"Blade Runner\", the URL you'd see in your browser window\nwould be:\n\nhttp://www.imdb.com/find?s=all&q=Blade+Runner\n\nTo run the same search with LWP, you'd use this idiom, which involves the URI class:\n\nuse URI;\nmy $url = URI->new( 'http://www.imdb.com/find' );\n# makes an object representing the URL\n\n$url->queryform(  # And here the form data pairs:\n'q' => 'Blade Runner',\n's' => 'all',\n);\n\nmy $response = $browser->get($url);\n\nSee chapter 5 of *Perl & LWP* for a longer discussion of HTML forms and of form data, and\nchapters 6 through 9 for a longer discussion of extracting data from HTML.\n"
                },
                {
                    "name": "Absolutizing URLs",
                    "content": "The URI class that we just mentioned above provides all sorts of methods for accessing and\nmodifying parts of URLs (such as asking sort of URL it is with \"$url->scheme\", and asking what\nhost it refers to with \"$url->host\", and so on, as described in the docs for the URI class.\nHowever, the methods of most immediate interest are the \"queryform\" method seen above, and now\nthe \"newabs\" method for taking a probably-relative URL string (like \"../foo.html\") and getting\nback an absolute URL (like \"http://www.perl.com/stuff/foo.html\"), as shown here:\n\nuse URI;\n$abs = URI->newabs($mayberelative, $base);\n\nFor example, consider this program that matches URLs in the HTML list of new modules in CPAN:\n\nuse strict;\nuse warnings;\nuse LWP;\nmy $browser = LWP::UserAgent->new;\n\nmy $url = 'http://www.cpan.org/RECENT.html';\nmy $response = $browser->get($url);\ndie \"Can't get $url -- \", $response->statusline\nunless $response->issuccess;\n\nmy $html = $response->decodedcontent;\nwhile( $html =~ m/<A HREF=\\\"(.*?)\\\"/g ) {\nprint \"$1\\n\";\n}\n\nWhen run, it emits output that starts out something like this:\n\nMIRRORING.FROM\nRECENT\nRECENT.html\nauthors/00whois.html\nauthors/01mailrc.txt.gz\nauthors/id/A/AA/AASSAD/CHECKSUMS\n...\n\nHowever, if you actually want to have those be absolute URLs, you can use the URI module's\n\"newabs\" method, by changing the \"while\" loop to this:\n\nwhile( $html =~ m/<A HREF=\\\"(.*?)\\\"/g ) {\nprint URI->newabs( $1, $response->base ) ,\"\\n\";\n}\n\n(The \"$response->base\" method from HTTP::Message is for returning what URL should be used for\nresolving relative URLs -- it's usually just the same as the URL that you requested.)\n\nThat program then emits nicely absolute URLs:\n\nhttp://www.cpan.org/MIRRORING.FROM\nhttp://www.cpan.org/RECENT\nhttp://www.cpan.org/RECENT.html\nhttp://www.cpan.org/authors/00whois.html\nhttp://www.cpan.org/authors/01mailrc.txt.gz\nhttp://www.cpan.org/authors/id/A/AA/AASSAD/CHECKSUMS\n...\n\nSee chapter 4 of *Perl & LWP* for a longer discussion of URI objects.\n\nOf course, using a regexp to match hrefs is a bit simplistic, and for more robust programs,\nyou'll probably want to use an HTML-parsing module like HTML::LinkExtor or HTML::TokeParser or\neven maybe HTML::TreeBuilder.\n"
                },
                {
                    "name": "Other Browser Attributes",
                    "content": "LWP::UserAgent objects have many attributes for controlling how they work. Here are a few\nnotable ones:\n\n*   \"$browser->timeout(15);\"\n\nThis sets this browser object to give up on requests that don't answer within 15 seconds.\n\n*   \"$browser->protocolsallowed( [ 'http', 'gopher'] );\"\n\nThis sets this browser object to not speak any protocols other than HTTP and gopher. If it\ntries accessing any other kind of URL (like an \"ftp:\" or \"mailto:\" or \"news:\" URL), then it\nwon't actually try connecting, but instead will immediately return an error code 500, with a\nmessage like \"Access to 'ftp' URIs has been disabled\".\n\n*   \"use LWP::ConnCache; $browser->conncache(LWP::ConnCache->new());\"\n\nThis tells the browser object to try using the HTTP/1.1 \"Keep-Alive\" feature, which speeds\nup requests by reusing the same socket connection for multiple requests to the same server.\n\n*   \"$browser->agent( 'SomeName/1.23 (more info here maybe)' )\"\n\nThis changes how the browser object will identify itself in the default \"User-Agent\" line is\nits HTTP requests. By default, it'll send \"libwww-perl/*versionnumber*\", like\n\"libwww-perl/5.65\". You can change that to something more descriptive like this:\n\n$browser->agent( 'SomeName/3.14 (contact@robotplexus.int)' );\n\nOr if need be, you can go in disguise, like this:\n\n$browser->agent( 'Mozilla/4.0 (compatible; MSIE 5.12; MacPowerPC)' );\n\n*   \"push @{ $ua->requestsredirectable }, 'POST';\"\n\nThis tells this browser to obey redirection responses to POST requests (like most modern\ninteractive browsers), even though the HTTP RFC says that should not normally be done.\n\nFor more options and information, see the full documentation for LWP::UserAgent.\n"
                },
                {
                    "name": "Writing Polite Robots",
                    "content": "If you want to make sure that your LWP-based program respects robots.txt files and doesn't make\ntoo many requests too fast, you can use the LWP::RobotUA class instead of the LWP::UserAgent\nclass.\n\nLWP::RobotUA class is just like LWP::UserAgent, and you can use it like so:\n\nuse LWP::RobotUA;\nmy $browser = LWP::RobotUA->new('YourSuperBot/1.34', 'you@yoursite.com');\n# Your bot's name and your email address\n\nmy $response = $browser->get($url);\n\nBut HTTP::RobotUA adds these features:\n\n*   If the robots.txt on $url's server forbids you from accessing $url, then the $browser object\n(assuming it's of class LWP::RobotUA) won't actually request it, but instead will give you\nback (in $response) a 403 error with a message \"Forbidden by robots.txt\". That is, if you\nhave this line:\n\ndie \"$url -- \", $response->statusline, \"\\nAborted\"\nunless $response->issuccess;\n\nthen the program would die with an error message like this:\n\nhttp://whatever.site.int/pith/x.html -- 403 Forbidden by robots.txt\nAborted at whateverprogram.pl line 1234\n\n*   If this $browser object sees that the last time it talked to $url's server was too recently,\nthen it will pause (via \"sleep\") to avoid making too many requests too often. How long it\nwill pause for, is by default one minute -- but you can control it with the\n\"$browser->delay( *minutes* )\" attribute.\n\nFor example, this code:\n\n$browser->delay( 7/60 );\n\n...means that this browser will pause when it needs to avoid talking to any given server\nmore than once every 7 seconds.\n\nFor more options and information, see the full documentation for LWP::RobotUA.\n"
                },
                {
                    "name": "Using Proxies",
                    "content": "In some cases, you will want to (or will have to) use proxies for accessing certain sites and/or\nusing certain protocols. This is most commonly the case when your LWP program is running (or\ncould be running) on a machine that is behind a firewall.\n\nTo make a browser object use proxies that are defined in the usual environment variables\n(\"HTTPPROXY\", etc.), just call the \"envproxy\" on a user-agent object before you go making any\nrequests on it. Specifically:\n\nuse LWP::UserAgent;\nmy $browser = LWP::UserAgent->new;\n\n# And before you go making any requests:\n$browser->envproxy;\n\nFor more information on proxy parameters, see the LWP::UserAgent documentation, specifically the\n\"proxy\", \"envproxy\", and \"noproxy\" methods.\n\nHTTP Authentication\nMany web sites restrict access to documents by using \"HTTP Authentication\". This isn't just any\nform of \"enter your password\" restriction, but is a specific mechanism where the HTTP server\nsends the browser an HTTP code that says \"That document is part of a protected 'realm', and you\ncan access it only if you re-request it and add some special authorization headers to your\nrequest\".\n\nFor example, the Unicode.org admins stop email-harvesting bots from harvesting the contents of\ntheir mailing list archives, by protecting them with HTTP Authentication, and then publicly\nstating the username and password (at \"http://www.unicode.org/mail-arch/\") -- namely username\n\"unicode-ml\" and password \"unicode\".\n\nFor example, consider this URL, which is part of the protected area of the web site:\n\nhttp://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html\n\nIf you access that with a browser, you'll get a prompt like \"Enter username and password for\n'Unicode-MailList-Archives' at server 'www.unicode.org'\".\n\nIn LWP, if you just request that URL, like this:\n\nuse LWP;\nmy $browser = LWP::UserAgent->new;\n\nmy $url =\n'http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html';\nmy $response = $browser->get($url);\n\ndie \"Error: \", $response->header('WWW-Authenticate') || 'Error accessing',\n#  ('WWW-Authenticate' is the realm-name)\n\"\\n \", $response->statusline, \"\\n at $url\\n Aborting\"\nunless $response->issuccess;\n\nThen you'll get this error:\n\nError: Basic realm=\"Unicode-MailList-Archives\"\n401 Authorization Required\nat http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html\nAborting at auth1.pl line 9.  [or wherever]\n\n...because the $browser doesn't know any the username and password for that realm\n(\"Unicode-MailList-Archives\") at that host (\"www.unicode.org\"). The simplest way to let the\nbrowser know about this is to use the \"credentials\" method to let it know about a username and\npassword that it can try using for that realm at that host. The syntax is:\n\n$browser->credentials(\n'servername:portnumber',\n'realm-name',\n'username' => 'password'\n);\n\nIn most cases, the port number is 80, the default TCP/IP port for HTTP; and you usually call the\n\"credentials\" method before you make any requests. For example:\n\n$browser->credentials(\n'reports.mybazouki.com:80',\n'webserverusagereports',\n'plinky' => 'banjo123'\n);\n\nSo if we add the following to the program above, right after the \"$browser =\nLWP::UserAgent->new;\" line...\n\n$browser->credentials(  # add this to our $browser 's \"key ring\"\n'www.unicode.org:80',\n'Unicode-MailList-Archives',\n'unicode-ml' => 'unicode'\n);\n\n...then when we run it, the request succeeds, instead of causing the \"die\" to be called.\n"
                },
                {
                    "name": "Accessing HTTPS URLs",
                    "content": "When you access an HTTPS URL, it'll work for you just like an HTTP URL would -- if your LWP\ninstallation has HTTPS support (via an appropriate Secure Sockets Layer library). For example:\n\nuse LWP;\nmy $url = 'https://www.paypal.com/';   # Yes, HTTPS!\nmy $browser = LWP::UserAgent->new;\nmy $response = $browser->get($url);\ndie \"Error at $url\\n \", $response->statusline, \"\\n Aborting\"\nunless $response->issuccess;\nprint \"Whee, it worked!  I got that \",\n$response->contenttype, \" document!\\n\";\n\nIf your LWP installation doesn't have HTTPS support set up, then the response will be\nunsuccessful, and you'll get this error message:\n\nError at https://www.paypal.com/\n501 Protocol scheme 'https' is not supported\nAborting at paypal.pl line 7.   [or whatever program and line]\n\nIf your LWP installation *does* have HTTPS support installed, then the response should be\nsuccessful, and you should be able to consult $response just like with any normal HTTP response.\n\nFor information about installing HTTPS support for your LWP installation, see the helpful\nREADME.SSL file that comes in the libwww-perl distribution.\n"
                },
                {
                    "name": "Getting Large Documents",
                    "content": "When you're requesting a large (or at least potentially large) document, a problem with the\nnormal way of using the request methods (like \"$response = $browser->get($url)\") is that the\nresponse object in memory will have to hold the whole document -- *in memory*. If the response\nis a thirty megabyte file, this is likely to be quite an imposition on this process's memory\nusage.\n\nA notable alternative is to have LWP save the content to a file on disk, instead of saving it up\nin memory. This is the syntax to use:\n\n$response = $ua->get($url,\n':contentfile' => $filespec,\n);\n\nFor example,\n\n$response = $ua->get('http://search.cpan.org/',\n':contentfile' => '/tmp/sco.html'\n);\n\nWhen you use this \":contentfile\" option, the $response will have all the normal header lines,\nbut \"$response->content\" will be empty. Errors writing to the content file (for example due to\npermission denied or the filesystem being full) will be reported via the \"Client-Aborted\" or\n\"X-Died\" response headers, and not the \"issuccess\" method:\n\nif ($response->header('Client-Aborted') eq 'die') {\n# handle error ...\n\nNote that this \":contentfile\" option isn't supported under older versions of LWP, so you should\nconsider adding \"use LWP 5.66;\" to check the LWP version, if you think your program might run on\nsystems with older versions.\n\nIf you need to be compatible with older LWP versions, then use this syntax, which does the same\nthing:\n\nuse HTTP::Request::Common;\n$response = $ua->request( GET($url), $filespec );\n"
                }
            ]
        },
        "SEE ALSO": {
            "content": "Remember, this article is just the most rudimentary introduction to LWP -- to learn more about\nLWP and LWP-related tasks, you really must read from the following:\n\n*   LWP::Simple -- simple functions for getting/heading/mirroring URLs\n\n*   LWP -- overview of the libwww-perl modules\n\n*   LWP::UserAgent -- the class for objects that represent \"virtual browsers\"\n\n*   HTTP::Response -- the class for objects that represent the response to a LWP response, as in\n\"$response = $browser->get(...)\"\n\n*   HTTP::Message and HTTP::Headers -- classes that provide more methods to HTTP::Response.\n\n*   URI -- class for objects that represent absolute or relative URLs\n\n*   URI::Escape -- functions for URL-escaping and URL-unescaping strings (like turning \"this &\nthat\" to and from \"this%20%26%20that\").\n\n*   HTML::Entities -- functions for HTML-escaping and HTML-unescaping strings (like turning \"C.\n& E. Brontë\" to and from \"C. &amp; E. Bront&euml;\")\n\n*   HTML::TokeParser and HTML::TreeBuilder -- classes for parsing HTML\n\n*   HTML::LinkExtor -- class for finding links in HTML documents\n\n*   The book *Perl & LWP* by Sean M. Burke. O'Reilly & Associates, 2002. ISBN: 0-596-00178-9,\n<http://oreilly.com/catalog/perllwp/>. The whole book is also available free online:\n<http://lwp.interglacial.com>.\n",
            "subsections": []
        },
        "COPYRIGHT": {
            "content": "Copyright 2002, Sean M. Burke. You can redistribute this document and/or modify it, but only\nunder the same terms as Perl itself.\n",
            "subsections": []
        },
        "AUTHOR": {
            "content": "Sean M. Burke \"sburke@cpan.org\"\n",
            "subsections": []
        }
    },
    "summary": "lwptut -- An LWP Tutorial",
    "flags": [],
    "examples": [],
    "see_also": []
}