{
    "content": [
        {
            "type": "text",
            "text": "# LWP::RobotUA (perldoc)\n\n## NAME\n\nLWP::RobotUA - a class for well-behaved Web robots\n\n## SYNOPSIS\n\nuse LWP::RobotUA;\nmy $ua = LWP::RobotUA->new('my-robot/0.1', 'me@foo.com');\n$ua->delay(10);  # be very nice -- max one hit every ten minutes!\n...\n# Then just use it just like a normal LWP::UserAgent:\nmy $response = $ua->get('http://whatever.int/...');\n...\n\n## DESCRIPTION\n\nThis class implements a user agent that is suitable for robot applications. Robots should be\nnice to the servers they visit. They should consult the /robots.txt file to ensure that they are\nwelcomed and they should not make requests too frequently.\n\n## Sections\n\n- **NAME**\n- **SYNOPSIS**\n- **DESCRIPTION**\n- **METHODS**\n- **SEE ALSO**\n- **COPYRIGHT**\n\nUse structuredContent.sections for detailed options, examples, and full documentation.\n"
        }
    ],
    "structuredContent": {
        "command": "LWP::RobotUA",
        "section": "",
        "mode": "perldoc",
        "summary": "LWP::RobotUA - a class for well-behaved Web robots",
        "synopsis": "use LWP::RobotUA;\nmy $ua = LWP::RobotUA->new('my-robot/0.1', 'me@foo.com');\n$ua->delay(10);  # be very nice -- max one hit every ten minutes!\n...\n# Then just use it just like a normal LWP::UserAgent:\nmy $response = $ua->get('http://whatever.int/...');\n...",
        "tldr_summary": null,
        "tldr_examples": [],
        "tldr_source": null,
        "flags": [],
        "examples": [],
        "see_also": [],
        "section_outline": [
            {
                "name": "NAME",
                "lines": 2,
                "subsections": []
            },
            {
                "name": "SYNOPSIS",
                "lines": 9,
                "subsections": []
            },
            {
                "name": "DESCRIPTION",
                "lines": 12,
                "subsections": []
            },
            {
                "name": "METHODS",
                "lines": 58,
                "subsections": []
            },
            {
                "name": "SEE ALSO",
                "lines": 2,
                "subsections": []
            },
            {
                "name": "COPYRIGHT",
                "lines": 5,
                "subsections": []
            }
        ],
        "sections": {
            "NAME": {
                "content": "LWP::RobotUA - a class for well-behaved Web robots\n",
                "subsections": []
            },
            "SYNOPSIS": {
                "content": "use LWP::RobotUA;\nmy $ua = LWP::RobotUA->new('my-robot/0.1', 'me@foo.com');\n$ua->delay(10);  # be very nice -- max one hit every ten minutes!\n...\n\n# Then just use it just like a normal LWP::UserAgent:\nmy $response = $ua->get('http://whatever.int/...');\n...\n",
                "subsections": []
            },
            "DESCRIPTION": {
                "content": "This class implements a user agent that is suitable for robot applications. Robots should be\nnice to the servers they visit. They should consult the /robots.txt file to ensure that they are\nwelcomed and they should not make requests too frequently.\n\nBut before you consider writing a robot, take a look at <http://www.robotstxt.org/>.\n\nWhen you use an *LWP::RobotUA* object as your user agent, then you do not really have to think\nabout these things yourself; \"robots.txt\" files are automatically consulted and obeyed, the\nserver isn't queried too rapidly, and so on. Just send requests as you do when you are using a\nnormal *LWP::UserAgent* object (using \"$ua->get(...)\", \"$ua->head(...)\", \"$ua->request(...)\",\netc.), and this special agent will make sure you are nice.\n",
                "subsections": []
            },
            "METHODS": {
                "content": "The LWP::RobotUA is a sub-class of LWP::UserAgent and implements the same methods. In addition\nthe following methods are provided:\n\nnew\nmy $ua = LWP::RobotUA->new( %options )\nmy $ua = LWP::RobotUA->new( $agent, $from )\nmy $ua = LWP::RobotUA->new( $agent, $from, $rules )\n\nThe LWP::UserAgent options \"agent\" and \"from\" are mandatory. The options \"delay\", \"usesleep\"\nand \"rules\" initialize attributes private to the RobotUA. If \"rules\" are not provided, then\nWWW::RobotRules is instantiated providing an internal database of robots.txt.\n\nIt is also possible to just pass the value of \"agent\", \"from\" and optionally \"rules\" as plain\npositional arguments.\n\ndelay\nmy $delay = $ua->delay;\n$ua->delay( $minutes );\n\nGet/set the minimum delay between requests to the same server, in *minutes*. The default is 1\nminute. Note that this number doesn't have to be an integer; for example, this sets the delay to\n10 seconds:\n\n$ua->delay(10/60);\n\nusesleep\nmy $bool = $ua->usesleep;\n$ua->usesleep( $boolean );\n\nGet/set a value indicating whether the UA should \"sleep\" in LWP::RobotUA if requests arrive too\nfast, defined as \"$ua->delay\" minutes not passed since last request to the given server. The\ndefault is true. If this value is false then an internal \"SERVICEUNAVAILABLE\" response will be\ngenerated. It will have a \"Retry-After\" header that indicates when it is OK to send another\nrequest to this server.\n\nrules\nmy $rules = $ua->rules;\n$ua->rules( $rules );\n\nSet/get which *WWW::RobotRules* object to use.\n\nnovisits\nmy $num = $ua->novisits( $netloc )\n\nReturns the number of documents fetched from this server host. Yeah I know, this method should\nprobably have been named \"numvisits\" or something like that. :-(\n\nhostwait\nmy $num = $ua->hostwait( $netloc )\n\nReturns the number of *seconds* (from now) you must wait before you can make a new request to\nthis host.\n\nasstring\nmy $string = $ua->asstring;\n\nReturns a string that describes the state of the UA. Mainly useful for debugging.\n",
                "subsections": []
            },
            "SEE ALSO": {
                "content": "LWP::UserAgent, WWW::RobotRules\n",
                "subsections": []
            },
            "COPYRIGHT": {
                "content": "Copyright 1996-2004 Gisle Aas.\n\nThis library is free software; you can redistribute it and/or modify it under the same terms as\nPerl itself.\n",
                "subsections": []
            }
        }
    }
}