{ "content": [ { "type": "text", "text": "# wg (info)\n\n## Sections\n\n- **Wget 1.21.2**\n- **1 Overview**\n- **2 Invoking** (13 subsections)\n- **3 Recursive Download**\n- **4 Following Links** (5 subsections)\n- **5 Time-Stamping** (3 subsections)\n- **6 Startup File** (4 subsections)\n- **7 Examples** (3 subsections)\n- **8 Various** (8 subsections)\n- **9 Appendices** (3 subsections)\n- **Appendix A Copying this manual** (2 subsections)\n- **Concept Index**\n\nUse structuredContent.sections for detailed options, examples, and full documentation.\n" } ], "structuredContent": { "command": "wg", "section": "", "mode": "info", "summary": null, "synopsis": null, "tldr_summary": null, "tldr_examples": [], "tldr_source": null, "flags": [], "examples": [], "see_also": [], "section_outline": [ { "name": "Wget 1.21.2", "lines": 27, "subsections": [] }, { "name": "1 Overview", "lines": 70, "subsections": [] }, { "name": "2 Invoking", "lines": 29, "subsections": [ { "name": "2.1 URL Format", "lines": 62 }, { "name": "2.2 Option Syntax", "lines": 59 }, { "name": "2.3 Basic Startup Options", "lines": 23 }, { "name": "2.4 Logging and Input File Options", "lines": 131 }, { "name": "2.5 Download Options", "lines": 598 }, { "name": "2.6 Directory Options", "lines": 62 }, { "name": "2.7 HTTP Options", "lines": 419 }, { "name": "2.8 HTTPS (SSL/TLS) Options", "lines": 264 }, { "name": "2.9 FTP Options", "lines": 91 }, { "name": "2.10 FTPS Options", "lines": 40 }, { "name": "2.11 Recursive Retrieval Options", "lines": 209 }, { "name": "2.12 Recursive Accept/Reject Options", "lines": 96 }, { "name": "2.13 Exit Status", "lines": 42 } ] }, { "name": "3 Recursive Download", "lines": 60, "subsections": [] }, { "name": "4 Following Links", "lines": 21, "subsections": [ { "name": "4.1 Spanning Hosts", "lines": 48 }, { "name": "4.2 Types of Files", "lines": 95 }, { "name": "4.3 Directory-Based Limits", "lines": 68 }, { "name": "4.4 Relative Links", "lines": 23 }, { "name": "4.5 Following FTP Links", "lines": 17 } ] }, { "name": "5 Time-Stamping", "lines": 41, "subsections": [ { "name": "5.1 Time-Stamping Usage", "lines": 44 }, { "name": "5.2 HTTP Time-Stamping Internals", "lines": 30 }, { "name": "5.3 FTP Time-Stamping Internals", "lines": 26 } ] }, { "name": "6 Startup File", "lines": 22, "subsections": [ { "name": "6.1 Wgetrc Location", "lines": 16 }, { "name": "6.2 Wgetrc Syntax", "lines": 20 }, { "name": "6.3 Wgetrc Commands", "lines": 485 }, { "name": "6.4 Sample Wgetrc", "lines": 150 } ] }, { "name": "7 Examples", "lines": 11, "subsections": [ { "name": "7.1 Simple Usage", "lines": 36 }, { "name": "7.2 Advanced Usage", "lines": 95 }, { "name": "7.3 Very Advanced Usage", "lines": 33 } ] }, { "name": "8 Various", "lines": 15, "subsections": [ { "name": "8.1 Proxies", "lines": 66 }, { "name": "8.2 Distribution", "lines": 7 }, { "name": "8.3 Web Site", "lines": 6 }, { "name": "8.4 Mailing Lists", "lines": 34 }, { "name": "8.5 Internet Relay Chat", "lines": 5 }, { "name": "8.6 Reporting Bugs", "lines": 52 }, { "name": "8.7 Portability", "lines": 29 }, { "name": "8.8 Signals", "lines": 16 } ] }, { "name": "9 Appendices", "lines": 10, "subsections": [ { "name": "9.1 Robot Exclusion", "lines": 71 }, { "name": "9.2 Security Considerations", "lines": 23 }, { "name": "9.3 Contributors", "lines": 126 } ] }, { "name": "Appendix A Copying this manual", "lines": 6, "subsections": [ { "name": "A.1 GNU Free Documentation License", "lines": 446 }, { "name": "ADDENDUM: How to use this License for your documents", "lines": 30 } ] }, { "name": "Concept Index", "lines": 292, "subsections": [] } ], "sections": { "Wget 1.21.2": { "content": "This file documents the GNU Wget utility for downloading network data.\n\nCopyright (C) 1996-2011, 2015, 2018-2021 Free Software Foundation, Inc.\n\nPermission is granted to copy, distribute and/or modify this document\nunder the terms of the GNU Free Documentation License, Version 1.3 or\nany later version published by the Free Software Foundation; with no\nInvariant Sections, with no Front-Cover Texts, and with no Back-Cover\nTexts. A copy of the license is included in the section entitled \"GNU\nFree Documentation License\".\n\n* Menu:\n\n* Overview:: Features of Wget.\n* Invoking:: Wget command-line arguments.\n* Recursive Download:: Downloading interlinked pages.\n* Following Links:: The available methods of chasing links.\n* Time-Stamping:: Mirroring according to time-stamps.\n* Startup File:: Wget's initialization file.\n* Examples:: Examples of usage.\n* Various:: The stuff that doesn't fit anywhere else.\n* Appendices:: Some useful references.\n* Copying this manual:: You may give out copies of this manual.\n* Concept Index:: Topics covered by this manual.\n\nFile: wget.info, Node: Overview, Next: Invoking, Prev: Top, Up: Top\n", "subsections": [] }, "1 Overview": { "content": "GNU Wget is a free utility for non-interactive download of files from\nthe Web. It supports HTTP, HTTPS, and FTP protocols, as well as\nretrieval through HTTP proxies.\n\nThis chapter is a partial overview of Wget's features.\n\n* Wget is non-interactive, meaning that it can work in the\nbackground, while the user is not logged on. This allows you to\nstart a retrieval and disconnect from the system, letting Wget\nfinish the work. By contrast, most of the Web browsers require\nconstant user's presence, which can be a great hindrance when\ntransferring a lot of data.\n\n* Wget can follow links in HTML, XHTML, and CSS pages, to create\nlocal versions of remote web sites, fully recreating the directory\nstructure of the original site. This is sometimes referred to as\n\"recursive downloading.\" While doing that, Wget respects the Robot\nExclusion Standard ('/robots.txt'). Wget can be instructed to\nconvert the links in downloaded files to point at the local files,\nfor offline viewing.\n\n* File name wildcard matching and recursive mirroring of directories\nare available when retrieving via FTP. Wget can read the\ntime-stamp information given by both HTTP and FTP servers, and\nstore it locally. Thus Wget can see if the remote file has changed\nsince last retrieval, and automatically retrieve the new version if\nit has. This makes Wget suitable for mirroring of FTP sites, as\nwell as home pages.\n\n* Wget has been designed for robustness over slow or unstable network\nconnections; if a download fails due to a network problem, it will\nkeep retrying until the whole file has been retrieved. If the\nserver supports regetting, it will instruct the server to continue\nthe download from where it left off.\n\n* Wget supports proxy servers, which can lighten the network load,\nspeed up retrieval and provide access behind firewalls. Wget uses\nthe passive FTP downloading by default, active FTP being an option.\n\n* Wget supports IP version 6, the next generation of IP. IPv6 is\nautodetected at compile-time, and can be disabled at either build\nor run time. Binaries built with IPv6 support work well in both\nIPv4-only and dual family environments.\n\n* Built-in features offer mechanisms to tune which links you wish to\nfollow (*note Following Links::).\n\n* The progress of individual downloads is traced using a progress\ngauge. Interactive downloads are tracked using a\n\"thermometer\"-style gauge, whereas non-interactive ones are traced\nwith dots, each dot representing a fixed amount of data received\n(1KB by default). Either gauge can be customized to your\npreferences.\n\n* Most of the features are fully configurable, either through command\nline options, or via the initialization file '.wgetrc' (*note\nStartup File::). Wget allows you to define \"global\" startup files\n('/etc/wgetrc' by default) for site settings. You can also specify\nthe location of a startup file with the -config option. To disable\nthe reading of config files, use -no-config. If both -config and\n-no-config are given, -no-config is ignored.\n\n* Finally, GNU Wget is free software. This means that everyone may\nuse it, redistribute it and/or modify it under the terms of the GNU\nGeneral Public License, as published by the Free Software\nFoundation (see the file 'COPYING' that came with GNU Wget, for\ndetails).\n\nFile: wget.info, Node: Invoking, Next: Recursive Download, Prev: Overview, Up: Top\n", "subsections": [] }, "2 Invoking": { "content": "By default, Wget is very simple to invoke. The basic syntax is:\n\nwget [OPTION]... [URL]...\n\nWget will simply download all the URLs specified on the command line.\nURL is a \"Uniform Resource Locator\", as defined below.\n\nHowever, you may wish to change some of the default parameters of\nWget. You can do it two ways: permanently, adding the appropriate\ncommand to '.wgetrc' (*note Startup File::), or specifying it on the\ncommand line.\n\n* Menu:\n\n* URL Format::\n* Option Syntax::\n* Basic Startup Options::\n* Logging and Input File Options::\n* Download Options::\n* Directory Options::\n* HTTP Options::\n* HTTPS (SSL/TLS) Options::\n* FTP Options::\n* Recursive Retrieval Options::\n* Recursive Accept/Reject Options::\n* Exit Status::\n\nFile: wget.info, Node: URL Format, Next: Option Syntax, Prev: Invoking, Up: Invoking\n", "subsections": [ { "name": "2.1 URL Format", "content": "\"URL\" is an acronym for Uniform Resource Locator. A uniform resource\nlocator is a compact string representation for a resource available via\nthe Internet. Wget recognizes the URL syntax as per RFC1738. This is\nthe most widely used form (square brackets denote optional parts):\n\nhttp://host[:port]/directory/file\nftp://host[:port]/directory/file\n\nYou can also encode your username and password within a URL:\n\nftp://user:password@host/path\nhttp://user:password@host/path\n\nEither USER or PASSWORD, or both, may be left out. If you leave out\neither the HTTP username or password, no authentication will be sent.\nIf you leave out the FTP username, 'anonymous' will be used. If you\nleave out the FTP password, your email address will be supplied as a\ndefault password.(1)\n\n*Important Note*: if you specify a password-containing URL on the\ncommand line, the username and password will be plainly visible to all\nusers on the system, by way of 'ps'. On multi-user systems, this is a\nbig security risk. To work around it, use 'wget -i -' and feed the URLs\nto Wget's standard input, each on a separate line, terminated by 'C-d'.\n\nYou can encode unsafe characters in a URL as '%xy', 'xy' being the\nhexadecimal representation of the character's ASCII value. Some common\nunsafe characters include '%' (quoted as '%25'), ':' (quoted as '%3A'),\nand '@' (quoted as '%40'). Refer to RFC1738 for a comprehensive list of\nunsafe characters.\n\nWget also supports the 'type' feature for FTP URLs. By default, FTP\ndocuments are retrieved in the binary mode (type 'i'), which means that\nthey are downloaded unchanged. Another useful mode is the 'a' (\"ASCII\")\nmode, which converts the line delimiters between the different operating\nsystems, and is thus useful for text files. Here is an example:\n\nftp://host/directory/file;type=a\n\nTwo alternative variants of URL specification are also supported,\nbecause of historical (hysterical?) reasons and their widespreaded use.\n\nFTP-only syntax (supported by 'NcFTP'):\nhost:/dir/file\n\nHTTP-only syntax (introduced by 'Netscape'):\nhost[:port]/dir/file\n\nThese two alternative forms are deprecated, and may cease being\nsupported in the future.\n\nIf you do not understand the difference between these notations, or\ndo not know which one to use, just use the plain ordinary format you use\nwith your favorite browser, like 'Lynx' or 'Netscape'.\n\n---------- Footnotes ----------\n\n(1) If you have a '.netrc' file in your home directory, password will\nalso be searched for there.\n\nFile: wget.info, Node: Option Syntax, Next: Basic Startup Options, Prev: URL Format, Up: Invoking\n" }, { "name": "2.2 Option Syntax", "content": "Since Wget uses GNU getopt to process command-line arguments, every\noption has a long form along with the short one. Long options are more\nconvenient to remember, but take time to type. You may freely mix\ndifferent option styles, or specify options after the command-line\narguments. Thus you may write:\n\nwget -r --tries=10 http://fly.srk.fer.hr/ -o log\n\nThe space between the option accepting an argument and the argument\nmay be omitted. Instead of '-o log' you can write '-olog'.\n\nYou may put several options that do not require arguments together,\nlike:\n\nwget -drc URL\n\nThis is completely equivalent to:\n\nwget -d -r -c URL\n\nSince the options can be specified after the arguments, you may\nterminate them with '--'. So the following will try to download URL\n'-x', reporting failure to 'log':\n\nwget -o log -- -x\n\nThe options that accept comma-separated lists all respect the\nconvention that specifying an empty list clears its value. This can be\nuseful to clear the '.wgetrc' settings. For instance, if your '.wgetrc'\nsets 'excludedirectories' to '/cgi-bin', the following example will\nfirst reset it, and then set it to exclude '/~nobody' and '/~somebody'.\nYou can also clear the lists in '.wgetrc' (*note Wgetrc Syntax::).\n\nwget -X \"\" -X /~nobody,/~somebody\n\nMost options that do not accept arguments are \"boolean\" options, so\nnamed because their state can be captured with a yes-or-no (\"boolean\")\nvariable. For example, '--follow-ftp' tells Wget to follow FTP links\nfrom HTML files and, on the other hand, '--no-glob' tells it not to\nperform file globbing on FTP URLs. A boolean option is either\n\"affirmative\" or \"negative\" (beginning with '--no'). All such options\nshare several properties.\n\nUnless stated otherwise, it is assumed that the default behavior is\nthe opposite of what the option accomplishes. For example, the\ndocumented existence of '--follow-ftp' assumes that the default is to\nnot follow FTP links from HTML pages.\n\nAffirmative options can be negated by prepending the '--no-' to the\noption name; negative options can be negated by omitting the '--no-'\nprefix. This might seem superfluous--if the default for an affirmative\noption is to not do something, then why provide a way to explicitly turn\nit off? But the startup file may in fact change the default. For\ninstance, using 'followftp = on' in '.wgetrc' makes Wget follow FTP\nlinks by default, and using '--no-follow-ftp' is the only way to restore\nthe factory default from the command line.\n\nFile: wget.info, Node: Basic Startup Options, Next: Logging and Input File Options, Prev: Option Syntax, Up: Invoking\n" }, { "name": "2.3 Basic Startup Options", "content": "'-V'\n'--version'\nDisplay the version of Wget.\n\n'-h'\n'--help'\nPrint a help message describing all of Wget's command-line options.\n\n'-b'\n'--background'\nGo to background immediately after startup. If no output file is\nspecified via the '-o', output is redirected to 'wget-log'.\n\n'-e COMMAND'\n'--execute COMMAND'\nExecute COMMAND as if it were a part of '.wgetrc' (*note Startup\nFile::). A command thus invoked will be executed after the\ncommands in '.wgetrc', thus taking precedence over them. If you\nneed to specify more than one wgetrc command, use multiple\ninstances of '-e'.\n\nFile: wget.info, Node: Logging and Input File Options, Next: Download Options, Prev: Basic Startup Options, Up: Invoking\n" }, { "name": "2.4 Logging and Input File Options", "content": "'-o LOGFILE'\n'--output-file=LOGFILE'\nLog all messages to LOGFILE. The messages are normally reported to\nstandard error.\n\n'-a LOGFILE'\n'--append-output=LOGFILE'\nAppend to LOGFILE. This is the same as '-o', only it appends to\nLOGFILE instead of overwriting the old log file. If LOGFILE does\nnot exist, a new file is created.\n\n'-d'\n'--debug'\nTurn on debug output, meaning various information important to the\ndevelopers of Wget if it does not work properly. Your system\nadministrator may have chosen to compile Wget without debug\nsupport, in which case '-d' will not work. Please note that\ncompiling with debug support is always safe--Wget compiled with the\ndebug support will not print any debug info unless requested with\n'-d'. *Note Reporting Bugs::, for more information on how to use\n'-d' for sending bug reports.\n\n'-q'\n'--quiet'\nTurn off Wget's output.\n\n'-v'\n'--verbose'\nTurn on verbose output, with all the available data. The default\noutput is verbose.\n\n'-nv'\n'--no-verbose'\nTurn off verbose without being completely quiet (use '-q' for\nthat), which means that error messages and basic information still\nget printed.\n\n'--report-speed=TYPE'\nOutput bandwidth as TYPE. The only accepted value is 'bits'.\n\n'-i FILE'\n'--input-file=FILE'\nRead URLs from a local or external FILE. If '-' is specified as\nFILE, URLs are read from the standard input. (Use './-' to read\nfrom a file literally named '-'.)\n\nIf this function is used, no URLs need be present on the command\nline. If there are URLs both on the command line and in an input\nfile, those on the command lines will be the first ones to be\nretrieved. If '--force-html' is not specified, then FILE should\nconsist of a series of URLs, one per line.\n\nHowever, if you specify '--force-html', the document will be\nregarded as 'html'. In that case you may have problems with\nrelative links, which you can solve either by adding '' to the documents or by specifying '--base=URL' on the\ncommand line.\n\nIf the FILE is an external one, the document will be automatically\ntreated as 'html' if the Content-Type matches 'text/html'.\nFurthermore, the FILE's location will be implicitly used as base\nhref if none was specified.\n\n'--input-metalink=FILE'\nDownloads files covered in local Metalink FILE. Metalink version 3\nand 4 are supported.\n\n'--keep-badhash'\nKeeps downloaded Metalink's files with a bad hash. It appends\n.badhash to the name of Metalink's files which have a checksum\nmismatch, except without overwriting existing files.\n\n'--metalink-over-http'\nIssues HTTP HEAD request instead of GET and extracts Metalink\nmetadata from response headers. Then it switches to Metalink\ndownload. If no valid Metalink metadata is found, it falls back to\nordinary HTTP download. Enables 'Content-Type:\napplication/metalink4+xml' files download/processing.\n\n'--metalink-index=NUMBER'\nSet the Metalink 'application/metalink4+xml' metaurl ordinal\nNUMBER. From 1 to the total number of \"application/metalink4+xml\"\navailable. Specify 0 or 'inf' to choose the first good one.\nMetaurls, such as those from a '--metalink-over-http', may have\nbeen sorted by priority key's value; keep this in mind to choose\nthe right NUMBER.\n\n'--preferred-location'\nSet preferred location for Metalink resources. This has effect if\nmultiple resources with same priority are available.\n\n'--xattr'\nEnable use of file system's extended attributes to save the\noriginal URL and the Referer HTTP header value if used.\n\nBe aware that the URL might contain private information like access\ntokens or credentials.\n\n'-F'\n'--force-html'\nWhen input is read from a file, force it to be treated as an HTML\nfile. This enables you to retrieve relative links from existing\nHTML files on your local disk, by adding '' to\nHTML, or using the '--base' command-line option.\n\n'-B URL'\n'--base=URL'\nResolves relative links using URL as the point of reference, when\nreading links from an HTML file specified via the\n'-i'/'--input-file' option (together with '--force-html', or when\nthe input file was fetched remotely from a server describing it as\nHTML). This is equivalent to the presence of a 'BASE' tag in the\nHTML input file, with URL as the value for the 'href' attribute.\n\nFor instance, if you specify 'http://foo/bar/a.html' for URL, and\nWget reads '../baz/b.html' from the input file, it would be\nresolved to 'http://foo/baz/b.html'.\n\n'--config=FILE'\nSpecify the location of a startup file you wish to use instead of\nthe default one(s). Use -no-config to disable reading of config\nfiles. If both -config and -no-config are given, -no-config is\nignored.\n\n'--rejected-log=LOGFILE'\nLogs all URL rejections to LOGFILE as comma separated values. The\nvalues include the reason of rejection, the URL and the parent URL\nit was found in.\n\nFile: wget.info, Node: Download Options, Next: Directory Options, Prev: Logging and Input File Options, Up: Invoking\n" }, { "name": "2.5 Download Options", "content": "'--bind-address=ADDRESS'\nWhen making client TCP/IP connections, bind to ADDRESS on the local\nmachine. ADDRESS may be specified as a hostname or IP address.\nThis option can be useful if your machine is bound to multiple IPs.\n\n'--bind-dns-address=ADDRESS'\n[libcares only] This address overrides the route for DNS requests.\nIf you ever need to circumvent the standard settings from\n/etc/resolv.conf, this option together with '--dns-servers' is your\nfriend. ADDRESS must be specified either as IPv4 or IPv6 address.\nWget needs to be built with libcares for this option to be\navailable.\n\n'--dns-servers=ADDRESSES'\n[libcares only] The given address(es) override the standard\nnameserver addresses, e.g. as configured in /etc/resolv.conf.\nADDRESSES may be specified either as IPv4 or IPv6 addresses,\ncomma-separated. Wget needs to be built with libcares for this\noption to be available.\n\n'-t NUMBER'\n'--tries=NUMBER'\nSet number of tries to NUMBER. Specify 0 or 'inf' for infinite\nretrying. The default is to retry 20 times, with the exception of\nfatal errors like \"connection refused\" or \"not found\" (404), which\nare not retried.\n\n'-O FILE'\n'--output-document=FILE'\nThe documents will not be written to the appropriate files, but all\nwill be concatenated together and written to FILE. If '-' is used\nas FILE, documents will be printed to standard output, disabling\nlink conversion. (Use './-' to print to a file literally named\n'-'.)\n\nUse of '-O' is not intended to mean simply \"use the name FILE\ninstead of the one in the URL;\" rather, it is analogous to shell\nredirection: 'wget -O file http://foo' is intended to work like\n'wget -O - http://foo > file'; 'file' will be truncated\nimmediately, and all downloaded content will be written there.\n\nFor this reason, '-N' (for timestamp-checking) is not supported in\ncombination with '-O': since FILE is always newly created, it will\nalways have a very new timestamp. A warning will be issued if this\ncombination is used.\n\nSimilarly, using '-r' or '-p' with '-O' may not work as you expect:\nWget won't just download the first file to FILE and then download\nthe rest to their normal names: all downloaded content will be\nplaced in FILE. This was disabled in version 1.11, but has been\nreinstated (with a warning) in 1.11.2, as there are some cases\nwhere this behavior can actually have some use.\n\nA combination with '-nc' is only accepted if the given output file\ndoes not exist.\n\nNote that a combination with '-k' is only permitted when\ndownloading a single document, as in that case it will just convert\nall relative URIs to external ones; '-k' makes no sense for\nmultiple URIs when they're all being downloaded to a single file;\n'-k' can be used only when the output is a regular file.\n\n'-nc'\n'--no-clobber'\nIf a file is downloaded more than once in the same directory,\nWget's behavior depends on a few options, including '-nc'. In\ncertain cases, the local file will be \"clobbered\", or overwritten,\nupon repeated download. In other cases it will be preserved.\n\nWhen running Wget without '-N', '-nc', '-r', or '-p', downloading\nthe same file in the same directory will result in the original\ncopy of FILE being preserved and the second copy being named\n'FILE.1'. If that file is downloaded yet again, the third copy\nwill be named 'FILE.2', and so on. (This is also the behavior with\n'-nd', even if '-r' or '-p' are in effect.) When '-nc' is\nspecified, this behavior is suppressed, and Wget will refuse to\ndownload newer copies of 'FILE'. Therefore, \"'no-clobber'\" is\nactually a misnomer in this mode--it's not clobbering that's\nprevented (as the numeric suffixes were already preventing\nclobbering), but rather the multiple version saving that's\nprevented.\n\nWhen running Wget with '-r' or '-p', but without '-N', '-nd', or\n'-nc', re-downloading a file will result in the new copy simply\noverwriting the old. Adding '-nc' will prevent this behavior,\ninstead causing the original version to be preserved and any newer\ncopies on the server to be ignored.\n\nWhen running Wget with '-N', with or without '-r' or '-p', the\ndecision as to whether or not to download a newer copy of a file\ndepends on the local and remote timestamp and size of the file\n(*note Time-Stamping::). '-nc' may not be specified at the same\ntime as '-N'.\n\nA combination with '-O'/'--output-document' is only accepted if the\ngiven output file does not exist.\n\nNote that when '-nc' is specified, files with the suffixes '.html'\nor '.htm' will be loaded from the local disk and parsed as if they\nhad been retrieved from the Web.\n\n'--backups=BACKUPS'\nBefore (over)writing a file, back up an existing file by adding a\n'.1' suffix ('1' on VMS) to the file name. Such backup files are\nrotated to '.2', '.3', and so on, up to BACKUPS (and lost beyond\nthat).\n\n'--no-netrc'\nDo not try to obtain credentials from '.netrc' file. By default\n'.netrc' file is searched for credentials in case none have been\npassed on command line and authentication is required.\n\n'-c'\n'--continue'\nContinue getting a partially-downloaded file. This is useful when\nyou want to finish up a download started by a previous instance of\nWget, or by another program. For instance:\n\nwget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z\n\nIf there is a file named 'ls-lR.Z' in the current directory, Wget\nwill assume that it is the first portion of the remote file, and\nwill ask the server to continue the retrieval from an offset equal\nto the length of the local file.\n\nNote that you don't need to specify this option if you just want\nthe current invocation of Wget to retry downloading a file should\nthe connection be lost midway through. This is the default\nbehavior. '-c' only affects resumption of downloads started\nprior to this invocation of Wget, and whose local files are still\nsitting around.\n\nWithout '-c', the previous example would just download the remote\nfile to 'ls-lR.Z.1', leaving the truncated 'ls-lR.Z' file alone.\n\nIf you use '-c' on a non-empty file, and the server does not\nsupport continued downloading, Wget will restart the download from\nscratch and overwrite the existing file entirely.\n\nBeginning with Wget 1.7, if you use '-c' on a file which is of\nequal size as the one on the server, Wget will refuse to download\nthe file and print an explanatory message. The same happens when\nthe file is smaller on the server than locally (presumably because\nit was changed on the server since your last download\nattempt)--because \"continuing\" is not meaningful, no download\noccurs.\n\nOn the other side of the coin, while using '-c', any file that's\nbigger on the server than locally will be considered an incomplete\ndownload and only '(length(remote) - length(local))' bytes will be\ndownloaded and tacked onto the end of the local file. This\nbehavior can be desirable in certain cases--for instance, you can\nuse 'wget -c' to download just the new portion that's been appended\nto a data collection or log file.\n\nHowever, if the file is bigger on the server because it's been\nchanged, as opposed to just appended to, you'll end up with a\ngarbled file. Wget has no way of verifying that the local file is\nreally a valid prefix of the remote file. You need to be\nespecially careful of this when using '-c' in conjunction with\n'-r', since every file will be considered as an \"incomplete\ndownload\" candidate.\n\nAnother instance where you'll get a garbled file if you try to use\n'-c' is if you have a lame HTTP proxy that inserts a \"transfer\ninterrupted\" string into the local file. In the future a\n\"rollback\" option may be added to deal with this case.\n\nNote that '-c' only works with FTP servers and with HTTP servers\nthat support the 'Range' header.\n\n'--start-pos=OFFSET'\nStart downloading at zero-based position OFFSET. Offset may be\nexpressed in bytes, kilobytes with the 'k' suffix, or megabytes\nwith the 'm' suffix, etc.\n\n'--start-pos' has higher precedence over '--continue'. When\n'--start-pos' and '--continue' are both specified, wget will emit a\nwarning then proceed as if '--continue' was absent.\n\nServer support for continued download is required, otherwise\n'--start-pos' cannot help. See '-c' for details.\n\n'--progress=TYPE'\nSelect the type of the progress indicator you wish to use. Legal\nindicators are \"dot\" and \"bar\".\n\nThe \"bar\" indicator is used by default. It draws an ASCII progress\nbar graphics (a.k.a \"thermometer\" display) indicating the status of\nretrieval. If the output is not a TTY, the \"dot\" bar will be used\nby default.\n\nUse '--progress=dot' to switch to the \"dot\" display. It traces the\nretrieval by printing dots on the screen, each dot representing a\nfixed amount of downloaded data.\n\nThe progress TYPE can also take one or more parameters. The\nparameters vary based on the TYPE selected. Parameters to TYPE are\npassed by appending them to the type sperated by a colon (:) like\nthis: '--progress=TYPE:PARAMETER1:PARAMETER2'.\n\nWhen using the dotted retrieval, you may set the \"style\" by\nspecifying the type as 'dot:STYLE'. Different styles assign\ndifferent meaning to one dot. With the 'default' style each dot\nrepresents 1K, there are ten dots in a cluster and 50 dots in a\nline. The 'binary' style has a more \"computer\"-like orientation--8K\ndots, 16-dots clusters and 48 dots per line (which makes for 384K\nlines). The 'mega' style is suitable for downloading large\nfiles--each dot represents 64K retrieved, there are eight dots in a\ncluster, and 48 dots on each line (so each line contains 3M). If\n'mega' is not enough then you can use the 'giga' style--each dot\nrepresents 1M retrieved, there are eight dots in a cluster, and 32\ndots on each line (so each line contains 32M).\n\nWith '--progress=bar', there are currently two possible parameters,\nFORCE and NOSCROLL.\n\nWhen the output is not a TTY, the progress bar always falls back to\n\"dot\", even if '--progress=bar' was passed to Wget during\ninvocation. This behaviour can be overridden and the \"bar\" output\nforced by using the \"force\" parameter as '--progress=bar:force'.\n\nBy default, the 'bar' style progress bar scroll the name of the\nfile from left to right for the file being downloaded if the\nfilename exceeds the maximum length allotted for its display. In\ncertain cases, such as with '--progress=bar:force', one may not\nwant the scrolling filename in the progress bar. By passing the\n\"noscroll\" parameter, Wget can be forced to display as much of the\nfilename as possible without scrolling through it.\n\nNote that you can set the default style using the 'progress'\ncommand in '.wgetrc'. That setting may be overridden from the\ncommand line. For example, to force the bar output without\nscrolling, use '--progress=bar:force:noscroll'.\n\n'--show-progress'\nForce wget to display the progress bar in any verbosity.\n\nBy default, wget only displays the progress bar in verbose mode.\nOne may however, want wget to display the progress bar on screen in\nconjunction with any other verbosity modes like '--no-verbose' or\n'--quiet'. This is often a desired a property when invoking wget\nto download several small/large files. In such a case, wget could\nsimply be invoked with this parameter to get a much cleaner output\non the screen.\n\nThis option will also force the progress bar to be printed to\n'stderr' when used alongside the '--output-file' option.\n\n'-N'\n'--timestamping'\nTurn on time-stamping. *Note Time-Stamping::, for details.\n\n'--no-if-modified-since'\nDo not send If-Modified-Since header in '-N' mode. Send\npreliminary HEAD request instead. This has only effect in '-N'\nmode.\n\n'--no-use-server-timestamps'\nDon't set the local file's timestamp by the one on the server.\n\nBy default, when a file is downloaded, its timestamps are set to\nmatch those from the remote file. This allows the use of\n'--timestamping' on subsequent invocations of wget. However, it is\nsometimes useful to base the local file's timestamp on when it was\nactually downloaded; for that purpose, the\n'--no-use-server-timestamps' option has been provided.\n\n'-S'\n'--server-response'\nPrint the headers sent by HTTP servers and responses sent by FTP\nservers.\n\n'--spider'\nWhen invoked with this option, Wget will behave as a Web \"spider\",\nwhich means that it will not download the pages, just check that\nthey are there. For example, you can use Wget to check your\nbookmarks:\n\nwget --spider --force-html -i bookmarks.html\n\nThis feature needs much more work for Wget to get close to the\nfunctionality of real web spiders.\n\n'-T seconds'\n'--timeout=SECONDS'\nSet the network timeout to SECONDS seconds. This is equivalent to\nspecifying '--dns-timeout', '--connect-timeout', and\n'--read-timeout', all at the same time.\n\nWhen interacting with the network, Wget can check for timeout and\nabort the operation if it takes too long. This prevents anomalies\nlike hanging reads and infinite connects. The only timeout enabled\nby default is a 900-second read timeout. Setting a timeout to 0\ndisables it altogether. Unless you know what you are doing, it is\nbest not to change the default timeout settings.\n\nAll timeout-related options accept decimal values, as well as\nsubsecond values. For example, '0.1' seconds is a legal (though\nunwise) choice of timeout. Subsecond timeouts are useful for\nchecking server response times or for testing network latency.\n\n'--dns-timeout=SECONDS'\nSet the DNS lookup timeout to SECONDS seconds. DNS lookups that\ndon't complete within the specified time will fail. By default,\nthere is no timeout on DNS lookups, other than that implemented by\nsystem libraries.\n\n'--connect-timeout=SECONDS'\nSet the connect timeout to SECONDS seconds. TCP connections that\ntake longer to establish will be aborted. By default, there is no\nconnect timeout, other than that implemented by system libraries.\n\n'--read-timeout=SECONDS'\nSet the read (and write) timeout to SECONDS seconds. The \"time\" of\nthis timeout refers to \"idle time\": if, at any point in the\ndownload, no data is received for more than the specified number of\nseconds, reading fails and the download is restarted. This option\ndoes not directly affect the duration of the entire download.\n\nOf course, the remote server may choose to terminate the connection\nsooner than this option requires. The default read timeout is 900\nseconds.\n\n'--limit-rate=AMOUNT'\nLimit the download speed to AMOUNT bytes per second. Amount may be\nexpressed in bytes, kilobytes with the 'k' suffix, or megabytes\nwith the 'm' suffix. For example, '--limit-rate=20k' will limit\nthe retrieval rate to 20KB/s. This is useful when, for whatever\nreason, you don't want Wget to consume the entire available\nbandwidth.\n\nThis option allows the use of decimal numbers, usually in\nconjunction with power suffixes; for example, '--limit-rate=2.5k'\nis a legal value.\n\nNote that Wget implements the limiting by sleeping the appropriate\namount of time after a network read that took less time than\nspecified by the rate. Eventually this strategy causes the TCP\ntransfer to slow down to approximately the specified rate.\nHowever, it may take some time for this balance to be achieved, so\ndon't be surprised if limiting the rate doesn't work well with very\nsmall files.\n\n'-w SECONDS'\n'--wait=SECONDS'\nWait the specified number of seconds between the retrievals. Use\nof this option is recommended, as it lightens the server load by\nmaking the requests less frequent. Instead of in seconds, the time\ncan be specified in minutes using the 'm' suffix, in hours using\n'h' suffix, or in days using 'd' suffix.\n\nSpecifying a large value for this option is useful if the network\nor the destination host is down, so that Wget can wait long enough\nto reasonably expect the network error to be fixed before the\nretry. The waiting interval specified by this function is\ninfluenced by '--random-wait', which see.\n\n'--waitretry=SECONDS'\nIf you don't want Wget to wait between every retrieval, but only\nbetween retries of failed downloads, you can use this option. Wget\nwill use \"linear backoff\", waiting 1 second after the first failure\non a given file, then waiting 2 seconds after the second failure on\nthat file, up to the maximum number of SECONDS you specify.\n\nBy default, Wget will assume a value of 10 seconds.\n\n'--random-wait'\nSome web sites may perform log analysis to identify retrieval\nprograms such as Wget by looking for statistically significant\nsimilarities in the time between requests. This option causes the\ntime between requests to vary between 0.5 and 1.5 * WAIT seconds,\nwhere WAIT was specified using the '--wait' option, in order to\nmask Wget's presence from such analysis.\n\nA 2001 article in a publication devoted to development on a popular\nconsumer platform provided code to perform this analysis on the\nfly. Its author suggested blocking at the class C address level to\nensure automated retrieval programs were blocked despite changing\nDHCP-supplied addresses.\n\nThe '--random-wait' option was inspired by this ill-advised\nrecommendation to block many unrelated users from a web site due to\nthe actions of one.\n\n'--no-proxy'\nDon't use proxies, even if the appropriate '*proxy' environment\nvariable is defined.\n\n*Note Proxies::, for more information about the use of proxies with\nWget.\n\n'-Q QUOTA'\n'--quota=QUOTA'\nSpecify download quota for automatic retrievals. The value can be\nspecified in bytes (default), kilobytes (with 'k' suffix), or\nmegabytes (with 'm' suffix).\n\nNote that quota will never affect downloading a single file. So if\nyou specify 'wget -Q10k https://example.com/ls-lR.gz', all of the\n'ls-lR.gz' will be downloaded. The same goes even when several\nURLs are specified on the command-line. The quota is checked only\nat the end of each downloaded file, so it will never result in a\npartially downloaded file. Thus you may safely type 'wget -Q2m -i\nsites'--download will be aborted after the file that exhausts the\nquota is completely downloaded.\n\nSetting quota to 0 or to 'inf' unlimits the download quota.\n\n'--no-dns-cache'\nTurn off caching of DNS lookups. Normally, Wget remembers the IP\naddresses it looked up from DNS so it doesn't have to repeatedly\ncontact the DNS server for the same (typically small) set of hosts\nit retrieves from. This cache exists in memory only; a new Wget\nrun will contact DNS again.\n\nHowever, it has been reported that in some situations it is not\ndesirable to cache host names, even for the duration of a\nshort-running application like Wget. With this option Wget issues\na new DNS lookup (more precisely, a new call to 'gethostbyname' or\n'getaddrinfo') each time it makes a new connection. Please note\nthat this option will not affect caching that might be performed\nby the resolving library or by an external caching layer, such as\nNSCD.\n\nIf you don't understand exactly what this option does, you probably\nwon't need it.\n\n'--restrict-file-names=MODES'\nChange which characters found in remote URLs must be escaped during\ngeneration of local filenames. Characters that are \"restricted\" by\nthis option are escaped, i.e. replaced with '%HH', where 'HH' is\nthe hexadecimal number that corresponds to the restricted\ncharacter. This option may also be used to force all alphabetical\ncases to be either lower- or uppercase.\n\nBy default, Wget escapes the characters that are not valid or safe\nas part of file names on your operating system, as well as control\ncharacters that are typically unprintable. This option is useful\nfor changing these defaults, perhaps because you are downloading to\na non-native partition, or because you want to disable escaping of\nthe control characters, or you want to further restrict characters\nto only those in the ASCII range of values.\n\nThe MODES are a comma-separated set of text values. The acceptable\nvalues are 'unix', 'windows', 'nocontrol', 'ascii', 'lowercase',\nand 'uppercase'. The values 'unix' and 'windows' are mutually\nexclusive (one will override the other), as are 'lowercase' and\n'uppercase'. Those last are special cases, as they do not change\nthe set of characters that would be escaped, but rather force local\nfile paths to be converted either to lower- or uppercase.\n\nWhen \"unix\" is specified, Wget escapes the character '/' and the\ncontrol characters in the ranges 0-31 and 128-159. This is the\ndefault on Unix-like operating systems.\n\nWhen \"windows\" is given, Wget escapes the characters '\\', '|', '/',\n':', '?', '\"', '*', '<', '>', and the control characters in the\nranges 0-31 and 128-159. In addition to this, Wget in Windows mode\nuses '+' instead of ':' to separate host and port in local file\nnames, and uses '@' instead of '?' to separate the query portion of\nthe file name from the rest. Therefore, a URL that would be saved\nas 'www.xemacs.org:4300/search.pl?input=blah' in Unix mode would be\nsaved as 'www.xemacs.org+4300/search.pl@input=blah' in Windows\nmode. This mode is the default on Windows.\n\nIf you specify 'nocontrol', then the escaping of the control\ncharacters is also switched off. This option may make sense when\nyou are downloading URLs whose names contain UTF-8 characters, on a\nsystem which can save and display filenames in UTF-8 (some possible\nbyte values used in UTF-8 byte sequences fall in the range of\nvalues designated by Wget as \"controls\").\n\nThe 'ascii' mode is used to specify that any bytes whose values are\noutside the range of ASCII characters (that is, greater than 127)\nshall be escaped. This can be useful when saving filenames whose\nencoding does not match the one used locally.\n\n'-4'\n'--inet4-only'\n'-6'\n'--inet6-only'\nForce connecting to IPv4 or IPv6 addresses. With '--inet4-only' or\n'-4', Wget will only connect to IPv4 hosts, ignoring AAAA records\nin DNS, and refusing to connect to IPv6 addresses specified in\nURLs. Conversely, with '--inet6-only' or '-6', Wget will only\nconnect to IPv6 hosts and ignore A records and IPv4 addresses.\n\nNeither options should be needed normally. By default, an\nIPv6-aware Wget will use the address family specified by the host's\nDNS record. If the DNS responds with both IPv4 and IPv6 addresses,\nWget will try them in sequence until it finds one it can connect\nto. (Also see '--prefer-family' option described below.)\n\nThese options can be used to deliberately force the use of IPv4 or\nIPv6 address families on dual family systems, usually to aid\ndebugging or to deal with broken network configuration. Only one\nof '--inet6-only' and '--inet4-only' may be specified at the same\ntime. Neither option is available in Wget compiled without IPv6\nsupport.\n\n'--prefer-family=none/IPv4/IPv6'\nWhen given a choice of several addresses, connect to the addresses\nwith specified address family first. The address order returned by\nDNS is used without change by default.\n\nThis avoids spurious errors and connect attempts when accessing\nhosts that resolve to both IPv6 and IPv4 addresses from IPv4\nnetworks. For example, 'www.kame.net' resolves to\n'2001:200:0:8002:203:47ff:fea5:3085' and to '203.178.141.194'.\nWhen the preferred family is 'IPv4', the IPv4 address is used\nfirst; when the preferred family is 'IPv6', the IPv6 address is\nused first; if the specified value is 'none', the address order\nreturned by DNS is used without change.\n\nUnlike '-4' and '-6', this option doesn't inhibit access to any\naddress family, it only changes the order in which the addresses\nare accessed. Also note that the reordering performed by this\noption is \"stable\"--it doesn't affect order of addresses of the same\nfamily. That is, the relative order of all IPv4 addresses and of\nall IPv6 addresses remains intact in all cases.\n\n'--retry-connrefused'\nConsider \"connection refused\" a transient error and try again.\nNormally Wget gives up on a URL when it is unable to connect to the\nsite because failure to connect is taken as a sign that the server\nis not running at all and that retries would not help. This option\nis for mirroring unreliable sites whose servers tend to disappear\nfor short periods of time.\n\n'--user=USER'\n'--password=PASSWORD'\nSpecify the username USER and password PASSWORD for both FTP and\nHTTP file retrieval. These parameters can be overridden using the\n'--ftp-user' and '--ftp-password' options for FTP connections and\nthe '--http-user' and '--http-password' options for HTTP\nconnections.\n\n'--ask-password'\nPrompt for a password for each connection established. Cannot be\nspecified when '--password' is being used, because they are\nmutually exclusive.\n\n'--use-askpass=COMMAND'\nPrompt for a user and password using the specified command. If no\ncommand is specified then the command in the environment variable\nWGETASKPASS is used. If WGETASKPASS is not set then the command\nin the environment variable SSHASKPASS is used.\n\nYou can set the default command for use-askpass in the '.wgetrc'.\nThat setting may be overridden from the command line.\n\n'--no-iri'\n\nTurn off internationalized URI (IRI) support. Use '--iri' to turn\nit on. IRI support is activated by default.\n\nYou can set the default state of IRI support using the 'iri'\ncommand in '.wgetrc'. That setting may be overridden from the\ncommand line.\n\n'--local-encoding=ENCODING'\n\nForce Wget to use ENCODING as the default system encoding. That\naffects how Wget converts URLs specified as arguments from locale\nto UTF-8 for IRI support.\n\nWget use the function 'nllanginfo()' and then the 'CHARSET'\nenvironment variable to get the locale. If it fails, ASCII is\nused.\n\nYou can set the default local encoding using the 'localencoding'\ncommand in '.wgetrc'. That setting may be overridden from the\ncommand line.\n\n'--remote-encoding=ENCODING'\n\nForce Wget to use ENCODING as the default remote server encoding.\nThat affects how Wget converts URIs found in files from remote\nencoding to UTF-8 during a recursive fetch. This options is only\nuseful for IRI support, for the interpretation of non-ASCII\ncharacters.\n\nFor HTTP, remote encoding can be found in HTTP 'Content-Type'\nheader and in HTML 'Content-Type http-equiv' meta tag.\n\nYou can set the default encoding using the 'remoteencoding' command\nin '.wgetrc'. That setting may be overridden from the command\nline.\n\n'--unlink'\n\nForce Wget to unlink file instead of clobbering existing file.\nThis option is useful for downloading to the directory with\nhardlinks.\n\nFile: wget.info, Node: Directory Options, Next: HTTP Options, Prev: Download Options, Up: Invoking\n" }, { "name": "2.6 Directory Options", "content": "'-nd'\n'--no-directories'\nDo not create a hierarchy of directories when retrieving\nrecursively. With this option turned on, all files will get saved\nto the current directory, without clobbering (if a name shows up\nmore than once, the filenames will get extensions '.n').\n\n'-x'\n'--force-directories'\nThe opposite of '-nd'--create a hierarchy of directories, even if\none would not have been created otherwise. E.g. 'wget -x\nhttp://fly.srk.fer.hr/robots.txt' will save the downloaded file to\n'fly.srk.fer.hr/robots.txt'.\n\n'-nH'\n'--no-host-directories'\nDisable generation of host-prefixed directories. By default,\ninvoking Wget with '-r http://fly.srk.fer.hr/' will create a\nstructure of directories beginning with 'fly.srk.fer.hr/'. This\noption disables such behavior.\n\n'--protocol-directories'\nUse the protocol name as a directory component of local file names.\nFor example, with this option, 'wget -r http://HOST' will save to\n'http/HOST/...' rather than just to 'HOST/...'.\n\n'--cut-dirs=NUMBER'\nIgnore NUMBER directory components. This is useful for getting a\nfine-grained control over the directory where recursive retrieval\nwill be saved.\n\nTake, for example, the directory at\n'ftp://ftp.xemacs.org/pub/xemacs/'. If you retrieve it with '-r',\nit will be saved locally under 'ftp.xemacs.org/pub/xemacs/'. While\nthe '-nH' option can remove the 'ftp.xemacs.org/' part, you are\nstill stuck with 'pub/xemacs'. This is where '--cut-dirs' comes in\nhandy; it makes Wget not \"see\" NUMBER remote directory components.\nHere are several examples of how '--cut-dirs' option works.\n\nNo options -> ftp.xemacs.org/pub/xemacs/\n-nH -> pub/xemacs/\n-nH --cut-dirs=1 -> xemacs/\n-nH --cut-dirs=2 -> .\n\n--cut-dirs=1 -> ftp.xemacs.org/xemacs/\n...\n\nIf you just want to get rid of the directory structure, this option\nis similar to a combination of '-nd' and '-P'. However, unlike\n'-nd', '--cut-dirs' does not lose with subdirectories--for instance,\nwith '-nH --cut-dirs=1', a 'beta/' subdirectory will be placed to\n'xemacs/beta', as one would expect.\n\n'-P PREFIX'\n'--directory-prefix=PREFIX'\nSet directory prefix to PREFIX. The \"directory prefix\" is the\ndirectory where all other files and subdirectories will be saved\nto, i.e. the top of the retrieval tree. The default is '.' (the\ncurrent directory).\n\nFile: wget.info, Node: HTTP Options, Next: HTTPS (SSL/TLS) Options, Prev: Directory Options, Up: Invoking\n" }, { "name": "2.7 HTTP Options", "content": "'--default-page=NAME'\nUse NAME as the default file name when it isn't known (i.e., for\nURLs that end in a slash), instead of 'index.html'.\n\n'-E'\n'--adjust-extension'\nIf a file of type 'application/xhtml+xml' or 'text/html' is\ndownloaded and the URL does not end with the regexp\n'\\.[Hh][Tt][Mm][Ll]?', this option will cause the suffix '.html' to\nbe appended to the local filename. This is useful, for instance,\nwhen you're mirroring a remote site that uses '.asp' pages, but you\nwant the mirrored pages to be viewable on your stock Apache server.\nAnother good use for this is when you're downloading CGI-generated\nmaterials. A URL like 'http://site.com/article.cgi?25' will be\nsaved as 'article.cgi?25.html'.\n\nNote that filenames changed in this way will be re-downloaded every\ntime you re-mirror a site, because Wget can't tell that the local\n'X.html' file corresponds to remote URL 'X' (since it doesn't yet\nknow that the URL produces output of type 'text/html' or\n'application/xhtml+xml'.\n\nAs of version 1.12, Wget will also ensure that any downloaded files\nof type 'text/css' end in the suffix '.css', and the option was\nrenamed from '--html-extension', to better reflect its new\nbehavior. The old option name is still acceptable, but should now\nbe considered deprecated.\n\nAs of version 1.19.2, Wget will also ensure that any downloaded\nfiles with a 'Content-Encoding' of 'br', 'compress', 'deflate' or\n'gzip' end in the suffix '.br', '.Z', '.zlib' and '.gz'\nrespectively.\n\nAt some point in the future, this option may well be expanded to\ninclude suffixes for other types of content, including content\ntypes that are not parsed by Wget.\n\n'--http-user=USER'\n'--http-password=PASSWORD'\nSpecify the username USER and password PASSWORD on an HTTP server.\nAccording to the type of the challenge, Wget will encode them using\neither the 'basic' (insecure), the 'digest', or the Windows 'NTLM'\nauthentication scheme.\n\nAnother way to specify username and password is in the URL itself\n(*note URL Format::). Either method reveals your password to\nanyone who bothers to run 'ps'. To prevent the passwords from\nbeing seen, use the '--use-askpass' or store them in '.wgetrc' or\n'.netrc', and make sure to protect those files from other users\nwith 'chmod'. If the passwords are really important, do not leave\nthem lying in those files either--edit the files and delete them\nafter Wget has started the download.\n\n'--no-http-keep-alive'\nTurn off the \"keep-alive\" feature for HTTP downloads. Normally,\nWget asks the server to keep the connection open so that, when you\ndownload more than one document from the same server, they get\ntransferred over the same TCP connection. This saves time and at\nthe same time reduces the load on the server.\n\nThis option is useful when, for some reason, persistent\n(keep-alive) connections don't work for you, for example due to a\nserver bug or due to the inability of server-side scripts to cope\nwith the connections.\n\n'--no-cache'\nDisable server-side cache. In this case, Wget will send the remote\nserver appropriate directives ('Cache-Control: no-cache' and\n'Pragma: no-cache') to get the file from the remote service, rather\nthan returning the cached version. This is especially useful for\nretrieving and flushing out-of-date documents on proxy servers.\n\nCaching is allowed by default.\n\n'--no-cookies'\nDisable the use of cookies. Cookies are a mechanism for\nmaintaining server-side state. The server sends the client a\ncookie using the 'Set-Cookie' header, and the client responds with\nthe same cookie upon further requests. Since cookies allow the\nserver owners to keep track of visitors and for sites to exchange\nthis information, some consider them a breach of privacy. The\ndefault is to use cookies; however, storing cookies is not on by\ndefault.\n\n'--load-cookies FILE'\nLoad cookies from FILE before the first HTTP retrieval. FILE is a\ntextual file in the format originally used by Netscape's\n'cookies.txt' file.\n\nYou will typically use this option when mirroring sites that\nrequire that you be logged in to access some or all of their\ncontent. The login process typically works by the web server\nissuing an HTTP cookie upon receiving and verifying your\ncredentials. The cookie is then resent by the browser when\naccessing that part of the site, and so proves your identity.\n\nMirroring such a site requires Wget to send the same cookies your\nbrowser sends when communicating with the site. This is achieved\nby '--load-cookies'--simply point Wget to the location of the\n'cookies.txt' file, and it will send the same cookies your browser\nwould send in the same situation. Different browsers keep textual\ncookie files in different locations:\n\nNetscape 4.x.\nThe cookies are in '~/.netscape/cookies.txt'.\n\nMozilla and Netscape 6.x.\nMozilla's cookie file is also named 'cookies.txt', located\nsomewhere under '~/.mozilla', in the directory of your\nprofile. The full path usually ends up looking somewhat like\n'~/.mozilla/default/SOME-WEIRD-STRING/cookies.txt'.\n\nInternet Explorer.\nYou can produce a cookie file Wget can use by using the File\nmenu, Import and Export, Export Cookies. This has been tested\nwith Internet Explorer 5; it is not guaranteed to work with\nearlier versions.\n\nOther browsers.\nIf you are using a different browser to create your cookies,\n'--load-cookies' will only work if you can locate or produce a\ncookie file in the Netscape format that Wget expects.\n\nIf you cannot use '--load-cookies', there might still be an\nalternative. If your browser supports a \"cookie manager\", you can\nuse it to view the cookies used when accessing the site you're\nmirroring. Write down the name and value of the cookie, and\nmanually instruct Wget to send those cookies, bypassing the\n\"official\" cookie support:\n\nwget --no-cookies --header \"Cookie: NAME=VALUE\"\n\n'--save-cookies FILE'\nSave cookies to FILE before exiting. This will not save cookies\nthat have expired or that have no expiry time (so-called \"session\ncookies\"), but also see '--keep-session-cookies'.\n\n'--keep-session-cookies'\nWhen specified, causes '--save-cookies' to also save session\ncookies. Session cookies are normally not saved because they are\nmeant to be kept in memory and forgotten when you exit the browser.\nSaving them is useful on sites that require you to log in or to\nvisit the home page before you can access some pages. With this\noption, multiple Wget runs are considered a single browser session\nas far as the site is concerned.\n\nSince the cookie file format does not normally carry session\ncookies, Wget marks them with an expiry timestamp of 0. Wget's\n'--load-cookies' recognizes those as session cookies, but it might\nconfuse other browsers. Also note that cookies so loaded will be\ntreated as other session cookies, which means that if you want\n'--save-cookies' to preserve them again, you must use\n'--keep-session-cookies' again.\n\n'--ignore-length'\nUnfortunately, some HTTP servers (CGI programs, to be more precise)\nsend out bogus 'Content-Length' headers, which makes Wget go wild,\nas it thinks not all the document was retrieved. You can spot this\nsyndrome if Wget retries getting the same document again and again,\neach time claiming that the (otherwise normal) connection has\nclosed on the very same byte.\n\nWith this option, Wget will ignore the 'Content-Length' header--as\nif it never existed.\n\n'--header=HEADER-LINE'\nSend HEADER-LINE along with the rest of the headers in each HTTP\nrequest. The supplied header is sent as-is, which means it must\ncontain name and value separated by colon, and must not contain\nnewlines.\n\nYou may define more than one additional header by specifying\n'--header' more than once.\n\nwget --header='Accept-Charset: iso-8859-2' \\\n--header='Accept-Language: hr' \\\nhttp://fly.srk.fer.hr/\n\nSpecification of an empty string as the header value will clear all\nprevious user-defined headers.\n\nAs of Wget 1.10, this option can be used to override headers\notherwise generated automatically. This example instructs Wget to\nconnect to localhost, but to specify 'foo.bar' in the 'Host'\nheader:\n\nwget --header=\"Host: foo.bar\" http://localhost/\n\nIn versions of Wget prior to 1.10 such use of '--header' caused\nsending of duplicate headers.\n\n'--compression=TYPE'\nChoose the type of compression to be used. Legal values are\n'auto', 'gzip' and 'none'.\n\nIf 'auto' or 'gzip' are specified, Wget asks the server to compress\nthe file using the gzip compression format. If the server\ncompresses the file and responds with the 'Content-Encoding' header\nfield set appropriately, the file will be decompressed\nautomatically.\n\nIf 'none' is specified, wget will not ask the server to compress\nthe file and will not decompress any server responses. This is the\ndefault.\n\nCompression support is currently experimental. In case it is\nturned on, please report any bugs to 'bug-wget@gnu.org'.\n\n'--max-redirect=NUMBER'\nSpecifies the maximum number of redirections to follow for a\nresource. The default is 20, which is usually far more than\nnecessary. However, on those occasions where you want to allow\nmore (or fewer), this is the option to use.\n\n'--proxy-user=USER'\n'--proxy-password=PASSWORD'\nSpecify the username USER and password PASSWORD for authentication\non a proxy server. Wget will encode them using the 'basic'\nauthentication scheme.\n\nSecurity considerations similar to those with '--http-password'\npertain here as well.\n\n'--referer=URL'\nInclude 'Referer: URL' header in HTTP request. Useful for\nretrieving documents with server-side processing that assume they\nare always being retrieved by interactive web browsers and only\ncome out properly when Referer is set to one of the pages that\npoint to them.\n\n'--save-headers'\nSave the headers sent by the HTTP server to the file, preceding the\nactual contents, with an empty line as the separator.\n\n'-U AGENT-STRING'\n'--user-agent=AGENT-STRING'\nIdentify as AGENT-STRING to the HTTP server.\n\nThe HTTP protocol allows the clients to identify themselves using a\n'User-Agent' header field. This enables distinguishing the WWW\nsoftware, usually for statistical purposes or for tracing of\nprotocol violations. Wget normally identifies as 'Wget/VERSION',\nVERSION being the current version number of Wget.\n\nHowever, some sites have been known to impose the policy of\ntailoring the output according to the 'User-Agent'-supplied\ninformation. While this is not such a bad idea in theory, it has\nbeen abused by servers denying information to clients other than\n(historically) Netscape or, more frequently, Microsoft Internet\nExplorer. This option allows you to change the 'User-Agent' line\nissued by Wget. Use of this option is discouraged, unless you\nreally know what you are doing.\n\nSpecifying empty user agent with '--user-agent=\"\"' instructs Wget\nnot to send the 'User-Agent' header in HTTP requests.\n\n'--post-data=STRING'\n'--post-file=FILE'\nUse POST as the method for all HTTP requests and send the specified\ndata in the request body. '--post-data' sends STRING as data,\nwhereas '--post-file' sends the contents of FILE. Other than that,\nthey work in exactly the same way. In particular, they both\nexpect content of the form 'key1=value1&key2=value2', with\npercent-encoding for special characters; the only difference is\nthat one expects its content as a command-line parameter and the\nother accepts its content from a file. In particular,\n'--post-file' is not for transmitting files as form attachments:\nthose must appear as 'key=value' data (with appropriate\npercent-coding) just like everything else. Wget does not currently\nsupport 'multipart/form-data' for transmitting POST data; only\n'application/x-www-form-urlencoded'. Only one of '--post-data' and\n'--post-file' should be specified.\n\nPlease note that wget does not require the content to be of the\nform 'key1=value1&key2=value2', and neither does it test for it.\nWget will simply transmit whatever data is provided to it. Most\nservers however expect the POST data to be in the above format when\nprocessing HTML Forms.\n\nWhen sending a POST request using the '--post-file' option, Wget\ntreats the file as a binary file and will send every character in\nthe POST request without stripping trailing newline or formfeed\ncharacters. Any other control characters in the text will also be\nsent as-is in the POST request.\n\nPlease be aware that Wget needs to know the size of the POST data\nin advance. Therefore the argument to '--post-file' must be a\nregular file; specifying a FIFO or something like '/dev/stdin'\nwon't work. It's not quite clear how to work around this\nlimitation inherent in HTTP/1.0. Although HTTP/1.1 introduces\n\"chunked\" transfer that doesn't require knowing the request length\nin advance, a client can't use chunked unless it knows it's talking\nto an HTTP/1.1 server. And it can't know that until it receives a\nresponse, which in turn requires the request to have been completed\n- a chicken-and-egg problem.\n\nNote: As of version 1.15 if Wget is redirected after the POST\nrequest is completed, its behaviour will depend on the response\ncode returned by the server. In case of a 301 Moved Permanently,\n302 Moved Temporarily or 307 Temporary Redirect, Wget will, in\naccordance with RFC2616, continue to send a POST request. In case\na server wants the client to change the Request method upon\nredirection, it should send a 303 See Other response code.\n\nThis example shows how to log in to a server using POST and then\nproceed to download the desired pages, presumably only accessible\nto authorized users:\n\n# Log in to the server. This can be done only once.\nwget --save-cookies cookies.txt \\\n--post-data 'user=foo&password=bar' \\\nhttp://example.com/auth.php\n\n# Now grab the page or pages we care about.\nwget --load-cookies cookies.txt \\\n-p http://example.com/interesting/article.php\n\nIf the server is using session cookies to track user\nauthentication, the above will not work because '--save-cookies'\nwill not save them (and neither will browsers) and the\n'cookies.txt' file will be empty. In that case use\n'--keep-session-cookies' along with '--save-cookies' to force\nsaving of session cookies.\n\n'--method=HTTP-METHOD'\nFor the purpose of RESTful scripting, Wget allows sending of other\nHTTP Methods without the need to explicitly set them using\n'--header=Header-Line'. Wget will use whatever string is passed to\nit after '--method' as the HTTP Method to the server.\n\n'--body-data=DATA-STRING'\n'--body-file=DATA-FILE'\nMust be set when additional data needs to be sent to the server\nalong with the Method specified using '--method'. '--body-data'\nsends STRING as data, whereas '--body-file' sends the contents of\nFILE. Other than that, they work in exactly the same way.\n\nCurrently, '--body-file' is not for transmitting files as a\nwhole. Wget does not currently support 'multipart/form-data' for\ntransmitting data; only 'application/x-www-form-urlencoded'. In\nthe future, this may be changed so that wget sends the\n'--body-file' as a complete file instead of sending its contents to\nthe server. Please be aware that Wget needs to know the contents\nof BODY Data in advance, and hence the argument to '--body-file'\nshould be a regular file. See '--post-file' for a more detailed\nexplanation. Only one of '--body-data' and '--body-file' should be\nspecified.\n\nIf Wget is redirected after the request is completed, Wget will\nsuspend the current method and send a GET request till the\nredirection is completed. This is true for all redirection\nresponse codes except 307 Temporary Redirect which is used to\nexplicitly specify that the request method should not change.\nAnother exception is when the method is set to 'POST', in which\ncase the redirection rules specified under '--post-data' are\nfollowed.\n\n'--content-disposition'\n\nIf this is set to on, experimental (not fully-functional) support\nfor 'Content-Disposition' headers is enabled. This can currently\nresult in extra round-trips to the server for a 'HEAD' request, and\nis known to suffer from a few bugs, which is why it is not\ncurrently enabled by default.\n\nThis option is useful for some file-downloading CGI programs that\nuse 'Content-Disposition' headers to describe what the name of a\ndownloaded file should be.\n\nWhen combined with '--metalink-over-http' and\n'--trust-server-names', a 'Content-Type: application/metalink4+xml'\nfile is named using the 'Content-Disposition' filename field, if\navailable.\n\n'--content-on-error'\n\nIf this is set to on, wget will not skip the content when the\nserver responds with a http status code that indicates error.\n\n'--trust-server-names'\n\nIf this is set, on a redirect, the local file name will be based on\nthe redirection URL. By default the local file name is based on the\noriginal URL. When doing recursive retrieving this can be helpful\nbecause in many web sites redirected URLs correspond to an\nunderlying file structure, while link URLs do not.\n\n'--auth-no-challenge'\n\nIf this option is given, Wget will send Basic HTTP authentication\ninformation (plaintext username and password) for all requests,\njust like Wget 1.10.2 and prior did by default.\n\nUse of this option is not recommended, and is intended only to\nsupport some few obscure servers, which never send HTTP\nauthentication challenges, but accept unsolicited auth info, say,\nin addition to form-based authentication.\n\n'--retry-on-host-error'\nConsider host errors, such as \"Temporary failure in name\nresolution\", as non-fatal, transient errors.\n\n'--retry-on-http-error=CODE[,CODE,...]'\nConsider given HTTP response codes as non-fatal, transient errors.\nSupply a comma-separated list of 3-digit HTTP response codes as\nargument. Useful to work around special circumstances where\nretries are required, but the server responds with an error code\nnormally not retried by Wget. Such errors might be 503 (Service\nUnavailable) and 429 (Too Many Requests). Retries enabled by this\noption are performed subject to the normal retry timing and retry\ncount limitations of Wget.\n\nUsing this option is intended to support special use cases only and\nis generally not recommended, as it can force retries even in cases\nwhere the server is actually trying to decrease its load. Please\nuse wisely and only if you know what you are doing.\n\nFile: wget.info, Node: HTTPS (SSL/TLS) Options, Next: FTP Options, Prev: HTTP Options, Up: Invoking\n" }, { "name": "2.8 HTTPS (SSL/TLS) Options", "content": "To support encrypted HTTP (HTTPS) downloads, Wget must be compiled with\nan external SSL library. The current default is GnuTLS. In addition,\nWget also supports HSTS (HTTP Strict Transport Security). If Wget is\ncompiled without SSL support, none of these options are available.\n\n'--secure-protocol=PROTOCOL'\nChoose the secure protocol to be used. Legal values are 'auto',\n'SSLv2', 'SSLv3', 'TLSv1', 'TLSv11', 'TLSv12', 'TLSv13' and\n'PFS'. If 'auto' is used, the SSL library is given the liberty of\nchoosing the appropriate protocol automatically, which is achieved\nby sending a TLSv1 greeting. This is the default.\n\nSpecifying 'SSLv2', 'SSLv3', 'TLSv1', 'TLSv11', 'TLSv12' or\n'TLSv13' forces the use of the corresponding protocol. This is\nuseful when talking to old and buggy SSL server implementations\nthat make it hard for the underlying SSL library to choose the\ncorrect protocol version. Fortunately, such servers are quite\nrare.\n\nSpecifying 'PFS' enforces the use of the so-called Perfect Forward\nSecurity cipher suites. In short, PFS adds security by creating a\none-time key for each SSL connection. It has a bit more CPU impact\non client and server. We use known to be secure ciphers (e.g. no\nMD4) and the TLS protocol. This mode also explicitly excludes\nnon-PFS key exchange methods, such as RSA.\n\n'--https-only'\nWhen in recursive mode, only HTTPS links are followed.\n\n'--ciphers'\nSet the cipher list string. Typically this string sets the cipher\nsuites and other SSL/TLS options that the user wish should be used,\nin a set order of preference (GnuTLS calls it 'priority string').\nThis string will be fed verbatim to the SSL/TLS engine (OpenSSL or\nGnuTLS) and hence its format and syntax is dependent on that. Wget\nwill not process or manipulate it in any way. Refer to the OpenSSL\nor GnuTLS documentation for more information.\n\n'--no-check-certificate'\nDon't check the server certificate against the available\ncertificate authorities. Also don't require the URL host name to\nmatch the common name presented by the certificate.\n\nAs of Wget 1.10, the default is to verify the server's certificate\nagainst the recognized certificate authorities, breaking the SSL\nhandshake and aborting the download if the verification fails.\nAlthough this provides more secure downloads, it does break\ninteroperability with some sites that worked with previous Wget\nversions, particularly those using self-signed, expired, or\notherwise invalid certificates. This option forces an \"insecure\"\nmode of operation that turns the certificate verification errors\ninto warnings and allows you to proceed.\n\nIf you encounter \"certificate verification\" errors or ones saying\nthat \"common name doesn't match requested host name\", you can use\nthis option to bypass the verification and proceed with the\ndownload. Only use this option if you are otherwise convinced of\nthe site's authenticity, or if you really don't care about the\nvalidity of its certificate. It is almost always a bad idea not\nto check the certificates when transmitting confidential or\nimportant data. For self-signed/internal certificates, you should\ndownload the certificate and verify against that instead of forcing\nthis insecure mode. If you are really sure of not desiring any\ncertificate verification, you can specify -check-certificate=quiet\nto tell wget to not print any warning about invalid certificates,\nalbeit in most cases this is the wrong thing to do.\n\n'--certificate=FILE'\nUse the client certificate stored in FILE. This is needed for\nservers that are configured to require certificates from the\nclients that connect to them. Normally a certificate is not\nrequired and this switch is optional.\n\n'--certificate-type=TYPE'\nSpecify the type of the client certificate. Legal values are 'PEM'\n(assumed by default) and 'DER', also known as 'ASN1'.\n\n'--private-key=FILE'\nRead the private key from FILE. This allows you to provide the\nprivate key in a file separate from the certificate.\n\n'--private-key-type=TYPE'\nSpecify the type of the private key. Accepted values are 'PEM'\n(the default) and 'DER'.\n\n'--ca-certificate=FILE'\nUse FILE as the file with the bundle of certificate authorities\n(\"CA\") to verify the peers. The certificates must be in PEM\nformat.\n\nWithout this option Wget looks for CA certificates at the\nsystem-specified locations, chosen at OpenSSL installation time.\n\n'--ca-directory=DIRECTORY'\nSpecifies directory containing CA certificates in PEM format. Each\nfile contains one CA certificate, and the file name is based on a\nhash value derived from the certificate. This is achieved by\nprocessing a certificate directory with the 'crehash' utility\nsupplied with OpenSSL. Using '--ca-directory' is more efficient\nthan '--ca-certificate' when many certificates are installed\nbecause it allows Wget to fetch certificates on demand.\n\nWithout this option Wget looks for CA certificates at the\nsystem-specified locations, chosen at OpenSSL installation time.\n\n'--crl-file=FILE'\nSpecifies a CRL file in FILE. This is needed for certificates that\nhave been revocated by the CAs.\n\n'--pinnedpubkey=file/hashes'\nTells wget to use the specified public key file (or hashes) to\nverify the peer. This can be a path to a file which contains a\nsingle public key in PEM or DER format, or any number of base64\nencoded sha256 hashes preceded by \"sha256//\" and separated by \";\"\n\nWhen negotiating a TLS or SSL connection, the server sends a\ncertificate indicating its identity. A public key is extracted\nfrom this certificate and if it does not exactly match the public\nkey(s) provided to this option, wget will abort the connection\nbefore sending or receiving any data.\n\n'--random-file=FILE'\n[OpenSSL and LibreSSL only] Use FILE as the source of random data\nfor seeding the pseudo-random number generator on systems without\n'/dev/urandom'.\n\nOn such systems the SSL library needs an external source of\nrandomness to initialize. Randomness may be provided by EGD (see\n'--egd-file' below) or read from an external source specified by\nthe user. If this option is not specified, Wget looks for random\ndata in '$RANDFILE' or, if that is unset, in '$HOME/.rnd'.\n\nIf you're getting the \"Could not seed OpenSSL PRNG; disabling SSL.\"\nerror, you should provide random data using some of the methods\ndescribed above.\n\n'--egd-file=FILE'\n[OpenSSL only] Use FILE as the EGD socket. EGD stands for \"Entropy\nGathering Daemon\", a user-space program that collects data from\nvarious unpredictable system sources and makes it available to\nother programs that might need it. Encryption software, such as\nthe SSL library, needs sources of non-repeating randomness to seed\nthe random number generator used to produce cryptographically\nstrong keys.\n\nOpenSSL allows the user to specify his own source of entropy using\nthe 'RANDFILE' environment variable. If this variable is unset,\nor if the specified file does not produce enough randomness,\nOpenSSL will read random data from EGD socket specified using this\noption.\n\nIf this option is not specified (and the equivalent startup command\nis not used), EGD is never contacted. EGD is not needed on modern\nUnix systems that support '/dev/urandom'.\n\n'--no-hsts'\nWget supports HSTS (HTTP Strict Transport Security, RFC 6797) by\ndefault. Use '--no-hsts' to make Wget act as a non-HSTS-compliant\nUA. As a consequence, Wget would ignore all the\n'Strict-Transport-Security' headers, and would not enforce any\nexisting HSTS policy.\n\n'--hsts-file=FILE'\nBy default, Wget stores its HSTS database in '~/.wget-hsts'. You\ncan use '--hsts-file' to override this. Wget will use the supplied\nfile as the HSTS database. Such file must conform to the correct\nHSTS database format used by Wget. If Wget cannot parse the\nprovided file, the behaviour is unspecified.\n\nThe Wget's HSTS database is a plain text file. Each line contains\nan HSTS entry (ie. a site that has issued a\n'Strict-Transport-Security' header and that therefore has specified\na concrete HSTS policy to be applied). Lines starting with a dash\n('#') are ignored by Wget. Please note that in spite of this\nconvenient human-readability hand-hacking the HSTS database is\ngenerally not a good idea.\n\nAn HSTS entry line consists of several fields separated by one or\nmore whitespace:\n\n' SP [] SP SP SP\n'\n\nThe HOSTNAME and PORT fields indicate the hostname and port to\nwhich the given HSTS policy applies. The PORT field may be zero,\nand it will, in most of the cases. That means that the port number\nwill not be taken into account when deciding whether such HSTS\npolicy should be applied on a given request (only the hostname will\nbe evaluated). When PORT is different to zero, both the target\nhostname and the port will be evaluated and the HSTS policy will\nonly be applied if both of them match. This feature has been\nincluded for testing/development purposes only. The Wget testsuite\n(in 'testenv/') creates HSTS databases with explicit ports with the\npurpose of ensuring Wget's correct behaviour. Applying HSTS\npolicies to ports other than the default ones is discouraged by RFC\n6797 (see Appendix B \"Differences between HSTS Policy and\nSame-Origin Policy\"). Thus, this functionality should not be used\nin production environments and PORT will typically be zero. The\nlast three fields do what they are expected to. The field\nINCLUDESUBDOMAINS can either be '1' or '0' and it signals whether\nthe subdomains of the target domain should be part of the given\nHSTS policy as well. The CREATED and MAX-AGE fields hold the\ntimestamp values of when such entry was created (first seen by\nWget) and the HSTS-defined value 'max-age', which states how long\nshould that HSTS policy remain active, measured in seconds elapsed\nsince the timestamp stored in CREATED. Once that time has passed,\nthat HSTS policy will no longer be valid and will eventually be\nremoved from the database.\n\nIf you supply your own HSTS database via '--hsts-file', be aware\nthat Wget may modify the provided file if any change occurs between\nthe HSTS policies requested by the remote servers and those in the\nfile. When Wget exits, it effectively updates the HSTS database by\nrewriting the database file with the new entries.\n\nIf the supplied file does not exist, Wget will create one. This\nfile will contain the new HSTS entries. If no HSTS entries were\ngenerated (no 'Strict-Transport-Security' headers were sent by any\nof the servers) then no file will be created, not even an empty\none. This behaviour applies to the default database file\n('~/.wget-hsts') as well: it will not be created until some server\nenforces an HSTS policy.\n\nCare is taken not to override possible changes made by other Wget\nprocesses at the same time over the HSTS database. Before dumping\nthe updated HSTS entries on the file, Wget will re-read it and\nmerge the changes.\n\nUsing a custom HSTS database and/or modifying an existing one is\ndiscouraged. For more information about the potential security\nthreats arose from such practice, see section 14 \"Security\nConsiderations\" of RFC 6797, specially section 14.9 \"Creative\nManipulation of HSTS Policy Store\".\n\n'--warc-file=FILE'\nUse FILE as the destination WARC file.\n\n'--warc-header=STRING'\nUse STRING into as the warcinfo record.\n\n'--warc-max-size=SIZE'\nSet the maximum size of the WARC files to SIZE.\n\n'--warc-cdx'\nWrite CDX index files.\n\n'--warc-dedup=FILE'\nDo not store records listed in this CDX file.\n\n'--no-warc-compression'\nDo not compress WARC files with GZIP.\n\n'--no-warc-digests'\nDo not calculate SHA1 digests.\n\n'--no-warc-keep-log'\nDo not store the log file in a WARC record.\n\n'--warc-tempdir=DIR'\nSpecify the location for temporary files created by the WARC\nwriter.\n\nFile: wget.info, Node: FTP Options, Next: Recursive Retrieval Options, Prev: HTTPS (SSL/TLS) Options, Up: Invoking\n" }, { "name": "2.9 FTP Options", "content": "'--ftp-user=USER'\n'--ftp-password=PASSWORD'\nSpecify the username USER and password PASSWORD on an FTP server.\nWithout this, or the corresponding startup option, the password\ndefaults to '-wget@', normally used for anonymous FTP.\n\nAnother way to specify username and password is in the URL itself\n(*note URL Format::). Either method reveals your password to\nanyone who bothers to run 'ps'. To prevent the passwords from\nbeing seen, store them in '.wgetrc' or '.netrc', and make sure to\nprotect those files from other users with 'chmod'. If the\npasswords are really important, do not leave them lying in those\nfiles either--edit the files and delete them after Wget has started\nthe download.\n\n'--no-remove-listing'\nDon't remove the temporary '.listing' files generated by FTP\nretrievals. Normally, these files contain the raw directory\nlistings received from FTP servers. Not removing them can be\nuseful for debugging purposes, or when you want to be able to\neasily check on the contents of remote server directories (e.g. to\nverify that a mirror you're running is complete).\n\nNote that even though Wget writes to a known filename for this\nfile, this is not a security hole in the scenario of a user making\n'.listing' a symbolic link to '/etc/passwd' or something and asking\n'root' to run Wget in his or her directory. Depending on the\noptions used, either Wget will refuse to write to '.listing',\nmaking the globbing/recursion/time-stamping operation fail, or the\nsymbolic link will be deleted and replaced with the actual\n'.listing' file, or the listing will be written to a\n'.listing.NUMBER' file.\n\nEven though this situation isn't a problem, though, 'root' should\nnever run Wget in a non-trusted user's directory. A user could do\nsomething as simple as linking 'index.html' to '/etc/passwd' and\nasking 'root' to run Wget with '-N' or '-r' so the file will be\noverwritten.\n\n'--no-glob'\nTurn off FTP globbing. Globbing refers to the use of shell-like\nspecial characters (\"wildcards\"), like '*', '?', '[' and ']' to\nretrieve more than one file from the same directory at once, like:\n\nwget ftp://gnjilux.srk.fer.hr/*.msg\n\nBy default, globbing will be turned on if the URL contains a\nglobbing character. This option may be used to turn globbing on or\noff permanently.\n\nYou may have to quote the URL to protect it from being expanded by\nyour shell. Globbing makes Wget look for a directory listing,\nwhich is system-specific. This is why it currently works only with\nUnix FTP servers (and the ones emulating Unix 'ls' output).\n\n'--no-passive-ftp'\nDisable the use of the \"passive\" FTP transfer mode. Passive FTP\nmandates that the client connect to the server to establish the\ndata connection rather than the other way around.\n\nIf the machine is connected to the Internet directly, both passive\nand active FTP should work equally well. Behind most firewall and\nNAT configurations passive FTP has a better chance of working.\nHowever, in some rare firewall configurations, active FTP actually\nworks when passive FTP doesn't. If you suspect this to be the\ncase, use this option, or set 'passiveftp=off' in your init file.\n\n'--preserve-permissions'\nPreserve remote file permissions instead of permissions set by\numask.\n\n'--retr-symlinks'\nBy default, when retrieving FTP directories recursively and a\nsymbolic link is encountered, the symbolic link is traversed and\nthe pointed-to files are retrieved. Currently, Wget does not\ntraverse symbolic links to directories to download them\nrecursively, though this feature may be added in the future.\n\nWhen '--retr-symlinks=no' is specified, the linked-to file is not\ndownloaded. Instead, a matching symbolic link is created on the\nlocal filesystem. The pointed-to file will not be retrieved unless\nthis recursive retrieval would have encountered it separately and\ndownloaded it anyway. This option poses a security risk where a\nmalicious FTP Server may cause Wget to write to files outside of\nthe intended directories through a specially crafted .LISTING file.\n\nNote that when retrieving a file (not a directory) because it was\nspecified on the command-line, rather than because it was recursed\nto, this option has no effect. Symbolic links are always traversed\nin this case.\n" }, { "name": "2.10 FTPS Options", "content": "'--ftps-implicit'\nThis option tells Wget to use FTPS implicitly. Implicit FTPS\nconsists of initializing SSL/TLS from the very beginning of the\ncontrol connection. This option does not send an 'AUTH TLS'\ncommand: it assumes the server speaks FTPS and directly starts an\nSSL/TLS connection. If the attempt is successful, the session\ncontinues just like regular FTPS ('PBSZ' and 'PROT' are sent,\netc.). Implicit FTPS is no longer a requirement for FTPS\nimplementations, and thus many servers may not support it. If\n'--ftps-implicit' is passed and no explicit port number specified,\nthe default port for implicit FTPS, 990, will be used, instead of\nthe default port for the \"normal\" (explicit) FTPS which is the same\nas that of FTP, 21.\n\n'--no-ftps-resume-ssl'\nDo not resume the SSL/TLS session in the data channel. When\nstarting a data connection, Wget tries to resume the SSL/TLS\nsession previously started in the control connection. SSL/TLS\nsession resumption avoids performing an entirely new handshake by\nreusing the SSL/TLS parameters of a previous session. Typically,\nthe FTPS servers want it that way, so Wget does this by default.\nUnder rare circumstances however, one might want to start an\nentirely new SSL/TLS session in every data connection. This is\nwhat '--no-ftps-resume-ssl' is for.\n\n'--ftps-clear-data-connection'\nAll the data connections will be in plain text. Only the control\nconnection will be under SSL/TLS. Wget will send a 'PROT C' command\nto achieve this, which must be approved by the server.\n\n'--ftps-fallback-to-ftp'\nFall back to FTP if FTPS is not supported by the target server.\nFor security reasons, this option is not asserted by default. The\ndefault behaviour is to exit with an error. If a server does not\nsuccessfully reply to the initial 'AUTH TLS' command, or in the\ncase of implicit FTPS, if the initial SSL/TLS connection attempt is\nrejected, it is considered that such server does not support FTPS.\n\nFile: wget.info, Node: Recursive Retrieval Options, Next: Recursive Accept/Reject Options, Prev: FTP Options, Up: Invoking\n" }, { "name": "2.11 Recursive Retrieval Options", "content": "'-r'\n'--recursive'\nTurn on recursive retrieving. *Note Recursive Download::, for more\ndetails. The default maximum depth is 5.\n\n'-l DEPTH'\n'--level=DEPTH'\nSet the maximum number of subdirectories that Wget will recurse\ninto to DEPTH. In order to prevent one from accidentally\ndownloading very large websites when using recursion this is\nlimited to a depth of 5 by default, i.e., it will traverse at most\n5 directories deep starting from the provided URL. Set '-l 0' or\n'-l inf' for infinite recursion depth.\n\nwget -r -l 0 http://SITE/1.html\n\nIdeally, one would expect this to download just '1.html'. but\nunfortunately this is not the case, because '-l 0' is equivalent to\n'-l inf'--that is, infinite recursion. To download a single HTML\npage (or a handful of them), specify them all on the command line\nand leave away '-r' and '-l'. To download the essential items to\nview a single HTML page, see 'page requisites'.\n\n'--delete-after'\nThis option tells Wget to delete every single file it downloads,\nafter having done so. It is useful for pre-fetching popular\npages through a proxy, e.g.:\n\nwget -r -nd --delete-after http://whatever.com/~popular/page/\n\nThe '-r' option is to retrieve recursively, and '-nd' to not create\ndirectories.\n\nNote that '--delete-after' deletes files on the local machine. It\ndoes not issue the 'DELE' command to remote FTP sites, for\ninstance. Also note that when '--delete-after' is specified,\n'--convert-links' is ignored, so '.orig' files are simply not\ncreated in the first place.\n\n'-k'\n'--convert-links'\nAfter the download is complete, convert the links in the document\nto make them suitable for local viewing. This affects not only the\nvisible hyperlinks, but any part of the document that links to\nexternal content, such as embedded images, links to style sheets,\nhyperlinks to non-HTML content, etc.\n\nEach link will be changed in one of the two ways:\n\n* The links to files that have been downloaded by Wget will be\nchanged to refer to the file they point to as a relative link.\n\nExample: if the downloaded file '/foo/doc.html' links to\n'/bar/img.gif', also downloaded, then the link in 'doc.html'\nwill be modified to point to '../bar/img.gif'. This kind of\ntransformation works reliably for arbitrary combinations of\ndirectories.\n\n* The links to files that have not been downloaded by Wget will\nbe changed to include host name and absolute path of the\nlocation they point to.\n\nExample: if the downloaded file '/foo/doc.html' links to\n'/bar/img.gif' (or to '../bar/img.gif'), then the link in\n'doc.html' will be modified to point to\n'http://HOSTNAME/bar/img.gif'.\n\nBecause of this, local browsing works reliably: if a linked file\nwas downloaded, the link will refer to its local name; if it was\nnot downloaded, the link will refer to its full Internet address\nrather than presenting a broken link. The fact that the former\nlinks are converted to relative links ensures that you can move the\ndownloaded hierarchy to another directory.\n\nNote that only at the end of the download can Wget know which links\nhave been downloaded. Because of that, the work done by '-k' will\nbe performed at the end of all the downloads.\n\n'--convert-file-only'\nThis option converts only the filename part of the URLs, leaving\nthe rest of the URLs untouched. This filename part is sometimes\nreferred to as the \"basename\", although we avoid that term here in\norder not to cause confusion.\n\nIt works particularly well in conjunction with\n'--adjust-extension', although this coupling is not enforced. It\nproves useful to populate Internet caches with files downloaded\nfrom different hosts.\n\nExample: if some link points to '//foo.com/bar.cgi?xyz' with\n'--adjust-extension' asserted and its local destination is intended\nto be './foo.com/bar.cgi?xyz.css', then the link would be converted\nto '//foo.com/bar.cgi?xyz.css'. Note that only the filename part\nhas been modified. The rest of the URL has been left untouched,\nincluding the net path ('//') which would otherwise be processed by\nWget and converted to the effective scheme (ie. 'http://').\n\n'-K'\n'--backup-converted'\nWhen converting a file, back up the original version with a '.orig'\nsuffix. Affects the behavior of '-N' (*note HTTP Time-Stamping\nInternals::).\n\n'-m'\n'--mirror'\nTurn on options suitable for mirroring. This option turns on\nrecursion and time-stamping, sets infinite recursion depth and\nkeeps FTP directory listings. It is currently equivalent to '-r -N\n-l inf --no-remove-listing'.\n\n'-p'\n'--page-requisites'\nThis option causes Wget to download all the files that are\nnecessary to properly display a given HTML page. This includes\nsuch things as inlined images, sounds, and referenced stylesheets.\n\nOrdinarily, when downloading a single HTML page, any requisite\ndocuments that may be needed to display it properly are not\ndownloaded. Using '-r' together with '-l' can help, but since Wget\ndoes not ordinarily distinguish between external and inlined\ndocuments, one is generally left with \"leaf documents\" that are\nmissing their requisites.\n\nFor instance, say document '1.html' contains an '' tag\nreferencing '1.gif' and an '' tag pointing to external document\n'2.html'. Say that '2.html' is similar but that its image is\n'2.gif' and it links to '3.html'. Say this continues up to some\narbitrarily high number.\n\nIf one executes the command:\n\nwget -r -l 2 http://SITE/1.html\n\nthen '1.html', '1.gif', '2.html', '2.gif', and '3.html' will be\ndownloaded. As you can see, '3.html' is without its requisite\n'3.gif' because Wget is simply counting the number of hops (up to\n2) away from '1.html' in order to determine where to stop the\nrecursion. However, with this command:\n\nwget -r -l 2 -p http://SITE/1.html\n\nall the above files and '3.html''s requisite '3.gif' will be\ndownloaded. Similarly,\n\nwget -r -l 1 -p http://SITE/1.html\n\nwill cause '1.html', '1.gif', '2.html', and '2.gif' to be\ndownloaded. One might think that:\n\nwget -r -l 0 -p http://SITE/1.html\n\nwould download just '1.html' and '1.gif', but unfortunately this is\nnot the case, because '-l 0' is equivalent to '-l inf'--that is,\ninfinite recursion. To download a single HTML page (or a handful\nof them, all specified on the command-line or in a '-i' URL input\nfile) and its (or their) requisites, simply leave off '-r' and\n'-l':\n\nwget -p http://SITE/1.html\n\nNote that Wget will behave as if '-r' had been specified, but only\nthat single page and its requisites will be downloaded. Links from\nthat page to external documents will not be followed. Actually, to\ndownload a single page and all its requisites (even if they exist\non separate websites), and make sure the lot displays properly\nlocally, this author likes to use a few options in addition to\n'-p':\n\nwget -E -H -k -K -p http://SITE/DOCUMENT\n\nTo finish off this topic, it's worth knowing that Wget's idea of an\nexternal document link is any URL specified in an '' tag, an\n'' tag, or a '' tag other than ''.\n\n'--strict-comments'\nTurn on strict parsing of HTML comments. The default is to\nterminate comments at the first occurrence of '-->'.\n\nAccording to specifications, HTML comments are expressed as SGML\n\"declarations\". Declaration is special markup that begins with\n'', such as '', that may contain\ncomments between a pair of '--' delimiters. HTML comments are\n\"empty declarations\", SGML declarations without any non-comment\ntext. Therefore, '' is a valid comment, and so is\n'', but '' is not.\n\nOn the other hand, most HTML writers don't perceive comments as\nanything other than text delimited with '', which is\nnot quite the same. For example, something like ''\nworks as a valid comment as long as the number of dashes is a\nmultiple of four (!). If not, the comment technically lasts until\nthe next '--', which may be at the other end of the document.\nBecause of this, many popular browsers completely ignore the\nspecification and implement what users have come to expect:\ncomments delimited with ''.\n\nUntil version 1.9, Wget interpreted comments strictly, which\nresulted in missing links in many web pages that displayed fine in\nbrowsers, but had the misfortune of containing non-compliant\ncomments. Beginning with version 1.9, Wget has joined the ranks of\nclients that implements \"naive\" comments, terminating each comment\nat the first occurrence of '-->'.\n\nIf, for whatever reason, you want strict comment parsing, use this\noption to turn it on.\n\nFile: wget.info, Node: Recursive Accept/Reject Options, Next: Exit Status, Prev: Recursive Retrieval Options, Up: Invoking\n" }, { "name": "2.12 Recursive Accept/Reject Options", "content": "'-A ACCLIST --accept ACCLIST'\n'-R REJLIST --reject REJLIST'\nSpecify comma-separated lists of file name suffixes or patterns to\naccept or reject (*note Types of Files::). Note that if any of the\nwildcard characters, '*', '?', '[' or ']', appear in an element of\nACCLIST or REJLIST, it will be treated as a pattern, rather than a\nsuffix. In this case, you have to enclose the pattern into quotes\nto prevent your shell from expanding it, like in '-A \"*.mp3\"' or\n'-A '*.mp3''.\n\n'--accept-regex URLREGEX'\n'--reject-regex URLREGEX'\nSpecify a regular expression to accept or reject the complete URL.\n\n'--regex-type REGEXTYPE'\nSpecify the regular expression type. Possible types are 'posix' or\n'pcre'. Note that to be able to use 'pcre' type, wget has to be\ncompiled with libpcre support.\n\n'-D DOMAIN-LIST'\n'--domains=DOMAIN-LIST'\nSet domains to be followed. DOMAIN-LIST is a comma-separated list\nof domains. Note that it does not turn on '-H'.\n\n'--exclude-domains DOMAIN-LIST'\nSpecify the domains that are not to be followed (*note Spanning\nHosts::).\n\n'--follow-ftp'\nFollow FTP links from HTML documents. Without this option, Wget\nwill ignore all the FTP links.\n\n'--follow-tags=LIST'\nWget has an internal table of HTML tag / attribute pairs that it\nconsiders when looking for linked documents during a recursive\nretrieval. If a user wants only a subset of those tags to be\nconsidered, however, he or she should be specify such tags in a\ncomma-separated LIST with this option.\n\n'--ignore-tags=LIST'\nThis is the opposite of the '--follow-tags' option. To skip\ncertain HTML tags when recursively looking for documents to\ndownload, specify them in a comma-separated LIST.\n\nIn the past, this option was the best bet for downloading a single\npage and its requisites, using a command-line like:\n\nwget --ignore-tags=a,area -H -k -K -r http://SITE/DOCUMENT\n\nHowever, the author of this option came across a page with tags\nlike '' and came to the realization that\nspecifying tags to ignore was not enough. One can't just tell Wget\nto ignore '', because then stylesheets will not be\ndownloaded. Now the best bet for downloading a single page and its\nrequisites is the dedicated '--page-requisites' option.\n\n'--ignore-case'\nIgnore case when matching files and directories. This influences\nthe behavior of -R, -A, -I, and -X options, as well as globbing\nimplemented when downloading from FTP sites. For example, with\nthis option, '-A \"*.txt\"' will match 'file1.txt', but also\n'file2.TXT', 'file3.TxT', and so on. The quotes in the example are\nto prevent the shell from expanding the pattern.\n\n'-H'\n'--span-hosts'\nEnable spanning across hosts when doing recursive retrieving (*note\nSpanning Hosts::).\n\n'-L'\n'--relative'\nFollow relative links only. Useful for retrieving a specific home\npage without any distractions, not even those from the same hosts\n(*note Relative Links::).\n\n'-I LIST'\n'--include-directories=LIST'\nSpecify a comma-separated list of directories you wish to follow\nwhen downloading (*note Directory-Based Limits::). Elements of\nLIST may contain wildcards.\n\n'-X LIST'\n'--exclude-directories=LIST'\nSpecify a comma-separated list of directories you wish to exclude\nfrom download (*note Directory-Based Limits::). Elements of LIST\nmay contain wildcards.\n\n'-np'\n'--no-parent'\nDo not ever ascend to the parent directory when retrieving\nrecursively. This is a useful option, since it guarantees that\nonly the files below a certain hierarchy will be downloaded.\n*Note Directory-Based Limits::, for more details.\n\nFile: wget.info, Node: Exit Status, Prev: Recursive Accept/Reject Options, Up: Invoking\n" }, { "name": "2.13 Exit Status", "content": "Wget may return one of several error codes if it encounters problems.\n\n0\nNo problems occurred.\n\n1\nGeneric error code.\n\n2\nParse error--for instance, when parsing command-line options, the\n'.wgetrc' or '.netrc'...\n\n3\nFile I/O error.\n\n4\nNetwork failure.\n\n5\nSSL verification failure.\n\n6\nUsername/password authentication failure.\n\n7\nProtocol errors.\n\n8\nServer issued an error response.\n\nWith the exceptions of 0 and 1, the lower-numbered exit codes take\nprecedence over higher-numbered ones, when multiple types of errors are\nencountered.\n\nIn versions of Wget prior to 1.12, Wget's exit status tended to be\nunhelpful and inconsistent. Recursive downloads would virtually always\nreturn 0 (success), regardless of any issues encountered, and\nnon-recursive fetches only returned the status corresponding to the most\nrecently-attempted download.\n\nFile: wget.info, Node: Recursive Download, Next: Following Links, Prev: Invoking, Up: Top\n" } ] }, "3 Recursive Download": { "content": "GNU Wget is capable of traversing parts of the Web (or a single HTTP or\nFTP server), following links and directory structure. We refer to this\nas to \"recursive retrieval\", or \"recursion\".\n\nWith HTTP URLs, Wget retrieves and parses the HTML or CSS from the\ngiven URL, retrieving the files the document refers to, through markup\nlike 'href' or 'src', or CSS URI values specified using the 'url()'\nfunctional notation. If the freshly downloaded file is also of type\n'text/html', 'application/xhtml+xml', or 'text/css', it will be parsed\nand followed further.\n\nRecursive retrieval of HTTP and HTML/CSS content is \"breadth-first\".\nThis means that Wget first downloads the requested document, then the\ndocuments linked from that document, then the documents linked by them,\nand so on. In other words, Wget first downloads the documents at depth\n1, then those at depth 2, and so on until the specified maximum depth.\n\nThe maximum \"depth\" to which the retrieval may descend is specified\nwith the '-l' option. The default maximum depth is five layers.\n\nWhen retrieving an FTP URL recursively, Wget will retrieve all the\ndata from the given directory tree (including the subdirectories up to\nthe specified depth) on the remote server, creating its mirror image\nlocally. FTP retrieval is also limited by the 'depth' parameter.\nUnlike HTTP recursion, FTP recursion is performed depth-first.\n\nBy default, Wget will create a local directory tree, corresponding to\nthe one found on the remote server.\n\nRecursive retrieving can find a number of applications, the most\nimportant of which is mirroring. It is also useful for WWW\npresentations, and any other opportunities where slow network\nconnections should be bypassed by storing the files locally.\n\nYou should be warned that recursive downloads can overload the remote\nservers. Because of that, many administrators frown upon them and may\nban access from your site if they detect very fast downloads of big\namounts of content. When downloading from Internet servers, consider\nusing the '-w' option to introduce a delay between accesses to the\nserver. The download will take a while longer, but the server\nadministrator will not be alarmed by your rudeness.\n\nOf course, recursive download may cause problems on your machine. If\nleft to run unchecked, it can easily fill up the disk. If downloading\nfrom local network, it can also take bandwidth on the system, as well as\nconsume memory and CPU.\n\nTry to specify the criteria that match the kind of download you are\ntrying to achieve. If you want to download only one page, use\n'--page-requisites' without any additional recursion. If you want to\ndownload things under one directory, use '-np' to avoid downloading\nthings from other directories. If you want to download all the files\nfrom one directory, use '-l 1' to make sure the recursion depth never\nexceeds one. *Note Following Links::, for more information about this.\n\nRecursive retrieval should be used with care. Don't say you were not\nwarned.\n\nFile: wget.info, Node: Following Links, Next: Time-Stamping, Prev: Recursive Download, Up: Top\n", "subsections": [] }, "4 Following Links": { "content": "When retrieving recursively, one does not wish to retrieve loads of\nunnecessary data. Most of the time the users bear in mind exactly what\nthey want to download, and want Wget to follow only specific links.\n\nFor example, if you wish to download the music archive from\n'fly.srk.fer.hr', you will not want to download all the home pages that\nhappen to be referenced by an obscure part of the archive.\n\nWget possesses several mechanisms that allows you to fine-tune which\nlinks it will follow.\n\n* Menu:\n\n* Spanning Hosts:: (Un)limiting retrieval based on host name.\n* Types of Files:: Getting only certain files.\n* Directory-Based Limits:: Getting only certain directories.\n* Relative Links:: Follow relative links only.\n* FTP Links:: Following FTP links.\n\nFile: wget.info, Node: Spanning Hosts, Next: Types of Files, Prev: Following Links, Up: Following Links\n", "subsections": [ { "name": "4.1 Spanning Hosts", "content": "Wget's recursive retrieval normally refuses to visit hosts different\nthan the one you specified on the command line. This is a reasonable\ndefault; without it, every retrieval would have the potential to turn\nyour Wget into a small version of google.\n\nHowever, visiting different hosts, or \"host spanning,\" is sometimes a\nuseful option. Maybe the images are served from a different server.\nMaybe you're mirroring a site that consists of pages interlinked between\nthree servers. Maybe the server has two equivalent names, and the HTML\npages refer to both interchangeably.\n\nSpan to any host--'-H'\n\nThe '-H' option turns on host spanning, thus allowing Wget's\nrecursive run to visit any host referenced by a link. Unless\nsufficient recursion-limiting criteria are applied depth, these\nforeign hosts will typically link to yet more hosts, and so on\nuntil Wget ends up sucking up much more data than you have\nintended.\n\nLimit spanning to certain domains--'-D'\n\nThe '-D' option allows you to specify the domains that will be\nfollowed, thus limiting the recursion only to the hosts that belong\nto these domains. Obviously, this makes sense only in conjunction\nwith '-H'. A typical example would be downloading the contents of\n'www.example.com', but allowing downloads from\n'images.example.com', etc.:\n\nwget -rH -Dexample.com http://www.example.com/\n\nYou can specify more than one address by separating them with a\ncomma, e.g. '-Ddomain1.com,domain2.com'.\n\nKeep download off certain domains--'--exclude-domains'\n\nIf there are domains you want to exclude specifically, you can do\nit with '--exclude-domains', which accepts the same type of\narguments of '-D', but will exclude all the listed domains. For\nexample, if you want to download all the hosts from 'foo.edu'\ndomain, with the exception of 'sunsite.foo.edu', you can do it like\nthis:\n\nwget -rH -Dfoo.edu --exclude-domains sunsite.foo.edu \\\nhttp://www.foo.edu/\n\nFile: wget.info, Node: Types of Files, Next: Directory-Based Limits, Prev: Spanning Hosts, Up: Following Links\n" }, { "name": "4.2 Types of Files", "content": "When downloading material from the web, you will often want to restrict\nthe retrieval to only certain file types. For example, if you are\ninterested in downloading GIFs, you will not be overjoyed to get loads\nof PostScript documents, and vice versa.\n\nWget offers two options to deal with this problem. Each option\ndescription lists a short name, a long name, and the equivalent command\nin '.wgetrc'.\n\n'-A ACCLIST'\n'--accept ACCLIST'\n'accept = ACCLIST'\n'--accept-regex URLREGEX'\n'accept-regex = URLREGEX'\nThe argument to '--accept' option is a list of file suffixes or\npatterns that Wget will download during recursive retrieval. A\nsuffix is the ending part of a file, and consists of \"normal\"\nletters, e.g. 'gif' or '.jpg'. A matching pattern contains\nshell-like wildcards, e.g. 'books*' or 'zelazny*196[0-9]*'.\n\nSo, specifying 'wget -A gif,jpg' will make Wget download only the\nfiles ending with 'gif' or 'jpg', i.e. GIFs and JPEGs. On the\nother hand, 'wget -A \"zelazny*196[0-9]*\"' will download only files\nbeginning with 'zelazny' and containing numbers from 1960 to 1969\nanywhere within. Look up the manual of your shell for a\ndescription of how pattern matching works.\n\nOf course, any number of suffixes and patterns can be combined into\na comma-separated list, and given as an argument to '-A'.\n\nThe argument to '--accept-regex' option is a regular expression\nwhich is matched against the complete URL.\n\n'-R REJLIST'\n'--reject REJLIST'\n'reject = REJLIST'\n'--reject-regex URLREGEX'\n'reject-regex = URLREGEX'\nThe '--reject' option works the same way as '--accept', only its\nlogic is the reverse; Wget will download all files except the\nones matching the suffixes (or patterns) in the list.\n\nSo, if you want to download a whole page except for the cumbersome\nMPEGs and .AU files, you can use 'wget -R mpg,mpeg,au'.\nAnalogously, to download all files except the ones beginning with\n'bjork', use 'wget -R \"bjork*\"'. The quotes are to prevent\nexpansion by the shell.\n\nThe argument to '--accept-regex' option is a regular expression which\nis matched against the complete URL.\n\nThe '-A' and '-R' options may be combined to achieve even better\nfine-tuning of which files to retrieve. E.g. 'wget -A \"*zelazny*\" -R\n.ps' will download all the files having 'zelazny' as a part of their\nname, but not the PostScript files.\n\nNote that these two options do not affect the downloading of HTML\nfiles (as determined by a '.htm' or '.html' filename prefix). This\nbehavior may not be desirable for all users, and may be changed for\nfuture versions of Wget.\n\nNote, too, that query strings (strings at the end of a URL beginning\nwith a question mark ('?') are not included as part of the filename for\naccept/reject rules, even though these will actually contribute to the\nname chosen for the local file. It is expected that a future version of\nWget will provide an option to allow matching against query strings.\n\nFinally, it's worth noting that the accept/reject lists are matched\ntwice against downloaded files: once against the URL's filename\nportion, to determine if the file should be downloaded in the first\nplace; then, after it has been accepted and successfully downloaded, the\nlocal file's name is also checked against the accept/reject lists to see\nif it should be removed. The rationale was that, since '.htm' and\n'.html' files are always downloaded regardless of accept/reject rules,\nthey should be removed after being downloaded and scanned for links,\nif they did match the accept/reject lists. However, this can lead to\nunexpected results, since the local filenames can differ from the\noriginal URL filenames in the following ways, all of which can change\nwhether an accept/reject rule matches:\n\n* If the local file already exists and '--no-directories' was\nspecified, a numeric suffix will be appended to the original name.\n* If '--adjust-extension' was specified, the local filename might\nhave '.html' appended to it. If Wget is invoked with '-E -A.php',\na filename such as 'index.php' will match be accepted, but upon\ndownload will be named 'index.php.html', which no longer matches,\nand so the file will be deleted.\n* Query strings do not contribute to URL matching, but are included\nin local filenames, and so do contribute to filename matching.\n\nThis behavior, too, is considered less-than-desirable, and may change in\na future version of Wget.\n\nFile: wget.info, Node: Directory-Based Limits, Next: Relative Links, Prev: Types of Files, Up: Following Links\n" }, { "name": "4.3 Directory-Based Limits", "content": "Regardless of other link-following facilities, it is often useful to\nplace the restriction of what files to retrieve based on the directories\nthose files are placed in. There can be many reasons for this--the home\npages may be organized in a reasonable directory structure; or some\ndirectories may contain useless information, e.g. '/cgi-bin' or '/dev'\ndirectories.\n\nWget offers three different options to deal with this requirement.\nEach option description lists a short name, a long name, and the\nequivalent command in '.wgetrc'.\n\n'-I LIST'\n'--include LIST'\n'includedirectories = LIST'\n'-I' option accepts a comma-separated list of directories included\nin the retrieval. Any other directories will simply be ignored.\nThe directories are absolute paths.\n\nSo, if you wish to download from 'http://host/people/bozo/'\nfollowing only links to bozo's colleagues in the '/people'\ndirectory and the bogus scripts in '/cgi-bin', you can specify:\n\nwget -I /people,/cgi-bin http://host/people/bozo/\n\n'-X LIST'\n'--exclude LIST'\n'excludedirectories = LIST'\n'-X' option is exactly the reverse of '-I'--this is a list of\ndirectories excluded from the download. E.g. if you do not want\nWget to download things from '/cgi-bin' directory, specify '-X\n/cgi-bin' on the command line.\n\nThe same as with '-A'/'-R', these two options can be combined to\nget a better fine-tuning of downloading subdirectories. E.g. if\nyou want to load all the files from '/pub' hierarchy except for\n'/pub/worthless', specify '-I/pub -X/pub/worthless'.\n\n'-np'\n'--no-parent'\n'noparent = on'\nThe simplest, and often very useful way of limiting directories is\ndisallowing retrieval of the links that refer to the hierarchy\n\"above\" than the beginning directory, i.e. disallowing ascent to\nthe parent directory/directories.\n\nThe '--no-parent' option (short '-np') is useful in this case.\nUsing it guarantees that you will never leave the existing\nhierarchy. Supposing you issue Wget with:\n\nwget -r --no-parent http://somehost/~luzer/my-archive/\n\nYou may rest assured that none of the references to\n'/~his-girls-homepage/' or '/~luzer/all-my-mpegs/' will be\nfollowed. Only the archive you are interested in will be\ndownloaded. Essentially, '--no-parent' is similar to\n'-I/~luzer/my-archive', only it handles redirections in a more\nintelligent fashion.\n\n*Note* that, for HTTP (and HTTPS), the trailing slash is very\nimportant to '--no-parent'. HTTP has no concept of a\n\"directory\"--Wget relies on you to indicate what's a directory and\nwhat isn't. In 'http://foo/bar/', Wget will consider 'bar' to be a\ndirectory, while in 'http://foo/bar' (no trailing slash), 'bar'\nwill be considered a filename (so '--no-parent' would be\nmeaningless, as its parent is '/').\n\nFile: wget.info, Node: Relative Links, Next: FTP Links, Prev: Directory-Based Limits, Up: Following Links\n" }, { "name": "4.4 Relative Links", "content": "When '-L' is turned on, only the relative links are ever followed.\nRelative links are here defined those that do not refer to the web\nserver root. For example, these links are relative:\n\n\n\n\n\nThese links are not relative:\n\n\n\n\n\nUsing this option guarantees that recursive retrieval will not span\nhosts, even without '-H'. In simple cases it also allows downloads to\n\"just work\" without having to convert links.\n\nThis option is probably not very useful and might be removed in a\nfuture release.\n\nFile: wget.info, Node: FTP Links, Prev: Relative Links, Up: Following Links\n" }, { "name": "4.5 Following FTP Links", "content": "The rules for FTP are somewhat specific, as it is necessary for them to\nbe. FTP links in HTML documents are often included for purposes of\nreference, and it is often inconvenient to download them by default.\n\nTo have FTP links followed from HTML documents, you need to specify\nthe '--follow-ftp' option. Having done that, FTP links will span hosts\nregardless of '-H' setting. This is logical, as FTP links rarely point\nto the same host where the HTTP server resides. For similar reasons,\nthe '-L' options has no effect on such downloads. On the other hand,\ndomain acceptance ('-D') and suffix rules ('-A' and '-R') apply\nnormally.\n\nAlso note that followed links to FTP directories will not be\nretrieved recursively further.\n\nFile: wget.info, Node: Time-Stamping, Next: Startup File, Prev: Following Links, Up: Top\n" } ] }, "5 Time-Stamping": { "content": "One of the most important aspects of mirroring information from the\nInternet is updating your archives.\n\nDownloading the whole archive again and again, just to replace a few\nchanged files is expensive, both in terms of wasted bandwidth and money,\nand the time to do the update. This is why all the mirroring tools\noffer the option of incremental updating.\n\nSuch an updating mechanism means that the remote server is scanned in\nsearch of \"new\" files. Only those new files will be downloaded in the\nplace of the old ones.\n\nA file is considered new if one of these two conditions are met:\n\n1. A file of that name does not already exist locally.\n\n2. A file of that name does exist, but the remote file was modified\nmore recently than the local file.\n\nTo implement this, the program needs to be aware of the time of last\nmodification of both local and remote files. We call this information\nthe \"time-stamp\" of a file.\n\nThe time-stamping in GNU Wget is turned on using '--timestamping'\n('-N') option, or through 'timestamping = on' directive in '.wgetrc'.\nWith this option, for each file it intends to download, Wget will check\nwhether a local file of the same name exists. If it does, and the\nremote file is not newer, Wget will not download it.\n\nIf the local file does not exist, or the sizes of the files do not\nmatch, Wget will download the remote file no matter what the time-stamps\nsay.\n\n* Menu:\n\n* Time-Stamping Usage::\n* HTTP Time-Stamping Internals::\n* FTP Time-Stamping Internals::\n\nFile: wget.info, Node: Time-Stamping Usage, Next: HTTP Time-Stamping Internals, Prev: Time-Stamping, Up: Time-Stamping\n", "subsections": [ { "name": "5.1 Time-Stamping Usage", "content": "The usage of time-stamping is simple. Say you would like to download a\nfile so that it keeps its date of modification.\n\nwget -S http://www.gnu.ai.mit.edu/\n\nA simple 'ls -l' shows that the time stamp on the local file equals\nthe state of the 'Last-Modified' header, as returned by the server. As\nyou can see, the time-stamping info is preserved locally, even without\n'-N' (at least for HTTP).\n\nSeveral days later, you would like Wget to check if the remote file\nhas changed, and download it if it has.\n\nwget -N http://www.gnu.ai.mit.edu/\n\nWget will ask the server for the last-modified date. If the local\nfile has the same timestamp as the server, or a newer one, the remote\nfile will not be re-fetched. However, if the remote file is more\nrecent, Wget will proceed to fetch it.\n\nThe same goes for FTP. For example:\n\nwget \"ftp://ftp.ifi.uio.no/pub/emacs/gnus/*\"\n\n(The quotes around that URL are to prevent the shell from trying to\ninterpret the '*'.)\n\nAfter download, a local directory listing will show that the\ntimestamps match those on the remote server. Reissuing the command with\n'-N' will make Wget re-fetch only the files that have been modified\nsince the last download.\n\nIf you wished to mirror the GNU archive every week, you would use a\ncommand like the following, weekly:\n\nwget --timestamping -r ftp://ftp.gnu.org/pub/gnu/\n\nNote that time-stamping will only work for files for which the server\ngives a timestamp. For HTTP, this depends on getting a 'Last-Modified'\nheader. For FTP, this depends on getting a directory listing with dates\nin a format that Wget can parse (*note FTP Time-Stamping Internals::).\n\nFile: wget.info, Node: HTTP Time-Stamping Internals, Next: FTP Time-Stamping Internals, Prev: Time-Stamping Usage, Up: Time-Stamping\n" }, { "name": "5.2 HTTP Time-Stamping Internals", "content": "Time-stamping in HTTP is implemented by checking of the 'Last-Modified'\nheader. If you wish to retrieve the file 'foo.html' through HTTP, Wget\nwill check whether 'foo.html' exists locally. If it doesn't, 'foo.html'\nwill be retrieved unconditionally.\n\nIf the file does exist locally, Wget will first check its local\ntime-stamp (similar to the way 'ls -l' checks it), and then send a\n'HEAD' request to the remote server, demanding the information on the\nremote file.\n\nThe 'Last-Modified' header is examined to find which file was\nmodified more recently (which makes it \"newer\"). If the remote file is\nnewer, it will be downloaded; if it is older, Wget will give up.(1)\n\nWhen '--backup-converted' ('-K') is specified in conjunction with\n'-N', server file 'X' is compared to local file 'X.orig', if extant,\nrather than being compared to local file 'X', which will always differ\nif it's been converted by '--convert-links' ('-k').\n\nArguably, HTTP time-stamping should be implemented using the\n'If-Modified-Since' request.\n\n---------- Footnotes ----------\n\n(1) As an additional check, Wget will look at the 'Content-Length'\nheader, and compare the sizes; if they are not the same, the remote file\nwill be downloaded no matter what the time-stamp says.\n\nFile: wget.info, Node: FTP Time-Stamping Internals, Prev: HTTP Time-Stamping Internals, Up: Time-Stamping\n" }, { "name": "5.3 FTP Time-Stamping Internals", "content": "In theory, FTP time-stamping works much the same as HTTP, only FTP has\nno headers--time-stamps must be ferreted out of directory listings.\n\nIf an FTP download is recursive or uses globbing, Wget will use the\nFTP 'LIST' command to get a file listing for the directory containing\nthe desired file(s). It will try to analyze the listing, treating it\nlike Unix 'ls -l' output, extracting the time-stamps. The rest is\nexactly the same as for HTTP. Note that when retrieving individual\nfiles from an FTP server without using globbing or recursion, listing\nfiles will not be downloaded (and thus files will not be time-stamped)\nunless '-N' is specified.\n\nAssumption that every directory listing is a Unix-style listing may\nsound extremely constraining, but in practice it is not, as many\nnon-Unix FTP servers use the Unixoid listing format because most (all?)\nof the clients understand it. Bear in mind that RFC959 defines no\nstandard way to get a file list, let alone the time-stamps. We can only\nhope that a future standard will define this.\n\nAnother non-standard solution includes the use of 'MDTM' command that\nis supported by some FTP servers (including the popular 'wu-ftpd'),\nwhich returns the exact time of the specified file. Wget may support\nthis command in the future.\n\nFile: wget.info, Node: Startup File, Next: Examples, Prev: Time-Stamping, Up: Top\n" } ] }, "6 Startup File": { "content": "Once you know how to change default settings of Wget through command\nline arguments, you may wish to make some of those settings permanent.\nYou can do that in a convenient way by creating the Wget startup\nfile--'.wgetrc'.\n\nBesides '.wgetrc' is the \"main\" initialization file, it is convenient\nto have a special facility for storing passwords. Thus Wget reads and\ninterprets the contents of '$HOME/.netrc', if it finds it. You can find\n'.netrc' format in your system manuals.\n\nWget reads '.wgetrc' upon startup, recognizing a limited set of\ncommands.\n\n* Menu:\n\n* Wgetrc Location:: Location of various wgetrc files.\n* Wgetrc Syntax:: Syntax of wgetrc.\n* Wgetrc Commands:: List of available commands.\n* Sample Wgetrc:: A wgetrc example.\n\nFile: wget.info, Node: Wgetrc Location, Next: Wgetrc Syntax, Prev: Startup File, Up: Startup File\n", "subsections": [ { "name": "6.1 Wgetrc Location", "content": "When initializing, Wget will look for a \"global\" startup file,\n'/etc/wgetrc' by default and read commands from there, if it exists.\n\nThen it will look for the user's file. If the environmental variable\n'WGETRC' is set, Wget will try to load that file. Failing that, no\nfurther attempts will be made.\n\nIf 'WGETRC' is not set, Wget will try to load '$HOME/.wgetrc'.\n\nThe fact that user's settings are loaded after the system-wide ones\nmeans that in case of collision user's wgetrc overrides the\nsystem-wide wgetrc (in '//etc/wgetrc' by default). Fascist admins,\naway!\n\nFile: wget.info, Node: Wgetrc Syntax, Next: Wgetrc Commands, Prev: Wgetrc Location, Up: Startup File\n" }, { "name": "6.2 Wgetrc Syntax", "content": "The syntax of a wgetrc command is simple:\n\nvariable = value\n\nThe \"variable\" will also be called \"command\". Valid \"values\" are\ndifferent for different commands.\n\nThe commands are case-, underscore- and minus-insensitive. Thus\n'DIrPrefiX', 'DIr-PrefiX' and 'dirprefix' are the same. Empty lines,\nlines beginning with '#' and lines containing white-space only are\ndiscarded.\n\nCommands that expect a comma-separated list will clear the list on an\nempty command. So, if you wish to reset the rejection list specified in\nglobal 'wgetrc', you can do it with:\n\nreject =\n\nFile: wget.info, Node: Wgetrc Commands, Next: Sample Wgetrc, Prev: Wgetrc Syntax, Up: Startup File\n" }, { "name": "6.3 Wgetrc Commands", "content": "The complete set of commands is listed below. Legal values are listed\nafter the '='. Simple Boolean values can be set or unset using 'on' and\n'off' or '1' and '0'.\n\nSome commands take pseudo-arbitrary values. ADDRESS values can be\nhostnames or dotted-quad IP addresses. N can be any positive integer,\nor 'inf' for infinity, where appropriate. STRING values can be any\nnon-empty string.\n\nMost of these commands have direct command-line equivalents. Also,\nany wgetrc command can be specified on the command line using the\n'--execute' switch (*note Basic Startup Options::.)\n\naccept/reject = STRING\nSame as '-A'/'-R' (*note Types of Files::).\n\naddhostdir = on/off\nEnable/disable host-prefixed file names. '-nH' disables it.\n\naskpassword = on/off\nPrompt for a password for each connection established. Cannot be\nspecified when '--password' is being used, because they are\nmutually exclusive. Equivalent to '--ask-password'.\n\nauthnochallenge = on/off\nIf this option is given, Wget will send Basic HTTP authentication\ninformation (plaintext username and password) for all requests.\nSee '--auth-no-challenge'.\n\nbackground = on/off\nEnable/disable going to background--the same as '-b' (which enables\nit).\n\nbackupconverted = on/off\nEnable/disable saving pre-converted files with the suffix\n'.orig'--the same as '-K' (which enables it).\n\nbackups = NUMBER\nUse up to NUMBER backups for a file. Backups are rotated by adding\nan incremental counter that starts at '1'. The default is '0'.\n\nbase = STRING\nConsider relative URLs in input files (specified via the 'input'\ncommand or the '--input-file'/'-i' option, together with\n'forcehtml' or '--force-html') as being relative to STRING--the\nsame as '--base=STRING'.\n\nbindaddress = ADDRESS\nBind to ADDRESS, like the '--bind-address=ADDRESS'.\n\ncacertificate = FILE\nSet the certificate authority bundle file to FILE. The same as\n'--ca-certificate=FILE'.\n\ncadirectory = DIRECTORY\nSet the directory used for certificate authorities. The same as\n'--ca-directory=DIRECTORY'.\n\ncache = on/off\nWhen set to off, disallow server-caching. See the '--no-cache'\noption.\n\ncertificate = FILE\nSet the client certificate file name to FILE. The same as\n'--certificate=FILE'.\n\ncertificatetype = STRING\nSpecify the type of the client certificate, legal values being\n'PEM' (the default) and 'DER' (aka ASN1). The same as\n'--certificate-type=STRING'.\n\ncheckcertificate = on/off\nIf this is set to off, the server certificate is not checked\nagainst the specified client authorities. The default is \"on\".\nThe same as '--check-certificate'.\n\nconnecttimeout = N\nSet the connect timeout--the same as '--connect-timeout'.\n\ncontentdisposition = on/off\nTurn on recognition of the (non-standard) 'Content-Disposition'\nHTTP header--if set to 'on', the same as '--content-disposition'.\n\ntrustservernames = on/off\nIf set to on, construct the local file name from redirection URLs\nrather than original URLs.\n\ncontinue = on/off\nIf set to on, force continuation of preexistent partially retrieved\nfiles. See '-c' before setting it.\n\nconvertlinks = on/off\nConvert non-relative links locally. The same as '-k'.\n\ncookies = on/off\nWhen set to off, disallow cookies. See the '--cookies' option.\n\ncutdirs = N\nIgnore N remote directory components. Equivalent to\n'--cut-dirs=N'.\n\ndebug = on/off\nDebug mode, same as '-d'.\n\ndefaultpage = STRING\nDefault page name--the same as '--default-page=STRING'.\n\ndeleteafter = on/off\nDelete after download--the same as '--delete-after'.\n\ndirprefix = STRING\nTop of directory tree--the same as '-P STRING'.\n\ndirstruct = on/off\nTurning dirstruct on or off--the same as '-x' or '-nd',\nrespectively.\n\ndnscache = on/off\nTurn DNS caching on/off. Since DNS caching is on by default, this\noption is normally used to turn it off and is equivalent to\n'--no-dns-cache'.\n\ndnstimeout = N\nSet the DNS timeout--the same as '--dns-timeout'.\n\ndomains = STRING\nSame as '-D' (*note Spanning Hosts::).\n\ndotbytes = N\nSpecify the number of bytes \"contained\" in a dot, as seen\nthroughout the retrieval (1024 by default). You can postfix the\nvalue with 'k' or 'm', representing kilobytes and megabytes,\nrespectively. With dot settings you can tailor the dot retrieval\nto suit your needs, or you can use the predefined \"styles\" (*note\nDownload Options::).\n\ndotspacing = N\nSpecify the number of dots in a single cluster (10 by default).\n\ndotsinline = N\nSpecify the number of dots that will be printed in each line\nthroughout the retrieval (50 by default).\n\negdfile = FILE\nUse STRING as the EGD socket file name. The same as\n'--egd-file=FILE'.\n\nexcludedirectories = STRING\nSpecify a comma-separated list of directories you wish to exclude\nfrom download--the same as '-X STRING' (*note Directory-Based\nLimits::).\n\nexcludedomains = STRING\nSame as '--exclude-domains=STRING' (*note Spanning Hosts::).\n\nfollowftp = on/off\nFollow FTP links from HTML documents--the same as '--follow-ftp'.\n\nfollowtags = STRING\nOnly follow certain HTML tags when doing a recursive retrieval,\njust like '--follow-tags=STRING'.\n\nforcehtml = on/off\nIf set to on, force the input filename to be regarded as an HTML\ndocument--the same as '-F'.\n\nftppassword = STRING\nSet your FTP password to STRING. Without this setting, the\npassword defaults to '-wget@', which is a useful default for\nanonymous FTP access.\n\nThis command used to be named 'passwd' prior to Wget 1.10.\n\nftpproxy = STRING\nUse STRING as FTP proxy, instead of the one specified in\nenvironment.\n\nftpuser = STRING\nSet FTP user to STRING.\n\nThis command used to be named 'login' prior to Wget 1.10.\n\nglob = on/off\nTurn globbing on/off--the same as '--glob' and '--no-glob'.\n\nheader = STRING\nDefine a header for HTTP downloads, like using '--header=STRING'.\n\ncompression = STRING\nChoose the compression type to be used. Legal values are 'auto'\n(the default), 'gzip', and 'none'. The same as\n'--compression=STRING'.\n\nadjustextension = on/off\nAdd a '.html' extension to 'text/html' or 'application/xhtml+xml'\nfiles that lack one, a '.css' extension to 'text/css' files that\nlack one, and a '.br', '.Z', '.zlib' or '.gz' to compressed files\nlike '-E'. Previously named 'htmlextension' (still acceptable,\nbut deprecated).\n\nhttpkeepalive = on/off\nTurn the keep-alive feature on or off (defaults to on). Turning it\noff is equivalent to '--no-http-keep-alive'.\n\nhttppassword = STRING\nSet HTTP password, equivalent to '--http-password=STRING'.\n\nhttpproxy = STRING\nUse STRING as HTTP proxy, instead of the one specified in\nenvironment.\n\nhttpuser = STRING\nSet HTTP user to STRING, equivalent to '--http-user=STRING'.\n\nhttpsonly = on/off\nWhen in recursive mode, only HTTPS links are followed (defaults to\noff).\n\nhttpsproxy = STRING\nUse STRING as HTTPS proxy, instead of the one specified in\nenvironment.\n\nignorecase = on/off\nWhen set to on, match files and directories case insensitively; the\nsame as '--ignore-case'.\n\nignorelength = on/off\nWhen set to on, ignore 'Content-Length' header; the same as\n'--ignore-length'.\n\nignoretags = STRING\nIgnore certain HTML tags when doing a recursive retrieval, like\n'--ignore-tags=STRING'.\n\nincludedirectories = STRING\nSpecify a comma-separated list of directories you wish to follow\nwhen downloading--the same as '-I STRING'.\n\niri = on/off\nWhen set to on, enable internationalized URI (IRI) support; the\nsame as '--iri'.\n\ninet4only = on/off\nForce connecting to IPv4 addresses, off by default. You can put\nthis in the global init file to disable Wget's attempts to resolve\nand connect to IPv6 hosts. Available only if Wget was compiled\nwith IPv6 support. The same as '--inet4-only' or '-4'.\n\ninet6only = on/off\nForce connecting to IPv6 addresses, off by default. Available only\nif Wget was compiled with IPv6 support. The same as '--inet6-only'\nor '-6'.\n\ninput = FILE\nRead the URLs from STRING, like '-i FILE'.\n\nkeepsessioncookies = on/off\nWhen specified, causes 'savecookies = on' to also save session\ncookies. See '--keep-session-cookies'.\n\nlimitrate = RATE\nLimit the download speed to no more than RATE bytes per second.\nThe same as '--limit-rate=RATE'.\n\nloadcookies = FILE\nLoad cookies from FILE. See '--load-cookies FILE'.\n\nlocalencoding = ENCODING\nForce Wget to use ENCODING as the default system encoding. See\n'--local-encoding'.\n\nlogfile = FILE\nSet logfile to FILE, the same as '-o FILE'.\n\nmaxredirect = NUMBER\nSpecifies the maximum number of redirections to follow for a\nresource. See '--max-redirect=NUMBER'.\n\nmirror = on/off\nTurn mirroring on/off. The same as '-m'.\n\nnetrc = on/off\nTurn reading netrc on or off.\n\nnoclobber = on/off\nSame as '-nc'.\n\nnoparent = on/off\nDisallow retrieving outside the directory hierarchy, like\n'--no-parent' (*note Directory-Based Limits::).\n\nnoproxy = STRING\nUse STRING as the comma-separated list of domains to avoid in proxy\nloading, instead of the one specified in environment.\n\noutputdocument = FILE\nSet the output filename--the same as '-O FILE'.\n\npagerequisites = on/off\nDownload all ancillary documents necessary for a single HTML page\nto display properly--the same as '-p'.\n\npassiveftp = on/off\nChange setting of passive FTP, equivalent to the '--passive-ftp'\noption.\n\npassword = STRING\nSpecify password STRING for both FTP and HTTP file retrieval. This\ncommand can be overridden using the 'ftppassword' and\n'httppassword' command for FTP and HTTP respectively.\n\npostdata = STRING\nUse POST as the method for all HTTP requests and send STRING in the\nrequest body. The same as '--post-data=STRING'.\n\npostfile = FILE\nUse POST as the method for all HTTP requests and send the contents\nof FILE in the request body. The same as '--post-file=FILE'.\n\npreferfamily = none/IPv4/IPv6\nWhen given a choice of several addresses, connect to the addresses\nwith specified address family first. The address order returned by\nDNS is used without change by default. The same as\n'--prefer-family', which see for a detailed discussion of why this\nis useful.\n\nprivatekey = FILE\nSet the private key file to FILE. The same as\n'--private-key=FILE'.\n\nprivatekeytype = STRING\nSpecify the type of the private key, legal values being 'PEM' (the\ndefault) and 'DER' (aka ASN1). The same as\n'--private-type=STRING'.\n\nprogress = STRING\nSet the type of the progress indicator. Legal types are 'dot' and\n'bar'. Equivalent to '--progress=STRING'.\n\nprotocoldirectories = on/off\nWhen set, use the protocol name as a directory component of local\nfile names. The same as '--protocol-directories'.\n\nproxypassword = STRING\nSet proxy authentication password to STRING, like\n'--proxy-password=STRING'.\n\nproxyuser = STRING\nSet proxy authentication user name to STRING, like\n'--proxy-user=STRING'.\n\nquiet = on/off\nQuiet mode--the same as '-q'.\n\nquota = QUOTA\nSpecify the download quota, which is useful to put in the global\n'wgetrc'. When download quota is specified, Wget will stop\nretrieving after the download sum has become greater than quota.\nThe quota can be specified in bytes (default), kbytes 'k' appended)\nor mbytes ('m' appended). Thus 'quota = 5m' will set the quota to\n5 megabytes. Note that the user's startup file overrides system\nsettings.\n\nrandomfile = FILE\nUse FILE as a source of randomness on systems lacking\n'/dev/random'.\n\nrandomwait = on/off\nTurn random between-request wait times on or off. The same as\n'--random-wait'.\n\nreadtimeout = N\nSet the read (and write) timeout--the same as '--read-timeout=N'.\n\nreclevel = N\nRecursion level (depth)--the same as '-l N'.\n\nrecursive = on/off\nRecursive on/off--the same as '-r'.\n\nreferer = STRING\nSet HTTP 'Referer:' header just like '--referer=STRING'. (Note\nthat it was the folks who wrote the HTTP spec who got the spelling\nof \"referrer\" wrong.)\n\nrelativeonly = on/off\nFollow only relative links--the same as '-L' (*note Relative\nLinks::).\n\nremoteencoding = ENCODING\nForce Wget to use ENCODING as the default remote server encoding.\nSee '--remote-encoding'.\n\nremovelisting = on/off\nIf set to on, remove FTP listings downloaded by Wget. Setting it\nto off is the same as '--no-remove-listing'.\n\nrestrictfilenames = unix/windows\nRestrict the file names generated by Wget from URLs. See\n'--restrict-file-names' for a more detailed description.\n\nretrsymlinks = on/off\nWhen set to on, retrieve symbolic links as if they were plain\nfiles; the same as '--retr-symlinks'.\n\nretryconnrefused = on/off\nWhen set to on, consider \"connection refused\" a transient error--the\nsame as '--retry-connrefused'.\n\nrobots = on/off\nSpecify whether the norobots convention is respected by Wget, \"on\"\nby default. This switch controls both the '/robots.txt' and the\n'nofollow' aspect of the spec. *Note Robot Exclusion::, for more\ndetails about this. Be sure you know what you are doing before\nturning this off.\n\nsavecookies = FILE\nSave cookies to FILE. The same as '--save-cookies FILE'.\n\nsaveheaders = on/off\nSame as '--save-headers'.\n\nsecureprotocol = STRING\nChoose the secure protocol to be used. Legal values are 'auto'\n(the default), 'SSLv2', 'SSLv3', and 'TLSv1'. The same as\n'--secure-protocol=STRING'.\n\nserverresponse = on/off\nChoose whether or not to print the HTTP and FTP server\nresponses--the same as '-S'.\n\nshowalldnsentries = on/off\nWhen a DNS name is resolved, show all the IP addresses, not just\nthe first three.\n\nspanhosts = on/off\nSame as '-H'.\n\nspider = on/off\nSame as '--spider'.\n\nstrictcomments = on/off\nSame as '--strict-comments'.\n\ntimeout = N\nSet all applicable timeout values to N, the same as '-T N'.\n\ntimestamping = on/off\nTurn timestamping on/off. The same as '-N' (*note\nTime-Stamping::).\n\nuseservertimestamps = on/off\nIf set to 'off', Wget won't set the local file's timestamp by the\none on the server (same as '--no-use-server-timestamps').\n\ntries = N\nSet number of retries per URL--the same as '-t N'.\n\nuseproxy = on/off\nWhen set to off, don't use proxy even when proxy-related\nenvironment variables are set. In that case it is the same as\nusing '--no-proxy'.\n\nuser = STRING\nSpecify username STRING for both FTP and HTTP file retrieval. This\ncommand can be overridden using the 'ftpuser' and 'httpuser'\ncommand for FTP and HTTP respectively.\n\nuseragent = STRING\nUser agent identification sent to the HTTP Server--the same as\n'--user-agent=STRING'.\n\nverbose = on/off\nTurn verbose on/off--the same as '-v'/'-nv'.\n\nwait = N\nWait N seconds between retrievals--the same as '-w N'.\n\nwaitretry = N\nWait up to N seconds between retries of failed retrievals only--the\nsame as '--waitretry=N'. Note that this is turned on by default in\nthe global 'wgetrc'.\n\nFile: wget.info, Node: Sample Wgetrc, Prev: Wgetrc Commands, Up: Startup File\n" }, { "name": "6.4 Sample Wgetrc", "content": "This is the sample initialization file, as given in the distribution.\nIt is divided in two section--one for global usage (suitable for global\nstartup file), and one for local usage (suitable for '$HOME/.wgetrc').\nBe careful about the things you change.\n\nNote that almost all the lines are commented out. For a command to\nhave any effect, you must remove the '#' character at the beginning of\nits line.\n\n###\n### Sample Wget initialization file .wgetrc\n###\n\n## You can use this file to change the default behaviour of wget or to\n## avoid having to type many many command-line options. This file does\n## not contain a comprehensive list of commands -- look at the manual\n## to find out what you can put into this file. You can find this here:\n## $ info wget.info 'Startup File'\n## Or online here:\n## https://www.gnu.org/software/wget/manual/wget.html#Startup-File\n##\n## Wget initialization file can reside in /etc/wgetrc\n## (global, for all users) or $HOME/.wgetrc (for a single user).\n##\n## To use the settings in this file, you will have to uncomment them,\n## as well as change them, in most cases, as the values on the\n## commented-out lines are the default values (e.g. \"off\").\n##\n## Command are case-, underscore- and minus-insensitive.\n## For example ftpproxy, ftp-proxy and ftpproxy are the same.\n\n\n##\n## Global settings (useful for setting up in /etc/wgetrc).\n## Think well before you change them, since they may reduce wget's\n## functionality, and make it behave contrary to the documentation:\n##\n\n# You can set retrieve quota for beginners by specifying a value\n# optionally followed by 'K' (kilobytes) or 'M' (megabytes). The\n# default quota is unlimited.\n#quota = inf\n\n# You can lower (or raise) the default number of retries when\n# downloading a file (default is 20).\n#tries = 20\n\n# Lowering the maximum depth of the recursive retrieval is handy to\n# prevent newbies from going too \"deep\" when they unwittingly start\n# the recursive retrieval. The default is 5.\n#reclevel = 5\n\n# By default Wget uses \"passive FTP\" transfer where the client\n# initiates the data connection to the server rather than the other\n# way around. That is required on systems behind NAT where the client\n# computer cannot be easily reached from the Internet. However, some\n# firewalls software explicitly supports active FTP and in fact has\n# problems supporting passive transfer. If you are in such\n# environment, use \"passiveftp = off\" to revert to active FTP.\n#passiveftp = off\npassiveftp = on\n\n# The \"wait\" command below makes Wget wait between every connection.\n# If, instead, you want Wget to wait only between retries of failed\n# downloads, set waitretry to maximum number of seconds to wait (Wget\n# will use \"linear backoff\", waiting 1 second after the first failure\n# on a file, 2 seconds after the second failure, etc. up to this max).\n#waitretry = 10\n\n\n##\n## Local settings (for a user to set in his $HOME/.wgetrc). It is\n## *highly* undesirable to put these settings in the global file, since\n## they are potentially dangerous to \"normal\" users.\n##\n## Even when setting up your own ~/.wgetrc, you should know what you\n## are doing before doing so.\n##\n\n# Set this to on to use timestamping by default:\n#timestamping = off\n\n# It is a good idea to make Wget send your email address in a `From:'\n# header with your request (so that server administrators can contact\n# you in case of errors). Wget does *not* send `From:' by default.\n#header = From: Your Name \n\n# You can set up other headers, like Accept-Language. Accept-Language\n# is *not* sent by default.\n#header = Accept-Language: en\n\n# You can set the default proxies for Wget to use for http, https, and ftp.\n# They will override the value in the environment.\n#httpsproxy = http://proxy.yoyodyne.com:18023/\n#httpproxy = http://proxy.yoyodyne.com:18023/\n#ftpproxy = http://proxy.yoyodyne.com:18023/\n\n# If you do not want to use proxy at all, set this to off.\n#useproxy = on\n\n# You can customize the retrieval outlook. Valid options are default,\n# binary, mega and micro.\n#dotstyle = default\n\n# Setting this to off makes Wget not download /robots.txt. Be sure to\n# know *exactly* what /robots.txt is and how it is used before changing\n# the default!\n#robots = on\n\n# It can be useful to make Wget wait between connections. Set this to\n# the number of seconds you want Wget to wait.\n#wait = 0\n\n# You can force creating directory structure, even if a single is being\n# retrieved, by setting this to on.\n#dirstruct = off\n\n# You can turn on recursive retrieving by default (don't do this if\n# you are not sure you know what it means) by setting this to on.\n#recursive = off\n\n# To always back up file X as X.orig before converting its links (due\n# to -k / --convert-links / convertlinks = on having been specified),\n# set this variable to on:\n#backupconverted = off\n\n# To have Wget follow FTP links from HTML files by default, set this\n# to on:\n#followftp = off\n\n# To try ipv6 addresses first:\n#prefer-family = IPv6\n\n# Set default IRI support state\n#iri = off\n\n# Force the default system encoding\n#localencoding = UTF-8\n\n# Force the default remote server encoding\n#remoteencoding = UTF-8\n\n# Turn on to prevent following non-HTTPS links when in recursive mode\n#httpsonly = off\n\n# Tune HTTPS security (auto, SSLv2, SSLv3, TLSv1, PFS)\n#secureprotocol = auto\n\nFile: wget.info, Node: Examples, Next: Various, Prev: Startup File, Up: Top\n" } ] }, "7 Examples": { "content": "The examples are divided into three sections loosely based on their\ncomplexity.\n\n* Menu:\n\n* Simple Usage:: Simple, basic usage of the program.\n* Advanced Usage:: Advanced tips.\n* Very Advanced Usage:: The hairy stuff.\n\nFile: wget.info, Node: Simple Usage, Next: Advanced Usage, Prev: Examples, Up: Examples\n", "subsections": [ { "name": "7.1 Simple Usage", "content": "* Say you want to download a URL. Just type:\n\nwget http://fly.srk.fer.hr/\n\n* But what will happen if the connection is slow, and the file is\nlengthy? The connection will probably fail before the whole file\nis retrieved, more than once. In this case, Wget will try getting\nthe file until it either gets the whole of it, or exceeds the\ndefault number of retries (this being 20). It is easy to change\nthe number of tries to 45, to insure that the whole file will\narrive safely:\n\nwget --tries=45 http://fly.srk.fer.hr/jpg/flyweb.jpg\n\n* Now let's leave Wget to work in the background, and write its\nprogress to log file 'log'. It is tiring to type '--tries', so we\nshall use '-t'.\n\nwget -t 45 -o log http://fly.srk.fer.hr/jpg/flyweb.jpg &\n\nThe ampersand at the end of the line makes sure that Wget works in\nthe background. To unlimit the number of retries, use '-t inf'.\n\n* The usage of FTP is as simple. Wget will take care of login and\npassword.\n\nwget ftp://gnjilux.srk.fer.hr/welcome.msg\n\n* If you specify a directory, Wget will retrieve the directory\nlisting, parse it and convert it to HTML. Try:\n\nwget ftp://ftp.gnu.org/pub/gnu/\nlinks index.html\n\nFile: wget.info, Node: Advanced Usage, Next: Very Advanced Usage, Prev: Simple Usage, Up: Examples\n" }, { "name": "7.2 Advanced Usage", "content": "* You have a file that contains the URLs you want to download? Use\nthe '-i' switch:\n\nwget -i FILE\n\nIf you specify '-' as file name, the URLs will be read from\nstandard input.\n\n* Create a five levels deep mirror image of the GNU web site, with\nthe same directory structure the original has, with only one try\nper document, saving the log of the activities to 'gnulog':\n\nwget -r https://www.gnu.org/ -o gnulog\n\n* The same as the above, but convert the links in the downloaded\nfiles to point to local files, so you can view the documents\noff-line:\n\nwget --convert-links -r https://www.gnu.org/ -o gnulog\n\n* Retrieve only one HTML page, but make sure that all the elements\nneeded for the page to be displayed, such as inline images and\nexternal style sheets, are also downloaded. Also make sure the\ndownloaded page references the downloaded links.\n\nwget -p --convert-links http://www.example.com/dir/page.html\n\nThe HTML page will be saved to 'www.example.com/dir/page.html', and\nthe images, stylesheets, etc., somewhere under 'www.example.com/',\ndepending on where they were on the remote server.\n\n* The same as the above, but without the 'www.example.com/'\ndirectory. In fact, I don't want to have all those random server\ndirectories anyway--just save all those files under a 'download/'\nsubdirectory of the current directory.\n\nwget -p --convert-links -nH -nd -Pdownload \\\nhttp://www.example.com/dir/page.html\n\n* Retrieve the index.html of 'www.lycos.com', showing the original\nserver headers:\n\nwget -S http://www.lycos.com/\n\n* Save the server headers with the file, perhaps for post-processing.\n\nwget --save-headers http://www.lycos.com/\nmore index.html\n\n* Retrieve the first two levels of 'wuarchive.wustl.edu', saving them\nto '/tmp'.\n\nwget -r -l2 -P/tmp ftp://wuarchive.wustl.edu/\n\n* You want to download all the GIFs from a directory on an HTTP\nserver. You tried 'wget http://www.example.com/dir/*.gif', but\nthat didn't work because HTTP retrieval does not support globbing.\nIn that case, use:\n\nwget -r -l1 --no-parent -A.gif http://www.example.com/dir/\n\nMore verbose, but the effect is the same. '-r -l1' means to\nretrieve recursively (*note Recursive Download::), with maximum\ndepth of 1. '--no-parent' means that references to the parent\ndirectory are ignored (*note Directory-Based Limits::), and\n'-A.gif' means to download only the GIF files. '-A \"*.gif\"' would\nhave worked too.\n\n* Suppose you were in the middle of downloading, when Wget was\ninterrupted. Now you do not want to clobber the files already\npresent. It would be:\n\nwget -nc -r https://www.gnu.org/\n\n* If you want to encode your own username and password to HTTP or\nFTP, use the appropriate URL syntax (*note URL Format::).\n\nwget ftp://hniksic:mypassword@unix.example.com/.emacs\n\nNote, however, that this usage is not advisable on multi-user\nsystems because it reveals your password to anyone who looks at the\noutput of 'ps'.\n\n* You would like the output documents to go to standard output\ninstead of to files?\n\nwget -O - http://jagor.srce.hr/ http://www.srce.hr/\n\nYou can also combine the two options and make pipelines to retrieve\nthe documents from remote hotlists:\n\nwget -O - http://cool.list.com/ | wget --force-html -i -\n\nFile: wget.info, Node: Very Advanced Usage, Prev: Advanced Usage, Up: Examples\n" }, { "name": "7.3 Very Advanced Usage", "content": "* If you wish Wget to keep a mirror of a page (or FTP\nsubdirectories), use '--mirror' ('-m'), which is the shorthand for\n'-r -l inf -N'. You can put Wget in the crontab file asking it to\nrecheck a site each Sunday:\n\ncrontab\n0 0 * * 0 wget --mirror https://www.gnu.org/ -o /home/me/weeklog\n\n* In addition to the above, you want the links to be converted for\nlocal viewing. But, after having read this manual, you know that\nlink conversion doesn't play well with timestamping, so you also\nwant Wget to back up the original HTML files before the conversion.\nWget invocation would look like this:\n\nwget --mirror --convert-links --backup-converted \\\nhttps://www.gnu.org/ -o /home/me/weeklog\n\n* But you've also noticed that local viewing doesn't work all that\nwell when HTML files are saved under extensions other than '.html',\nperhaps because they were served as 'index.cgi'. So you'd like\nWget to rename all the files served with content-type 'text/html'\nor 'application/xhtml+xml' to 'NAME.html'.\n\nwget --mirror --convert-links --backup-converted \\\n--html-extension -o /home/me/weeklog \\\nhttps://www.gnu.org/\n\nOr, with less typing:\n\nwget -m -k -K -E https://www.gnu.org/ -o /home/me/weeklog\n\nFile: wget.info, Node: Various, Next: Appendices, Prev: Examples, Up: Top\n" } ] }, "8 Various": { "content": "This chapter contains all the stuff that could not fit anywhere else.\n\n* Menu:\n\n* Proxies:: Support for proxy servers.\n* Distribution:: Getting the latest version.\n* Web Site:: GNU Wget's presence on the World Wide Web.\n* Mailing Lists:: Wget mailing list for announcements and discussion.\n* Internet Relay Chat:: Wget's presence on IRC.\n* Reporting Bugs:: How and where to report bugs.\n* Portability:: The systems Wget works on.\n* Signals:: Signal-handling performed by Wget.\n\nFile: wget.info, Node: Proxies, Next: Distribution, Prev: Various, Up: Various\n", "subsections": [ { "name": "8.1 Proxies", "content": "\"Proxies\" are special-purpose HTTP servers designed to transfer data\nfrom remote servers to local clients. One typical use of proxies is\nlightening network load for users behind a slow connection. This is\nachieved by channeling all HTTP and FTP requests through the proxy which\ncaches the transferred data. When a cached resource is requested again,\nproxy will return the data from cache. Another use for proxies is for\ncompanies that separate (for security reasons) their internal networks\nfrom the rest of Internet. In order to obtain information from the Web,\ntheir users connect and retrieve remote data using an authorized proxy.\n\nWget supports proxies for both HTTP and FTP retrievals. The standard\nway to specify proxy location, which Wget recognizes, is using the\nfollowing environment variables:\n\n'httpproxy'\n'httpsproxy'\nIf set, the 'httpproxy' and 'httpsproxy' variables should contain\nthe URLs of the proxies for HTTP and HTTPS connections\nrespectively.\n\n'ftpproxy'\nThis variable should contain the URL of the proxy for FTP\nconnections. It is quite common that 'httpproxy' and 'ftpproxy'\nare set to the same URL.\n\n'noproxy'\nThis variable should contain a comma-separated list of domain\nextensions proxy should not be used for. For instance, if the\nvalue of 'noproxy' is '.mit.edu', proxy will not be used to\nretrieve documents from MIT.\n\nIn addition to the environment variables, proxy location and settings\nmay be specified from within Wget itself.\n\n'--no-proxy'\n'proxy = on/off'\nThis option and the corresponding command may be used to suppress\nthe use of proxy, even if the appropriate environment variables are\nset.\n\n'httpproxy = URL'\n'httpsproxy = URL'\n'ftpproxy = URL'\n'noproxy = STRING'\nThese startup file variables allow you to override the proxy\nsettings specified by the environment.\n\nSome proxy servers require authorization to enable you to use them.\nThe authorization consists of \"username\" and \"password\", which must be\nsent by Wget. As with HTTP authorization, several authentication\nschemes exist. For proxy authorization only the 'Basic' authentication\nscheme is currently implemented.\n\nYou may specify your username and password either through the proxy\nURL or through the command-line options. Assuming that the company's\nproxy is located at 'proxy.company.com' at port 8001, a proxy URL\nlocation containing authorization data might look like this:\n\nhttp://hniksic:mypassword@proxy.company.com:8001/\n\nAlternatively, you may use the 'proxy-user' and 'proxy-password'\noptions, and the equivalent '.wgetrc' settings 'proxyuser' and\n'proxypassword' to set the proxy username and password.\n\nFile: wget.info, Node: Distribution, Next: Web Site, Prev: Proxies, Up: Various\n" }, { "name": "8.2 Distribution", "content": "Like all GNU utilities, the latest version of Wget can be found at the\nmaster GNU archive site ftp.gnu.org, and its mirrors. For example, Wget\n1.21.2 can be found at\n\n\nFile: wget.info, Node: Web Site, Next: Mailing Lists, Prev: Distribution, Up: Various\n" }, { "name": "8.3 Web Site", "content": "The official web site for GNU Wget is at\n. However, most useful information\nresides at \"The Wget Wgiki\", .\n\nFile: wget.info, Node: Mailing Lists, Next: Internet Relay Chat, Prev: Web Site, Up: Various\n" }, { "name": "8.4 Mailing Lists", "content": "The primary mailinglist for discussion, bug-reports, or questions about\nGNU Wget is at . To subscribe, send an email to\n, or visit\n.\n\nYou do not need to subscribe to send a message to the list; however,\nplease note that unsubscribed messages are moderated, and may take a\nwhile before they hit the list--*usually around a day*. If you want your\nmessage to show up immediately, please subscribe to the list before\nposting. Archives for the list may be found at\n.\n\nAn NNTP/Usenettish gateway is also available via Gmane\n(http://gmane.org/about.php). You can see the Gmane archives at\n. Note that the\nGmane archives conveniently include messages from both the current list,\nand the previous one. Messages also show up in the Gmane archives\nsooner than they do at .\n\n\nPreviously, the mailing list was used as the main\ndiscussion list, and another list, was used\nfor submitting and discussing patches to GNU Wget.\n\nMessages from are archived at\n and at\n (which also\ncontinues to archive the current list, ).\n\nMessages from are archived at\n.\n\nFile: wget.info, Node: Internet Relay Chat, Next: Reporting Bugs, Prev: Mailing Lists, Up: Various\n" }, { "name": "8.5 Internet Relay Chat", "content": "In addition to the mailinglists, we also have a support channel set up\nvia IRC at 'irc.freenode.org', '#wget'. Come check it out!\n\nFile: wget.info, Node: Reporting Bugs, Next: Portability, Prev: Internet Relay Chat, Up: Various\n" }, { "name": "8.6 Reporting Bugs", "content": "You are welcome to submit bug reports via the GNU Wget bug tracker (see\n) or to our\nmailing list .\n\nVisit to get more\ninfo (how to subscribe, list archives, ...).\n\nBefore actually submitting a bug report, please try to follow a few\nsimple guidelines.\n\n1. Please try to ascertain that the behavior you see really is a bug.\nIf Wget crashes, it's a bug. If Wget does not behave as\ndocumented, it's a bug. If things work strange, but you are not\nsure about the way they are supposed to work, it might well be a\nbug, but you might want to double-check the documentation and the\nmailing lists (*note Mailing Lists::).\n\n2. Try to repeat the bug in as simple circumstances as possible. E.g.\nif Wget crashes while downloading 'wget -rl0 -kKE -t5 --no-proxy\nhttp://example.com -o /tmp/log', you should try to see if the crash\nis repeatable, and if will occur with a simpler set of options.\nYou might even try to start the download at the page where the\ncrash occurred to see if that page somehow triggered the crash.\n\nAlso, while I will probably be interested to know the contents of\nyour '.wgetrc' file, just dumping it into the debug message is\nprobably a bad idea. Instead, you should first try to see if the\nbug repeats with '.wgetrc' moved out of the way. Only if it turns\nout that '.wgetrc' settings affect the bug, mail me the relevant\nparts of the file.\n\n3. Please start Wget with '-d' option and send us the resulting output\n(or relevant parts thereof). If Wget was compiled without debug\nsupport, recompile it--it is much easier to trace bugs with debug\nsupport on.\n\nNote: please make sure to remove any potentially sensitive\ninformation from the debug log before sending it to the bug\naddress. The '-d' won't go out of its way to collect sensitive\ninformation, but the log will contain a fairly complete\ntranscript of Wget's communication with the server, which may\ninclude passwords and pieces of downloaded data. Since the bug\naddress is publicly archived, you may assume that all bug reports\nare visible to the public.\n\n4. If Wget has crashed, try to run it in a debugger, e.g. 'gdb `which\nwget` core' and type 'where' to get the backtrace. This may not\nwork if the system administrator has disabled core files, but it is\nsafe to try.\n\nFile: wget.info, Node: Portability, Next: Signals, Prev: Reporting Bugs, Up: Various\n" }, { "name": "8.7 Portability", "content": "Like all GNU software, Wget works on the GNU system. However, since it\nuses GNU Autoconf for building and configuring, and mostly avoids using\n\"special\" features of any particular Unix, it should compile (and work)\non all common Unix flavors.\n\nVarious Wget versions have been compiled and tested under many kinds\nof Unix systems, including GNU/Linux, Solaris, SunOS 4.x, Mac OS X, OSF\n(aka Digital Unix or Tru64), Ultrix, *BSD, IRIX, AIX, and others. Some\nof those systems are no longer in widespread use and may not be able to\nsupport recent versions of Wget. If Wget fails to compile on your\nsystem, we would like to know about it.\n\nThanks to kind contributors, this version of Wget compiles and works\non 32-bit Microsoft Windows platforms. It has been compiled\nsuccessfully using MS Visual C++ 6.0, Watcom, Borland C, and GCC\ncompilers. Naturally, it is crippled of some features available on\nUnix, but it should work as a substitute for people stuck with Windows.\nNote that Windows-specific portions of Wget are not guaranteed to be\nsupported in the future, although this has been the case in practice for\nmany years now. All questions and problems in Windows usage should be\nreported to Wget mailing list at where the volunteers\nwho maintain the Windows-related features might look at them.\n\nSupport for building on MS-DOS via DJGPP has been contributed by\nGisle Vanem; a port to VMS is maintained by Steven Schweda, and is\navailable at .\n\nFile: wget.info, Node: Signals, Prev: Portability, Up: Various\n" }, { "name": "8.8 Signals", "content": "Since the purpose of Wget is background work, it catches the hangup\nsignal ('SIGHUP') and ignores it. If the output was on standard output,\nit will be redirected to a file named 'wget-log'. Otherwise, 'SIGHUP'\nis ignored. This is convenient when you wish to redirect the output of\nWget after having started it.\n\n$ wget http://www.gnus.org/dist/gnus.tar.gz &\n...\n$ kill -HUP %%\nSIGHUP received, redirecting output to `wget-log'.\n\nOther than that, Wget will not try to interfere with signals in any\nway. 'C-c', 'kill -TERM' and 'kill -KILL' should kill it alike.\n\nFile: wget.info, Node: Appendices, Next: Copying this manual, Prev: Various, Up: Top\n" } ] }, "9 Appendices": { "content": "This chapter contains some references I consider useful.\n\n* Menu:\n\n* Robot Exclusion:: Wget's support for RES.\n* Security Considerations:: Security with Wget.\n* Contributors:: People who helped.\n\nFile: wget.info, Node: Robot Exclusion, Next: Security Considerations, Prev: Appendices, Up: Appendices\n", "subsections": [ { "name": "9.1 Robot Exclusion", "content": "It is extremely easy to make Wget wander aimlessly around a web site,\nsucking all the available data in progress. 'wget -r SITE', and you're\nset. Great? Not for the server admin.\n\nAs long as Wget is only retrieving static pages, and doing it at a\nreasonable rate (see the '--wait' option), there's not much of a\nproblem. The trouble is that Wget can't tell the difference between the\nsmallest static page and the most demanding CGI. A site I know has a\nsection handled by a CGI Perl script that converts Info files to HTML on\nthe fly. The script is slow, but works well enough for human users\nviewing an occasional Info file. However, when someone's recursive Wget\ndownload stumbles upon the index page that links to all the Info files\nthrough the script, the system is brought to its knees without providing\nanything useful to the user (This task of converting Info files could be\ndone locally and access to Info documentation for all installed GNU\nsoftware on a system is available from the 'info' command).\n\nTo avoid this kind of accident, as well as to preserve privacy for\ndocuments that need to be protected from well-behaved robots, the\nconcept of \"robot exclusion\" was invented. The idea is that the server\nadministrators and document authors can specify which portions of the\nsite they wish to protect from robots and those they will permit access.\n\nThe most popular mechanism, and the de facto standard supported by\nall the major robots, is the \"Robots Exclusion Standard\" (RES) written\nby Martijn Koster et al. in 1994. It specifies the format of a text\nfile containing directives that instruct the robots which URL paths to\navoid. To be found by the robots, the specifications must be placed in\n'/robots.txt' in the server root, which the robots are expected to\ndownload and parse.\n\nAlthough Wget is not a web robot in the strictest sense of the word,\nit can download large parts of the site without the user's intervention\nto download an individual page. Because of that, Wget honors RES when\ndownloading recursively. For instance, when you issue:\n\nwget -r http://www.example.com/\n\nFirst the index of 'www.example.com' will be downloaded. If Wget\nfinds that it wants to download more documents from that server, it will\nrequest 'http://www.example.com/robots.txt' and, if found, use it for\nfurther downloads. 'robots.txt' is loaded only once per each server.\n\nUntil version 1.8, Wget supported the first version of the standard,\nwritten by Martijn Koster in 1994 and available at\n. As of version 1.8, Wget has\nsupported the additional directives specified in the internet draft\n'' titled \"A Method for Web Robots Control\".\nThe draft, which has as far as I know never made to an RFC, is available\nat .\n\nThis manual no longer includes the text of the Robot Exclusion\nStandard.\n\nThe second, less known mechanism, enables the author of an individual\ndocument to specify whether they want the links from the file to be\nfollowed by a robot. This is achieved using the 'META' tag, like this:\n\n\n\nThis is explained in some detail at\n. Wget supports this method of\nrobot exclusion in addition to the usual '/robots.txt' exclusion.\n\nIf you know what you are doing and really really wish to turn off the\nrobot exclusion, set the 'robots' variable to 'off' in your '.wgetrc'.\nYou can achieve the same effect from the command line using the '-e'\nswitch, e.g. 'wget -e robots=off URL...'.\n\nFile: wget.info, Node: Security Considerations, Next: Contributors, Prev: Robot Exclusion, Up: Appendices\n" }, { "name": "9.2 Security Considerations", "content": "When using Wget, you must be aware that it sends unencrypted passwords\nthrough the network, which may present a security problem. Here are the\nmain issues, and some solutions.\n\n1. The passwords on the command line are visible using 'ps'. The best\nway around it is to use 'wget -i -' and feed the URLs to Wget's\nstandard input, each on a separate line, terminated by 'C-d'.\nAnother workaround is to use '.netrc' to store passwords; however,\nstoring unencrypted passwords is also considered a security risk.\n\n2. Using the insecure \"basic\" authentication scheme, unencrypted\npasswords are transmitted through the network routers and gateways.\n\n3. The FTP passwords are also in no way encrypted. There is no good\nsolution for this at the moment.\n\n4. Although the \"normal\" output of Wget tries to hide the passwords,\ndebugging logs show them, in all forms. This problem is avoided by\nbeing careful when you send debug logs (yes, even when you send\nthem to me).\n\nFile: wget.info, Node: Contributors, Prev: Security Considerations, Up: Appendices\n" }, { "name": "9.3 Contributors", "content": "GNU Wget was written by Hrvoje Nik??i?? ,\n\nHowever, the development of Wget could never have gone as far as it\nhas, were it not for the help of many people, either with bug reports,\nfeature proposals, patches, or letters saying \"Thanks!\".\n\nSpecial thanks goes to the following people (no particular order):\n\n* Dan Harkless--contributed a lot of code and documentation of\nextremely high quality, as well as the '--page-requisites' and\nrelated options. He was the principal maintainer for some time and\nreleased Wget 1.6.\n\n* Ian Abbott--contributed bug fixes, Windows-related fixes, and\nprovided a prototype implementation of the breadth-first recursive\ndownload. Co-maintained Wget during the 1.8 release cycle.\n\n* The dotsrc.org crew, in particular Karsten Thygesen--donated system\nresources such as the mailing list, web space, FTP space, and\nversion control repositories, along with a lot of time to make\nthese actually work. Christian Reiniger was of invaluable help\nwith setting up Subversion.\n\n* Heiko Herold--provided high-quality Windows builds and contributed\nbug and build reports for many years.\n\n* Shawn McHorse--bug reports and patches.\n\n* Kaveh R. Ghazi--on-the-fly 'ansi2knr'-ization. Lots of portability\nfixes.\n\n* Gordon Matzigkeit--'.netrc' support.\n\n* Zlatko ??alu??i??, Tomislav Vujec and Dra??en Ka??ar--feature suggestions\nand \"philosophical\" discussions.\n\n* Darko Budor--initial port to Windows.\n\n* Antonio Rosella--help and suggestions, plus the initial Italian\ntranslation.\n\n* Tomislav Petrovi??, Mario Miko??evi??--many bug reports and\nsuggestions.\n\n* Franc,is Pinard--many thorough bug reports and discussions.\n\n* Karl Eichwalder--lots of help with internationalization, Makefile\nlayout and many other things.\n\n* Junio Hamano--donated support for Opie and HTTP 'Digest'\nauthentication.\n\n* Mauro Tortonesi--improved IPv6 support, adding support for dual\nfamily systems. Refactored and enhanced FTP IPv6 code. Maintained\nGNU Wget from 2004-2007.\n\n* Christopher G. Lewis--maintenance of the Windows version of GNU\nWGet.\n\n* Gisle Vanem--many helpful patches and improvements, especially for\nWindows and MS-DOS support.\n\n* Ralf Wildenhues--contributed patches to convert Wget to use Automake\nas part of its build process, and various bugfixes.\n\n* Steven Schubiger--Many helpful patches, bugfixes and improvements.\nNotably, conversion of Wget to use the Gnulib quotes and quoteargs\nmodules, and the addition of password prompts at the console, via\nthe Gnulib getpasswd-gnu module.\n\n* Ted Mielczarek--donated support for CSS.\n\n* Saint Xavier--Support for IRIs (RFC 3987).\n\n* Tim Ru\"hsen--Loads of helpful patches, especially fuzzing support and\nContinuous Integration. Maintainer since 2014.\n\n* Darshit Shah--Many helpful patches. Community support on various\nplatforms. Maintainer since 2014.\n\n* People who provided donations for development--including Brian\nGough.\n\nThe following people have provided patches, bug/build reports, useful\nsuggestions, beta testing services, fan mail and all the other things\nthat make maintenance so much fun:\n\nTim Adam, Adrian Aichner, Martin Baehr, Dieter Baron, Roger Beeman,\nDan Berger, T. Bharath, Christian Biere, Paul Bludov, Daniel Bodea, Mark\nBoyns, John Burden, Julien Buty, Wanderlei Cavassin, Gilles Cedoc, Tim\nCharron, Noel Cragg, Kristijan ??onka??, John Daily, Andreas Damm, Ahmon\nDancy, Andrew Davison, Bertrand Demiddelaer, Alexander Dergachev, Andrew\nDeryabin, Ulrich Drepper, Marc Duponcheel, Damir D??eko, Alan Eldridge,\nHans-Andreas Engel, Aleksandar Erkalovi??, Andy Eskilsson, Jo??o Ferreira,\nChristian Fraenkel, David Fritz, Mike Frysinger, Charles C. Fu,\nFUJISHIMA Satsuki, Masashi Fujita, Howard Gayle, Marcel Gerrits, Lemble\nGregory, Hans Grobler, Alain Guibert, Mathieu Guillaume, Aaron Hawley,\nJochen Hein, Karl Heuer, Madhusudan Hosaagrahara, HIROSE Masaaki, Ulf\nHarnhammar, Gregor Hoffleit, Erik Magnus Hulthen, Richard Huveneers,\nJonas Jensen, Larry Jones, Simon Josefsson, Mario Juri??, Hack Kampbj??rn,\nConst Kaplinsky, Goran Kezunovi??, Igor Khristophorov, Robert Kleine,\nKOJIMA Haime, Fila Kolodny, Alexander Kourakos, Martin Kraemer, Sami\nKrank, Jay Krell, ?????????? ???????????????????? (Simos KSenitellis), Christian\nLackas, Hrvoje Lacko, Daniel S. Lewart, Nicol??s Lichtmeier, Dave Love,\nAlexander V. Lukyanov, Thomas Lussnig, Andre Majorel, Aurelien Marchand,\nMatthew J. Mellon, Jordan Mendelson, Ted Mielczarek, Robert Millan, Lin\nZhe Min, Jan Minar, Tim Mooney, Keith Moore, Adam D. Moss, Simon Munton,\nCharlie Negyesi, R. K. Owen, Jim Paris, Kenny Parnell, Leonid Petrov,\nSimone Piunno, Andrew Pollock, Steve Pothier, Jan P??ikryl, Marin Purgar,\nCsaba R??duly, Keith Refson, Bill Richardson, Tyler Riddle, Tobias\nRingstrom, Jochen Roderburg, Juan Jose' Rodr??guez, Maciej W. Rozycki,\nEdward J. Sabol, Heinz Salzmann, Robert Schmidt, Nicolas Schodet, Benno\nSchulenberg, Andreas Schwab, Steven M. Schweda, Chris Seawood, Pranab\nShenoy, Dennis Smit, Toomas Soome, Tage Stabell-Kulo, Philip Stadermann,\nDaniel Stenberg, Sven Sternberger, Markus Strasser, John Summerfield,\nSzakacsits Szabolcs, Mike Thomas, Philipp Thomas, Mauro Tortonesi, Dave\nTurner, Gisle Vanem, Rabin Vincent, Russell Vincent, ??eljko Vrba,\nCharles G Waldman, Douglas E. Wegscheid, Ralf Wildenhues, Joshua David\nWilliams, Benjamin Wolsey, Saint Xavier, YAMAZAKI Makoto, Jasmin Zainul,\nBojan ??drnja, Kristijan Zimmer, Xin Zou.\n\nApologies to all who I accidentally left out, and many thanks to all\nthe subscribers of the Wget mailing list.\n\nFile: wget.info, Node: Copying this manual, Next: Concept Index, Prev: Appendices, Up: Top\n" } ] }, "Appendix A Copying this manual": { "content": "* Menu:\n\n* GNU Free Documentation License:: License for copying this manual.\n\nFile: wget.info, Node: GNU Free Documentation License, Prev: Copying this manual, Up: Copying this manual\n", "subsections": [ { "name": "A.1 GNU Free Documentation License", "content": "Version 1.3, 3 November 2008\n\nCopyright (C) 2000-2002, 2007-2008, 2015, 2018-2021 Free\nSoftware Foundation, Inc.\n\n\nEveryone is permitted to copy and distribute verbatim copies\nof this license document, but changing it is not allowed.\n\n0. PREAMBLE\n\nThe purpose of this License is to make a manual, textbook, or other\nfunctional and useful document \"free\" in the sense of freedom: to\nassure everyone the effective freedom to copy and redistribute it,\nwith or without modifying it, either commercially or\nnoncommercially. Secondarily, this License preserves for the\nauthor and publisher a way to get credit for their work, while not\nbeing considered responsible for modifications made by others.\n\nThis License is a kind of \"copyleft\", which means that derivative\nworks of the document must themselves be free in the same sense.\nIt complements the GNU General Public License, which is a copyleft\nlicense designed for free software.\n\nWe have designed this License in order to use it for manuals for\nfree software, because free software needs free documentation: a\nfree program should come with manuals providing the same freedoms\nthat the software does. But this License is not limited to\nsoftware manuals; it can be used for any textual work, regardless\nof subject matter or whether it is published as a printed book. We\nrecommend this License principally for works whose purpose is\ninstruction or reference.\n\n1. APPLICABILITY AND DEFINITIONS\n\nThis License applies to any manual or other work, in any medium,\nthat contains a notice placed by the copyright holder saying it can\nbe distributed under the terms of this License. Such a notice\ngrants a world-wide, royalty-free license, unlimited in duration,\nto use that work under the conditions stated herein. The\n\"Document\", below, refers to any such manual or work. Any member\nof the public is a licensee, and is addressed as \"you\". You accept\nthe license if you copy, modify or distribute the work in a way\nrequiring permission under copyright law.\n\nA \"Modified Version\" of the Document means any work containing the\nDocument or a portion of it, either copied verbatim, or with\nmodifications and/or translated into another language.\n\nA \"Secondary Section\" is a named appendix or a front-matter section\nof the Document that deals exclusively with the relationship of the\npublishers or authors of the Document to the Document's overall\nsubject (or to related matters) and contains nothing that could\nfall directly within that overall subject. (Thus, if the Document\nis in part a textbook of mathematics, a Secondary Section may not\nexplain any mathematics.) The relationship could be a matter of\nhistorical connection with the subject or with related matters, or\nof legal, commercial, philosophical, ethical or political position\nregarding them.\n\nThe \"Invariant Sections\" are certain Secondary Sections whose\ntitles are designated, as being those of Invariant Sections, in the\nnotice that says that the Document is released under this License.\nIf a section does not fit the above definition of Secondary then it\nis not allowed to be designated as Invariant. The Document may\ncontain zero Invariant Sections. If the Document does not identify\nany Invariant Sections then there are none.\n\nThe \"Cover Texts\" are certain short passages of text that are\nlisted, as Front-Cover Texts or Back-Cover Texts, in the notice\nthat says that the Document is released under this License. A\nFront-Cover Text may be at most 5 words, and a Back-Cover Text may\nbe at most 25 words.\n\nA \"Transparent\" copy of the Document means a machine-readable copy,\nrepresented in a format whose specification is available to the\ngeneral public, that is suitable for revising the document\nstraightforwardly with generic text editors or (for images composed\nof pixels) generic paint programs or (for drawings) some widely\navailable drawing editor, and that is suitable for input to text\nformatters or for automatic translation to a variety of formats\nsuitable for input to text formatters. A copy made in an otherwise\nTransparent file format whose markup, or absence of markup, has\nbeen arranged to thwart or discourage subsequent modification by\nreaders is not Transparent. An image format is not Transparent if\nused for any substantial amount of text. A copy that is not\n\"Transparent\" is called \"Opaque\".\n\nExamples of suitable formats for Transparent copies include plain\nASCII without markup, Texinfo input format, LaTeX input format,\nSGML or XML using a publicly available DTD, and standard-conforming\nsimple HTML, PostScript or PDF designed for human modification.\nExamples of transparent image formats include PNG, XCF and JPG.\nOpaque formats include proprietary formats that can be read and\nedited only by proprietary word processors, SGML or XML for which\nthe DTD and/or processing tools are not generally available, and\nthe machine-generated HTML, PostScript or PDF produced by some word\nprocessors for output purposes only.\n\nThe \"Title Page\" means, for a printed book, the title page itself,\nplus such following pages as are needed to hold, legibly, the\nmaterial this License requires to appear in the title page. For\nworks in formats which do not have any title page as such, \"Title\nPage\" means the text near the most prominent appearance of the\nwork's title, preceding the beginning of the body of the text.\n\nThe \"publisher\" means any person or entity that distributes copies\nof the Document to the public.\n\nA section \"Entitled XYZ\" means a named subunit of the Document\nwhose title either is precisely XYZ or contains XYZ in parentheses\nfollowing text that translates XYZ in another language. (Here XYZ\nstands for a specific section name mentioned below, such as\n\"Acknowledgements\", \"Dedications\", \"Endorsements\", or \"History\".)\nTo \"Preserve the Title\" of such a section when you modify the\nDocument means that it remains a section \"Entitled XYZ\" according\nto this definition.\n\nThe Document may include Warranty Disclaimers next to the notice\nwhich states that this License applies to the Document. These\nWarranty Disclaimers are considered to be included by reference in\nthis License, but only as regards disclaiming warranties: any other\nimplication that these Warranty Disclaimers may have is void and\nhas no effect on the meaning of this License.\n\n2. VERBATIM COPYING\n\nYou may copy and distribute the Document in any medium, either\ncommercially or noncommercially, provided that this License, the\ncopyright notices, and the license notice saying this License\napplies to the Document are reproduced in all copies, and that you\nadd no other conditions whatsoever to those of this License. You\nmay not use technical measures to obstruct or control the reading\nor further copying of the copies you make or distribute. However,\nyou may accept compensation in exchange for copies. If you\ndistribute a large enough number of copies you must also follow the\nconditions in section 3.\n\nYou may also lend copies, under the same conditions stated above,\nand you may publicly display copies.\n\n3. COPYING IN QUANTITY\n\nIf you publish printed copies (or copies in media that commonly\nhave printed covers) of the Document, numbering more than 100, and\nthe Document's license notice requires Cover Texts, you must\nenclose the copies in covers that carry, clearly and legibly, all\nthese Cover Texts: Front-Cover Texts on the front cover, and\nBack-Cover Texts on the back cover. Both covers must also clearly\nand legibly identify you as the publisher of these copies. The\nfront cover must present the full title with all words of the title\nequally prominent and visible. You may add other material on the\ncovers in addition. Copying with changes limited to the covers, as\nlong as they preserve the title of the Document and satisfy these\nconditions, can be treated as verbatim copying in other respects.\n\nIf the required texts for either cover are too voluminous to fit\nlegibly, you should put the first ones listed (as many as fit\nreasonably) on the actual cover, and continue the rest onto\nadjacent pages.\n\nIf you publish or distribute Opaque copies of the Document\nnumbering more than 100, you must either include a machine-readable\nTransparent copy along with each Opaque copy, or state in or with\neach Opaque copy a computer-network location from which the general\nnetwork-using public has access to download using public-standard\nnetwork protocols a complete Transparent copy of the Document, free\nof added material. If you use the latter option, you must take\nreasonably prudent steps, when you begin distribution of Opaque\ncopies in quantity, to ensure that this Transparent copy will\nremain thus accessible at the stated location until at least one\nyear after the last time you distribute an Opaque copy (directly or\nthrough your agents or retailers) of that edition to the public.\n\nIt is requested, but not required, that you contact the authors of\nthe Document well before redistributing any large number of copies,\nto give them a chance to provide you with an updated version of the\nDocument.\n\n4. MODIFICATIONS\n\nYou may copy and distribute a Modified Version of the Document\nunder the conditions of sections 2 and 3 above, provided that you\nrelease the Modified Version under precisely this License, with the\nModified Version filling the role of the Document, thus licensing\ndistribution and modification of the Modified Version to whoever\npossesses a copy of it. In addition, you must do these things in\nthe Modified Version:\n\nA. Use in the Title Page (and on the covers, if any) a title\ndistinct from that of the Document, and from those of previous\nversions (which should, if there were any, be listed in the\nHistory section of the Document). You may use the same title\nas a previous version if the original publisher of that\nversion gives permission.\n\nB. List on the Title Page, as authors, one or more persons or\nentities responsible for authorship of the modifications in\nthe Modified Version, together with at least five of the\nprincipal authors of the Document (all of its principal\nauthors, if it has fewer than five), unless they release you\nfrom this requirement.\n\nC. State on the Title page the name of the publisher of the\nModified Version, as the publisher.\n\nD. Preserve all the copyright notices of the Document.\n\nE. Add an appropriate copyright notice for your modifications\nadjacent to the other copyright notices.\n\nF. Include, immediately after the copyright notices, a license\nnotice giving the public permission to use the Modified\nVersion under the terms of this License, in the form shown in\nthe Addendum below.\n\nG. Preserve in that license notice the full lists of Invariant\nSections and required Cover Texts given in the Document's\nlicense notice.\n\nH. Include an unaltered copy of this License.\n\nI. Preserve the section Entitled \"History\", Preserve its Title,\nand add to it an item stating at least the title, year, new\nauthors, and publisher of the Modified Version as given on the\nTitle Page. If there is no section Entitled \"History\" in the\nDocument, create one stating the title, year, authors, and\npublisher of the Document as given on its Title Page, then add\nan item describing the Modified Version as stated in the\nprevious sentence.\n\nJ. Preserve the network location, if any, given in the Document\nfor public access to a Transparent copy of the Document, and\nlikewise the network locations given in the Document for\nprevious versions it was based on. These may be placed in the\n\"History\" section. You may omit a network location for a work\nthat was published at least four years before the Document\nitself, or if the original publisher of the version it refers\nto gives permission.\n\nK. For any section Entitled \"Acknowledgements\" or \"Dedications\",\nPreserve the Title of the section, and preserve in the section\nall the substance and tone of each of the contributor\nacknowledgements and/or dedications given therein.\n\nL. Preserve all the Invariant Sections of the Document, unaltered\nin their text and in their titles. Section numbers or the\nequivalent are not considered part of the section titles.\n\nM. Delete any section Entitled \"Endorsements\". Such a section\nmay not be included in the Modified Version.\n\nN. Do not retitle any existing section to be Entitled\n\"Endorsements\" or to conflict in title with any Invariant\nSection.\n\nO. Preserve any Warranty Disclaimers.\n\nIf the Modified Version includes new front-matter sections or\nappendices that qualify as Secondary Sections and contain no\nmaterial copied from the Document, you may at your option designate\nsome or all of these sections as invariant. To do this, add their\ntitles to the list of Invariant Sections in the Modified Version's\nlicense notice. These titles must be distinct from any other\nsection titles.\n\nYou may add a section Entitled \"Endorsements\", provided it contains\nnothing but endorsements of your Modified Version by various\nparties--for example, statements of peer review or that the text has\nbeen approved by an organization as the authoritative definition of\na standard.\n\nYou may add a passage of up to five words as a Front-Cover Text,\nand a passage of up to 25 words as a Back-Cover Text, to the end of\nthe list of Cover Texts in the Modified Version. Only one passage\nof Front-Cover Text and one of Back-Cover Text may be added by (or\nthrough arrangements made by) any one entity. If the Document\nalready includes a cover text for the same cover, previously added\nby you or by arrangement made by the same entity you are acting on\nbehalf of, you may not add another; but you may replace the old\none, on explicit permission from the previous publisher that added\nthe old one.\n\nThe author(s) and publisher(s) of the Document do not by this\nLicense give permission to use their names for publicity for or to\nassert or imply endorsement of any Modified Version.\n\n5. COMBINING DOCUMENTS\n\nYou may combine the Document with other documents released under\nthis License, under the terms defined in section 4 above for\nmodified versions, provided that you include in the combination all\nof the Invariant Sections of all of the original documents,\nunmodified, and list them all as Invariant Sections of your\ncombined work in its license notice, and that you preserve all\ntheir Warranty Disclaimers.\n\nThe combined work need only contain one copy of this License, and\nmultiple identical Invariant Sections may be replaced with a single\ncopy. If there are multiple Invariant Sections with the same name\nbut different contents, make the title of each such section unique\nby adding at the end of it, in parentheses, the name of the\noriginal author or publisher of that section if known, or else a\nunique number. Make the same adjustment to the section titles in\nthe list of Invariant Sections in the license notice of the\ncombined work.\n\nIn the combination, you must combine any sections Entitled\n\"History\" in the various original documents, forming one section\nEntitled \"History\"; likewise combine any sections Entitled\n\"Acknowledgements\", and any sections Entitled \"Dedications\". You\nmust delete all sections Entitled \"Endorsements.\"\n\n6. COLLECTIONS OF DOCUMENTS\n\nYou may make a collection consisting of the Document and other\ndocuments released under this License, and replace the individual\ncopies of this License in the various documents with a single copy\nthat is included in the collection, provided that you follow the\nrules of this License for verbatim copying of each of the documents\nin all other respects.\n\nYou may extract a single document from such a collection, and\ndistribute it individually under this License, provided you insert\na copy of this License into the extracted document, and follow this\nLicense in all other respects regarding verbatim copying of that\ndocument.\n\n7. AGGREGATION WITH INDEPENDENT WORKS\n\nA compilation of the Document or its derivatives with other\nseparate and independent documents or works, in or on a volume of a\nstorage or distribution medium, is called an \"aggregate\" if the\ncopyright resulting from the compilation is not used to limit the\nlegal rights of the compilation's users beyond what the individual\nworks permit. When the Document is included in an aggregate, this\nLicense does not apply to the other works in the aggregate which\nare not themselves derivative works of the Document.\n\nIf the Cover Text requirement of section 3 is applicable to these\ncopies of the Document, then if the Document is less than one half\nof the entire aggregate, the Document's Cover Texts may be placed\non covers that bracket the Document within the aggregate, or the\nelectronic equivalent of covers if the Document is in electronic\nform. Otherwise they must appear on printed covers that bracket\nthe whole aggregate.\n\n8. TRANSLATION\n\nTranslation is considered a kind of modification, so you may\ndistribute translations of the Document under the terms of section\n4. Replacing Invariant Sections with translations requires special\npermission from their copyright holders, but you may include\ntranslations of some or all Invariant Sections in addition to the\noriginal versions of these Invariant Sections. You may include a\ntranslation of this License, and all the license notices in the\nDocument, and any Warranty Disclaimers, provided that you also\ninclude the original English version of this License and the\noriginal versions of those notices and disclaimers. In case of a\ndisagreement between the translation and the original version of\nthis License or a notice or disclaimer, the original version will\nprevail.\n\nIf a section in the Document is Entitled \"Acknowledgements\",\n\"Dedications\", or \"History\", the requirement (section 4) to\nPreserve its Title (section 1) will typically require changing the\nactual title.\n\n9. TERMINATION\n\nYou may not copy, modify, sublicense, or distribute the Document\nexcept as expressly provided under this License. Any attempt\notherwise to copy, modify, sublicense, or distribute it is void,\nand will automatically terminate your rights under this License.\n\nHowever, if you cease all violation of this License, then your\nlicense from a particular copyright holder is reinstated (a)\nprovisionally, unless and until the copyright holder explicitly and\nfinally terminates your license, and (b) permanently, if the\ncopyright holder fails to notify you of the violation by some\nreasonable means prior to 60 days after the cessation.\n\nMoreover, your license from a particular copyright holder is\nreinstated permanently if the copyright holder notifies you of the\nviolation by some reasonable means, this is the first time you have\nreceived notice of violation of this License (for any work) from\nthat copyright holder, and you cure the violation prior to 30 days\nafter your receipt of the notice.\n\nTermination of your rights under this section does not terminate\nthe licenses of parties who have received copies or rights from you\nunder this License. If your rights have been terminated and not\npermanently reinstated, receipt of a copy of some or all of the\nsame material does not give you any rights to use it.\n\n10. FUTURE REVISIONS OF THIS LICENSE\n\nThe Free Software Foundation may publish new, revised versions of\nthe GNU Free Documentation License from time to time. Such new\nversions will be similar in spirit to the present version, but may\ndiffer in detail to address new problems or concerns. See\n.\n\nEach version of the License is given a distinguishing version\nnumber. If the Document specifies that a particular numbered\nversion of this License \"or any later version\" applies to it, you\nhave the option of following the terms and conditions either of\nthat specified version or of any later version that has been\npublished (not as a draft) by the Free Software Foundation. If the\nDocument does not specify a version number of this License, you may\nchoose any version ever published (not as a draft) by the Free\nSoftware Foundation. If the Document specifies that a proxy can\ndecide which future versions of this License can be used, that\nproxy's public statement of acceptance of a version permanently\nauthorizes you to choose that version for the Document.\n\n11. RELICENSING\n\n\"Massive Multiauthor Collaboration Site\" (or \"MMC Site\") means any\nWorld Wide Web server that publishes copyrightable works and also\nprovides prominent facilities for anybody to edit those works. A\npublic wiki that anybody can edit is an example of such a server.\nA \"Massive Multiauthor Collaboration\" (or \"MMC\") contained in the\nsite means any set of copyrightable works thus published on the MMC\nsite.\n\n\"CC-BY-SA\" means the Creative Commons Attribution-Share Alike 3.0\nlicense published by Creative Commons Corporation, a not-for-profit\ncorporation with a principal place of business in San Francisco,\nCalifornia, as well as future copyleft versions of that license\npublished by that same organization.\n\n\"Incorporate\" means to publish or republish a Document, in whole or\nin part, as part of another Document.\n\nAn MMC is \"eligible for relicensing\" if it is licensed under this\nLicense, and if all works that were first published under this\nLicense somewhere other than this MMC, and subsequently\nincorporated in whole or in part into the MMC, (1) had no cover\ntexts or invariant sections, and (2) were thus incorporated prior\nto November 1, 2008.\n\nThe operator of an MMC Site may republish an MMC contained in the\nsite under CC-BY-SA on the same site at any time before August 1,\n2009, provided the MMC is eligible for relicensing.\n" }, { "name": "ADDENDUM: How to use this License for your documents", "content": "To use this License in a document you have written, include a copy of\nthe License in the document and put the following copyright and license\nnotices just after the title page:\n\nCopyright (C) YEAR YOUR NAME.\nPermission is granted to copy, distribute and/or modify this document\nunder the terms of the GNU Free Documentation License, Version 1.3\nor any later version published by the Free Software Foundation;\nwith no Invariant Sections, no Front-Cover Texts, and no Back-Cover\nTexts. A copy of the license is included in the section entitled ``GNU\nFree Documentation License''.\n\nIf you have Invariant Sections, Front-Cover Texts and Back-Cover\nTexts, replace the \"with...Texts.\" line with this:\n\nwith the Invariant Sections being LIST THEIR TITLES, with\nthe Front-Cover Texts being LIST, and with the Back-Cover Texts\nbeing LIST.\n\nIf you have Invariant Sections without Cover Texts, or some other\ncombination of the three, merge those two alternatives to suit the\nsituation.\n\nIf your document contains nontrivial examples of program code, we\nrecommend releasing these examples in parallel under your choice of free\nsoftware license, such as the GNU General Public License, to permit\ntheir use in free software.\n\nFile: wget.info, Node: Concept Index, Prev: Copying this manual, Up: Top\n" } ] }, "Concept Index": { "content": "* Menu:\n\n* #wget: Internet Relay Chat. (line 6)\n* .css extension: HTTP Options. (line 10)\n* .html extension: HTTP Options. (line 10)\n* .listing files, removing: FTP Options. (line 21)\n* .netrc: Startup File. (line 6)\n* .wgetrc: Startup File. (line 6)\n* accept directories: Directory-Based Limits.\n(line 17)\n* accept suffixes: Types of Files. (line 15)\n* accept wildcards: Types of Files. (line 15)\n* append to log: Logging and Input File Options.\n(line 11)\n* arguments: Invoking. (line 6)\n* authentication: Download Options. (line 536)\n* authentication <1>: HTTP Options. (line 43)\n* authentication <2>: HTTP Options. (line 393)\n* authentication credentials: Download Options. (line 113)\n* backing up converted files: Recursive Retrieval Options.\n(line 103)\n* backing up files: Download Options. (line 107)\n* bandwidth, limit: Download Options. (line 330)\n* base for relative links in input file: Logging and Input File Options.\n(line 111)\n* bind address: Download Options. (line 6)\n* bind DNS address: Download Options. (line 11)\n* bug reports: Reporting Bugs. (line 6)\n* bugs: Reporting Bugs. (line 6)\n* cache: HTTP Options. (line 71)\n* caching of DNS lookups: Download Options. (line 415)\n* case fold: Recursive Accept/Reject Options.\n(line 62)\n* client DNS address: Download Options. (line 11)\n* client IP address: Download Options. (line 6)\n* clobbering, file: Download Options. (line 68)\n* command line: Invoking. (line 6)\n* comments, HTML: Recursive Retrieval Options.\n(line 181)\n* connect timeout: Download Options. (line 314)\n* Content On Error: HTTP Options. (line 380)\n* Content-Disposition: HTTP Options. (line 363)\n* Content-Encoding, choose: HTTP Options. (line 197)\n* Content-Length, ignore: HTTP Options. (line 160)\n* continue retrieval: Download Options. (line 118)\n* continue retrieval <1>: Download Options. (line 177)\n* contributors: Contributors. (line 6)\n* conversion of links: Recursive Retrieval Options.\n(line 45)\n* cookies: HTTP Options. (line 80)\n* cookies, loading: HTTP Options. (line 90)\n* cookies, saving: HTTP Options. (line 138)\n* cookies, session: HTTP Options. (line 143)\n* cut directories: Directory Options. (line 32)\n* debug: Logging and Input File Options.\n(line 17)\n* default page name: HTTP Options. (line 6)\n* delete after retrieval: Recursive Retrieval Options.\n(line 29)\n* directories: Directory-Based Limits.\n(line 6)\n* directories, exclude: Directory-Based Limits.\n(line 30)\n* directories, include: Directory-Based Limits.\n(line 17)\n* directory limits: Directory-Based Limits.\n(line 6)\n* directory prefix: Directory Options. (line 59)\n* DNS cache: Download Options. (line 415)\n* DNS IP address, client, DNS: Download Options. (line 11)\n* DNS IP address, client, DNS <1>: Download Options. (line 19)\n* DNS server: Download Options. (line 19)\n* DNS timeout: Download Options. (line 308)\n* dot style: Download Options. (line 189)\n* downloading multiple times: Download Options. (line 68)\n* EGD: HTTPS (SSL/TLS) Options.\n(line 142)\n* entropy, specifying source of: HTTPS (SSL/TLS) Options.\n(line 127)\n* examples: Examples. (line 6)\n* exclude directories: Directory-Based Limits.\n(line 30)\n* execute wgetrc command: Basic Startup Options.\n(line 19)\n* FDL, GNU Free Documentation License: GNU Free Documentation License.\n(line 6)\n* features: Overview. (line 6)\n* file names, restrict: Download Options. (line 434)\n* file permissions: FTP Options. (line 73)\n* filling proxy cache: Recursive Retrieval Options.\n(line 29)\n* follow FTP links: Recursive Accept/Reject Options.\n(line 34)\n* following ftp links: FTP Links. (line 6)\n* following links: Following Links. (line 6)\n* force html: Logging and Input File Options.\n(line 104)\n* ftp authentication: FTP Options. (line 6)\n* ftp password: FTP Options. (line 6)\n* ftp time-stamping: FTP Time-Stamping Internals.\n(line 6)\n* ftp user: FTP Options. (line 6)\n* globbing, toggle: FTP Options. (line 45)\n* hangup: Signals. (line 6)\n* header, add: HTTP Options. (line 171)\n* hosts, spanning: Spanning Hosts. (line 6)\n* HSTS: HTTPS (SSL/TLS) Options.\n(line 161)\n* HTML comments: Recursive Retrieval Options.\n(line 181)\n* http password: HTTP Options. (line 43)\n* http referer: HTTP Options. (line 229)\n* http time-stamping: HTTP Time-Stamping Internals.\n(line 6)\n* http user: HTTP Options. (line 43)\n* idn support: Download Options. (line 558)\n* ignore case: Recursive Accept/Reject Options.\n(line 62)\n* ignore length: HTTP Options. (line 160)\n* include directories: Directory-Based Limits.\n(line 17)\n* incomplete downloads: Download Options. (line 118)\n* incomplete downloads <1>: Download Options. (line 177)\n* incremental updating: Time-Stamping. (line 6)\n* index.html: HTTP Options. (line 6)\n* input-file: Logging and Input File Options.\n(line 46)\n* input-metalink: Logging and Input File Options.\n(line 69)\n* Internet Relay Chat: Internet Relay Chat. (line 6)\n* invoking: Invoking. (line 6)\n* IP address, client: Download Options. (line 6)\n* IPv6: Download Options. (line 484)\n* IRC: Internet Relay Chat. (line 6)\n* iri support: Download Options. (line 558)\n* Keep-Alive, turning off: HTTP Options. (line 59)\n* keep-badhash: Logging and Input File Options.\n(line 73)\n* latest version: Distribution. (line 6)\n* limit bandwidth: Download Options. (line 330)\n* link conversion: Recursive Retrieval Options.\n(line 45)\n* links: Following Links. (line 6)\n* list: Mailing Lists. (line 5)\n* loading cookies: HTTP Options. (line 90)\n* local encoding: Download Options. (line 567)\n* location of wgetrc: Wgetrc Location. (line 6)\n* log file: Logging and Input File Options.\n(line 6)\n* mailing list: Mailing Lists. (line 6)\n* metalink-index: Logging and Input File Options.\n(line 85)\n* metalink-over-http: Logging and Input File Options.\n(line 78)\n* mirroring: Very Advanced Usage. (line 6)\n* no parent: Directory-Based Limits.\n(line 43)\n* no-clobber: Download Options. (line 68)\n* nohup: Invoking. (line 6)\n* number of tries: Download Options. (line 26)\n* offset: Download Options. (line 177)\n* operating systems: Portability. (line 6)\n* option syntax: Option Syntax. (line 6)\n* Other HTTP Methods: HTTP Options. (line 330)\n* output file: Logging and Input File Options.\n(line 6)\n* overview: Overview. (line 6)\n* page requisites: Recursive Retrieval Options.\n(line 116)\n* passive ftp: FTP Options. (line 61)\n* password: Download Options. (line 536)\n* pause: Download Options. (line 350)\n* Persistent Connections, disabling: HTTP Options. (line 59)\n* portability: Portability. (line 6)\n* POST: HTTP Options. (line 262)\n* preferred-location: Logging and Input File Options.\n(line 93)\n* progress indicator: Download Options. (line 189)\n* proxies: Proxies. (line 6)\n* proxy: Download Options. (line 391)\n* proxy <1>: HTTP Options. (line 71)\n* proxy authentication: HTTP Options. (line 220)\n* proxy filling: Recursive Retrieval Options.\n(line 29)\n* proxy password: HTTP Options. (line 220)\n* proxy user: HTTP Options. (line 220)\n* quiet: Logging and Input File Options.\n(line 28)\n* quota: Download Options. (line 398)\n* random wait: Download Options. (line 373)\n* randomness, specifying source of: HTTPS (SSL/TLS) Options.\n(line 127)\n* rate, limit: Download Options. (line 330)\n* read timeout: Download Options. (line 319)\n* recursion: Recursive Download. (line 6)\n* recursive download: Recursive Download. (line 6)\n* redirect: HTTP Options. (line 214)\n* redirecting output: Advanced Usage. (line 89)\n* referer, http: HTTP Options. (line 229)\n* reject directories: Directory-Based Limits.\n(line 30)\n* reject suffixes: Types of Files. (line 39)\n* reject wildcards: Types of Files. (line 39)\n* relative links: Relative Links. (line 6)\n* remote encoding: Download Options. (line 581)\n* reporting bugs: Reporting Bugs. (line 6)\n* required images, downloading: Recursive Retrieval Options.\n(line 116)\n* resume download: Download Options. (line 118)\n* resume download <1>: Download Options. (line 177)\n* retries: Download Options. (line 26)\n* retries, waiting between: Download Options. (line 364)\n* retrieving: Recursive Download. (line 6)\n* robot exclusion: Robot Exclusion. (line 6)\n* robots.txt: Robot Exclusion. (line 6)\n* sample wgetrc: Sample Wgetrc. (line 6)\n* saving cookies: HTTP Options. (line 138)\n* security: Security Considerations.\n(line 6)\n* server maintenance: Robot Exclusion. (line 6)\n* server response, print: Download Options. (line 274)\n* server response, save: HTTP Options. (line 236)\n* session cookies: HTTP Options. (line 143)\n* signal handling: Signals. (line 6)\n* spanning hosts: Spanning Hosts. (line 6)\n* specify config: Logging and Input File Options.\n(line 124)\n* spider: Download Options. (line 279)\n* SSL: HTTPS (SSL/TLS) Options.\n(line 6)\n* SSL certificate: HTTPS (SSL/TLS) Options.\n(line 73)\n* SSL certificate authority: HTTPS (SSL/TLS) Options.\n(line 99)\n* SSL certificate type, specify: HTTPS (SSL/TLS) Options.\n(line 79)\n* SSL certificate, check: HTTPS (SSL/TLS) Options.\n(line 44)\n* SSL CRL, certificate revocation list: HTTPS (SSL/TLS) Options.\n(line 111)\n* SSL protocol, choose: HTTPS (SSL/TLS) Options.\n(line 11)\n* SSL Public Key Pin: HTTPS (SSL/TLS) Options.\n(line 115)\n* start position: Download Options. (line 177)\n* startup: Startup File. (line 6)\n* startup file: Startup File. (line 6)\n* suffixes, accept: Types of Files. (line 15)\n* suffixes, reject: Types of Files. (line 39)\n* symbolic links, retrieving: FTP Options. (line 77)\n* syntax of options: Option Syntax. (line 6)\n* syntax of wgetrc: Wgetrc Syntax. (line 6)\n* tag-based recursive pruning: Recursive Accept/Reject Options.\n(line 38)\n* time-stamping: Time-Stamping. (line 6)\n* time-stamping usage: Time-Stamping Usage. (line 6)\n* timeout: Download Options. (line 290)\n* timeout, connect: Download Options. (line 314)\n* timeout, DNS: Download Options. (line 308)\n* timeout, read: Download Options. (line 319)\n* timestamping: Time-Stamping. (line 6)\n* tries: Download Options. (line 26)\n* Trust server names: HTTP Options. (line 385)\n* types of files: Types of Files. (line 6)\n* unlink: Download Options. (line 596)\n* updating the archives: Time-Stamping. (line 6)\n* URL: URL Format. (line 6)\n* URL syntax: URL Format. (line 6)\n* usage, time-stamping: Time-Stamping Usage. (line 6)\n* user: Download Options. (line 536)\n* user-agent: HTTP Options. (line 240)\n* various: Various. (line 6)\n* verbose: Logging and Input File Options.\n(line 32)\n* wait: Download Options. (line 350)\n* wait, random: Download Options. (line 373)\n* waiting between retries: Download Options. (line 364)\n* WARC: HTTPS (SSL/TLS) Options.\n(line 240)\n* web site: Web Site. (line 6)\n* Wget as spider: Download Options. (line 279)\n* wgetrc: Startup File. (line 6)\n* wgetrc commands: Wgetrc Commands. (line 6)\n* wgetrc location: Wgetrc Location. (line 6)\n* wgetrc syntax: Wgetrc Syntax. (line 6)\n* wildcards, accept: Types of Files. (line 15)\n* wildcards, reject: Types of Files. (line 39)\n* Windows file names: Download Options. (line 434)\n* xattr: Logging and Input File Options.\n(line 97)\n\n", "subsections": [] } } } }