{
    "content": [
        {
            "type": "text",
            "text": "# qsfilter2(1) (man)\n\n**Summary:** qsfilter2  - an utility to generate modqos request line rules out from existing access/audit log data.\n\n**Synopsis:** qsfilter2 -i <path> [-c <path>] [-d <num>] [-h] [-b <num>] [-p|-s|-m|-o] [-l <len>] [-n] [-e]\n[-u 'uni'] [-k <prefix>] [-t] [-f <path>] [-v 0|1|2]\n\n## Flags\n\n| Flag | Long | Arg | Description |\n|------|------|-----|-------------|\n| -i | — | <path> | Input file containing request URIs. The URIs for this file have to be extracted from the servers access logs. Each line  |\n| -c | — | <path> | modqos configuration file defining QSDenyRequestLine and QSPermitUri directives. qsfilter2 generates rules from access l |\n| -d | — | <num> | Depth (sub locations) of the path string which is defined as a literal string. Default is 1. -h Always use a string repr |\n| -b | — | <num> | Replaces url pattern by the regular expression when detecting a base64/hex encoded string. Detecting sensibility is defi |\n| -l | — | <len> | Outsizes the query length by the defined length ({0,size+len}), default is 10. -n Disables redundant rules elimination.  |\n| -k | — | <prefix> | Prefix used to generate rule identifiers (QSF by default). -t Calculates the maximal latency per request (worst case) us |\n| -f | — | <path> | Filters the input by the provided path (prefix) only processing matching lines. |\n| -v | — | <level> | Verbose mode. (0=silent, 1=rule source, 2=detailed). Default is 1. Don't use rules you haven't checked the request data  |\n\n## Examples\n\n- `qsfilter2 -i loc.txt -c httpd.conf -m -e`\n- `...`\n- `# ADD line 1: /aaa/index.do`\n- `# 003 ^(/[a-zA-Z0-9\\-]+)+[/]?\\.?[a-zA-Z]{0,4}$`\n- `# ADD line 3: /aaa/view?page=1`\n- `# --- ^[/a-zA-Z0-9]+/view\\?(page=[0-9]+)?$`\n- `# ADD line 4: /aaa/edit?document=1`\n- `# 004 ^[/a-zA-Z]+/edit\\?((document)(=[0-9]*)*[&]?)*$`\n- `# ADD line 5: /aaa/edit?image=1.jpg`\n- `# 005 ^[/a-zA-Z]+/edit\\?((image)(=[0-9\\.a-zA-Z]*)*[&]?)*$`\n- `...`\n- `QSPermitUri +QSF001 deny \"^[/a-zA-Z]+/edit\\?((document|image)(=[0-9\\.a-zA-Z]*)*[&]?)*$\"`\n- `QSPermitUri +QSF002 deny \"^[/a-zA-Z0-9]+/view\\?(page=[0-9]+)?$\"`\n- `QSPermitUri +QSF003 deny \"^(/[a-zA-Z0-9\\-]+)+[/]?\\.?[a-zA-Z]{0,4}$\"`\n\n## See Also\n\n- qsdt(1)\n- qsexec(1)\n- qsgeo(1)\n- qsgrep(1)\n- qshead(1)\n- qslog(1)\n- qslogger(1)\n- qspng(1)\n- qsre(1)\n- qsrespeed(1)\n- qsrotate(1)\n- qssign(1)\n- qstail(1)\n\n## Section Outline\n\n- **NAME** (3 lines)\n- **SYNOPSIS** (3 lines)\n- **DESCRIPTION** (14 lines)\n- **OPTIONS** (1 lines) — 8 subsections\n  - -i <path> (17 lines)\n  - -c <path> (14 lines)\n  - -d <num> (6 lines)\n  - -b <num> (12 lines)\n  - -l <len> (10 lines)\n  - -k <prefix> (4 lines)\n  - -f <path> (2 lines)\n  - -v <level> (4 lines)\n- **OUTPUT** (10 lines)\n- **EXAMPLE** (16 lines)\n- **SEE ALSO** (3 lines)\n- **AUTHOR** (5 lines)\n\n## Full Content\n\n### NAME\n\nqsfilter2  - an utility to generate modqos request line rules out from existing access/audit\nlog data.\n\n### SYNOPSIS\n\nqsfilter2 -i <path> [-c <path>] [-d <num>] [-h] [-b <num>] [-p|-s|-m|-o] [-l <len>] [-n] [-e]\n[-u 'uni'] [-k <prefix>] [-t] [-f <path>] [-v 0|1|2]\n\n### DESCRIPTION\n\nmodqos  implements  a  request filter which validates each request line. The module supports\nboth, negative and positive security model. The QSDeny* directives are used to  specify  re‐\nquest  line  patterns  which  are not allowed to access the server (negative security model /\nblacklist). These rules are used to restrict access to certain resources which should not  be\navailable to users or to protect the server from malicious patterns. The QSPermit* rules im‐\nplement a positive security model (whitelist). These directives are used  to  define  allowed\nrequest  line  patterns.  Request which do not match any of these patterns are not allowed to\naccess the server.\n\nqsfilter2 is an audit log analyzer used to generate filter rules (perl compatible regular ex‐\npressions)  which  may  be  used by modqos to deny access for suspect requests (QSPermitUri\nrules). It parses existing audit log files in order to generate request patterns covering all\nallowed requests.\n\n### OPTIONS\n\n#### -i <path>\n\nInput  file  containing request URIs. The URIs for this file have to be extracted from\nthe servers access logs. Each line of the input file contains a request URI  consiting\nof a path and and query.\nExample:\n/aaa/index.do\n/aaa/edit?image=1.jpg\n/aaa/image/1.jpg\n/aaa/view?page=1\n/aaa/edit?document=1\n\nThese  access  log  data must include current request URIs but also request lines from\nprevious rule generation steps. It must also include request lines which  cover  manu‐\nally  generated  rules. You may use the 'qos-path' and 'qos-query' variables to create\nan audit log containing all request data (path and query/body data). Example: 'Custom‐\nLog  auditlog %{qos-path}n%{qos-query}n'. See also http://mod-qos.sourceforge.net#qs‐\nfiltersample about the module settings.\n\n#### -c <path>\n\nmodqos configuration file defining QSDenyRequestLine  and  QSPermitUri  directives.\nqsfilter2 generates rules from access log data automatically. Manually generated rules\n(QSPermitUri) may be provided from this file. Note: each manual rule must  be  repre‐\nsented by a request URI in the input data (-i) in order to make sure not to be deleted\nby the rule optimisation algorithm. QSDeny* rules from this file are used  to  filter\nrequest lines which should not be used for whitelist rule generation.\nExample:\n# manually defined whitelist rule:\nQSPermitUri +view deny \"^[/a-zA-Z0-9]+/view\\?(page=[0-9]+)?$\"\n# filter unwanted request line patterns:\nQSDenyRequestLine +printable deny \".*[\\x00-\\x19].*\"\n\n#### -d <num>\n\nDepth (sub locations) of the path string which is defined as a literal string. Default\nis 1.\n\n-h     Always use a string representing the handler name in the path even the  url  does  not\nhave a query. See also -d option.\n\n#### -b <num>\n\nReplaces  url  pattern  by  the regular expression when detecting a base64/hex encoded\nstring. Detecting sensibility is defined by a numeric value.  You  should  use  values\nhigher than 5 (default) or 0 to disable this function.\n\n-p     Represents query by pcre only (no literal strings).\n\n-s     Uses one single pcre for the whole query string.\n\n-m     Uses one pcre for multiple query values (recommended mode).\n\n-o     Does not care the order of query parameters.\n\n#### -l <len>\n\nOutsizes the query length by the defined length ({0,size+len}), default is 10.\n\n-n     Disables redundant rules elimination.\n\n-e     Exit on error.\n\n-u 'uni'\nEnables  additional  decoding  methods. Use the same settings as you have used for the\nQSDecoding directive.\n\n#### -k <prefix>\n\nPrefix used to generate rule identifiers (QSF by default).\n\n-t     Calculates the maximal latency per request (worst case) using the generated rules.\n\n#### -f <path>\n\nFilters the input by the provided path (prefix) only processing matching lines.\n\n#### -v <level>\n\nVerbose mode. (0=silent, 1=rule source, 2=detailed). Default is 1. Don't use rules you\nhaven't  checked  the  request data used to generate it! Level 1 is highly recommended\n(as long as you don't have created the log data using your own web crawler).\n\n### OUTPUT\n\nThe output of qsfilter2 is written to stdout. The output contains the generated  QSPermitUri\ndirectives but also information about the source which has been used to generate these rules.\nIt is very important to check the validity of each request line which has been used to calcu‐\nlate  the QSPermitUri rules. Each request line which has been used to generate a new rule is\nshown in the output prefixed by \"ADD line <line number>:\".  These  request  lines  should  be\nstored  and  reused at any later rule generation (add them to the URI input file). The subse‐\nquent line shows the generated rule. At the end of data processing a list  of  all  generated\nQSPermitUri  rules  is shown. These directives may be used withn the configuration file used\nby modqos.\n\n### EXAMPLE\n\nqsfilter2 -i loc.txt -c httpd.conf -m -e\n...\n# ADD line 1: /aaa/index.do\n# 003 ^(/[a-zA-Z0-9\\-]+)+[/]?\\.?[a-zA-Z]{0,4}$\n# ADD line 3: /aaa/view?page=1\n# --- ^[/a-zA-Z0-9]+/view\\?(page=[0-9]+)?$\n# ADD line 4: /aaa/edit?document=1\n# 004 ^[/a-zA-Z]+/edit\\?((document)(=[0-9]*)*[&]?)*$\n# ADD line 5: /aaa/edit?image=1.jpg\n# 005 ^[/a-zA-Z]+/edit\\?((image)(=[0-9\\.a-zA-Z]*)*[&]?)*$\n...\nQSPermitUri +QSF001 deny \"^[/a-zA-Z]+/edit\\?((document|image)(=[0-9\\.a-zA-Z]*)*[&]?)*$\"\nQSPermitUri +QSF002 deny \"^[/a-zA-Z0-9]+/view\\?(page=[0-9]+)?$\"\nQSPermitUri +QSF003 deny \"^(/[a-zA-Z0-9\\-]+)+[/]?\\.?[a-zA-Z]{0,4}$\"\n\n### SEE ALSO\n\nqsdt(1), qsexec(1), qsgeo(1), qsgrep(1), qshead(1), qslog(1), qslogger(1), qspng(1), qsre(1),\nqsrespeed(1), qsrotate(1), qssign(1), qstail(1)\n\n### AUTHOR\n\nPascal Buchbinder, http://mod-qos.sourceforge.net/\n\n\n\nmodqos utilities 11.63                       May 2019                                  QSFILTER2(1)\n\n"
        }
    ],
    "structuredContent": {
        "command": "qsfilter2",
        "section": "1",
        "mode": "man",
        "summary": "qsfilter2  - an utility to generate modqos request line rules out from existing access/audit log data.",
        "synopsis": "qsfilter2 -i <path> [-c <path>] [-d <num>] [-h] [-b <num>] [-p|-s|-m|-o] [-l <len>] [-n] [-e]\n[-u 'uni'] [-k <prefix>] [-t] [-f <path>] [-v 0|1|2]",
        "tldr_summary": null,
        "tldr_examples": [],
        "tldr_source": null,
        "flags": [
            {
                "flag": "-i",
                "long": null,
                "arg": "<path>",
                "description": "Input file containing request URIs. The URIs for this file have to be extracted from the servers access logs. Each line of the input file contains a request URI consiting of a path and and query. Example: /aaa/index.do /aaa/edit?image=1.jpg /aaa/image/1.jpg /aaa/view?page=1 /aaa/edit?document=1 These access log data must include current request URIs but also request lines from previous rule generation steps. It must also include request lines which cover manu‐ ally generated rules. You may use the 'qos-path' and 'qos-query' variables to create an audit log containing all request data (path and query/body data). Example: 'Custom‐ Log auditlog %{qos-path}n%{qos-query}n'. See also http://mod-qos.sourceforge.net#qs‐ filtersample about the module settings."
            },
            {
                "flag": "-c",
                "long": null,
                "arg": "<path>",
                "description": "modqos configuration file defining QSDenyRequestLine and QSPermitUri directives. qsfilter2 generates rules from access log data automatically. Manually generated rules (QSPermitUri) may be provided from this file. Note: each manual rule must be repre‐ sented by a request URI in the input data (-i) in order to make sure not to be deleted by the rule optimisation algorithm. QSDeny* rules from this file are used to filter request lines which should not be used for whitelist rule generation. Example: # manually defined whitelist rule: QSPermitUri +view deny \"^[/a-zA-Z0-9]+/view\\?(page=[0-9]+)?$\" # filter unwanted request line patterns: QSDenyRequestLine +printable deny \".*[\\x00-\\x19].*\""
            },
            {
                "flag": "-d",
                "long": null,
                "arg": "<num>",
                "description": "Depth (sub locations) of the path string which is defined as a literal string. Default is 1. -h Always use a string representing the handler name in the path even the url does not have a query. See also -d option."
            },
            {
                "flag": "-b",
                "long": null,
                "arg": "<num>",
                "description": "Replaces url pattern by the regular expression when detecting a base64/hex encoded string. Detecting sensibility is defined by a numeric value. You should use values higher than 5 (default) or 0 to disable this function. -p Represents query by pcre only (no literal strings). -s Uses one single pcre for the whole query string. -m Uses one pcre for multiple query values (recommended mode). -o Does not care the order of query parameters."
            },
            {
                "flag": "-l",
                "long": null,
                "arg": "<len>",
                "description": "Outsizes the query length by the defined length ({0,size+len}), default is 10. -n Disables redundant rules elimination. -e Exit on error. -u 'uni' Enables additional decoding methods. Use the same settings as you have used for the QSDecoding directive."
            },
            {
                "flag": "-k",
                "long": null,
                "arg": "<prefix>",
                "description": "Prefix used to generate rule identifiers (QSF by default). -t Calculates the maximal latency per request (worst case) using the generated rules."
            },
            {
                "flag": "-f",
                "long": null,
                "arg": "<path>",
                "description": "Filters the input by the provided path (prefix) only processing matching lines."
            },
            {
                "flag": "-v",
                "long": null,
                "arg": "<level>",
                "description": "Verbose mode. (0=silent, 1=rule source, 2=detailed). Default is 1. Don't use rules you haven't checked the request data used to generate it! Level 1 is highly recommended (as long as you don't have created the log data using your own web crawler)."
            }
        ],
        "examples": [
            "qsfilter2 -i loc.txt -c httpd.conf -m -e",
            "...",
            "# ADD line 1: /aaa/index.do",
            "# 003 ^(/[a-zA-Z0-9\\-]+)+[/]?\\.?[a-zA-Z]{0,4}$",
            "# ADD line 3: /aaa/view?page=1",
            "# --- ^[/a-zA-Z0-9]+/view\\?(page=[0-9]+)?$",
            "# ADD line 4: /aaa/edit?document=1",
            "# 004 ^[/a-zA-Z]+/edit\\?((document)(=[0-9]*)*[&]?)*$",
            "# ADD line 5: /aaa/edit?image=1.jpg",
            "# 005 ^[/a-zA-Z]+/edit\\?((image)(=[0-9\\.a-zA-Z]*)*[&]?)*$",
            "...",
            "QSPermitUri +QSF001 deny \"^[/a-zA-Z]+/edit\\?((document|image)(=[0-9\\.a-zA-Z]*)*[&]?)*$\"",
            "QSPermitUri +QSF002 deny \"^[/a-zA-Z0-9]+/view\\?(page=[0-9]+)?$\"",
            "QSPermitUri +QSF003 deny \"^(/[a-zA-Z0-9\\-]+)+[/]?\\.?[a-zA-Z]{0,4}$\""
        ],
        "see_also": [
            {
                "name": "qsdt",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qsdt/1/json"
            },
            {
                "name": "qsexec",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qsexec/1/json"
            },
            {
                "name": "qsgeo",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qsgeo/1/json"
            },
            {
                "name": "qsgrep",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qsgrep/1/json"
            },
            {
                "name": "qshead",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qshead/1/json"
            },
            {
                "name": "qslog",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qslog/1/json"
            },
            {
                "name": "qslogger",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qslogger/1/json"
            },
            {
                "name": "qspng",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qspng/1/json"
            },
            {
                "name": "qsre",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qsre/1/json"
            },
            {
                "name": "qsrespeed",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qsrespeed/1/json"
            },
            {
                "name": "qsrotate",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qsrotate/1/json"
            },
            {
                "name": "qssign",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qssign/1/json"
            },
            {
                "name": "qstail",
                "section": "1",
                "url": "https://www.chedong.com/phpMan.php/man/qstail/1/json"
            }
        ],
        "section_outline": [
            {
                "name": "NAME",
                "lines": 3,
                "subsections": []
            },
            {
                "name": "SYNOPSIS",
                "lines": 3,
                "subsections": []
            },
            {
                "name": "DESCRIPTION",
                "lines": 14,
                "subsections": []
            },
            {
                "name": "OPTIONS",
                "lines": 1,
                "subsections": [
                    {
                        "name": "-i <path>",
                        "lines": 17,
                        "flag": "-i",
                        "arg": "<path>"
                    },
                    {
                        "name": "-c <path>",
                        "lines": 14,
                        "flag": "-c",
                        "arg": "<path>"
                    },
                    {
                        "name": "-d <num>",
                        "lines": 6,
                        "flag": "-d",
                        "arg": "<num>"
                    },
                    {
                        "name": "-b <num>",
                        "lines": 12,
                        "flag": "-b",
                        "arg": "<num>"
                    },
                    {
                        "name": "-l <len>",
                        "lines": 10,
                        "flag": "-l",
                        "arg": "<len>"
                    },
                    {
                        "name": "-k <prefix>",
                        "lines": 4,
                        "flag": "-k",
                        "arg": "<prefix>"
                    },
                    {
                        "name": "-f <path>",
                        "lines": 2,
                        "flag": "-f",
                        "arg": "<path>"
                    },
                    {
                        "name": "-v <level>",
                        "lines": 4,
                        "flag": "-v",
                        "arg": "<level>"
                    }
                ]
            },
            {
                "name": "OUTPUT",
                "lines": 10,
                "subsections": []
            },
            {
                "name": "EXAMPLE",
                "lines": 16,
                "subsections": []
            },
            {
                "name": "SEE ALSO",
                "lines": 3,
                "subsections": []
            },
            {
                "name": "AUTHOR",
                "lines": 5,
                "subsections": []
            }
        ]
    }
}