man > Expr(3pm)

Found in /usr/share/perl/5.38/pod/perlfaq6.pod
  How can I hope to use regular expressions without creating illegible and unmaintainable code?
    Three techniques can make regular expressions maintainable and
    understandable.

    Comments Outside the Regex
        Describe what you're doing and how you're doing it, using normal
        Perl comments.

            # turn the line into the first word, a colon, and the
            # number of characters on the rest of the line
            s/^(\w+)(.*)/ lc($1) . ":" . length($2) /meg;

    Comments Inside the Regex
        The "/x" modifier causes whitespace to be ignored in a regex pattern
        (except in a character class and a few other places), and also
        allows you to use normal comments there, too. As you can imagine,
        whitespace and comments help a lot.

        "/x" lets you turn this:

            s{<(?:[^>'"]*|".*?"|'.*?')+>}{}gs;

        into this:

            s{ <                    # opening angle bracket
                (?:                 # Non-backreffing grouping paren
                    [^>'"] *        # 0 or more things that are neither > nor ' nor "
                        |           #    or else
                    ".*?"           # a section between double quotes (stingy match)
                        |           #    or else
                    '.*?'           # a section between single quotes (stingy match)
                ) +                 #   all occurring one or more times
                >                   # closing angle bracket
            }{}gsx;                 # replace with nothing, i.e. delete

        It's still not quite so clear as prose, but it is very useful for
        describing the meaning of each part of the pattern.

    Different Delimiters
        While we normally think of patterns as being delimited with "/"
        characters, they can be delimited by almost any character. perlre
        describes this. For example, the "s///" above uses braces as
        delimiters. Selecting another delimiter can avoid quoting the
        delimiter within the pattern:

            s/\/usr\/local/\/usr\/share/g;    # bad delimiter choice
            s#/usr/local#/usr/share#g;        # better

        Using logically paired delimiters can be even more readable:

            s{/usr/local/}{/usr/share}g;      # better still

  I put a regular expression into $/ but it didn't work. What's wrong?
    $/ has to be a string. You can use these examples if you really need to
    do this.

    If you have File::Stream, this is easy.

        use File::Stream;

        my $stream = File::Stream->new(
            $filehandle,
            separator => qr/\s*,\s*/,
            );

        print "$_\n" while <$stream>;

    If you don't have File::Stream, you have to do a little more work.

    You can use the four-argument form of sysread to continually add to a
    buffer. After you add to the buffer, you check if you have a complete
    line (using your regular expression).

        local $_ = "";
        while( sysread FH, $_, 8192, length ) {
            while( s/^((?s).*?)your_pattern// ) {
                my $record = $1;
                # do stuff here.
            }
        }

    You can do the same thing with foreach and a match using the c flag and
    the \G anchor, if you do not mind your entire file being in memory at
    the end.

        local $_ = "";
        while( sysread FH, $_, 8192, length ) {
            foreach my $record ( m/\G((?s).*?)your_pattern/gc ) {
                # do stuff here.
            }
            substr( $_, 0, pos ) = "" if pos;
        }

  How do I use a regular expression to strip C-style comments from a file?
    While this actually can be done, it's much harder than you'd think. For
    example, this one-liner

        perl -0777 -pe 's{/\*.*?\*/}{}gs' foo.c

    will work in many but not all cases. You see, it's too simple-minded for
    certain kinds of C programs, in particular, those with what appear to be
    comments in quoted strings. For that, you'd need something like this,
    created by Jeffrey Friedl and later modified by Fred Curtis.

        $/ = undef;
        $_ = <>;
        s#/\*[^*]*\*+([^/*][^*]*\*+)*/|("(\\.|[^"\\])*"|'(\\.|[^'\\])*'|.[^/"'\\]*)#defined $2 ? $2 : ""#gse;
        print;

    This could, of course, be more legibly written with the "/x" modifier,
    adding whitespace and comments. Here it is expanded, courtesy of Fred
    Curtis.

        s{
           /\*         ##  Start of /* ... */ comment
           [^*]*\*+    ##  Non-* followed by 1-or-more *'s
           (
             [^/*][^*]*\*+
           )*          ##  0-or-more things which don't start with /
                       ##    but do end with '*'
           /           ##  End of /* ... */ comment

         |         ##     OR  various things which aren't comments:

           (
             "           ##  Start of " ... " string
             (
               \\.           ##  Escaped char
             |               ##    OR
               [^"\\]        ##  Non "\
             )*
             "           ##  End of " ... " string

           |         ##     OR

             '           ##  Start of ' ... ' string
             (
               \\.           ##  Escaped char
             |               ##    OR
               [^'\\]        ##  Non '\
             )*
             '           ##  End of ' ... ' string

           |         ##     OR

             .           ##  Anything other char
             [^/"'\\]*   ##  Chars which doesn't start a comment, string or escape
           )
         }{defined $2 ? $2 : ""}gxse;

    A slight modification also removes C++ comments, possibly spanning
    multiple lines using a continuation character:

     s#/\*[^*]*\*+([^/*][^*]*\*+)*/|//([^\\]|[^\n][\n]?)*?\n|("(\\.|[^"\\])*"|'(\\.|[^'\\])*'|.[^/"'\\]*)#defined $3 ? $3 : ""#gse;

  Can I use Perl regular expressions to match balanced text?
    (contributed by brian d foy)

    Your first try should probably be the Text::Balanced module, which is in
    the Perl standard library since Perl 5.8. It has a variety of functions
    to deal with tricky text. The Regexp::Common module can also help by
    providing canned patterns you can use.

    As of Perl 5.10, you can match balanced text with regular expressions
    using recursive patterns. Before Perl 5.10, you had to resort to various
    tricks such as using Perl code in "(??{})" sequences.

    Here's an example using a recursive regular expression. The goal is to
    capture all of the text within angle brackets, including the text in
    nested angle brackets. This sample text has two "major" groups: a group
    with one level of nesting and a group with two levels of nesting. There
    are five total groups in angle brackets:

        I have some <brackets in <nested brackets> > and
        <another group <nested once <nested twice> > >
        and that's it.

    The regular expression to match the balanced text uses two new (to Perl
    5.10) regular expression features. These are covered in perlre and this
    example is a modified version of one in that documentation.

    First, adding the new possessive "+" to any quantifier finds the longest
    match and does not backtrack. That's important since you want to handle
    any angle brackets through the recursion, not backtracking. The group
    "[^<>]++" finds one or more non-angle brackets without backtracking.

    Second, the new "(?PARNO)" refers to the sub-pattern in the particular
    capture group given by "PARNO". In the following regex, the first
    capture group finds (and remembers) the balanced text, and you need that
    same pattern within the first buffer to get past the nested text. That's
    the recursive part. The "(?1)" uses the pattern in the outer capture
    group as an independent part of the regex.

    Putting it all together, you have:

        #!/usr/local/bin/perl5.10.0

        my $string =<<"HERE";
        I have some <brackets in <nested brackets> > and
        <another group <nested once <nested twice> > >
        and that's it.
        HERE

        my @groups = $string =~ m/
                (                   # start of capture group 1
                <                   # match an opening angle bracket
                    (?:
                        [^<>]++     # one or more non angle brackets, non backtracking
                          |
                        (?1)        # found < or >, so recurse to capture group 1
                    )*
                >                   # match a closing angle bracket
                )                   # end of capture group 1
                /xg;

        $" = "\n\t";
        print "Found:\n\t@groups\n";

    The output shows that Perl found the two major groups:

        Found:
            <brackets in <nested brackets> >
            <another group <nested once <nested twice> > >

    With a little extra work, you can get all of the groups in angle
    brackets even if they are in other angle brackets too. Each time you get
    a balanced match, remove its outer delimiter (that's the one you just
    matched so don't match it again) and add it to a queue of strings to
    process. Keep doing that until you get no matches:

        #!/usr/local/bin/perl5.10.0

        my @queue =<<"HERE";
        I have some <brackets in <nested brackets> > and
        <another group <nested once <nested twice> > >
        and that's it.
        HERE

        my $regex = qr/
                (                   # start of bracket 1
                <                   # match an opening angle bracket
                    (?:
                        [^<>]++     # one or more non angle brackets, non backtracking
                          |
                        (?1)        # recurse to bracket 1
                    )*
                >                   # match a closing angle bracket
                )                   # end of bracket 1
                /x;

        $" = "\n\t";

        while( @queue ) {
            my $string = shift @queue;

            my @groups = $string =~ m/$regex/g;
            print "Found:\n\t@groups\n\n" if @groups;

            unshift @queue, map { s/^<//; s/>$//; $_ } @groups;
        }

    The output shows all of the groups. The outermost matches show up first
    and the nested matches show up later:

        Found:
            <brackets in <nested brackets> >
            <another group <nested once <nested twice> > >

        Found:
            <nested brackets>

        Found:
            <nested once <nested twice> >

        Found:
            <nested twice>

  How do I efficiently match many regular expressions at once?
    (contributed by brian d foy)

    You want to avoid compiling a regular expression every time you want to
    match it. In this example, perl must recompile the regular expression
    for every iteration of the "foreach" loop since $pattern can change:

        my @patterns = qw( fo+ ba[rz] );

        LINE: while( my $line = <> ) {
            foreach my $pattern ( @patterns ) {
                if( $line =~ m/\b$pattern\b/i ) {
                    print $line;
                    next LINE;
                }
            }
        }

    The "qr//" operator compiles a regular expression, but doesn't apply it.
    When you use the pre-compiled version of the regex, perl does less work.
    In this example, I inserted a "map" to turn each pattern into its
    pre-compiled form. The rest of the script is the same, but faster:

        my @patterns = map { qr/\b$_\b/i } qw( fo+ ba[rz] );

        LINE: while( my $line = <> ) {
            foreach my $pattern ( @patterns ) {
                if( $line =~ m/$pattern/ ) {
                    print $line;
                    next LINE;
                }
            }
        }

    In some cases, you may be able to make several patterns into a single
    regular expression. Beware of situations that require backtracking
    though. In this example, the regex is only compiled once because $regex
    doesn't change between iterations:

        my $regex = join '|', qw( fo+ ba[rz] );

        while( my $line = <> ) {
            print if $line =~ m/\b(?:$regex)\b/i;
        }

    The function "list2re" in Data::Munge on CPAN can also be used to form a
    single regex that matches a list of literal strings (not regexes).

    For more details on regular expression efficiency, see *Mastering
    Regular Expressions* by Jeffrey Friedl. He explains how the regular
    expressions engine works and why some patterns are surprisingly
    inefficient. Once you understand how perl applies regular expressions,
    you can tune them for individual situations.

  What good is "\G" in a regular expression?
    You use the "\G" anchor to start the next match on the same string where
    the last match left off. The regular expression engine cannot skip over
    any characters to find the next match with this anchor, so "\G" is
    similar to the beginning of string anchor, "^". The "\G" anchor is
    typically used with the "g" modifier. It uses the value of pos() as the
    position to start the next match. As the match operator makes successive
    matches, it updates pos() with the position of the next character past
    the last match (or the first character of the next match, depending on
    how you like to look at it). Each string has its own pos() value.

    Suppose you want to match all of consecutive pairs of digits in a string
    like "1122a44" and stop matching when you encounter non-digits. You want
    to match 11 and 22 but the letter "a" shows up between 22 and 44 and you
    want to stop at "a". Simply matching pairs of digits skips over the "a"
    and still matches 44.

        $_ = "1122a44";
        my @pairs = m/(\d\d)/g;   # qw( 11 22 44 )

    If you use the "\G" anchor, you force the match after 22 to start with
    the "a". The regular expression cannot match there since it does not
    find a digit, so the next match fails and the match operator returns the
    pairs it already found.

        $_ = "1122a44";
        my @pairs = m/\G(\d\d)/g; # qw( 11 22 )

    You can also use the "\G" anchor in scalar context. You still need the
    "g" modifier.

        $_ = "1122a44";
        while( m/\G(\d\d)/g ) {
            print "Found $1\n";
        }

    After the match fails at the letter "a", perl resets pos() and the next
    match on the same string starts at the beginning.

        $_ = "1122a44";
        while( m/\G(\d\d)/g ) {
            print "Found $1\n";
        }

        print "Found $1 after while" if m/(\d\d)/g; # finds "11"

    You can disable pos() resets on fail with the "c" modifier, documented
    in perlop and perlreref. Subsequent matches start where the last
    successful match ended (the value of pos()) even if a match on the same
    string has failed in the meantime. In this case, the match after the
    while() loop starts at the "a" (where the last match stopped), and since
    it does not use any anchor it can skip over the "a" to find 44.

        $_ = "1122a44";
        while( m/\G(\d\d)/gc ) {
            print "Found $1\n";
        }

        print "Found $1 after while" if m/(\d\d)/g; # finds "44"

    Typically you use the "\G" anchor with the "c" modifier when you want to
    try a different match if one fails, such as in a tokenizer. Jeffrey
    Friedl offers this example which works in 5.004 or later.

        while (<>) {
            chomp;
            PARSER: {
                m/ \G( \d+\b    )/gcx   && do { print "number: $1\n";  redo; };
                m/ \G( \w+      )/gcx   && do { print "word:   $1\n";  redo; };
                m/ \G( \s+      )/gcx   && do { print "space:  $1\n";  redo; };
                m/ \G( [^\w\d]+ )/gcx   && do { print "other:  $1\n";  redo; };
            }
        }

    For each line, the "PARSER" loop first tries to match a series of digits
    followed by a word boundary. This match has to start at the place the
    last match left off (or the beginning of the string on the first match).
    Since "m/ \G( \d+\b )/gcx" uses the "c" modifier, if the string does not
    match that regular expression, perl does not reset pos() and the next
    match starts at the same position to try a different pattern.

  How do I match a regular expression that's in a variable?
    (contributed by brian d foy)

    We don't have to hard-code patterns into the match operator (or anything
    else that works with regular expressions). We can put the pattern in a
    variable for later use.

    The match operator is a double quote context, so you can interpolate
    your variable just like a double quoted string. In this case, you read
    the regular expression as user input and store it in $regex. Once you
    have the pattern in $regex, you use that variable in the match operator.

        chomp( my $regex = <STDIN> );

        if( $string =~ m/$regex/ ) { ... }

    Any regular expression special characters in $regex are still special,
    and the pattern still has to be valid or Perl will complain. For
    instance, in this pattern there is an unpaired parenthesis.

        my $regex = "Unmatched ( paren";

        "Two parens to bind them all" =~ m/$regex/;

    When Perl compiles the regular expression, it treats the parenthesis as
    the start of a memory match. When it doesn't find the closing
    parenthesis, it complains:

        Unmatched ( in regex; marked by <-- HERE in m/Unmatched ( <-- HERE  paren/ at script line 3.

    You can get around this in several ways depending on our situation.
    First, if you don't want any of the characters in the string to be
    special, you can escape them with "quotemeta" before you use the string.

        chomp( my $regex = <STDIN> );
        $regex = quotemeta( $regex );

        if( $string =~ m/$regex/ ) { ... }

    You can also do this directly in the match operator using the "\Q" and
    "\E" sequences. The "\Q" tells Perl where to start escaping special
    characters, and the "\E" tells it where to stop (see perlop for more
    details).

        chomp( my $regex = <STDIN> );

        if( $string =~ m/\Q$regex\E/ ) { ... }

    Alternately, you can use "qr//", the regular expression quote operator
    (see perlop for more details). It quotes and perhaps compiles the
    pattern, and you can apply regular expression flags to the pattern.

        chomp( my $input = <STDIN> );

        my $regex = qr/$input/is;

        $string =~ m/$regex/  # same as m/$input/is;

    You might also want to trap any errors by wrapping an "eval" block
    around the whole thing.

        chomp( my $input = <STDIN> );

        eval {
            if( $string =~ m/\Q$input\E/ ) { ... }
        };
        warn $@ if $@;

    Or...

        my $regex = eval { qr/$input/is };
        if( defined $regex ) {
            $string =~ m/$regex/;
        }
        else {
            warn $@;
        }