{ "content": [ { "type": "text", "text": "# perlunicode (man)\n\n## NAME\n\nperlunicode - Unicode support in Perl\n\n## DESCRIPTION\n\nIf you haven't already, before reading this document, you should become familiar with both\nperlunitut and perluniintro.\n\n## Sections\n\n- **NAME**\n- **DESCRIPTION** (40 subsections)\n- **BUGS** (2 subsections)\n- **SEE ALSO**\n\nUse structuredContent.sections for detailed options, examples, and full documentation.\n" } ], "structuredContent": { "command": "perlunicode", "section": "", "mode": "man", "summary": "perlunicode - Unicode support in Perl", "synopsis": null, "tldr_summary": null, "tldr_examples": [], "tldr_source": null, "flags": [], "examples": [], "see_also": [], "section_outline": [ { "name": "NAME", "lines": 2, "subsections": [] }, { "name": "DESCRIPTION", "lines": 17, "subsections": [ { "name": "Important Caveats", "lines": 42 }, { "name": "Byte and Character Semantics", "lines": 78 }, { "name": "ASCII Rules versus Unicode Rules", "lines": 98 }, { "name": "Extended Grapheme Clusters (Logical characters)", "lines": 36 }, { "name": "Unicode Character Properties", "lines": 307 }, { "name": "\"\\p{All}\"", "lines": 5 }, { "name": "\"\\p{Alnum}\"", "lines": 2 }, { "name": "\"\\p{Any}\"", "lines": 3 }, { "name": "\"\\p{ASCII}\"", "lines": 3 }, { "name": "\"\\p{Assigned}\"", "lines": 3 }, { "name": "\"\\p{Blank}\"", "lines": 23 }, { "name": "\"\\p{Graph}\"", "lines": 3 }, { "name": "\"\\p{HorizSpace}\"", "lines": 6 }, { "name": "\"\\p{PerlSpace}\"", "lines": 5 }, { "name": "\"\\p{PerlWord}\"", "lines": 4 }, { "name": "\"\\p{Posix...}\"", "lines": 32 }, { "name": "\"\\p{Print}\"", "lines": 2 }, { "name": "\"\\p{SpacePerl}\"", "lines": 11 }, { "name": "\"\\p{Unicode}\"", "lines": 2 }, { "name": "\"\\p{VertSpace}\"", "lines": 2 }, { "name": "\"\\p{Word}\"", "lines": 2 }, { "name": "\"\\p{XPosix...}\"", "lines": 3 }, { "name": "Comparison of \"\\N{...}\" and \"\\p{name=...}\"", "lines": 31 }, { "name": "Wildcards in Property Values", "lines": 152 }, { "name": "User-Defined Character Properties", "lines": 119 }, { "name": "User-Defined Case Mappings (for serious hackers only)", "lines": 5 }, { "name": "Character Encodings for Input and Output", "lines": 2 }, { "name": "Unicode Regular Expression Support Level", "lines": 120 }, { "name": "Unicode Encodings", "lines": 133 }, { "name": "Noncharacter code points", "lines": 54 }, { "name": "Beyond Unicode code points", "lines": 90 }, { "name": "Security Implications of Unicode", "lines": 30 }, { "name": "Unicode in Perl on EBCDIC", "lines": 10 }, { "name": "Locales", "lines": 2 }, { "name": "When Unicode Does Not Happen", "lines": 28 }, { "name": "The \"Unicode Bug\"", "lines": 91 }, { "name": "Forcing Unicode in Perl (Or Unforcing Unicode in Perl)", "lines": 13 }, { "name": "Using Unicode in XS", "lines": 3 }, { "name": "Hacking Perl to work on earlier Unicode versions (for very serious hackers only)", "lines": 9 }, { "name": "Porting code from perl-5.6.X", "lines": 87 } ] }, { "name": "BUGS", "lines": 2, "subsections": [ { "name": "Interaction with Extensions", "lines": 55 }, { "name": "Speed", "lines": 12 } ] }, { "name": "SEE ALSO", "lines": 6, "subsections": [] } ], "sections": { "NAME": { "content": "perlunicode - Unicode support in Perl\n", "subsections": [] }, "DESCRIPTION": { "content": "If you haven't already, before reading this document, you should become familiar with both\nperlunitut and perluniintro.\n\nUnicode aims to UNI-fy the en-CODE-ings of all the world's character sets into a single\nStandard. For quite a few of the various coding standards that existed when Unicode was\nfirst created, converting from each to Unicode essentially meant adding a constant to each\ncode point in the original standard, and converting back meant just subtracting that same\nconstant. For ASCII and ISO-8859-1, the constant is 0. For ISO-8859-5, (Cyrillic) the\nconstant is 864; for Hebrew (ISO-8859-8), it's 1488; Thai (ISO-8859-11), 3424; and so forth.\nThis made it easy to do the conversions, and facilitated the adoption of Unicode.\n\nAnd it worked; nowadays, those legacy standards are rarely used. Most everyone uses Unicode.\n\nUnicode is a comprehensive standard. It specifies many things outside the scope of Perl,\nsuch as how to display sequences of characters. For a full discussion of all aspects of\nUnicode, see .\n", "subsections": [ { "name": "Important Caveats", "content": "Even though some of this section may not be understandable to you on first reading, we think\nit's important enough to highlight some of the gotchas before delving further, so here goes:\n\nUnicode support is an extensive requirement. While Perl does not implement the Unicode\nstandard or the accompanying technical reports from cover to cover, Perl does support many\nUnicode features.\n\nAlso, the use of Unicode may present security issues that aren't obvious, see \"Security\nImplications of Unicode\" below.\n\nSafest if you \"use feature 'unicodestrings'\"\nIn order to preserve backward compatibility, Perl does not turn on full internal Unicode\nsupport unless the pragma \"use feature 'unicodestrings'\" is specified. (This is\nautomatically selected if you \"use 5.012\" or higher.) Failure to do this can trigger\nunexpected surprises. See \"The \"Unicode Bug\"\" below.\n\nThis pragma doesn't affect I/O. Nor does it change the internal representation of\nstrings, only their interpretation. There are still several places where Unicode isn't\nfully supported, such as in filenames.\n\nInput and Output Layers\nUse the \":encoding(...)\" layer to read from and write to filehandles using the specified\nencoding. (See open.)\n\nYou must convert your non-ASCII, non-UTF-8 Perl scripts to be UTF-8.\nThe encoding module has been deprecated since perl 5.18 and the perl internals it\nrequires have been removed with perl 5.26.\n\n\"use utf8\" still needed to enable UTF-8 in scripts\nIf your Perl script is itself encoded in UTF-8, the \"use utf8\" pragma must be explicitly\nincluded to enable recognition of that (in string or regular expression literals, or in\nidentifier names). This is the only time when an explicit \"use  utf8\" is needed. (See\nutf8).\n\nIf a Perl script begins with the bytes that form the UTF-8 encoding of the Unicode BYTE\nORDER MARK (\"BOM\", see \"Unicode Encodings\"), those bytes are completely ignored.\n\nUTF-16 scripts autodetected\nIf a Perl script begins with the Unicode \"BOM\" (UTF-16LE, UTF16-BE), or if the script\nlooks like non-\"BOM\"-marked UTF-16 of either endianness, Perl will correctly read in the\nscript as the appropriate Unicode encoding.\n" }, { "name": "Byte and Character Semantics", "content": "Before Unicode, most encodings used 8 bits (a single byte) to encode each character. Thus a\ncharacter was a byte, and a byte was a character, and there could be only 256 or fewer\npossible characters. \"Byte Semantics\" in the title of this section refers to this behavior.\nThere was no need to distinguish between \"Byte\" and \"Character\".\n\nThen along comes Unicode which has room for over a million characters (and Perl allows for\neven more). This means that a character may require more than a single byte to represent it,\nand so the two terms are no longer equivalent. What matter are the characters as whole\nentities, and not usually the bytes that comprise them. That's what the term \"Character\nSemantics\" in the title of this section refers to.\n\nPerl had to change internally to decouple \"bytes\" from \"characters\". It is important that\nyou too change your ideas, if you haven't already, so that \"byte\" and \"character\" no longer\nmean the same thing in your mind.\n\nThe basic building block of Perl strings has always been a \"character\". The changes\nbasically come down to that the implementation no longer thinks that a character is always\njust a single byte.\n\nThere are various things to note:\n\n• String handling functions, for the most part, continue to operate in terms of characters.\n\"length()\", for example, returns the number of characters in a string, just as before.\nBut that number no longer is necessarily the same as the number of bytes in the string\n(there may be more bytes than characters). The other such functions include \"chop()\",\n\"chomp()\", \"substr()\", \"pos()\", \"index()\", \"rindex()\", \"sort()\", \"sprintf()\", and\n\"write()\".\n\nThe exceptions are:\n\n• the bit-oriented \"vec\"\n\n \n\n• the byte-oriented \"pack\"/\"unpack\" \"C\" format\n\nHowever, the \"W\" specifier does operate on whole characters, as does the \"U\"\nspecifier.\n\n• some operators that interact with the platform's operating system\n\nOperators dealing with filenames are examples.\n\n• when the functions are called from within the scope of the \"use bytes\" pragma\n\nLikely, you should use this only for debugging anyway.\n\n• Strings--including hash keys--and regular expression patterns may contain characters that\nhave ordinal values larger than 255.\n\nIf you use a Unicode editor to edit your program, Unicode characters may occur directly\nwithin the literal strings in UTF-8 encoding, or UTF-16. (The former requires a \"use\nutf8\", the latter may require a \"BOM\".)\n\n\"Creating Unicode\" in perluniintro gives other ways to place non-ASCII characters in your\nstrings.\n\n• The \"chr()\" and \"ord()\" functions work on whole characters.\n\n• Regular expressions match whole characters. For example, \".\" matches a whole character\ninstead of only a single byte.\n\n• The \"tr///\" operator translates whole characters. (Note that the \"tr///CU\" functionality\nhas been removed. For similar functionality to that, see \"pack('U0', ...)\" and\n\"pack('C0', ...)\").\n\n• \"scalar reverse()\" reverses by character rather than by byte.\n\n• The bit string operators, \"& | ^ ~\" and (starting in v5.22) \"&. |. ^. ~.\" can operate on\nbit strings encoded in UTF-8, but this can give unexpected results if any of the strings\ncontain code points above 0xFF. Starting in v5.28, it is a fatal error to have such an\noperand. Otherwise, the operation is performed on a non-UTF-8 copy of the operand. If\nyou're not sure about the encoding of a string, downgrade it before using any of these\noperators; you can use \"utf8::utf8downgrade()\".\n\nThe bottom line is that Perl has always practiced \"Character Semantics\", but with the advent\nof Unicode, that is now different than \"Byte Semantics\".\n" }, { "name": "ASCII Rules versus Unicode Rules", "content": "Before Unicode, when a character was a byte was a character, Perl knew only about the 128\ncharacters defined by ASCII, code points 0 through 127 (except for under \"use locale\"). That\nleft the code points 128 to 255 as unassigned, and available for whatever use a program might\nwant. The only semantics they have is their ordinal numbers, and that they are members of\nnone of the non-negative character classes. None are considered to match \"\\w\" for example,\nbut all match \"\\W\".\n\nUnicode, of course, assigns each of those code points a particular meaning (along with ones\nabove 255). To preserve backward compatibility, Perl only uses the Unicode meanings when\nthere is some indication that Unicode is what is intended; otherwise the non-ASCII code\npoints remain treated as if they are unassigned.\n\nHere are the ways that Perl knows that a string should be treated as Unicode:\n\n• Within the scope of \"use utf8\"\n\nIf the whole program is Unicode (signified by using 8-bit Unicode Transformation Format),\nthen all literal strings within it must be Unicode.\n\n• Within the scope of \"use feature 'unicodestrings'\"\n\nThis pragma was created so you can explicitly tell Perl that operations executed within\nits scope are to use Unicode rules. More operations are affected with newer perls. See\n\"The \"Unicode Bug\"\".\n\n• Within the scope of \"use 5.012\" or higher\n\nThis implicitly turns on \"use feature 'unicodestrings'\".\n\n• Within the scope of \"use locale 'notcharacters'\", or \"use locale\" and the current locale\nis a UTF-8 locale.\n\nThe former is defined to imply Unicode handling; and the latter indicates a Unicode\nlocale, hence a Unicode interpretation of all strings within it.\n\n• When the string contains a Unicode-only code point\n\nPerl has never accepted code points above 255 without them being Unicode, so their use\nimplies Unicode for the whole string.\n\n• When the string contains a Unicode named code point \"\\N{...}\"\n\nThe \"\\N{...}\" construct explicitly refers to a Unicode code point, even if it is one that\nis also in ASCII. Therefore the string containing it must be Unicode.\n\n• When the string has come from an external source marked as Unicode\n\nThe \"-C\" command line option can specify that certain inputs to the program are Unicode,\nand the values of this can be read by your Perl code, see \"${^UNICODE}\" in perlvar.\n\n• When the string has been upgraded to UTF-8\n\nThe function \"utf8::utf8upgrade()\" can be explicitly used to permanently (unless a\nsubsequent \"utf8::utf8downgrade()\" is called) cause a string to be treated as Unicode.\n\n• There are additional methods for regular expression patterns\n\nA pattern that is compiled with the \"/u\" or \"/a\" modifiers is treated as Unicode (though\nthere are some restrictions with \"/a\"). Under the \"/d\" and \"/l\" modifiers, there are\nseveral other indications for Unicode; see \"Character set modifiers\" in perlre.\n\nNote that all of the above are overridden within the scope of \"use bytes\"; but you should be\nusing this pragma only for debugging.\n\nNote also that some interactions with the platform's operating system never use Unicode\nrules.\n\nWhen Unicode rules are in effect:\n\n• Case translation operators use the Unicode case translation tables.\n\nNote that \"uc()\", or \"\\U\" in interpolated strings, translates to uppercase, while\n\"ucfirst\", or \"\\u\" in interpolated strings, translates to titlecase in languages that\nmake the distinction (which is equivalent to uppercase in languages without the\ndistinction).\n\nThere is a CPAN module, \"Unicode::Casing\", which allows you to define your own mappings\nto be used in \"lc()\", \"lcfirst()\", \"uc()\", \"ucfirst()\", and \"fc\" (or their double-quoted\nstring inlined versions such as \"\\U\"). (Prior to Perl 5.16, this functionality was\npartially provided in the Perl core, but suffered from a number of insurmountable\ndrawbacks, so the CPAN module was written instead.)\n\n• Character classes in regular expressions match based on the character properties\nspecified in the Unicode properties database.\n\n\"\\w\" can be used to match a Japanese ideograph, for instance; and \"[[:digit:]]\" a Bengali\nnumber.\n\n• Named Unicode properties, scripts, and block ranges may be used (like bracketed character\nclasses) by using the \"\\p{}\" \"matches property\" construct and the \"\\P{}\" negation,\n\"doesn't match property\".\n\nSee \"Unicode Character Properties\" for more details.\n\nYou can define your own character properties and use them in the regular expression with\nthe \"\\p{}\" or \"\\P{}\" construct. See \"User-Defined Character Properties\" for more\ndetails.\n" }, { "name": "Extended Grapheme Clusters (Logical characters)", "content": "Consider a character, say \"H\". It could appear with various marks around it, such as an\nacute accent, or a circumflex, or various hooks, circles, arrows, etc., above, below, to one\nside or the other, etc. There are many possibilities among the world's languages. The\nnumber of combinations is astronomical, and if there were a character for each combination,\nit would soon exhaust Unicode's more than a million possible characters. So Unicode took a\ndifferent approach: there is a character for the base \"H\", and a character for each of the\npossible marks, and these can be variously combined to get a final logical character. So a\nlogical character--what appears to be a single character--can be a sequence of more than one\nindividual characters. The Unicode standard calls these \"extended grapheme clusters\" (which\nis an improved version of the no-longer much used \"grapheme cluster\"); Perl furnishes the\n\"\\X\" regular expression construct to match such sequences in their entirety.\n\nBut Unicode's intent is to unify the existing character set standards and practices, and\nseveral pre-existing standards have single characters that mean the same thing as some of\nthese combinations, like ISO-8859-1, which has quite a few of them. For example, \"LATIN\nCAPITAL LETTER E WITH ACUTE\" was already in this standard when Unicode came along. Unicode\ntherefore added it to its repertoire as that single character. But this character is\nconsidered by Unicode to be equivalent to the sequence consisting of the character \"LATIN\nCAPITAL LETTER E\" followed by the character \"COMBINING ACUTE ACCENT\".\n\n\"LATIN CAPITAL LETTER E WITH ACUTE\" is called a \"pre-composed\" character, and its equivalence\nwith the \"E\" and the \"COMBINING ACCENT\" sequence is called canonical equivalence. All pre-\ncomposed characters are said to have a decomposition (into the equivalent sequence), and the\ndecomposition type is also called canonical. A string may be comprised as much as possible\nof precomposed characters, or it may be comprised of entirely decomposed characters. Unicode\ncalls these respectively, \"Normalization Form Composed\" (NFC) and \"Normalization Form\nDecomposed\". The \"Unicode::Normalize\" module contains functions that convert between the\ntwo. A string may also have both composed characters and decomposed characters; this module\ncan be used to make it all one or the other.\n\nYou may be presented with strings in any of these equivalent forms. There is currently\nnothing in Perl 5 that ignores the differences. So you'll have to specially handle it. The\nusual advice is to convert your inputs to \"NFD\" before processing further.\n\nFor more detailed information, see .\n" }, { "name": "Unicode Character Properties", "content": "(The only time that Perl considers a sequence of individual code points as a single logical\ncharacter is in the \"\\X\" construct, already mentioned above. Therefore \"character\" in this\ndiscussion means a single Unicode code point.)\n\nVery nearly all Unicode character properties are accessible through regular expressions by\nusing the \"\\p{}\" \"matches property\" construct and the \"\\P{}\" \"doesn't match property\" for its\nnegation.\n\nFor instance, \"\\p{Uppercase}\" matches any single character with the Unicode \"Uppercase\"\nproperty, while \"\\p{L}\" matches any character with a \"GeneralCategory\" of \"L\" (letter)\nproperty (see \"GeneralCategory\" below). Brackets are not required for single letter\nproperty names, so \"\\p{L}\" is equivalent to \"\\pL\".\n\nMore formally, \"\\p{Uppercase}\" matches any single character whose Unicode \"Uppercase\"\nproperty value is \"True\", and \"\\P{Uppercase}\" matches any character whose \"Uppercase\"\nproperty value is \"False\", and they could have been written as \"\\p{Uppercase=True}\" and\n\"\\p{Uppercase=False}\", respectively.\n\nThis formality is needed when properties are not binary; that is, if they can take on more\nvalues than just \"True\" and \"False\". For example, the \"BidiClass\" property (see\n\"Bidirectional Character Types\" below), can take on several different values, such as \"Left\",\n\"Right\", \"Whitespace\", and others. To match these, one needs to specify both the property\nname (\"BidiClass\"), AND the value being matched against (\"Left\", \"Right\", etc.). This is\ndone, as in the examples above, by having the two components separated by an equal sign (or\ninterchangeably, a colon), like \"\\p{BidiClass: Left}\".\n\nAll Unicode-defined character properties may be written in these compound forms of\n\"\\p{property=value}\" or \"\\p{property:value}\", but Perl provides some additional properties\nthat are written only in the single form, as well as single-form short-cuts for all binary\nproperties and certain others described below, in which you may omit the property name and\nthe equals or colon separator.\n\nMost Unicode character properties have at least two synonyms (or aliases if you prefer): a\nshort one that is easier to type and a longer one that is more descriptive and hence easier\nto understand. Thus the \"L\" and \"Letter\" properties above are equivalent and can be used\ninterchangeably. Likewise, \"Upper\" is a synonym for \"Uppercase\", and we could have written\n\"\\p{Uppercase}\" equivalently as \"\\p{Upper}\". Also, there are typically various synonyms for\nthe values the property can be. For binary properties, \"True\" has 3 synonyms: \"T\", \"Yes\",\nand \"Y\"; and \"False\" has correspondingly \"F\", \"No\", and \"N\". But be careful. A short form\nof a value for one property may not mean the same thing as the short form spelled the same\nfor another. Thus, for the \"GeneralCategory\" property, \"L\" means \"Letter\", but for the\n\"BidiClass\" property, \"L\" means \"Left\". A complete list of properties and synonyms is in\nperluniprops.\n\nUpper/lower case differences in property names and values are irrelevant; thus \"\\p{Upper}\"\nmeans the same thing as \"\\p{upper}\" or even \"\\p{UpPeR}\". Similarly, you can add or subtract\nunderscores anywhere in the middle of a word, so that these are also equivalent to\n\"\\p{Upper}\". And white space is generally irrelevant adjacent to non-word characters,\nsuch as the braces and the equals or colon separators, so \"\\p{ Upper }\" and \"\\p{\nUppercase : Y }\" are equivalent to these as well. In fact, white space and even hyphens can\nusually be added or deleted anywhere. So even \"\\p{ Up-per case = Yes}\" is equivalent. All\nthis is called \"loose-matching\" by Unicode. The \"name\" property has some restrictions on\nthis due to a few outlier names. Full details are given in\n.\n\nThe few places where stricter matching is used is in the middle of numbers, the \"name\"\nproperty, and in the Perl extension properties that begin or end with an underscore.\nStricter matching cares about white space (except adjacent to non-word characters), hyphens,\nand non-interior underscores.\n\nYou can also use negation in both \"\\p{}\" and \"\\P{}\" by introducing a caret (\"^\") between the\nfirst brace and the property name: \"\\p{^Tamil}\" is equal to \"\\P{Tamil}\".\n\nAlmost all properties are immune to case-insensitive matching. That is, adding a \"/i\"\nregular expression modifier does not change what they match. There are two sets that are\naffected. The first set is \"UppercaseLetter\", \"LowercaseLetter\", and \"TitlecaseLetter\",\nall of which match \"CasedLetter\" under \"/i\" matching. And the second set is \"Uppercase\",\n\"Lowercase\", and \"Titlecase\", all of which match \"Cased\" under \"/i\" matching. This set also\nincludes its subsets \"PosixUpper\" and \"PosixLower\" both of which under \"/i\" match\n\"PosixAlpha\". (The difference between these sets is that some things, such as Roman\nnumerals, come in both upper and lower case so they are \"Cased\", but aren't considered\nletters, so they aren't \"CasedLetter\"'s.)\n\nSee \"Beyond Unicode code points\" for special considerations when matching Unicode properties\nagainst non-Unicode code points.\n\nGGeenneerraallCCaatteeggoorryy\n\nEvery Unicode character is assigned a general category, which is the \"most usual\ncategorization of a character\" (from ).\n\nThe compound way of writing these is like \"\\p{GeneralCategory=Number}\" (short: \"\\p{gc:n}\").\nBut Perl furnishes shortcuts in which everything up through the equal or colon separator is\nomitted. So you can instead just write \"\\pN\".\n\nHere are the short and long forms of the values the \"General Category\" property can have:\n\nShort Long\n\nL Letter\nLC, L& CasedLetter (that is: [\\p{Ll}\\p{Lu}\\p{Lt}])\nLu UppercaseLetter\nLl LowercaseLetter\nLt TitlecaseLetter\nLm ModifierLetter\nLo OtherLetter\n\nM Mark\nMn NonspacingMark\nMc SpacingMark\nMe EnclosingMark\n\nN Number\nNd DecimalNumber (also Digit)\nNl LetterNumber\nNo OtherNumber\n\nP Punctuation (also Punct)\nPc ConnectorPunctuation\nPd DashPunctuation\nPs OpenPunctuation\nPe ClosePunctuation\nPi InitialPunctuation\n(may behave like Ps or Pe depending on usage)\nPf FinalPunctuation\n(may behave like Ps or Pe depending on usage)\nPo OtherPunctuation\n\nS Symbol\nSm MathSymbol\nSc CurrencySymbol\nSk ModifierSymbol\nSo OtherSymbol\n\nZ Separator\nZs SpaceSeparator\nZl LineSeparator\nZp ParagraphSeparator\n\nC Other\nCc Control (also Cntrl)\nCf Format\nCs Surrogate\nCo PrivateUse\nCn Unassigned\n\nSingle-letter properties match all characters in any of the two-letter sub-properties\nstarting with the same letter. \"LC\" and \"L&\" are special: both are aliases for the set\nconsisting of everything matched by \"Ll\", \"Lu\", and \"Lt\".\n\nBBiiddiirreeccttiioonnaall CChhaarraacctteerr TTyyppeess\n\nBecause scripts differ in their directionality (Hebrew and Arabic are written right to left,\nfor example) Unicode supplies a \"BidiClass\" property. Some of the values this property can\nhave are:\n\nValue Meaning\n\nL Left-to-Right\nLRE Left-to-Right Embedding\nLRO Left-to-Right Override\nR Right-to-Left\nAL Arabic Letter\nRLE Right-to-Left Embedding\nRLO Right-to-Left Override\nPDF Pop Directional Format\nEN European Number\nES European Separator\nET European Terminator\nAN Arabic Number\nCS Common Separator\nNSM Non-Spacing Mark\nBN Boundary Neutral\nB Paragraph Separator\nS Segment Separator\nWS Whitespace\nON Other Neutrals\n\nThis property is always written in the compound form. For example, \"\\p{BidiClass:R}\"\nmatches characters that are normally written right to left. Unlike the \"GeneralCategory\"\nproperty, this property can have more values added in a future Unicode release. Those listed\nabove comprised the complete set for many Unicode releases, but others were added in Unicode\n6.3; you can always find what the current ones are in perluniprops. And\n describes how to use them.\n\nSSccrriippttss\n\nThe world's languages are written in many different scripts. This sentence (unless you're\nreading it in translation) is written in Latin, while Russian is written in Cyrillic, and\nGreek is written in, well, Greek; Japanese mainly in Hiragana or Katakana. There are many\nmore.\n\nThe Unicode \"Script\" and \"ScriptExtensions\" properties give what script a given character is\nin. The \"ScriptExtensions\" property is an improved version of \"Script\", as demonstrated\nbelow. Either property can be specified with the compound form like \"\\p{Script=Hebrew}\"\n(short: \"\\p{sc=hebr}\"), or \"\\p{ScriptExtensions=Javanese}\" (short: \"\\p{scx=java}\"). In\naddition, Perl furnishes shortcuts for all \"ScriptExtensions\" property names. You can omit\neverything up through the equals (or colon), and simply write \"\\p{Latin}\" or \"\\P{Cyrillic}\".\n(This is not true for \"Script\", which is required to be written in the compound form. Prior\nto Perl v5.26, the single form returned the plain old \"Script\" version, but was changed\nbecause \"ScriptExtensions\" gives better results.)\n\nThe difference between these two properties involves characters that are used in multiple\nscripts. For example the digits '0' through '9' are used in many parts of the world. These\nare placed in a script named \"Common\". Other characters are used in just a few scripts. For\nexample, the \"KATAKANA-HIRAGANA DOUBLE HYPHEN\" is used in both Japanese scripts, Katakana and\nHiragana, but nowhere else. The \"Script\" property places all characters that are used in\nmultiple scripts in the \"Common\" script, while the \"ScriptExtensions\" property places those\nthat are used in only a few scripts into each of those scripts; while still using \"Common\"\nfor those used in many scripts. Thus both these match:\n\n\"0\" =~ /\\p{sc=Common}/ # Matches\n\"0\" =~ /\\p{scx=Common}/ # Matches\n\nand only the first of these match:\n\n\"\\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}\" =~ /\\p{sc=Common} # Matches\n\"\\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}\" =~ /\\p{scx=Common} # No match\n\nAnd only the last two of these match:\n\n\"\\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}\" =~ /\\p{sc=Hiragana} # No match\n\"\\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}\" =~ /\\p{sc=Katakana} # No match\n\"\\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}\" =~ /\\p{scx=Hiragana} # Matches\n\"\\N{KATAKANA-HIRAGANA DOUBLE HYPHEN}\" =~ /\\p{scx=Katakana} # Matches\n\n\"ScriptExtensions\" is thus an improved \"Script\", in which there are fewer characters in the\n\"Common\" script, and correspondingly more in other scripts. It is new in Unicode version\n6.0, and its data are likely to change significantly in later releases, as things get sorted\nout. New code should probably be using \"ScriptExtensions\" and not plain \"Script\". If you\ncompile perl with a Unicode release that doesn't have \"ScriptExtensions\", the single form\nPerl extensions will instead refer to the plain \"Script\" property. If you compile with a\nversion of Unicode that doesn't have the \"Script\" property, these extensions will not be\ndefined at all.\n\n(Actually, besides \"Common\", the \"Inherited\" script, contains characters that are used in\nmultiple scripts. These are modifier characters which inherit the script value of the\ncontrolling character. Some of these are used in many scripts, and so go into \"Inherited\" in\nboth \"Script\" and \"ScriptExtensions\". Others are used in just a few scripts, so are in\n\"Inherited\" in \"Script\", but not in \"ScriptExtensions\".)\n\nIt is worth stressing that there are several different sets of digits in Unicode that are\nequivalent to 0-9 and are matchable by \"\\d\" in a regular expression. If they are used in a\nsingle language only, they are in that language's \"Script\" and \"ScriptExtensions\". If they\nare used in more than one script, they will be in \"sc=Common\", but only if they are used in\nmany scripts should they be in \"scx=Common\".\n\nThe explanation above has omitted some detail; refer to UAX#24 \"Unicode Script Property\":\n.\n\nA complete list of scripts and their shortcuts is in perluniprops.\n\nUUssee ooff tthhee \"Is\" PPrreeffiixx\n\nFor backward compatibility (with ancient Perl 5.6), all properties writable without using the\ncompound form mentioned so far may have \"Is\" or \"Is\" prepended to their name, so\n\"\\P{IsLu}\", for example, is equal to \"\\P{Lu}\", and \"\\p{IsScript:Arabic}\" is equal to\n\"\\p{Arabic}\".\n\nBBlloocckkss\n\nIn addition to scripts, Unicode also defines blocks of characters. The difference between\nscripts and blocks is that the concept of scripts is closer to natural languages, while the\nconcept of blocks is more of an artificial grouping based on groups of Unicode characters\nwith consecutive ordinal values. For example, the \"Basic Latin\" block is all the characters\nwhose ordinals are between 0 and 127, inclusive; in other words, the ASCII characters. The\n\"Latin\" script contains some letters from this as well as several other blocks, like \"Latin-1\nSupplement\", \"Latin Extended-A\", etc., but it does not contain all the characters from those\nblocks. It does not, for example, contain the digits 0-9, because those digits are shared\nacross many scripts, and hence are in the \"Common\" script.\n\nFor more about scripts versus blocks, see UAX#24 \"Unicode Script Property\":\n\n\nThe \"ScriptExtensions\" or \"Script\" properties are likely to be the ones you want to use when\nprocessing natural language; the \"Block\" property may occasionally be useful in working with\nthe nuts and bolts of Unicode.\n\nBlock names are matched in the compound form, like \"\\p{Block: Arrows}\" or \"\\p{Blk=Hebrew}\".\nUnlike most other properties, only a few block names have a Unicode-defined short name.\n\nPerl also defines single form synonyms for the block property in cases where these do not\nconflict with something else. But don't use any of these, because they are unstable. Since\nthese are Perl extensions, they are subordinate to official Unicode property names; Unicode\ndoesn't know nor care about Perl's extensions. It may happen that a name that currently\nmeans the Perl extension will later be changed without warning to mean a different Unicode\nproperty in a future version of the perl interpreter that uses a later Unicode release, and\nyour code would no longer work. The extensions are mentioned here for completeness: Take\nthe block name and prefix it with one of: \"In\" (for example \"\\p{Blk=Arrows}\" can currently be\nwritten as \"\\p{InArrows}\"); or sometimes \"Is\" (like \"\\p{IsArrows}\"); or sometimes no prefix\nat all (\"\\p{Arrows}\"). As of this writing (Unicode 9.0) there are no conflicts with using\nthe \"In\" prefix, but there are plenty with the other two forms. For example,\n\"\\p{IsHebrew}\" and \"\\p{Hebrew}\" mean \"\\p{ScriptExtensions=Hebrew}\" which is NOT the same\nthing as \"\\p{Blk=Hebrew}\". Our advice used to be to use the \"In\" prefix as a single form\nway of specifying a block. But Unicode 8.0 added properties whose names begin with \"In\", and\nit's now clear that it's only luck that's so far prevented a conflict. Using \"In\" is only\nmarginally less typing than \"Blk:\", and the latter's meaning is clearer anyway, and\nguaranteed to never conflict. So don't take chances. Use \"\\p{Blk=foo}\" for new code. And\nbe sure that block is what you really really want to do. In most cases scripts are what you\nwant instead.\n\nA complete list of blocks is in perluniprops.\n\nOOtthheerr PPrrooppeerrttiieess\n\nThere are many more properties than the very basic ones described here. A complete list is\nin perluniprops.\n\nUnicode defines all its properties in the compound form, so all single-form properties are\nPerl extensions. Most of these are just synonyms for the Unicode ones, but some are genuine\nextensions, including several that are in the compound form. And quite a few of these are\nactually recommended by Unicode (in ).\n\nThis section gives some details on all extensions that aren't just synonyms for compound-form\nUnicode properties (for those properties, you'll have to refer to the Unicode Standard\n.\n" }, { "name": "\"\\p{All}\"", "content": "This matches every possible code point. It is equivalent to \"qr/./s\". Unlike all the\nother non-user-defined \"\\p{}\" property matches, no warning is ever generated if this is\nproperty is matched against a non-Unicode code point (see \"Beyond Unicode code points\"\nbelow).\n" }, { "name": "\"\\p{Alnum}\"", "content": "This matches any \"\\p{Alphabetic}\" or \"\\p{DecimalNumber}\" character.\n" }, { "name": "\"\\p{Any}\"", "content": "This matches any of the 1114112 Unicode code points. It is a synonym for\n\"\\p{Unicode}\".\n" }, { "name": "\"\\p{ASCII}\"", "content": "This matches any of the 128 characters in the US-ASCII character set, which is a subset\nof Unicode.\n" }, { "name": "\"\\p{Assigned}\"", "content": "This matches any assigned code point; that is, any code point whose general category is\nnot \"Unassigned\" (or equivalently, not \"Cn\").\n" }, { "name": "\"\\p{Blank}\"", "content": "This is the same as \"\\h\" and \"\\p{HorizSpace}\": A character that changes the spacing\nhorizontally.\n\n\"\\p{DecompositionType: NonCanonical}\" (Short: \"\\p{Dt=NonCanon}\")\nMatches a character that has a non-canonical decomposition.\n\nThe \"Extended Grapheme Clusters (Logical characters)\" section above talked about\ncanonical decompositions. However, many more characters have a different type of\ndecomposition, a \"compatible\" or \"non-canonical\" decomposition. The sequences that form\nthese decompositions are not considered canonically equivalent to the pre-composed\ncharacter. An example is the \"SUPERSCRIPT ONE\". It is somewhat like a regular digit 1,\nbut not exactly; its decomposition into the digit 1 is called a \"compatible\"\ndecomposition, specifically a \"super\" decomposition. There are several such\ncompatibility decompositions (see ), including one\ncalled \"compat\", which means some miscellaneous type of decomposition that doesn't fit\ninto the other decomposition categories that Unicode has chosen.\n\nNote that most Unicode characters don't have a decomposition, so their decomposition type\nis \"None\".\n\nFor your convenience, Perl has added the \"NonCanonical\" decomposition type to mean any\nof the several compatibility decompositions.\n" }, { "name": "\"\\p{Graph}\"", "content": "Matches any character that is graphic. Theoretically, this means a character that on a\nprinter would cause ink to be used.\n" }, { "name": "\"\\p{HorizSpace}\"", "content": "This is the same as \"\\h\" and \"\\p{Blank}\": a character that changes the spacing\nhorizontally.\n\n\"\\p{In=*}\"\nThis is a synonym for \"\\p{PresentIn=*}\"\n" }, { "name": "\"\\p{PerlSpace}\"", "content": "This is the same as \"\\s\", restricted to ASCII, namely \"[ \\f\\n\\r\\t]\" and starting in Perl\nv5.18, a vertical tab.\n\nMnemonic: Perl's (original) space\n" }, { "name": "\"\\p{PerlWord}\"", "content": "This is the same as \"\\w\", restricted to ASCII, namely \"[A-Za-z0-9]\"\n\nMnemonic: Perl's (original) word.\n" }, { "name": "\"\\p{Posix...}\"", "content": "There are several of these, which are equivalents, using the \"\\p{}\" notation, for Posix\nclasses and are described in \"POSIX Character Classes\" in perlrecharclass.\n\n\"\\p{PresentIn: *}\" (Short: \"\\p{In=*}\")\nThis property is used when you need to know in what Unicode version(s) a character is.\n\nThe \"*\" above stands for some Unicode version number, such as 1.1 or 12.0; or the \"*\" can\nalso be \"Unassigned\". This property will match the code points whose final disposition\nhas been settled as of the Unicode release given by the version number; \"\\p{PresentIn:\nUnassigned}\" will match those code points whose meaning has yet to be assigned.\n\nFor example, \"U+0041\" \"LATIN CAPITAL LETTER A\" was present in the very first Unicode\nrelease available, which is 1.1, so this property is true for all valid \"*\" versions. On\nthe other hand, \"U+1EFF\" was not assigned until version 5.1 when it became \"LATIN SMALL\nLETTER Y WITH LOOP\", so the only \"*\" that would match it are 5.1, 5.2, and later.\n\nUnicode furnishes the \"Age\" property from which this is derived. The problem with Age is\nthat a strict interpretation of it (which Perl takes) has it matching the precise release\na code point's meaning is introduced in. Thus \"U+0041\" would match only 1.1; and\n\"U+1EFF\" only 5.1. This is not usually what you want.\n\nSome non-Perl implementations of the Age property may change its meaning to be the same\nas the Perl \"PresentIn\" property; just be aware of that.\n\nAnother confusion with both these properties is that the definition is not that the code\npoint has been assigned, but that the meaning of the code point has been determined.\nThis is because 66 code points will always be unassigned, and so the \"Age\" for them is\nthe Unicode version in which the decision to make them so was made. For example,\n\"U+FDD0\" is to be permanently unassigned to a character, and the decision to do that was\nmade in version 3.1, so \"\\p{Age=3.1}\" matches this character, as also does\n\"\\p{PresentIn: 3.1}\" and up.\n" }, { "name": "\"\\p{Print}\"", "content": "This matches any character that is graphical or blank, except controls.\n" }, { "name": "\"\\p{SpacePerl}\"", "content": "This is the same as \"\\s\", including beyond ASCII.\n\nMnemonic: Space, as modified by Perl. (It doesn't include the vertical tab until v5.18,\nwhich both the Posix standard and Unicode consider white space.)\n\n\"\\p{Title}\" and \"\\p{Titlecase}\"\nUnder case-sensitive matching, these both match the same code points as \"\\p{General\nCategory=TitlecaseLetter}\" (\"\\p{gc=lt}\"). The difference is that under \"/i\" caseless\nmatching, these match the same as \"\\p{Cased}\", whereas \"\\p{gc=lt}\" matches\n\"\\p{CasedLetter\").\n" }, { "name": "\"\\p{Unicode}\"", "content": "This matches any of the 1114112 Unicode code points. \"\\p{Any}\".\n" }, { "name": "\"\\p{VertSpace}\"", "content": "This is the same as \"\\v\": A character that changes the spacing vertically.\n" }, { "name": "\"\\p{Word}\"", "content": "This is the same as \"\\w\", including over 100000 characters beyond ASCII.\n" }, { "name": "\"\\p{XPosix...}\"", "content": "There are several of these, which are the standard Posix classes extended to the full\nUnicode range. They are described in \"POSIX Character Classes\" in perlrecharclass.\n" }, { "name": "Comparison of \"\\N{...}\" and \"\\p{name=...}\"", "content": "Starting in Perl 5.32, you can specify a character by its name in regular expression patterns\nusing \"\\p{name=...}\". This is in addition to the longstanding method of using \"\\N{...}\".\nThe following summarizes the differences between these two:\n\n\\N{...} \\p{Name=...}\ncan interpolate only with eval yes [1]\ncustom names yes no [2]\nname aliases yes yes [3]\nnamed sequences yes yes [4]\nname value parsing exact Unicode loose [5]\n\n[1] The ability to interpolate means you can do something like\n\nqr/\\p{na=latin capital letter $which}/\n\nand specify $which elsewhere.\n\n[2] You can create your own names for characters, and override official ones when using\n\"\\N{...}\". See \"CUSTOM ALIASES\" in charnames.\n\n[3] Some characters have multiple names (synonyms).\n\n[4] Some particular sequences of characters are given a single name, in addition to their\nindividual ones.\n\n[5] Exact name value matching means you have to specify case, hyphens, underscores, and\nspaces precisely in the name you want. Loose matching follows the Unicode rules\n, where these are mostly\nirrelevant. Except for a few outlier character names, these are the same rules as are\nalready used for any other \"\\p{...}\" property.\n" }, { "name": "Wildcards in Property Values", "content": "Starting in Perl 5.30, it is possible to do something like this:\n\nqr!\\p{numericvalue=/\\A[0-5]\\z/}!\n\nor, by abbreviating and adding \"/x\",\n\nqr! \\p{nv= /(?x) \\A [0-5] \\z / }!\n\nThis matches all code points whose numeric value is one of 0, 1, 2, 3, 4, or 5. This\nparticular example could instead have been written as\n\nqr! \\A [ \\p{nv=0}\\p{nv=1}\\p{nv=2}\\p{nv=3}\\p{nv=4}\\p{nv=5} ] \\z !xx\n\nin earlier perls, so in this case this feature just makes things easier and shorter to write.\nIf we hadn't included the \"\\A\" and \"\\z\", these would have matched things like \"1/2\" because\nthat contains a 1 (as well as a 2). As written, it matches things like subscripts that have\nthese numeric values. If we only wanted the decimal digits with those numeric values, we\ncould say,\n\nqr! (?[ \\d & \\p{nv=/[0-5]/ ]) }!x\n\nThe \"\\d\" gets rid of needing to anchor the pattern, since it forces the result to only match\n\"[0-9]\", and the \"[0-5]\" further restricts it.\n\nThe text in the above examples enclosed between the \"/\" characters can be just about any\nregular expression. It is independent of the main pattern, so doesn't share any capturing\ngroups, etc. The delimiters for it must be ASCII punctuation, but it may NOT be delimited by\n\"{\", nor \"}\" nor contain a literal \"}\", as that delimits the end of the enclosing \"\\p{}\".\nLike any pattern, certain other delimiters are terminated by their mirror images. These are\n\"(\", \"\"[\"\", and \"<\". If the delimiter is any of \"-\", \"\", \"+\", or \"\\\", or is the same\ndelimiter as is used for the enclosing pattern, it must be preceded by a backslash escape,\nboth fore and aft.\n\nBeware of using \"$\" to indicate to match the end of the string. It can too easily be\ninterpreted as being a punctuation variable, like $/.\n\nNo modifiers may follow the final delimiter. Instead, use \"(?adlupimnsx-imnsx)\" in perlre\nand/or \"(?adluimnsx-imnsx:pattern)\" in perlre to specify modifiers. However, certain\nmodifiers are illegal in your wildcard subpattern. The only character set modifier\nspecifiable is \"/aa\"; any other character set, and \"-m\", and \"p\", and \"s\" are all illegal.\nSpecifying modifiers like \"qr/.../gc\" that aren't legal in the \"(?...)\" notation normally\nraise a warning, but with wildcard subpatterns, their use is an error. The \"m\" modifier is\nineffective; everything that matches will be a single line.\n\nBy default, your pattern is matched case-insensitively, as if \"/i\" had been specified. You\ncan change this by saying \"(?-i)\" in your pattern.\n\nThere are also certain operations that are illegal. You can't nest \"\\p{...}\" and \"\\P{...}\"\ncalls within a wildcard subpattern, and \"\\G\" doesn't make sense, so is also prohibited.\n\nAnd the \"*\" quantifier (or its equivalent \"(0,}\") is illegal.\n\nThis feature is not available when the left-hand side is prefixed by \"Is\", nor for any form\nthat is marked as \"Discouraged\" in \"Discouraged\" in perluniprops.\n\nThis experimental feature has been added to begin to implement\n. Using it will raise a (default-\non) warning in the \"experimental::unipropwildcards\" category. We reserve the right to\nchange its operation as we gain experience.\n\nYour subpattern can be just about anything, but for it to have some utility, it should match\nwhen called with either or both of a) the full name of the property value with underscores\n(and/or spaces in the Block property) and some things uppercase; or b) the property value in\nall lowercase with spaces and underscores squeezed out. For example,\n\nqr!\\p{Blk=/Old I.*/}!\nqr!\\p{Blk=/oldi.*/}!\n\nwould match the same things.\n\nAnother example that shows that within \"\\p{...}\", \"/x\" isn't needed to have spaces:\n\nqr!\\p{scx= /Hebrew|Greek/ }!\n\nTo be safe, we should have anchored the above example, to prevent matches for something like\n\"HebrewBraille\", but there aren't any script names like that, so far. A warning is issued\nif none of the legal values for a property are matched by your pattern. It's likely that a\nfuture release will raise a warning if your pattern ends up causing every possible code point\nto match.\n\nStarting in 5.32, the Name, Name Aliases, and Named Sequences properties are allowed to be\nmatched. They are considered to be a single combination property, just as has long been the\ncase for \"\\N{}\". Loose matching doesn't work in exactly the same way for these as it does\nfor the values of other properties. The rules are given in\n. As a result, Perl doesn't try\nloose matching for you, like it does in other properties. All letters in names are\nuppercase, but you can add \"(?i)\" to your subpattern to ignore case. If you're uncertain\nwhere a blank is, you can use \" ?\" in your subpattern. No character name contains an\nunderscore, so don't bother trying to match one. The use of hyphens is particularly\nproblematic; refer to the above link. But note that, as of Unicode 13.0, the only script in\nmodern usage which has weirdnesses with these is Tibetan; also the two Korean characters\nU+116C HANGUL JUNGSEONG OE and U+1180 HANGUL JUNGSEONG O-E. Unicode makes no promises to not\nadd hyphen-problematic names in the future.\n\nUsing wildcards on these is resource intensive, given the hundreds of thousands of legal\nnames that must be checked against.\n\nAn example of using Name property wildcards is\n\nqr!\\p{name=/(SMILING|GRINNING) FACE/}!\n\nAnother is\n\nqr/(?[ \\p{name=\\/CJK\\/} - \\p{ideographic} ])/\n\nwhich is the 200-ish (as of Unicode 13.0) CJK characters that aren't ideographs.\n\nThere are certain properties that wildcard subpatterns don't currently work with. These are:\n\nBidi Mirroring Glyph\nBidi Paired Bracket\nCase Folding\nDecomposition Mapping\nEquivalent Unified Ideograph\nLowercase Mapping\nNFKC Case Fold\nTitlecase Mapping\nUppercase Mapping\n\nNor is the \"@unicodeproperty@\" form implemented.\n\nHere's a complete example of matching IPV4 internet protocol addresses in any (single) script\n\nno warnings 'experimental::regexsets';\nno warnings 'experimental::unipropwildcards';\n\n# Can match a substring, so this intermediate regex needs to have\n# context or anchoring in its final use. Using nt=de yields decimal\n# digits. When specifying a subset of these, we must include \\d to\n# prevent things like U+00B2 SUPERSCRIPT TWO from matching\nmy $zerothrough255 =\nqr/ \\b (*sr: # All from same sript\n(?[ \\p{nv=0} & \\d ])* # Optional leading zeros\n( # Then one of:\n\\d{1,2} # 0 - 99\n| (?[ \\p{nv=1} & \\d ]) \\d{2} # 100 - 199\n| (?[ \\p{nv=2} & \\d ])\n( (?[ \\p{nv=:[0-4]:} & \\d ]) \\d # 200 - 249\n| (?[ \\p{nv=5} & \\d ])\n(?[ \\p{nv=:[0-5]:} & \\d ]) # 250 - 255\n)\n)\n)\n\\b\n/x;\n\nmy $ipv4 = qr/ \\A (*sr: $zerothrough255\n(?: [.] $zerothrough255 ) {3}\n)\n\\z\n/x;\n" }, { "name": "User-Defined Character Properties", "content": "You can define your own binary character properties by defining subroutines whose names begin\nwith \"In\" or \"Is\". (The experimental feature \"(?[ ])\" in perlre provides an alternative\nwhich allows more complex definitions.) The subroutines can be defined in any package. They\noverride any Unicode properties expressed as the same names. The user-defined properties can\nbe used in the regular expression \"\\p{}\" and \"\\P{}\" constructs; if you are using a user-\ndefined property from a package other than the one you are in, you must specify its package\nin the \"\\p{}\" or \"\\P{}\" construct.\n\n# assuming property IsForeign defined in Lang::\npackage main; # property package name required\nif ($txt =~ /\\p{Lang::IsForeign}+/) { ... }\n\npackage Lang; # property package name not required\nif ($txt =~ /\\p{IsForeign}+/) { ... }\n\nNote that the effect is compile-time and immutable once defined. However, the subroutines\nare passed a single parameter, which is 0 if case-sensitive matching is in effect and non-\nzero if caseless matching is in effect. The subroutine may return different values depending\non the value of the flag, and one set of values will immutably be in effect for all case-\nsensitive matches, and the other set for all case-insensitive matches.\n\nNote that if the regular expression is tainted, then Perl will die rather than calling the\nsubroutine when the name of the subroutine is determined by the tainted data.\n\nThe subroutines must return a specially-formatted string, with one or more newline-separated\nlines. Each line must be one of the following:\n\n• A single hexadecimal number denoting a code point to include.\n\n• Two hexadecimal numbers separated by horizontal whitespace (space or tabular characters)\ndenoting a range of code points to include. The second number must not be smaller than\nthe first.\n\n• Something to include, prefixed by \"+\": a built-in character property (prefixed by\n\"utf8::\") or a fully qualified (including package name) user-defined character property,\nto represent all the characters in that property; two hexadecimal code points for a\nrange; or a single hexadecimal code point.\n\n• Something to exclude, prefixed by \"-\": an existing character property (prefixed by\n\"utf8::\") or a fully qualified (including package name) user-defined character property,\nto represent all the characters in that property; two hexadecimal code points for a\nrange; or a single hexadecimal code point.\n\n• Something to negate, prefixed \"!\": an existing character property (prefixed by \"utf8::\")\nor a fully qualified (including package name) user-defined character property, to\nrepresent all the characters in that property; two hexadecimal code points for a range;\nor a single hexadecimal code point.\n\n• Something to intersect with, prefixed by \"&\": an existing character property (prefixed by\n\"utf8::\") or a fully qualified (including package name) user-defined character property,\nfor all the characters except the characters in the property; two hexadecimal code points\nfor a range; or a single hexadecimal code point.\n\nFor example, to define a property that covers both the Japanese syllabaries (hiragana and\nkatakana), you can define\n\nsub InKana {\nreturn <\n" }, { "name": "Character Encodings for Input and Output", "content": "See Encode.\n" }, { "name": "Unicode Regular Expression Support Level", "content": "The following list of Unicode supported features for regular expressions describes all\nfeatures currently directly supported by core Perl. The references to \"Level N\" and the\nsection numbers refer to UTS#18 \"Unicode Regular Expressions\"\n, version 18, October 2016.\n\nLevel 1 - Basic Unicode Support\n\nRL1.1 Hex Notation - Done [1]\nRL1.2 Properties - Done [2]\nRL1.2a Compatibility Properties - Done [3]\nRL1.3 Subtraction and Intersection - Experimental [4]\nRL1.4 Simple Word Boundaries - Done [5]\nRL1.5 Simple Loose Matches - Done [6]\nRL1.6 Line Boundaries - Partial [7]\nRL1.7 Supplementary Code Points - Done [8]\n\n[1] \"\\N{U+...}\" and \"\\x{...}\"\n[2] \"\\p{...}\" \"\\P{...}\". This requirement is for a minimal list of properties. Perl\nsupports these. See R2.7 for other properties.\n[3] Perl has \"\\d\" \"\\D\" \"\\s\" \"\\S\" \"\\w\" \"\\W\" \"\\X\" \"[:prop:]\" \"[:^prop:]\", plus all the\nproperties specified by .\nThese are described above in \"Other Properties\"\n[4] The experimental feature \"(?[...])\" starting in v5.18 accomplishes this.\n\nSee \"(?[ ])\" in perlre. If you don't want to use an experimental feature, you can use\none of the following:\n\n• Regular expression lookahead\n\nYou can mimic class subtraction using lookahead. For example, what UTS#18 might\nwrite as\n\n[{Block=Greek}-[{UNASSIGNED}]]\n\nin Perl can be written as:\n\n(?!\\p{Unassigned})\\p{Block=Greek}\n(?=\\p{Assigned})\\p{Block=Greek}\n\nBut in this particular example, you probably really want\n\n\\p{Greek}\n\nwhich will match assigned characters known to be part of the Greek script.\n\n• CPAN module \"Unicode::Regex::Set\"\n\nIt does implement the full UTS#18 grouping, intersection, union, and removal\n(subtraction) syntax.\n\n• \"User-Defined Character Properties\"\n\n\"+\" for union, \"-\" for removal (set-difference), \"&\" for intersection\n\n[5] \"\\b\" \"\\B\" meet most, but not all, the details of this requirement, but \"\\b{wb}\" and\n\"\\B{wb}\" do, as well as the stricter R2.3.\n[6] Note that Perl does Full case-folding in matching, not Simple:\n\nFor example \"U+1F88\" is equivalent to \"U+1F00 U+03B9\", instead of just \"U+1F80\". This\ndifference matters mainly for certain Greek capital letters with certain modifiers: the\nFull case-folding decomposes the letter, while the Simple case-folding would map it to a\nsingle character.\n\n[7] The reason this is considered to be only partially implemented is that Perl has\n\"qr/\\b{lb}/\" and \"Unicode::LineBreak\" that are conformant with UAX#14 \"Unicode Line\nBreaking Algorithm\" . The regular expression\nconstruct provides default behavior, while the heavier-weight module provides\ncustomizable line breaking.\n\nBut Perl treats \"\\n\" as the start- and end-line delimiter, whereas Unicode specifies more\ncharacters that should be so-interpreted.\n\nThese are:\n\nVT U+000B (\\v in C)\nFF U+000C (\\f)\nCR U+000D (\\r)\nNEL U+0085\nLS U+2028\nPS U+2029\n\n\"^\" and \"$\" in regular expression patterns are supposed to match all these, but don't.\nThese characters also don't, but should, affect \"<>\" $., and script line numbers.\n\nAlso, lines should not be split within \"CRLF\" (i.e. there is no empty line between \"\\r\"\nand \"\\n\"). For \"CRLF\", try the \":crlf\" layer (see PerlIO).\n\n[8] UTF-8/UTF-EBDDIC used in Perl allows not only \"U+10000\" to \"U+10FFFF\" but also beyond\n\"U+10FFFF\"\n\nLevel 2 - Extended Unicode Support\n\nRL2.1 Canonical Equivalents - Retracted [9]\nby Unicode\nRL2.2 Extended Grapheme Clusters and - Partial [10]\nCharacter Classes with Strings\nRL2.3 Default Word Boundaries - Done [11]\nRL2.4 Default Case Conversion - Done\nRL2.5 Name Properties - Done\nRL2.6 Wildcards in Property Values - Partial [12]\nRL2.7 Full Properties - Partial [13]\nRL2.8 Optional Properties - Partial [14]\n\n[9] Unicode has rewritten this portion of UTS#18 to say that getting canonical equivalence\n(see UAX#15 \"Unicode Normalization Forms\" ) is\nbasically to be done at the programmer level. Use NFD to write both your regular expressions\nand text to match them against (you can use Unicode::Normalize).\n[10] Perl has \"\\X\" and \"\\b{gcb}\". Unicode has retracted their \"Grapheme Cluster Mode\", and\nrecently added string properties, which Perl does not yet support.\n[11] see UAX#29 \"Unicode Text Segmentation\" ,\n[12] see \"Wildcards in Property Values\" above.\n[13] Perl supports all the properties in the Unicode Character Database (UCD). It does not\nyet support the listed properties that come from other Unicode sources.\n[14] The only optional property that Perl supports is Named Sequence. None of these\nproperties are in the UCD.\n\nLevel 3 - Tailored Support\n\nThis has been retracted by Unicode.\n" }, { "name": "Unicode Encodings", "content": "Unicode characters are assigned to code points, which are abstract numbers. To use these\nnumbers, various encodings are needed.\n\n• UTF-8\n\nUTF-8 is a variable-length (1 to 4 bytes), byte-order independent encoding. In most of\nPerl's documentation, including elsewhere in this document, the term \"UTF-8\" means also\n\"UTF-EBCDIC\". But in this section, \"UTF-8\" refers only to the encoding used on ASCII\nplatforms. It is a superset of 7-bit US-ASCII, so anything encoded in ASCII has the\nidentical representation when encoded in UTF-8.\n\nThe following table is from Unicode 3.2.\n\nCode Points 1st Byte 2nd Byte 3rd Byte 4th Byte\n\nU+0000..U+007F 00..7F\nU+0080..U+07FF * C2..DF 80..BF\nU+0800..U+0FFF E0 * A0..BF 80..BF\nU+1000..U+CFFF E1..EC 80..BF 80..BF\nU+D000..U+D7FF ED 80..9F 80..BF\nU+D800..U+DFFF +++++ utf16 surrogates, not legal utf8 +++++\nU+E000..U+FFFF EE..EF 80..BF 80..BF\nU+10000..U+3FFFF F0 * 90..BF 80..BF 80..BF\nU+40000..U+FFFFF F1..F3 80..BF 80..BF 80..BF\nU+100000..U+10FFFF F4 80..8F 80..BF 80..BF\n\nNote the gaps marked by \"*\" before several of the byte entries above. These are caused\nby legal UTF-8 avoiding non-shortest encodings: it is technically possible to\nUTF-8-encode a single code point in different ways, but that is explicitly forbidden, and\nthe shortest possible encoding should always be used (and that is what Perl does).\n\nAnother way to look at it is via bits:\n\nCode Points 1st Byte 2nd Byte 3rd Byte 4th Byte\n\n0aaaaaaa 0aaaaaaa\n00000bbbbbaaaaaa 110bbbbb 10aaaaaa\nccccbbbbbbaaaaaa 1110cccc 10bbbbbb 10aaaaaa\n00000dddccccccbbbbbbaaaaaa 11110ddd 10cccccc 10bbbbbb 10aaaaaa\n\nAs you can see, the continuation bytes all begin with \"10\", and the leading bits of the\nstart byte tell how many bytes there are in the encoded character.\n\nThe original UTF-8 specification allowed up to 6 bytes, to allow encoding of numbers up\nto \"0x7FFFFFFF\". Perl continues to allow those, and has extended that up to 13 bytes to\nencode code points up to what can fit in a 64-bit word. However, Perl will warn if you\noutput any of these as being non-portable; and under strict UTF-8 input protocols, they\nare forbidden. In addition, it is now illegal to use a code point larger than what a\nsigned integer variable on your system can hold. On 32-bit ASCII systems, this means\n\"0x7FFFFFFF\" is the legal maximum (much higher on 64-bit systems).\n\n• UTF-EBCDIC\n\nLike UTF-8, but EBCDIC-safe, in the way that UTF-8 is ASCII-safe. This means that all\nthe basic characters (which includes all those that have ASCII equivalents (like \"A\",\n\"0\", \"%\", etc.) are the same in both EBCDIC and UTF-EBCDIC.)\n\nUTF-EBCDIC is used on EBCDIC platforms. It generally requires more bytes to represent a\ngiven code point than UTF-8 does; the largest Unicode code points take 5 bytes to\nrepresent (instead of 4 in UTF-8), and, extended for 64-bit words, it uses 14 bytes\ninstead of 13 bytes in UTF-8.\n\n• UTF-16, UTF-16BE, UTF-16LE, Surrogates, and \"BOM\"'s (Byte Order Marks)\n\nThe followings items are mostly for reference and general Unicode knowledge, Perl doesn't\nuse these constructs internally.\n\nLike UTF-8, UTF-16 is a variable-width encoding, but where UTF-8 uses 8-bit code units,\nUTF-16 uses 16-bit code units. All code points occupy either 2 or 4 bytes in UTF-16:\ncode points \"U+0000..U+FFFF\" are stored in a single 16-bit unit, and code points\n\"U+10000..U+10FFFF\" in two 16-bit units. The latter case is using surrogates, the first\n16-bit unit being the high surrogate, and the second being the low surrogate.\n\nSurrogates are code points set aside to encode the \"U+10000..U+10FFFF\" range of Unicode\ncode points in pairs of 16-bit units. The high surrogates are the range \"U+D800..U+DBFF\"\nand the low surrogates are the range \"U+DC00..U+DFFF\". The surrogate encoding is\n\n$hi = ($uni - 0x10000) / 0x400 + 0xD800;\n$lo = ($uni - 0x10000) % 0x400 + 0xDC00;\n\nand the decoding is\n\n$uni = 0x10000 + ($hi - 0xD800) * 0x400 + ($lo - 0xDC00);\n\nBecause of the 16-bitness, UTF-16 is byte-order dependent. UTF-16 itself can be used for\nin-memory computations, but if storage or transfer is required either UTF-16BE (big-\nendian) or UTF-16LE (little-endian) encodings must be chosen.\n\nThis introduces another problem: what if you just know that your data is UTF-16, but you\ndon't know which endianness? Byte Order Marks, or \"BOM\"'s, are a solution to this. A\nspecial character has been reserved in Unicode to function as a byte order marker: the\ncharacter with the code point \"U+FEFF\" is the \"BOM\".\n\nThe trick is that if you read a \"BOM\", you will know the byte order, since if it was\nwritten on a big-endian platform, you will read the bytes \"0xFE 0xFF\", but if it was\nwritten on a little-endian platform, you will read the bytes \"0xFF 0xFE\". (And if the\noriginating platform was writing in ASCII platform UTF-8, you will read the bytes \"0xEF\n0xBB 0xBF\".)\n\nThe way this trick works is that the character with the code point \"U+FFFE\" is not\nsupposed to be in input streams, so the sequence of bytes \"0xFF 0xFE\" is unambiguously\n\"\"BOM\", represented in little-endian format\" and cannot be \"U+FFFE\", represented in big-\nendian format\".\n\nSurrogates have no meaning in Unicode outside their use in pairs to represent other code\npoints. However, Perl allows them to be represented individually internally, for example\nby saying \"chr(0xD801)\", so that all code points, not just those valid for open\ninterchange, are representable. Unicode does define semantics for them, such as their\n\"GeneralCategory\" is \"Cs\". But because their use is somewhat dangerous, Perl will warn\n(using the warning category \"surrogate\", which is a sub-category of \"utf8\") if an attempt\nis made to do things like take the lower case of one, or match case-insensitively, or to\noutput them. (But don't try this on Perls before 5.14.)\n\n• UTF-32, UTF-32BE, UTF-32LE\n\nThe UTF-32 family is pretty much like the UTF-16 family, except that the units are\n32-bit, and therefore the surrogate scheme is not needed. UTF-32 is a fixed-width\nencoding. The \"BOM\" signatures are \"0x00 0x00 0xFE 0xFF\" for BE and \"0xFF 0xFE 0x00\n0x00\" for LE.\n\n• UCS-2, UCS-4\n\nLegacy, fixed-width encodings defined by the ISO 10646 standard. UCS-2 is a 16-bit\nencoding. Unlike UTF-16, UCS-2 is not extensible beyond \"U+FFFF\", because it does not\nuse surrogates. UCS-4 is a 32-bit encoding, functionally identical to UTF-32 (the\ndifference being that UCS-4 forbids neither surrogates nor code points larger than\n\"0x10FFFF\").\n\n• UTF-7\n\nA seven-bit safe (non-eight-bit) encoding, which is useful if the transport or storage is\nnot eight-bit safe. Defined by RFC 2152.\n" }, { "name": "Noncharacter code points", "content": "66 code points are set aside in Unicode as \"noncharacter code points\". These all have the\n\"Unassigned\" (\"Cn\") \"GeneralCategory\", and no character will ever be assigned to any of\nthem. They are the 32 code points between \"U+FDD0\" and \"U+FDEF\" inclusive, and the 34 code\npoints:\n\nU+FFFE U+FFFF\nU+1FFFE U+1FFFF\nU+2FFFE U+2FFFF\n...\nU+EFFFE U+EFFFF\nU+FFFFE U+FFFFF\nU+10FFFE U+10FFFF\n\nUntil Unicode 7.0, the noncharacters were \"forbidden for use in open interchange of Unicode\ntext data\", so that code that processed those streams could use these code points as\nsentinels that could be mixed in with character data, and would always be distinguishable\nfrom that data. (Emphasis above and in the next paragraph are added in this document.)\n\nUnicode 7.0 changed the wording so that they are \"not recommended for use in open interchange\nof Unicode text data\". The 7.0 Standard goes on to say:\n\n\"If a noncharacter is received in open interchange, an application is not required to\ninterpret it in any way. It is good practice, however, to recognize it as a noncharacter\nand to take appropriate action, such as replacing it with \"U+FFFD\" replacement character,\nto indicate the problem in the text. It is not recommended to simply delete noncharacter\ncode points from such text, because of the potential security issues caused by deleting\nuninterpreted characters. (See conformance clause C7 in Section 3.2, Conformance\nRequirements, and Unicode Technical Report #36, \"Unicode Security Considerations\"\n).\"\n\nThis change was made because it was found that various commercial tools like editors, or for\nthings like source code control, had been written so that they would not handle program files\nthat used these code points, effectively precluding their use almost entirely! And that was\nnever the intent. They've always been meant to be usable within an application, or\ncooperating set of applications, at will.\n\nIf you're writing code, such as an editor, that is supposed to be able to handle any Unicode\ntext data, then you shouldn't be using these code points yourself, and instead allow them in\nthe input. If you need sentinels, they should instead be something that isn't legal Unicode.\nFor UTF-8 data, you can use the bytes 0xC1 and 0xC2 as sentinels, as they never appear in\nwell-formed UTF-8. (There are equivalents for UTF-EBCDIC). You can also store your Unicode\ncode points in integer variables and use negative values as sentinels.\n\nIf you're not writing such a tool, then whether you accept noncharacters as input is up to\nyou (though the Standard recommends that you not). If you do strict input stream checking\nwith Perl, these code points continue to be forbidden. This is to maintain backward\ncompatibility (otherwise potential security holes could open up, as an unsuspecting\napplication that was written assuming the noncharacters would be filtered out before getting\nto it, could now, without warning, start getting them). To do strict checking, you can use\nthe layer \":encoding('UTF-8')\".\n\nPerl continues to warn (using the warning category \"nonchar\", which is a sub-category of\n\"utf8\") if an attempt is made to output noncharacters.\n" }, { "name": "Beyond Unicode code points", "content": "The maximum Unicode code point is \"U+10FFFF\", and Unicode only defines operations on code\npoints up through that. But Perl works on code points up to the maximum permissible signed\nnumber available on the platform. However, Perl will not accept these from input streams\nunless lax rules are being used, and will warn (using the warning category \"nonunicode\",\nwhich is a sub-category of \"utf8\") if any are output.\n\nSince Unicode rules are not defined on these code points, if a Unicode-defined operation is\ndone on them, Perl uses what we believe are sensible rules, while generally warning, using\nthe \"nonunicode\" category. For example, \"uc(\"\\x{110000}\")\" will generate such a warning,\nreturning the input parameter as its result, since Perl defines the uppercase of every non-\nUnicode code point to be the code point itself. (All the case changing operations, not just\nuppercasing, work this way.)\n\nThe situation with matching Unicode properties in regular expressions, the \"\\p{}\" and \"\\P{}\"\nconstructs, against these code points is not as clear cut, and how these are handled has\nchanged as we've gained experience.\n\nOne possibility is to treat any match against these code points as undefined. But since Perl\ndoesn't have the concept of a match being undefined, it converts this to failing or \"FALSE\".\nThis is almost, but not quite, what Perl did from v5.14 (when use of these code points became\ngenerally reliable) through v5.18. The difference is that Perl treated all \"\\p{}\" matches as\nfailing, but all \"\\P{}\" matches as succeeding.\n\nOne problem with this is that it leads to unexpected, and confusing results in some cases:\n\nchr(0x110000) =~ \\p{ASCIIHexDigit=True} # Failed on <= v5.18\nchr(0x110000) =~ \\p{ASCIIHexDigit=False} # Failed! on <= v5.18\n\nThat is, it treated both matches as undefined, and converted that to false (raising a warning\non each). The first case is the expected result, but the second is likely counterintuitive:\n\"How could both be false when they are complements?\" Another problem was that the\nimplementation optimized many Unicode property matches down to already existing simpler,\nfaster operations, which don't raise the warning. We chose to not forgo those optimizations,\nwhich help the vast majority of matches, just to generate a warning for the unlikely event\nthat an above-Unicode code point is being matched against.\n\nAs a result of these problems, starting in v5.20, what Perl does is to treat non-Unicode code\npoints as just typical unassigned Unicode characters, and matches accordingly. (Note:\nUnicode has atypical unassigned code points. For example, it has noncharacter code points,\nand ones that, when they do get assigned, are destined to be written Right-to-left, as Arabic\nand Hebrew are. Perl assumes that no non-Unicode code point has any atypical properties.)\n\nPerl, in most cases, will raise a warning when matching an above-Unicode code point against a\nUnicode property when the result is \"TRUE\" for \"\\p{}\", and \"FALSE\" for \"\\P{}\". For example:\n\nchr(0x110000) =~ \\p{ASCIIHexDigit=True} # Fails, no warning\nchr(0x110000) =~ \\p{ASCIIHexDigit=False} # Succeeds, with warning\n\nIn both these examples, the character being matched is non-Unicode, so Unicode doesn't define\nhow it should match. It clearly isn't an ASCII hex digit, so the first example clearly\nshould fail, and so it does, with no warning. But it is arguable that the second example\nshould have an undefined, hence \"FALSE\", result. So a warning is raised for it.\n\nThus the warning is raised for many fewer cases than in earlier Perls, and only when what the\nresult is could be arguable. It turns out that none of the optimizations made by Perl (or\nare ever likely to be made) cause the warning to be skipped, so it solves both problems of\nPerl's earlier approach. The most commonly used property that is affected by this change is\n\"\\p{Unassigned}\" which is a short form for \"\\p{GeneralCategory=Unassigned}\". Starting in\nv5.20, all non-Unicode code points are considered \"Unassigned\". In earlier releases the\nmatches failed because the result was considered undefined.\n\nThe only place where the warning is not raised when it might ought to have been is if\noptimizations cause the whole pattern match to not even be attempted. For example, Perl may\nfigure out that for a string to match a certain regular expression pattern, the string has to\ncontain the substring \"foobar\". Before attempting the match, Perl may look for that\nsubstring, and if not found, immediately fail the match without actually trying it; so no\nwarning gets generated even if the string contains an above-Unicode code point.\n\nThis behavior is more \"Do what I mean\" than in earlier Perls for most applications. But it\ncatches fewer issues for code that needs to be strictly Unicode compliant. Therefore there\nis an additional mode of operation available to accommodate such code. This mode is enabled\nif a regular expression pattern is compiled within the lexical scope where the \"nonunicode\"\nwarning class has been made fatal, say by:\n\nuse warnings FATAL => \"nonunicode\"\n\n(see warnings). In this mode of operation, Perl will raise the warning for all matches\nagainst a non-Unicode code point (not just the arguable ones), and it skips the optimizations\nthat might cause the warning to not be output. (It currently still won't warn if the match\nisn't even attempted, like in the \"foobar\" example above.)\n\nIn summary, Perl now normally treats non-Unicode code points as typical Unicode unassigned\ncode points for regular expression matches, raising a warning only when it is arguable what\nthe result should be. However, if this warning has been made fatal, it isn't skipped.\n\nThere is one exception to all this. \"\\p{All}\" looks like a Unicode property, but it is a\nPerl extension that is defined to be true for all possible code points, Unicode or not, so no\nwarning is ever generated when matching this against a non-Unicode code point. (Prior to\nv5.20, it was an exact synonym for \"\\p{Any}\", matching code points 0 through 0x10FFFF.)\n" }, { "name": "Security Implications of Unicode", "content": "First, read Unicode Security Considerations .\n\nAlso, note the following:\n\n• Malformed UTF-8\n\nUTF-8 is very structured, so many combinations of bytes are invalid. In the past, Perl\ntried to soldier on and make some sense of invalid combinations, but this can lead to\nsecurity holes, so now, if the Perl core needs to process an invalid combination, it will\neither raise a fatal error, or will replace those bytes by the sequence that forms the\nUnicode REPLACEMENT CHARACTER, for which purpose Unicode created it.\n\nEvery code point can be represented by more than one possible syntactically valid UTF-8\nsequence. Early on, both Unicode and Perl considered any of these to be valid, but now,\nall sequences longer than the shortest possible one are considered to be malformed.\n\nUnicode considers many code points to be illegal, or to be avoided. Perl generally\naccepts them, once they have passed through any input filters that may try to exclude\nthem. These have been discussed above (see \"Surrogates\" under UTF-16 in \"Unicode\nEncodings\", \"Noncharacter code points\", and \"Beyond Unicode code points\").\n\n• Regular expression pattern matching may surprise you if you're not accustomed to Unicode.\nStarting in Perl 5.14, several pattern modifiers are available to control this, called\nthe character set modifiers. Details are given in \"Character set modifiers\" in perlre.\n\nAs discussed elsewhere, Perl has one foot (two hooves?) planted in each of two worlds: the\nold world of ASCII and single-byte locales, and the new world of Unicode, upgrading when\nnecessary. If your legacy code does not explicitly use Unicode, no automatic switch-over to\nUnicode should happen.\n" }, { "name": "Unicode in Perl on EBCDIC", "content": "Unicode is supported on EBCDIC platforms. See perlebcdic.\n\nUnless ASCII vs. EBCDIC issues are specifically being discussed, references to UTF-8 encoding\nin this document and elsewhere should be read as meaning UTF-EBCDIC on EBCDIC platforms. See\n\"Unicode and UTF\" in perlebcdic.\n\nBecause UTF-EBCDIC is so similar to UTF-8, the differences are mostly hidden from you;\n\"use utf8\" (and NOT something like \"use utfebcdic\") declares the script is in the platform's\n\"native\" 8-bit encoding of Unicode. (Similarly for the \":utf8\" layer.)\n" }, { "name": "Locales", "content": "See \"Unicode and UTF-8\" in perllocale\n" }, { "name": "When Unicode Does Not Happen", "content": "There are still many places where Unicode (in some encoding or another) could be given as\narguments or received as results, or both in Perl, but it is not, in spite of Perl having\nextensive ways to input and output in Unicode, and a few other \"entry points\" like the @ARGV\narray (which can sometimes be interpreted as UTF-8).\n\nThe following are such interfaces. Also, see \"The \"Unicode Bug\"\". For all of these\ninterfaces Perl currently (as of v5.16.0) simply assumes byte strings both as arguments and\nresults, or UTF-8 strings if the (deprecated) \"encoding\" pragma has been used.\n\nOne reason that Perl does not attempt to resolve the role of Unicode in these situations is\nthat the answers are highly dependent on the operating system and the file system(s). For\nexample, whether filenames can be in Unicode and in exactly what kind of encoding, is not\nexactly a portable concept. Similarly for \"qx\" and \"system\": how well will the \"command-line\ninterface\" (and which of them?) handle Unicode?\n\n• \"chdir\", \"chmod\", \"chown\", \"chroot\", \"exec\", \"link\", \"lstat\", \"mkdir\", \"rename\", \"rmdir\",\n\"stat\", \"symlink\", \"truncate\", \"unlink\", \"utime\", \"-X\"\n\n• %ENV\n\n• \"glob\" (aka the \"<*>\")\n\n• \"open\", \"opendir\", \"sysopen\"\n\n• \"qx\" (aka the backtick operator), \"system\"\n\n• \"readdir\", \"readlink\"\n" }, { "name": "The \"Unicode Bug\"", "content": "The term, \"Unicode bug\" has been applied to an inconsistency with the code points in the\n\"Latin-1 Supplement\" block, that is, between 128 and 255. Without a locale specified, unlike\nall other characters or code points, these characters can have very different semantics\ndepending on the rules in effect. (Characters whose code points are above 255 force Unicode\nrules; whereas the rules for ASCII characters are the same under both ASCII and Unicode\nrules.)\n\nUnder Unicode rules, these upper-Latin1 characters are interpreted as Unicode code points,\nwhich means they have the same semantics as Latin-1 (ISO-8859-1) and C1 controls.\n\nAs explained in \"ASCII Rules versus Unicode Rules\", under ASCII rules, they are considered to\nbe unassigned characters.\n\nThis can lead to unexpected results. For example, a string's semantics can suddenly change\nif a code point above 255 is appended to it, which changes the rules from ASCII to Unicode.\nAs an example, consider the following program and its output:\n\n$ perl -le'\nno feature \"unicodestrings\";\n$s1 = \"\\xC2\";\n$s2 = \"\\x{2660}\";\nfor ($s1, $s2, $s1.$s2) {\nprint /\\w/ || 0;\n}\n'\n0\n0\n1\n\nIf there's no \"\\w\" in \"s1\" nor in \"s2\", why does their concatenation have one?\n\nThis anomaly stems from Perl's attempt to not disturb older programs that didn't use Unicode,\nalong with Perl's desire to add Unicode support seamlessly. But the result turned out to not\nbe seamless. (By the way, you can choose to be warned when things like this happen. See\n\"encoding::warnings\".)\n\n\"use feature 'unicodestrings'\" was added, starting in Perl v5.12, to address this problem.\nIt affects these things:\n\n• Changing the case of a scalar, that is, using \"uc()\", \"ucfirst()\", \"lc()\", and\n\"lcfirst()\", or \"\\L\", \"\\U\", \"\\u\" and \"\\l\" in double-quotish contexts, such as regular\nexpression substitutions.\n\nUnder \"unicodestrings\" starting in Perl 5.12.0, Unicode rules are generally used. See\n\"lc\" in perlfunc for details on how this works in combination with various other pragmas.\n\n• Using caseless (\"/i\") regular expression matching.\n\nStarting in Perl 5.14.0, regular expressions compiled within the scope of\n\"unicodestrings\" use Unicode rules even when executed or compiled into larger regular\nexpressions outside the scope.\n\n• Matching any of several properties in regular expressions.\n\nThese properties are \"\\b\" (without braces), \"\\B\" (without braces), \"\\s\", \"\\S\", \"\\w\",\n\"\\W\", and all the Posix character classes except \"[[:ascii:]]\".\n\nStarting in Perl 5.14.0, regular expressions compiled within the scope of\n\"unicodestrings\" use Unicode rules even when executed or compiled into larger regular\nexpressions outside the scope.\n\n• In \"quotemeta\" or its inline equivalent \"\\Q\".\n\nStarting in Perl 5.16.0, consistent quoting rules are used within the scope of\n\"unicodestrings\", as described in \"quotemeta\" in perlfunc. Prior to that, or outside\nits scope, no code points above 127 are quoted in UTF-8 encoded strings, but in byte\nencoded strings, code points between 128-255 are always quoted.\n\n• In the \"..\" or range operator.\n\nStarting in Perl 5.26.0, the range operator on strings treats their lengths consistently\nwithin the scope of \"unicodestrings\". Prior to that, or outside its scope, it could\nproduce strings whose length in characters exceeded that of the right-hand side, where\nthe right-hand side took up more bytes than the correct range endpoint.\n\n• In \"split\"'s special-case whitespace splitting.\n\nStarting in Perl 5.28.0, the \"split\" function with a pattern specified as a string\ncontaining a single space handles whitespace characters consistently within the scope of\n\"unicodestrings\". Prior to that, or outside its scope, characters that are whitespace\naccording to Unicode rules but not according to ASCII rules were treated as field\ncontents rather than field separators when they appear in byte-encoded strings.\n\nYou can see from the above that the effect of \"unicodestrings\" increased over several Perl\nreleases. (And Perl's support for Unicode continues to improve; it's best to use the latest\navailable release in order to get the most complete and accurate results possible.) Note\nthat \"unicodestrings\" is automatically chosen if you \"use 5.012\" or higher.\n\nFor Perls earlier than those described above, or when a string is passed to a function\noutside the scope of \"unicodestrings\", see the next section.\n" }, { "name": "Forcing Unicode in Perl (Or Unforcing Unicode in Perl)", "content": "Sometimes (see \"When Unicode Does Not Happen\" or \"The \"Unicode Bug\"\") there are situations\nwhere you simply need to force a byte string into UTF-8, or vice versa. The standard module\nEncode can be used for this, or the low-level calls \"utf8::upgrade($bytestring)\" and\n\"utf8::downgrade($utf8string[, FAILOK])\".\n\nNote that \"utf8::downgrade()\" can fail if the string contains characters that don't fit into\na byte.\n\nCalling either function on a string that already is in the desired state is a no-op.\n\n\"ASCII Rules versus Unicode Rules\" gives all the ways that a string is made to use Unicode\nrules.\n" }, { "name": "Using Unicode in XS", "content": "See \"Unicode Support\" in perlguts for an introduction to Unicode at the XS level, and\n\"Unicode Support\" in perlapi for the API details.\n" }, { "name": "Hacking Perl to work on earlier Unicode versions (for very serious hackers only)", "content": "Perl by default comes with the latest supported Unicode version built-in, but the goal is to\nallow you to change to use any earlier one. In Perls v5.20 and v5.22, however, the earliest\nusable version is Unicode 5.1. Perl v5.18 and v5.24 are able to handle all earlier versions.\n\nDownload the files in the desired version of Unicode from the Unicode web site\n). These should replace the existing files in lib/unicore in the\nPerl source tree. Follow the instructions in README.perl in that directory to change some of\ntheir names, and then build perl (see INSTALL).\n" }, { "name": "Porting code from perl-5.6.X", "content": "Perls starting in 5.8 have a different Unicode model from 5.6. In 5.6 the programmer was\nrequired to use the \"utf8\" pragma to declare that a given scope expected to deal with Unicode\ndata and had to make sure that only Unicode data were reaching that scope. If you have code\nthat is working with 5.6, you will need some of the following adjustments to your code. The\nexamples are written such that the code will continue to work under 5.6, so you should be\nsafe to try them out.\n\n• A filehandle that should read or write UTF-8\n\nif ($] > 5.008) {\nbinmode $fh, \":encoding(UTF-8)\";\n}\n\n• A scalar that is going to be passed to some extension\n\nBe it \"Compress::Zlib\", \"Apache::Request\" or any extension that has no mention of Unicode\nin the manpage, you need to make sure that the UTF8 flag is stripped off. Note that at the\ntime of this writing (January 2012) the mentioned modules are not UTF-8-aware. Please\ncheck the documentation to verify if this is still true.\n\nif ($] > 5.008) {\nrequire Encode;\n$val = Encode::encode(\"UTF-8\", $val); # make octets\n}\n\n• A scalar we got back from an extension\n\nIf you believe the scalar comes back as UTF-8, you will most likely want the UTF8 flag\nrestored:\n\nif ($] > 5.008) {\nrequire Encode;\n$val = Encode::decode(\"UTF-8\", $val);\n}\n\n• Same thing, if you are really sure it is UTF-8\n\nif ($] > 5.008) {\nrequire Encode;\nEncode::utf8on($val);\n}\n\n• A wrapper for DBI \"fetchrowarray\" and \"fetchrowhashref\"\n\nWhen the database contains only UTF-8, a wrapper function or method is a convenient way to\nreplace all your \"fetchrowarray\" and \"fetchrowhashref\" calls. A wrapper function will\nalso make it easier to adapt to future enhancements in your database driver. Note that at\nthe time of this writing (January 2012), the DBI has no standardized way to deal with\nUTF-8 data. Please check the DBI documentation to verify if that is still true.\n\nsub fetchrow {\n# $what is one of fetchrow{array,hashref}\nmy($self, $sth, $what) = @;\nif ($] < 5.008) {\nreturn $sth->$what;\n} else {\nrequire Encode;\nif (wantarray) {\nmy @arr = $sth->$what;\nfor (@arr) {\ndefined && /[^\\000-\\177]/ && Encode::utf8on($);\n}\nreturn @arr;\n} else {\nmy $ret = $sth->$what;\nif (ref $ret) {\nfor my $k (keys %$ret) {\ndefined\n&& /[^\\000-\\177]/\n&& Encode::utf8on($) for $ret->{$k};\n}\nreturn $ret;\n} else {\ndefined && /[^\\000-\\177]/ && Encode::utf8on($) for $ret;\nreturn $ret;\n}\n}\n}\n}\n\n• A large scalar that you know can only contain ASCII\n\nScalars that contain only ASCII and are marked as UTF-8 are sometimes a drag to your\nprogram. If you recognize such a situation, just remove the UTF8 flag:\n\nutf8::downgrade($val) if $] > 5.008;\n" } ] }, "BUGS": { "content": "See also \"The \"Unicode Bug\"\" above.\n", "subsections": [ { "name": "Interaction with Extensions", "content": "When Perl exchanges data with an extension, the extension should be able to understand the\nUTF8 flag and act accordingly. If the extension doesn't recognize that flag, it's likely that\nthe extension will return incorrectly-flagged data.\n\nSo if you're working with Unicode data, consult the documentation of every module you're\nusing if there are any issues with Unicode data exchange. If the documentation does not talk\nabout Unicode at all, suspect the worst and probably look at the source to learn how the\nmodule is implemented. Modules written completely in Perl shouldn't cause problems. Modules\nthat directly or indirectly access code written in other programming languages are at risk.\n\nFor affected functions, the simple strategy to avoid data corruption is to always make the\nencoding of the exchanged data explicit. Choose an encoding that you know the extension can\nhandle. Convert arguments passed to the extensions to that encoding and convert results back\nfrom that encoding. Write wrapper functions that do the conversions for you, so you can later\nchange the functions when the extension catches up.\n\nTo provide an example, let's say the popular \"Foo::Bar::escapehtml\" function doesn't deal\nwith Unicode data yet. The wrapper function would convert the argument to raw UTF-8 and\nconvert the result back to Perl's internal representation like so:\n\nsub myescapehtml ($) {\nmy($what) = shift;\nreturn unless defined $what;\nEncode::decode(\"UTF-8\", Foo::Bar::escapehtml(\nEncode::encode(\"UTF-8\", $what)));\n}\n\nSometimes, when the extension does not convert data but just stores and retrieves it, you\nwill be able to use the otherwise dangerous \"Encode::utf8on()\" function. Let's say the\npopular \"Foo::Bar\" extension, written in C, provides a \"param\" method that lets you store and\nretrieve data according to these prototypes:\n\n$self->param($name, $value); # set a scalar\n$value = $self->param($name); # retrieve a scalar\n\nIf it does not yet provide support for any encoding, one could write a derived class with\nsuch a \"param\" method:\n\nsub param {\nmy($self,$name,$value) = @;\nutf8::upgrade($name); # make sure it is UTF-8 encoded\nif (defined $value) {\nutf8::upgrade($value); # make sure it is UTF-8 encoded\nreturn $self->SUPER::param($name,$value);\n} else {\nmy $ret = $self->SUPER::param($name);\nEncode::utf8on($ret); # we know, it is UTF-8 encoded\nreturn $ret;\n}\n}\n\nSome extensions provide filters on data entry/exit points, such as\n\"DBFile::filterstorekey\" and family. Look out for such filters in the documentation of\nyour extensions; they can make the transition to Unicode data much easier.\n" }, { "name": "Speed", "content": "Some functions are slower when working on UTF-8 encoded strings than on byte encoded strings.\nAll functions that need to hop over characters such as \"length()\", \"substr()\" or \"index()\",\nor matching regular expressions can work much faster when the underlying data are byte-\nencoded.\n\nIn Perl 5.8.0 the slowness was often quite spectacular; in Perl 5.8.1 a caching scheme was\nintroduced which improved the situation. In general, operations with UTF-8 encoded strings\nare still slower. As an example, the Unicode properties (character classes) like \"\\p{Nd}\" are\nknown to be quite a bit slower (5-20 times) than their simpler counterparts like \"[0-9]\"\n(then again, there are hundreds of Unicode characters matching \"Nd\" compared with the 10\nASCII characters matching \"[0-9]\").\n" } ] }, "SEE ALSO": { "content": "perlunitut, perluniintro, perluniprops, Encode, open, utf8, bytes, perlretut, \"${^UNICODE}\"\nin perlvar, ).\n\n\n\nperl v5.34.0 2025-07-25 PERLUNICODE(1)", "subsections": [] } } } }