{
    "mode": "man",
    "parameter": "perlunicook",
    "section": "1",
    "url": "https://www.chedong.com/phpMan.php/man/perlunicook/1/json",
    "generated": "2026-06-03T00:34:50Z",
    "sections": {
        "NAME": {
            "content": "perlunicook - cookbookish examples of handling Unicode in Perl\n",
            "subsections": []
        },
        "DESCRIPTION": {
            "content": "This manpage contains short recipes demonstrating how to handle common Unicode operations in\nPerl, plus one complete program at the end. Any undeclared variables in individual recipes\nare assumed to have a previous appropriate value in them.\n",
            "subsections": []
        },
        "EXAMPLES": {
            "content": "℞℞ 0: Standard preamble\nUnless otherwise notes, all examples below require this standard preamble to work correctly,\nwith the \"#!\" adjusted to work on your system:\n\n#!/usr/bin/env perl\n\nuse utf8;      # so literals and identifiers can be in UTF-8\nuse v5.12;     # or later to get \"unicodestrings\" feature\nuse strict;    # quote strings, declare variables\nuse warnings;  # on by default\nuse warnings  qw(FATAL utf8);    # fatalize encoding glitches\nuse open      qw(:std :encoding(UTF-8)); # undeclared streams in UTF-8\nuse charnames qw(:full :short);  # unneeded in v5.16\n\nThis does make even Unix programmers \"binmode\" your binary streams, or open them with \":raw\",\nbut that's the only way to get at them portably anyway.\n\nWARNING: \"use autodie\" (pre 2.26) and \"use open\" do not get along with each other.\n\n℞℞ 1: Generic Unicode-savvy filter\nAlways decompose on the way in, then recompose on the way out.\n\nuse Unicode::Normalize;\n\nwhile (<>) {\n$ = NFD($);   # decompose + reorder canonically\n...\n} continue {\nprint NFC($);  # recompose (where possible) + reorder canonically\n}\n\n℞℞ 2: Fine-tuning Unicode warnings\nAs of v5.14, Perl distinguishes three subclasses of UTF‑8 warnings.\n\nuse v5.14;                  # subwarnings unavailable any earlier\nno warnings \"nonchar\";      # the 66 forbidden non-characters\nno warnings \"surrogate\";    # UTF-16/CESU-8 nonsense\nno warnings \"nonunicode\";  # for codepoints over 0x10FFFF\n\n℞℞ 3: Declare source in utf8 for identifiers and literals\nWithout the all-critical \"use utf8\" declaration, putting UTF‑8 in your literals and\nidentifiers won’t work right.  If you used the standard preamble just given above, this\nalready happened.  If you did, you can do things like this:\n\nuse utf8;\n\nmy $measure   = \"Ångström\";\nmy @μsoft     = qw( cp852 cp1251 cp1252 );\nmy @ὑπέρμεγας = qw( ὑπέρ  μεγας );\nmy @鯉        = qw( koi8-f koi8-u koi8-r );\nmy $motto     = \"👪 💗 🐪\"; # FAMILY, GROWING HEART, DROMEDARY CAMEL\n\nIf you forget \"use utf8\", high bytes will be misunderstood as separate characters, and\nnothing will work right.\n\n℞℞ 4: Characters and their numbers\nThe \"ord\" and \"chr\" functions work transparently on all codepoints, not just on ASCII alone —\nnor in fact, not even just on Unicode alone.\n\n# ASCII characters\nord(\"A\")\nchr(65)\n\n# characters from the Basic Multilingual Plane\nord(\"Σ\")\nchr(0x3A3)\n\n# beyond the BMP\nord(\"𝑛\")               # MATHEMATICAL ITALIC SMALL N\nchr(0x1D45B)\n\n# beyond Unicode! (up to MAXINT)\nord(\"\\x{200000}\")\nchr(0x200000)\n\n℞℞ 5: Unicode literals by character number\nIn an interpolated literal, whether a double-quoted string or a regex, you may specify a\ncharacter by its number using the \"\\x{HHHHHH}\" escape.\n\nString: \"\\x{3a3}\"\nRegex:  /\\x{3a3}/\n\nString: \"\\x{1d45b}\"\nRegex:  /\\x{1d45b}/\n\n# even non-BMP ranges in regex work fine\n/[\\x{1D434}-\\x{1D467}]/\n\n℞℞ 6: Get character name by number\nuse charnames ();\nmy $name = charnames::viacode(0x03A3);\n\n℞℞ 7: Get character number by name\nuse charnames ();\nmy $number = charnames::vianame(\"GREEK CAPITAL LETTER SIGMA\");\n\n℞℞ 8: Unicode named characters\nUse the \"\\N{charname}\" notation to get the character by that name for use in interpolated\nliterals (double-quoted strings and regexes).  In v5.16, there is an implicit\n\nuse charnames qw(:full :short);\n\nBut prior to v5.16, you must be explicit about which set of charnames you want.  The \":full\"\nnames are the official Unicode character name, alias, or sequence, which all share a\nnamespace.\n\nuse charnames qw(:full :short latin greek);\n\n\"\\N{MATHEMATICAL ITALIC SMALL N}\"      # :full\n\"\\N{GREEK CAPITAL LETTER SIGMA}\"       # :full\n\nAnything else is a Perl-specific convenience abbreviation.  Specify one or more scripts by\nnames if you want short names that are script-specific.\n\n\"\\N{Greek:Sigma}\"                      # :short\n\"\\N{ae}\"                               #  latin\n\"\\N{epsilon}\"                          #  greek\n\nThe v5.16 release also supports a \":loose\" import for loose matching of character names,\nwhich works just like loose matching of property names: that is, it disregards case,\nwhitespace, and underscores:\n\n\"\\N{euro sign}\"                        # :loose (from v5.16)\n\nStarting in v5.32, you can also use\n\nqr/\\p{name=euro sign}/\n\nto get official Unicode named characters in regular expressions.  Loose matching is always\ndone for these.\n\n℞℞ 9: Unicode named sequences\nThese look just like character names but return multiple codepoints.  Notice the %vx vector-\nprint functionality in \"printf\".\n\nuse charnames qw(:full);\nmy $seq = \"\\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}\";\nprintf \"U+%v04X\\n\", $seq;\nU+0100.0300\n\n℞℞ 10: Custom named characters\nUse \":alias\" to give your own lexically scoped nicknames to existing characters, or even to\ngive unnamed private-use characters useful names.\n\nuse charnames \":full\", \":alias\" => {\necute => \"LATIN SMALL LETTER E WITH ACUTE\",\n\"APPLE LOGO\" => 0xF8FF, # private use character\n};\n\n\"\\N{ecute}\"\n\"\\N{APPLE LOGO}\"\n\n℞℞ 11: Names of CJK codepoints\nSinograms like “東京” come back with character names of \"CJK UNIFIED IDEOGRAPH-6771\" and \"CJK\nUNIFIED IDEOGRAPH-4EAC\", because their “names” vary.  The CPAN \"Unicode::Unihan\" module has a\nlarge database for decoding these (and a whole lot more), provided you know how to understand\nits output.\n\n# cpan -i Unicode::Unihan\nuse Unicode::Unihan;\nmy $str = \"東京\";\nmy $unhan = Unicode::Unihan->new;\nfor my $lang (qw(Mandarin Cantonese Korean JapaneseOn JapaneseKun)) {\nprintf \"CJK $str in %-12s is \", $lang;\nsay $unhan->$lang($str);\n}\n\nprints:\n\nCJK 東京 in Mandarin     is DONG1JING1\nCJK 東京 in Cantonese    is dung1ging1\nCJK 東京 in Korean       is TONGKYENG\nCJK 東京 in JapaneseOn   is TOUKYOU KEI KIN\nCJK 東京 in JapaneseKun  is HIGASHI AZUMAMIYAKO\n\nIf you have a specific romanization scheme in mind, use the specific module:\n\n# cpan -i Lingua::JA::Romanize::Japanese\nuse Lingua::JA::Romanize::Japanese;\nmy $k2r = Lingua::JA::Romanize::Japanese->new;\nmy $str = \"東京\";\nsay \"Japanese for $str is \", $k2r->chars($str);\n\nprints\n\nJapanese for 東京 is toukyou\n\n℞℞ 12: Explicit encode/decode\nOn rare occasion, such as a database read, you may be given encoded text you need to decode.\n\nuse Encode qw(encode decode);\n\nmy $chars = decode(\"shiftjis\", $bytes, 1);\n# OR\nmy $bytes = encode(\"MIME-Header-ISO2022JP\", $chars, 1);\n\nFor streams all in the same encoding, don't use encode/decode; instead set the file encoding\nwhen you open the file or immediately after with \"binmode\" as described later below.\n\n℞℞ 13: Decode program arguments as utf8\n$ perl -CA ...\nor\n$ export PERLUNICODE=A\nor\nuse Encode qw(decode);\n@ARGV = map { decode('UTF-8', $, 1) } @ARGV;\n\n℞℞ 14: Decode program arguments as locale encoding\n# cpan -i Encode::Locale\nuse Encode qw(locale);\nuse Encode::Locale;\n\n# use \"locale\" as an arg to encode/decode\n@ARGV = map { decode(locale => $, 1) } @ARGV;\n\n℞℞ 15: Declare STD{IN,OUT,ERR} to be utf8\nUse a command-line option, an environment variable, or else call \"binmode\" explicitly:\n\n$ perl -CS ...\nor\n$ export PERLUNICODE=S\nor\nuse open qw(:std :encoding(UTF-8));\nor\nbinmode(STDIN,  \":encoding(UTF-8)\");\nbinmode(STDOUT, \":utf8\");\nbinmode(STDERR, \":utf8\");\n\n℞℞ 16: Declare STD{IN,OUT,ERR} to be in locale encoding\n# cpan -i Encode::Locale\nuse Encode;\nuse Encode::Locale;\n\n# or as a stream for binmode or open\nbinmode STDIN,  \":encoding(consolein)\"  if -t STDIN;\nbinmode STDOUT, \":encoding(consoleout)\" if -t STDOUT;\nbinmode STDERR, \":encoding(consoleout)\" if -t STDERR;\n\n℞℞ 17: Make file I/O default to utf8\nFiles opened without an encoding argument will be in UTF-8:\n\n$ perl -CD ...\nor\n$ export PERLUNICODE=D\nor\nuse open qw(:encoding(UTF-8));\n\n℞℞ 18: Make all I/O and args default to utf8\n$ perl -CSDA ...\nor\n$ export PERLUNICODE=SDA\nor\nuse open qw(:std :encoding(UTF-8));\nuse Encode qw(decode);\n@ARGV = map { decode('UTF-8', $, 1) } @ARGV;\n\n℞℞ 19: Open file with specific encoding\nSpecify stream encoding.  This is the normal way to deal with encoded text, not by calling\nlow-level functions.\n\n# input file\nopen(my $infile, \"< :encoding(UTF-16)\", \"wintext\");\nOR\nopen(my $infile, \"<\", \"wintext\");\nbinmode($infile, \":encoding(UTF-16)\");\nTHEN\nmy $line = <$infile>;\n\n# output file\nopen($outfile, \"> :encoding(cp1252)\", \"wintext\");\nOR\nopen(my $outfile, \">\", \"wintext\");\nbinmode($outfile, \":encoding(cp1252)\");\nTHEN\nprint $outfile \"some text\\n\";\n\nMore layers than just the encoding can be specified here. For example, the incantation \":raw\n:encoding(UTF-16LE) :crlf\" includes implicit CRLF handling.\n\n℞℞ 20: Unicode casing\nUnicode casing is very different from ASCII casing.\n\nuc(\"henry ⅷ\")  # \"HENRY Ⅷ\"\nuc(\"tschüß\")   # \"TSCHÜSS\"  notice ß => SS\n\n# both are true:\n\"tschüß\"  =~ /TSCHÜSS/i   # notice ß => SS\n\"Σίσυφος\" =~ /ΣΊΣΥΦΟΣ/i   # notice Σ,σ,ς sameness\n\n℞℞ 21: Unicode case-insensitive comparisons\nAlso available in the CPAN Unicode::CaseFold module, the new \"fc\" “foldcase” function from\nv5.16 grants access to the same Unicode casefolding as the \"/i\" pattern modifier has always\nused:\n\nuse feature \"fc\"; # fc() function is from v5.16\n\n# sort case-insensitively\nmy @sorted = sort { fc($a) cmp fc($b) } @list;\n\n# both are true:\nfc(\"tschüß\")  eq fc(\"TSCHÜSS\")\nfc(\"Σίσυφος\") eq fc(\"ΣΊΣΥΦΟΣ\")\n\n℞℞ 22: Match Unicode linebreak sequence in regex\nA Unicode linebreak matches the two-character CRLF grapheme or any of seven vertical\nwhitespace characters.  Good for dealing with textfiles coming from different operating\nsystems.\n\n\\R\n\ns/\\R/\\n/g;  # normalize all linebreaks to \\n\n\n℞℞ 23: Get character category\nFind the general category of a numeric codepoint.\n\nuse Unicode::UCD qw(charinfo);\nmy $cat = charinfo(0x3A3)->{category};  # \"Lu\"\n\n℞℞ 24: Disabling Unicode-awareness in builtin charclasses\nDisable \"\\w\", \"\\b\", \"\\s\", \"\\d\", and the POSIX classes from working correctly on Unicode\neither in this scope, or in just one regex.\n\nuse v5.14;\nuse re \"/a\";\n\n# OR\n\nmy($num) = $str =~ /(\\d+)/a;\n\nOr use specific un-Unicode properties, like \"\\p{ahex}\" and \"\\p{POSIXDigit\"}.  Properties\nstill work normally no matter what charset modifiers (\"/d /u /l /a /aa\") should be effect.\n\n℞℞ 25: Match Unicode properties in regex with \\p, \\P\nThese all match a single codepoint with the given property.  Use \"\\P\" in place of \"\\p\" to\nmatch one codepoint lacking that property.\n\n\\pL, \\pN, \\pS, \\pP, \\pM, \\pZ, \\pC\n\\p{Sk}, \\p{Ps}, \\p{Lt}\n\\p{alpha}, \\p{upper}, \\p{lower}\n\\p{Latin}, \\p{Greek}\n\\p{scriptextensions=Latin}, \\p{scx=Greek}\n\\p{EastAsianWidth=Wide}, \\p{EA=W}\n\\p{LineBreak=Hyphen}, \\p{LB=HY}\n\\p{NumericValue=4}, \\p{NV=4}\n\n℞℞ 26: Custom character properties\nDefine at compile-time your own custom character properties for use in regexes.\n\n# using private-use characters\nsub InTengwar { \"E000\\tE07F\\n\" }\n\nif (/\\p{InTengwar}/) { ... }\n\n# blending existing properties\nsub IsGraecoRomanTitle {<<'ENDOFSET'}\n+utf8::IsLatin\n+utf8::IsGreek\n&utf8::IsTitle\nENDOFSET\n\nif (/\\p{IsGraecoRomanTitle}/ { ... }\n\n℞℞ 27: Unicode normalization\nTypically render into NFD on input and NFC on output. Using NFKC or NFKD functions improves\nrecall on searches, assuming you've already done to the same text to be searched. Note that\nthis is about much more than just pre- combined compatibility glyphs; it also reorders marks\naccording to their canonical combining classes and weeds out singletons.\n\nuse Unicode::Normalize;\nmy $nfd  = NFD($orig);\nmy $nfc  = NFC($orig);\nmy $nfkd = NFKD($orig);\nmy $nfkc = NFKC($orig);\n\n℞℞ 28: Convert non-ASCII Unicode numerics\nUnless you’ve used \"/a\" or \"/aa\", \"\\d\" matches more than ASCII digits only, but Perl’s\nimplicit string-to-number conversion does not current recognize these.  Here’s how to convert\nsuch strings manually.\n\nuse v5.14;  # needed for num() function\nuse Unicode::UCD qw(num);\nmy $str = \"got Ⅻ and ४५६७ and ⅞ and here\";\nmy @nums = ();\nwhile ($str =~ /(\\d+|\\N)/g) {  # not just ASCII!\npush @nums, num($1);\n}\nsay \"@nums\";   #     12      4567      0.875\n\nuse charnames qw(:full);\nmy $nv = num(\"\\N{RUMI DIGIT ONE}\\N{RUMI DIGIT TWO}\");\n\n℞℞ 29: Match Unicode grapheme cluster in regex\nProgrammer-visible “characters” are codepoints matched by \"/./s\", but user-visible\n“characters” are graphemes matched by \"/\\X/\".\n\n# Find vowel *plus* any combining diacritics,underlining,etc.\nmy $nfd = NFD($orig);\n$nfd =~ / (?=[aeiou]) \\X /xi\n\n℞℞ 30: Extract by grapheme instead of by codepoint (regex)\n# match and grab five first graphemes\nmy($firstfive) = $str =~ /^ ( \\X{5} ) /x;\n\n℞℞ 31: Extract by grapheme instead of by codepoint (substr)\n# cpan -i Unicode::GCString\nuse Unicode::GCString;\nmy $gcs = Unicode::GCString->new($str);\nmy $firstfive = $gcs->substr(0, 5);\n\n℞℞ 32: Reverse string by grapheme\nReversing by codepoint messes up diacritics, mistakenly converting \"crème brûlée\" into\n\"éel̂urb em̀erc\" instead of into \"eélûrb emèrc\"; so reverse by grapheme instead.  Both these\napproaches work right no matter what normalization the string is in:\n\n$str = join(\"\", reverse $str =~ /\\X/g);\n\n# OR: cpan -i Unicode::GCString\nuse Unicode::GCString;\n$str = reverse Unicode::GCString->new($str);\n\n℞℞ 33: String length in graphemes\nThe string \"brûlée\" has six graphemes but up to eight codepoints.  This counts by grapheme,\nnot by codepoint:\n\nmy $str = \"brûlée\";\nmy $count = 0;\nwhile ($str =~ /\\X/g) { $count++ }\n\n# OR: cpan -i Unicode::GCString\nuse Unicode::GCString;\nmy $gcs = Unicode::GCString->new($str);\nmy $count = $gcs->length;\n\n℞℞ 34: Unicode column-width for printing\nPerl’s \"printf\", \"sprintf\", and \"format\" think all codepoints take up 1 print column, but\nmany take 0 or 2.  Here to show that normalization makes no difference, we print out both\nforms:\n\nuse Unicode::GCString;\nuse Unicode::Normalize;\n\nmy @words = qw/crème brûlée/;\n@words = map { NFC($), NFD($) } @words;\n\nfor my $str (@words) {\nmy $gcs = Unicode::GCString->new($str);\nmy $cols = $gcs->columns;\nmy $pad = \" \" x (10 - $cols);\nsay str, $pad, \" |\";\n}\n\ngenerates this to show that it pads correctly no matter the normalization:\n\ncrème      |\ncrème      |\nbrûlée     |\nbrûlée     |\n\n℞℞ 35: Unicode collation\nText sorted by numeric codepoint follows no reasonable alphabetic order; use the UCA for\nsorting text.\n\nuse Unicode::Collate;\nmy $col = Unicode::Collate->new();\nmy @list = $col->sort(@oldlist);\n\nSee the ucsort program from the Unicode::Tussle CPAN module for a convenient command-line\ninterface to this module.\n\n℞℞ 36: Case- and accent-insensitive Unicode sort\nSpecify a collation strength of level 1 to ignore case and diacritics, only looking at the\nbasic character.\n\nuse Unicode::Collate;\nmy $col = Unicode::Collate->new(level => 1);\nmy @list = $col->sort(@oldlist);\n\n℞℞ 37: Unicode locale collation\nSome locales have special sorting rules.\n\n# either use v5.12, OR: cpan -i Unicode::Collate::Locale\nuse Unicode::Collate::Locale;\nmy $col = Unicode::Collate::Locale->new(locale => \"dephonebook\");\nmy @list = $col->sort(@oldlist);\n\nThe ucsort program mentioned above accepts a \"--locale\" parameter.\n\n℞℞ 38: Making \"cmp\" work on text instead of codepoints\nInstead of this:\n\n@srecs = sort {\n$b->{AGE}   <=>  $a->{AGE}\n||\n$a->{NAME}  cmp  $b->{NAME}\n} @recs;\n\nUse this:\n\nmy $coll = Unicode::Collate->new();\nfor my $rec (@recs) {\n$rec->{NAMEkey} = $coll->getSortKey( $rec->{NAME} );\n}\n@srecs = sort {\n$b->{AGE}       <=>  $a->{AGE}\n||\n$a->{NAMEkey}  cmp  $b->{NAMEkey}\n} @recs;\n\n℞℞ 39: Case- and accent-insensitive comparisons\nUse a collator object to compare Unicode text by character instead of by codepoint.\n\nuse Unicode::Collate;\nmy $es = Unicode::Collate->new(\nlevel => 1,\nnormalization => undef\n);\n\n# now both are true:\n$es->eq(\"García\",  \"GARCIA\" );\n$es->eq(\"Márquez\", \"MARQUEZ\");\n\n℞℞ 40: Case- and accent-insensitive locale comparisons\nSame, but in a specific locale.\n\nmy $de = Unicode::Collate::Locale->new(\nlocale => \"dephonebook\",\n);\n\n# now this is true:\n$de->eq(\"tschüß\", \"TSCHUESS\");  # notice ü => UE, ß => SS\n\n℞℞ 41: Unicode linebreaking\nBreak up text into lines according to Unicode rules.\n\n# cpan -i Unicode::LineBreak\nuse Unicode::LineBreak;\nuse charnames qw(:full);\n\nmy $para = \"This is a super\\N{HYPHEN}long string. \" x 20;\nmy $fmt = Unicode::LineBreak->new;\nprint $fmt->break($para), \"\\n\";\n\n℞℞ 42: Unicode text in DBM hashes, the tedious way\nUsing a regular Perl string as a key or value for a DBM hash will trigger a wide character\nexception if any codepoints won’t fit into a byte.  Here’s how to manually manage the\ntranslation:\n\nuse DBFile;\nuse Encode qw(encode decode);\ntie %dbhash, \"DBFile\", \"pathname\";\n\n# STORE\n\n# assume $unikey and $univalue are abstract Unicode strings\nmy $enckey   = encode(\"UTF-8\", $unikey, 1);\nmy $encvalue = encode(\"UTF-8\", $univalue, 1);\n$dbhash{$enckey} = $encvalue;\n\n# FETCH\n\n# assume $unikey holds a normal Perl string (abstract Unicode)\nmy $enckey   = encode(\"UTF-8\", $unikey, 1);\nmy $encvalue = $dbhash{$enckey};\nmy $univalue = decode(\"UTF-8\", $encvalue, 1);\n\n℞℞ 43: Unicode text in DBM hashes, the easy way\nHere’s how to implicitly manage the translation; all encoding and decoding is done\nautomatically, just as with streams that have a particular encoding attached to them:\n\nuse DBFile;\nuse DBMFilter;\n\nmy $dbobj = tie %dbhash, \"DBFile\", \"pathname\";\n$dbobj->FilterValue(\"utf8\");  # this is the magic bit\n\n# STORE\n\n# assume $unikey and $univalue are abstract Unicode strings\n$dbhash{$unikey} = $univalue;\n\n# FETCH\n\n# $unikey holds a normal Perl string (abstract Unicode)\nmy $univalue = $dbhash{$unikey};\n\n℞℞ 44: PROGRAM: Demo of Unicode collation and printing\nHere’s a full program showing how to make use of locale-sensitive sorting, Unicode casing,\nand managing print widths when some of the characters take up zero or two columns, not just\none column each time.  When run, the following program produces this nicely aligned output:\n\nCrème Brûlée....... €2.00\nÉclair............. €1.60\nFideuà............. €4.20\nHamburger.......... €6.00\nJamón Serrano...... €4.45\nLinguiça........... €7.00\nPâté............... €4.15\nPears.............. €2.00\nPêches............. €2.25\nSmørbrød........... €5.75\nSpätzle............ €5.50\nXoriço............. €3.00\nΓύρος.............. €6.50\n막걸리............. €4.00\nおもち............. €2.65\nお好み焼き......... €8.00\nシュークリーム..... €1.85\n寿司............... €9.99\n包子............... €7.50\n\nHere's that program; tested on v5.14.\n\n#!/usr/bin/env perl\n# umenu - demo sorting and printing of Unicode food\n#\n# (obligatory and increasingly long preamble)\n#\nuse utf8;\nuse v5.14;                       # for locale sorting\nuse strict;\nuse warnings;\nuse warnings  qw(FATAL utf8);    # fatalize encoding faults\nuse open      qw(:std :encoding(UTF-8)); # undeclared streams in UTF-8\nuse charnames qw(:full :short);  # unneeded in v5.16\n\n# std modules\nuse Unicode::Normalize;          # std perl distro as of v5.8\nuse List::Util qw(max);          # std perl distro as of v5.10\nuse Unicode::Collate::Locale;    # std perl distro as of v5.14\n\n# cpan modules\nuse Unicode::GCString;           # from CPAN\n\n# forward defs\nsub pad($$$);\nsub colwidth();\nsub entitle();\n\nmy %price = (\n\"γύρος\"             => 6.50, # gyros\n\"pears\"             => 2.00, # like um, pears\n\"linguiça\"          => 7.00, # spicy sausage, Portuguese\n\"xoriço\"            => 3.00, # chorizo sausage, Catalan\n\"hamburger\"         => 6.00, # burgermeister meisterburger\n\"éclair\"            => 1.60, # dessert, French\n\"smørbrød\"          => 5.75, # sandwiches, Norwegian\n\"spätzle\"           => 5.50, # Bayerisch noodles, little sparrows\n\"包子\"              => 7.50, # bao1 zi5, steamed pork buns, Mandarin\n\"jamón serrano\"     => 4.45, # country ham, Spanish\n\"pêches\"            => 2.25, # peaches, French\n\"シュークリーム\"    => 1.85, # cream-filled pastry like eclair\n\"막걸리\"            => 4.00, # makgeolli, Korean rice wine\n\"寿司\"              => 9.99, # sushi, Japanese\n\"おもち\"            => 2.65, # omochi, rice cakes, Japanese\n\"crème brûlée\"      => 2.00, # crema catalana\n\"fideuà\"            => 4.20, # more noodles, Valencian\n# (Catalan=fideuada)\n\"pâté\"              => 4.15, # gooseliver paste, French\n\"お好み焼き\"        => 8.00, # okonomiyaki, Japanese\n);\n\nmy $width = 5 + max map { colwidth } keys %price;\n\n# So the Asian stuff comes out in an order that someone\n# who reads those scripts won't freak out over; the\n# CJK stuff will be in JIS X 0208 order that way.\nmy $coll  = Unicode::Collate::Locale->new(locale => \"ja\");\n\nfor my $item ($coll->sort(keys %price)) {\nprint pad(entitle($item), $width, \".\");\nprintf \" €%.2f\\n\", $price{$item};\n}\n\nsub pad($$$) {\nmy($str, $width, $padchar) = @;\nreturn $str . ($padchar x ($width - colwidth($str)));\n}\n\nsub colwidth() {\nmy($str) = @;\nreturn Unicode::GCString->new($str)->columns;\n}\n\nsub entitle() {\nmy($str) = @;\n$str =~ s{ (?=\\pL)(\\S)     (\\S*) }\n{ ucfirst($1) . lc($2)  }xge;\nreturn $str;\n}\n",
            "subsections": []
        },
        "SEE ALSO": {
            "content": "See these manpages, some of which are CPAN modules: perlunicode, perluniprops, perlre,\nperlrecharclass, perluniintro, perlunitut, perlunifaq, PerlIO, DBFile, DBMFilter,\nDBMFilter::utf8, Encode, Encode::Locale, Unicode::UCD, Unicode::Normalize,\nUnicode::GCString, Unicode::LineBreak, Unicode::Collate, Unicode::Collate::Locale,\nUnicode::Unihan, Unicode::CaseFold, Unicode::Tussle, Lingua::JA::Romanize::Japanese,\nLingua::ZH::Romanize::Pinyin, Lingua::KO::Romanize::Hangul.\n\nThe Unicode::Tussle CPAN module includes many programs to help with working with Unicode,\nincluding these programs to fully or partly replace standard utilities: tcgrep instead of\negrep, uniquote instead of cat -v or hexdump, uniwc instead of wc, unilook instead of look,\nunifmt instead of fmt, and ucsort instead of sort.  For exploring Unicode character names and\ncharacter properties, see its uniprops, unichars, and uninames programs.  It also supplies\nthese programs, all of which are general filters that do Unicode-y things: unititle and\nunicaps; uniwide and uninarrow; unisupers and unisubs; nfd, nfc, nfkd, and nfkc; and uc, lc,\nand tc.\n\nFinally, see the published Unicode Standard (page numbers are from version 6.0.0), including\nthese specific annexes and technical reports:\n\n§3.13 Default Case Algorithms, page 113; §4.2  Case, pages 120–122; Case Mappings, page\n166–172, especially Caseless Matching starting on page 170.\nUAX #44: Unicode Character Database\nUTS #18: Unicode Regular Expressions\nUAX #15: Unicode Normalization Forms\nUTS #10: Unicode Collation Algorithm\nUAX #29: Unicode Text Segmentation\nUAX #14: Unicode Line Breaking Algorithm\nUAX #11: East Asian Width\n",
            "subsections": []
        },
        "AUTHOR": {
            "content": "Tom Christiansen <tchrist@perl.com> wrote this, with occasional kibbitzing from Larry Wall\nand Jeffrey Friedl in the background.\n",
            "subsections": []
        },
        "COPYRIGHT AND LICENCE": {
            "content": "Copyright © 2012 Tom Christiansen.\n\nThis program is free software; you may redistribute it and/or modify it under the same terms\nas Perl itself.\n\nMost of these examples taken from the current edition of the “Camel Book”; that is, from the\n4ᵗʰ Edition of Programming Perl, Copyright © 2012 Tom Christiansen <et al.>, 2012-02-13 by\nO’Reilly Media.  The code itself is freely redistributable, and you are encouraged to\ntransplant, fold, spindle, and mutilate any of the examples in this manpage however you\nplease for inclusion into your own programs without any encumbrance whatsoever.\nAcknowledgement via code comment is polite but not required.\n",
            "subsections": []
        },
        "REVISION HISTORY": {
            "content": "v1.0.0 – first public release, 2012-02-27\n\n\n\nperl v5.34.0                                 2025-07-25                               PERLUNICOOK(1)",
            "subsections": []
        }
    },
    "summary": "perlunicook - cookbookish examples of handling Unicode in Perl",
    "flags": [],
    "examples": [
        "℞℞ 0: Standard preamble",
        "Unless otherwise notes, all examples below require this standard preamble to work correctly,",
        "with the \"#!\" adjusted to work on your system:",
        "#!/usr/bin/env perl",
        "use utf8;      # so literals and identifiers can be in UTF-8",
        "use v5.12;     # or later to get \"unicodestrings\" feature",
        "use strict;    # quote strings, declare variables",
        "use warnings;  # on by default",
        "use warnings  qw(FATAL utf8);    # fatalize encoding glitches",
        "use open      qw(:std :encoding(UTF-8)); # undeclared streams in UTF-8",
        "use charnames qw(:full :short);  # unneeded in v5.16",
        "This does make even Unix programmers \"binmode\" your binary streams, or open them with \":raw\",",
        "but that's the only way to get at them portably anyway.",
        "WARNING: \"use autodie\" (pre 2.26) and \"use open\" do not get along with each other.",
        "℞℞ 1: Generic Unicode-savvy filter",
        "Always decompose on the way in, then recompose on the way out.",
        "use Unicode::Normalize;",
        "while (<>) {",
        "$ = NFD($);   # decompose + reorder canonically",
        "...",
        "} continue {",
        "print NFC($);  # recompose (where possible) + reorder canonically",
        "℞℞ 2: Fine-tuning Unicode warnings",
        "As of v5.14, Perl distinguishes three subclasses of UTF‑8 warnings.",
        "use v5.14;                  # subwarnings unavailable any earlier",
        "no warnings \"nonchar\";      # the 66 forbidden non-characters",
        "no warnings \"surrogate\";    # UTF-16/CESU-8 nonsense",
        "no warnings \"nonunicode\";  # for codepoints over 0x10FFFF",
        "℞℞ 3: Declare source in utf8 for identifiers and literals",
        "Without the all-critical \"use utf8\" declaration, putting UTF‑8 in your literals and",
        "identifiers won’t work right.  If you used the standard preamble just given above, this",
        "already happened.  If you did, you can do things like this:",
        "use utf8;",
        "my $measure   = \"Ångström\";",
        "my @μsoft     = qw( cp852 cp1251 cp1252 );",
        "my @ὑπέρμεγας = qw( ὑπέρ  μεγας );",
        "my @鯉        = qw( koi8-f koi8-u koi8-r );",
        "my $motto     = \"👪 💗 🐪\"; # FAMILY, GROWING HEART, DROMEDARY CAMEL",
        "If you forget \"use utf8\", high bytes will be misunderstood as separate characters, and",
        "nothing will work right.",
        "℞℞ 4: Characters and their numbers",
        "The \"ord\" and \"chr\" functions work transparently on all codepoints, not just on ASCII alone —",
        "nor in fact, not even just on Unicode alone.",
        "# ASCII characters",
        "ord(\"A\")",
        "chr(65)",
        "# characters from the Basic Multilingual Plane",
        "ord(\"Σ\")",
        "chr(0x3A3)",
        "# beyond the BMP",
        "ord(\"𝑛\")               # MATHEMATICAL ITALIC SMALL N",
        "chr(0x1D45B)",
        "# beyond Unicode! (up to MAXINT)",
        "ord(\"\\x{200000}\")",
        "chr(0x200000)",
        "℞℞ 5: Unicode literals by character number",
        "In an interpolated literal, whether a double-quoted string or a regex, you may specify a",
        "character by its number using the \"\\x{HHHHHH}\" escape.",
        "String: \"\\x{3a3}\"",
        "Regex:  /\\x{3a3}/",
        "String: \"\\x{1d45b}\"",
        "Regex:  /\\x{1d45b}/",
        "# even non-BMP ranges in regex work fine",
        "/[\\x{1D434}-\\x{1D467}]/",
        "℞℞ 6: Get character name by number",
        "use charnames ();",
        "my $name = charnames::viacode(0x03A3);",
        "℞℞ 7: Get character number by name",
        "use charnames ();",
        "my $number = charnames::vianame(\"GREEK CAPITAL LETTER SIGMA\");",
        "℞℞ 8: Unicode named characters",
        "Use the \"\\N{charname}\" notation to get the character by that name for use in interpolated",
        "literals (double-quoted strings and regexes).  In v5.16, there is an implicit",
        "use charnames qw(:full :short);",
        "But prior to v5.16, you must be explicit about which set of charnames you want.  The \":full\"",
        "names are the official Unicode character name, alias, or sequence, which all share a",
        "namespace.",
        "use charnames qw(:full :short latin greek);",
        "\"\\N{MATHEMATICAL ITALIC SMALL N}\"      # :full",
        "\"\\N{GREEK CAPITAL LETTER SIGMA}\"       # :full",
        "Anything else is a Perl-specific convenience abbreviation.  Specify one or more scripts by",
        "names if you want short names that are script-specific.",
        "\"\\N{Greek:Sigma}\"                      # :short",
        "\"\\N{ae}\"                               #  latin",
        "\"\\N{epsilon}\"                          #  greek",
        "The v5.16 release also supports a \":loose\" import for loose matching of character names,",
        "which works just like loose matching of property names: that is, it disregards case,",
        "whitespace, and underscores:",
        "\"\\N{euro sign}\"                        # :loose (from v5.16)",
        "Starting in v5.32, you can also use",
        "qr/\\p{name=euro sign}/",
        "to get official Unicode named characters in regular expressions.  Loose matching is always",
        "done for these.",
        "℞℞ 9: Unicode named sequences",
        "These look just like character names but return multiple codepoints.  Notice the %vx vector-",
        "print functionality in \"printf\".",
        "use charnames qw(:full);",
        "my $seq = \"\\N{LATIN CAPITAL LETTER A WITH MACRON AND GRAVE}\";",
        "printf \"U+%v04X\\n\", $seq;",
        "U+0100.0300",
        "℞℞ 10: Custom named characters",
        "Use \":alias\" to give your own lexically scoped nicknames to existing characters, or even to",
        "give unnamed private-use characters useful names.",
        "use charnames \":full\", \":alias\" => {",
        "ecute => \"LATIN SMALL LETTER E WITH ACUTE\",",
        "\"APPLE LOGO\" => 0xF8FF, # private use character",
        "};",
        "\"\\N{ecute}\"",
        "\"\\N{APPLE LOGO}\"",
        "℞℞ 11: Names of CJK codepoints",
        "Sinograms like “東京” come back with character names of \"CJK UNIFIED IDEOGRAPH-6771\" and \"CJK",
        "UNIFIED IDEOGRAPH-4EAC\", because their “names” vary.  The CPAN \"Unicode::Unihan\" module has a",
        "large database for decoding these (and a whole lot more), provided you know how to understand",
        "its output.",
        "# cpan -i Unicode::Unihan",
        "use Unicode::Unihan;",
        "my $str = \"東京\";",
        "my $unhan = Unicode::Unihan->new;",
        "for my $lang (qw(Mandarin Cantonese Korean JapaneseOn JapaneseKun)) {",
        "printf \"CJK $str in %-12s is \", $lang;",
        "say $unhan->$lang($str);",
        "prints:",
        "CJK 東京 in Mandarin     is DONG1JING1",
        "CJK 東京 in Cantonese    is dung1ging1",
        "CJK 東京 in Korean       is TONGKYENG",
        "CJK 東京 in JapaneseOn   is TOUKYOU KEI KIN",
        "CJK 東京 in JapaneseKun  is HIGASHI AZUMAMIYAKO",
        "If you have a specific romanization scheme in mind, use the specific module:",
        "# cpan -i Lingua::JA::Romanize::Japanese",
        "use Lingua::JA::Romanize::Japanese;",
        "my $k2r = Lingua::JA::Romanize::Japanese->new;",
        "my $str = \"東京\";",
        "say \"Japanese for $str is \", $k2r->chars($str);",
        "prints",
        "Japanese for 東京 is toukyou",
        "℞℞ 12: Explicit encode/decode",
        "On rare occasion, such as a database read, you may be given encoded text you need to decode.",
        "use Encode qw(encode decode);",
        "my $chars = decode(\"shiftjis\", $bytes, 1);",
        "# OR",
        "my $bytes = encode(\"MIME-Header-ISO2022JP\", $chars, 1);",
        "For streams all in the same encoding, don't use encode/decode; instead set the file encoding",
        "when you open the file or immediately after with \"binmode\" as described later below.",
        "℞℞ 13: Decode program arguments as utf8",
        "$ perl -CA ...",
        "or",
        "$ export PERLUNICODE=A",
        "or",
        "use Encode qw(decode);",
        "@ARGV = map { decode('UTF-8', $, 1) } @ARGV;",
        "℞℞ 14: Decode program arguments as locale encoding",
        "# cpan -i Encode::Locale",
        "use Encode qw(locale);",
        "use Encode::Locale;",
        "# use \"locale\" as an arg to encode/decode",
        "@ARGV = map { decode(locale => $, 1) } @ARGV;",
        "℞℞ 15: Declare STD{IN,OUT,ERR} to be utf8",
        "Use a command-line option, an environment variable, or else call \"binmode\" explicitly:",
        "$ perl -CS ...",
        "or",
        "$ export PERLUNICODE=S",
        "or",
        "use open qw(:std :encoding(UTF-8));",
        "or",
        "binmode(STDIN,  \":encoding(UTF-8)\");",
        "binmode(STDOUT, \":utf8\");",
        "binmode(STDERR, \":utf8\");",
        "℞℞ 16: Declare STD{IN,OUT,ERR} to be in locale encoding",
        "# cpan -i Encode::Locale",
        "use Encode;",
        "use Encode::Locale;",
        "# or as a stream for binmode or open",
        "binmode STDIN,  \":encoding(consolein)\"  if -t STDIN;",
        "binmode STDOUT, \":encoding(consoleout)\" if -t STDOUT;",
        "binmode STDERR, \":encoding(consoleout)\" if -t STDERR;",
        "℞℞ 17: Make file I/O default to utf8",
        "Files opened without an encoding argument will be in UTF-8:",
        "$ perl -CD ...",
        "or",
        "$ export PERLUNICODE=D",
        "or",
        "use open qw(:encoding(UTF-8));",
        "℞℞ 18: Make all I/O and args default to utf8",
        "$ perl -CSDA ...",
        "or",
        "$ export PERLUNICODE=SDA",
        "or",
        "use open qw(:std :encoding(UTF-8));",
        "use Encode qw(decode);",
        "@ARGV = map { decode('UTF-8', $, 1) } @ARGV;",
        "℞℞ 19: Open file with specific encoding",
        "Specify stream encoding.  This is the normal way to deal with encoded text, not by calling",
        "low-level functions.",
        "# input file",
        "open(my $infile, \"< :encoding(UTF-16)\", \"wintext\");",
        "OR",
        "open(my $infile, \"<\", \"wintext\");",
        "binmode($infile, \":encoding(UTF-16)\");",
        "THEN",
        "my $line = <$infile>;",
        "# output file",
        "open($outfile, \"> :encoding(cp1252)\", \"wintext\");",
        "OR",
        "open(my $outfile, \">\", \"wintext\");",
        "binmode($outfile, \":encoding(cp1252)\");",
        "THEN",
        "print $outfile \"some text\\n\";",
        "More layers than just the encoding can be specified here. For example, the incantation \":raw",
        ":encoding(UTF-16LE) :crlf\" includes implicit CRLF handling.",
        "℞℞ 20: Unicode casing",
        "Unicode casing is very different from ASCII casing.",
        "uc(\"henry ⅷ\")  # \"HENRY Ⅷ\"",
        "uc(\"tschüß\")   # \"TSCHÜSS\"  notice ß => SS",
        "# both are true:",
        "\"tschüß\"  =~ /TSCHÜSS/i   # notice ß => SS",
        "\"Σίσυφος\" =~ /ΣΊΣΥΦΟΣ/i   # notice Σ,σ,ς sameness",
        "℞℞ 21: Unicode case-insensitive comparisons",
        "Also available in the CPAN Unicode::CaseFold module, the new \"fc\" “foldcase” function from",
        "v5.16 grants access to the same Unicode casefolding as the \"/i\" pattern modifier has always",
        "used:",
        "use feature \"fc\"; # fc() function is from v5.16",
        "# sort case-insensitively",
        "my @sorted = sort { fc($a) cmp fc($b) } @list;",
        "# both are true:",
        "fc(\"tschüß\")  eq fc(\"TSCHÜSS\")",
        "fc(\"Σίσυφος\") eq fc(\"ΣΊΣΥΦΟΣ\")",
        "℞℞ 22: Match Unicode linebreak sequence in regex",
        "A Unicode linebreak matches the two-character CRLF grapheme or any of seven vertical",
        "whitespace characters.  Good for dealing with textfiles coming from different operating",
        "systems.",
        "\\R",
        "s/\\R/\\n/g;  # normalize all linebreaks to \\n",
        "℞℞ 23: Get character category",
        "Find the general category of a numeric codepoint.",
        "use Unicode::UCD qw(charinfo);",
        "my $cat = charinfo(0x3A3)->{category};  # \"Lu\"",
        "℞℞ 24: Disabling Unicode-awareness in builtin charclasses",
        "Disable \"\\w\", \"\\b\", \"\\s\", \"\\d\", and the POSIX classes from working correctly on Unicode",
        "either in this scope, or in just one regex.",
        "use v5.14;",
        "use re \"/a\";",
        "# OR",
        "my($num) = $str =~ /(\\d+)/a;",
        "Or use specific un-Unicode properties, like \"\\p{ahex}\" and \"\\p{POSIXDigit\"}.  Properties",
        "still work normally no matter what charset modifiers (\"/d /u /l /a /aa\") should be effect.",
        "℞℞ 25: Match Unicode properties in regex with \\p, \\P",
        "These all match a single codepoint with the given property.  Use \"\\P\" in place of \"\\p\" to",
        "match one codepoint lacking that property.",
        "\\pL, \\pN, \\pS, \\pP, \\pM, \\pZ, \\pC",
        "\\p{Sk}, \\p{Ps}, \\p{Lt}",
        "\\p{alpha}, \\p{upper}, \\p{lower}",
        "\\p{Latin}, \\p{Greek}",
        "\\p{scriptextensions=Latin}, \\p{scx=Greek}",
        "\\p{EastAsianWidth=Wide}, \\p{EA=W}",
        "\\p{LineBreak=Hyphen}, \\p{LB=HY}",
        "\\p{NumericValue=4}, \\p{NV=4}",
        "℞℞ 26: Custom character properties",
        "Define at compile-time your own custom character properties for use in regexes.",
        "# using private-use characters",
        "sub InTengwar { \"E000\\tE07F\\n\" }",
        "if (/\\p{InTengwar}/) { ... }",
        "# blending existing properties",
        "sub IsGraecoRomanTitle {<<'ENDOFSET'}",
        "+utf8::IsLatin",
        "+utf8::IsGreek",
        "&utf8::IsTitle",
        "ENDOFSET",
        "if (/\\p{IsGraecoRomanTitle}/ { ... }",
        "℞℞ 27: Unicode normalization",
        "Typically render into NFD on input and NFC on output. Using NFKC or NFKD functions improves",
        "recall on searches, assuming you've already done to the same text to be searched. Note that",
        "this is about much more than just pre- combined compatibility glyphs; it also reorders marks",
        "according to their canonical combining classes and weeds out singletons.",
        "use Unicode::Normalize;",
        "my $nfd  = NFD($orig);",
        "my $nfc  = NFC($orig);",
        "my $nfkd = NFKD($orig);",
        "my $nfkc = NFKC($orig);",
        "℞℞ 28: Convert non-ASCII Unicode numerics",
        "Unless you’ve used \"/a\" or \"/aa\", \"\\d\" matches more than ASCII digits only, but Perl’s",
        "implicit string-to-number conversion does not current recognize these.  Here’s how to convert",
        "such strings manually.",
        "use v5.14;  # needed for num() function",
        "use Unicode::UCD qw(num);",
        "my $str = \"got Ⅻ and ४५६७ and ⅞ and here\";",
        "my @nums = ();",
        "while ($str =~ /(\\d+|\\N)/g) {  # not just ASCII!",
        "push @nums, num($1);",
        "say \"@nums\";   #     12      4567      0.875",
        "use charnames qw(:full);",
        "my $nv = num(\"\\N{RUMI DIGIT ONE}\\N{RUMI DIGIT TWO}\");",
        "℞℞ 29: Match Unicode grapheme cluster in regex",
        "Programmer-visible “characters” are codepoints matched by \"/./s\", but user-visible",
        "“characters” are graphemes matched by \"/\\X/\".",
        "# Find vowel *plus* any combining diacritics,underlining,etc.",
        "my $nfd = NFD($orig);",
        "$nfd =~ / (?=[aeiou]) \\X /xi",
        "℞℞ 30: Extract by grapheme instead of by codepoint (regex)",
        "# match and grab five first graphemes",
        "my($firstfive) = $str =~ /^ ( \\X{5} ) /x;",
        "℞℞ 31: Extract by grapheme instead of by codepoint (substr)",
        "# cpan -i Unicode::GCString",
        "use Unicode::GCString;",
        "my $gcs = Unicode::GCString->new($str);",
        "my $firstfive = $gcs->substr(0, 5);",
        "℞℞ 32: Reverse string by grapheme",
        "Reversing by codepoint messes up diacritics, mistakenly converting \"crème brûlée\" into",
        "\"éel̂urb em̀erc\" instead of into \"eélûrb emèrc\"; so reverse by grapheme instead.  Both these",
        "approaches work right no matter what normalization the string is in:",
        "$str = join(\"\", reverse $str =~ /\\X/g);",
        "# OR: cpan -i Unicode::GCString",
        "use Unicode::GCString;",
        "$str = reverse Unicode::GCString->new($str);",
        "℞℞ 33: String length in graphemes",
        "The string \"brûlée\" has six graphemes but up to eight codepoints.  This counts by grapheme,",
        "not by codepoint:",
        "my $str = \"brûlée\";",
        "my $count = 0;",
        "while ($str =~ /\\X/g) { $count++ }",
        "# OR: cpan -i Unicode::GCString",
        "use Unicode::GCString;",
        "my $gcs = Unicode::GCString->new($str);",
        "my $count = $gcs->length;",
        "℞℞ 34: Unicode column-width for printing",
        "Perl’s \"printf\", \"sprintf\", and \"format\" think all codepoints take up 1 print column, but",
        "many take 0 or 2.  Here to show that normalization makes no difference, we print out both",
        "forms:",
        "use Unicode::GCString;",
        "use Unicode::Normalize;",
        "my @words = qw/crème brûlée/;",
        "@words = map { NFC($), NFD($) } @words;",
        "for my $str (@words) {",
        "my $gcs = Unicode::GCString->new($str);",
        "my $cols = $gcs->columns;",
        "my $pad = \" \" x (10 - $cols);",
        "say str, $pad, \" |\";",
        "generates this to show that it pads correctly no matter the normalization:",
        "crème      |",
        "crème      |",
        "brûlée     |",
        "brûlée     |",
        "℞℞ 35: Unicode collation",
        "Text sorted by numeric codepoint follows no reasonable alphabetic order; use the UCA for",
        "sorting text.",
        "use Unicode::Collate;",
        "my $col = Unicode::Collate->new();",
        "my @list = $col->sort(@oldlist);",
        "See the ucsort program from the Unicode::Tussle CPAN module for a convenient command-line",
        "interface to this module.",
        "℞℞ 36: Case- and accent-insensitive Unicode sort",
        "Specify a collation strength of level 1 to ignore case and diacritics, only looking at the",
        "basic character.",
        "use Unicode::Collate;",
        "my $col = Unicode::Collate->new(level => 1);",
        "my @list = $col->sort(@oldlist);",
        "℞℞ 37: Unicode locale collation",
        "Some locales have special sorting rules.",
        "# either use v5.12, OR: cpan -i Unicode::Collate::Locale",
        "use Unicode::Collate::Locale;",
        "my $col = Unicode::Collate::Locale->new(locale => \"dephonebook\");",
        "my @list = $col->sort(@oldlist);",
        "The ucsort program mentioned above accepts a \"--locale\" parameter.",
        "℞℞ 38: Making \"cmp\" work on text instead of codepoints",
        "Instead of this:",
        "@srecs = sort {",
        "$b->{AGE}   <=>  $a->{AGE}",
        "||",
        "$a->{NAME}  cmp  $b->{NAME}",
        "} @recs;",
        "Use this:",
        "my $coll = Unicode::Collate->new();",
        "for my $rec (@recs) {",
        "$rec->{NAMEkey} = $coll->getSortKey( $rec->{NAME} );",
        "@srecs = sort {",
        "$b->{AGE}       <=>  $a->{AGE}",
        "||",
        "$a->{NAMEkey}  cmp  $b->{NAMEkey}",
        "} @recs;",
        "℞℞ 39: Case- and accent-insensitive comparisons",
        "Use a collator object to compare Unicode text by character instead of by codepoint.",
        "use Unicode::Collate;",
        "my $es = Unicode::Collate->new(",
        "level => 1,",
        "normalization => undef",
        ");",
        "# now both are true:",
        "$es->eq(\"García\",  \"GARCIA\" );",
        "$es->eq(\"Márquez\", \"MARQUEZ\");",
        "℞℞ 40: Case- and accent-insensitive locale comparisons",
        "Same, but in a specific locale.",
        "my $de = Unicode::Collate::Locale->new(",
        "locale => \"dephonebook\",",
        ");",
        "# now this is true:",
        "$de->eq(\"tschüß\", \"TSCHUESS\");  # notice ü => UE, ß => SS",
        "℞℞ 41: Unicode linebreaking",
        "Break up text into lines according to Unicode rules.",
        "# cpan -i Unicode::LineBreak",
        "use Unicode::LineBreak;",
        "use charnames qw(:full);",
        "my $para = \"This is a super\\N{HYPHEN}long string. \" x 20;",
        "my $fmt = Unicode::LineBreak->new;",
        "print $fmt->break($para), \"\\n\";",
        "℞℞ 42: Unicode text in DBM hashes, the tedious way",
        "Using a regular Perl string as a key or value for a DBM hash will trigger a wide character",
        "exception if any codepoints won’t fit into a byte.  Here’s how to manually manage the",
        "translation:",
        "use DBFile;",
        "use Encode qw(encode decode);",
        "tie %dbhash, \"DBFile\", \"pathname\";",
        "# STORE",
        "# assume $unikey and $univalue are abstract Unicode strings",
        "my $enckey   = encode(\"UTF-8\", $unikey, 1);",
        "my $encvalue = encode(\"UTF-8\", $univalue, 1);",
        "$dbhash{$enckey} = $encvalue;",
        "# FETCH",
        "# assume $unikey holds a normal Perl string (abstract Unicode)",
        "my $enckey   = encode(\"UTF-8\", $unikey, 1);",
        "my $encvalue = $dbhash{$enckey};",
        "my $univalue = decode(\"UTF-8\", $encvalue, 1);",
        "℞℞ 43: Unicode text in DBM hashes, the easy way",
        "Here’s how to implicitly manage the translation; all encoding and decoding is done",
        "automatically, just as with streams that have a particular encoding attached to them:",
        "use DBFile;",
        "use DBMFilter;",
        "my $dbobj = tie %dbhash, \"DBFile\", \"pathname\";",
        "$dbobj->FilterValue(\"utf8\");  # this is the magic bit",
        "# STORE",
        "# assume $unikey and $univalue are abstract Unicode strings",
        "$dbhash{$unikey} = $univalue;",
        "# FETCH",
        "# $unikey holds a normal Perl string (abstract Unicode)",
        "my $univalue = $dbhash{$unikey};",
        "℞℞ 44: PROGRAM: Demo of Unicode collation and printing",
        "Here’s a full program showing how to make use of locale-sensitive sorting, Unicode casing,",
        "and managing print widths when some of the characters take up zero or two columns, not just",
        "one column each time.  When run, the following program produces this nicely aligned output:",
        "Crème Brûlée....... €2.00",
        "Éclair............. €1.60",
        "Fideuà............. €4.20",
        "Hamburger.......... €6.00",
        "Jamón Serrano...... €4.45",
        "Linguiça........... €7.00",
        "Pâté............... €4.15",
        "Pears.............. €2.00",
        "Pêches............. €2.25",
        "Smørbrød........... €5.75",
        "Spätzle............ €5.50",
        "Xoriço............. €3.00",
        "Γύρος.............. €6.50",
        "막걸리............. €4.00",
        "おもち............. €2.65",
        "お好み焼き......... €8.00",
        "シュークリーム..... €1.85",
        "寿司............... €9.99",
        "包子............... €7.50",
        "Here's that program; tested on v5.14.",
        "#!/usr/bin/env perl",
        "# umenu - demo sorting and printing of Unicode food",
        "# (obligatory and increasingly long preamble)",
        "use utf8;",
        "use v5.14;                       # for locale sorting",
        "use strict;",
        "use warnings;",
        "use warnings  qw(FATAL utf8);    # fatalize encoding faults",
        "use open      qw(:std :encoding(UTF-8)); # undeclared streams in UTF-8",
        "use charnames qw(:full :short);  # unneeded in v5.16",
        "# std modules",
        "use Unicode::Normalize;          # std perl distro as of v5.8",
        "use List::Util qw(max);          # std perl distro as of v5.10",
        "use Unicode::Collate::Locale;    # std perl distro as of v5.14",
        "# cpan modules",
        "use Unicode::GCString;           # from CPAN",
        "# forward defs",
        "sub pad($$$);",
        "sub colwidth();",
        "sub entitle();",
        "my %price = (",
        "\"γύρος\"             => 6.50, # gyros",
        "\"pears\"             => 2.00, # like um, pears",
        "\"linguiça\"          => 7.00, # spicy sausage, Portuguese",
        "\"xoriço\"            => 3.00, # chorizo sausage, Catalan",
        "\"hamburger\"         => 6.00, # burgermeister meisterburger",
        "\"éclair\"            => 1.60, # dessert, French",
        "\"smørbrød\"          => 5.75, # sandwiches, Norwegian",
        "\"spätzle\"           => 5.50, # Bayerisch noodles, little sparrows",
        "\"包子\"              => 7.50, # bao1 zi5, steamed pork buns, Mandarin",
        "\"jamón serrano\"     => 4.45, # country ham, Spanish",
        "\"pêches\"            => 2.25, # peaches, French",
        "\"シュークリーム\"    => 1.85, # cream-filled pastry like eclair",
        "\"막걸리\"            => 4.00, # makgeolli, Korean rice wine",
        "\"寿司\"              => 9.99, # sushi, Japanese",
        "\"おもち\"            => 2.65, # omochi, rice cakes, Japanese",
        "\"crème brûlée\"      => 2.00, # crema catalana",
        "\"fideuà\"            => 4.20, # more noodles, Valencian",
        "# (Catalan=fideuada)",
        "\"pâté\"              => 4.15, # gooseliver paste, French",
        "\"お好み焼き\"        => 8.00, # okonomiyaki, Japanese",
        ");",
        "my $width = 5 + max map { colwidth } keys %price;",
        "# So the Asian stuff comes out in an order that someone",
        "# who reads those scripts won't freak out over; the",
        "# CJK stuff will be in JIS X 0208 order that way.",
        "my $coll  = Unicode::Collate::Locale->new(locale => \"ja\");",
        "for my $item ($coll->sort(keys %price)) {",
        "print pad(entitle($item), $width, \".\");",
        "printf \" €%.2f\\n\", $price{$item};",
        "sub pad($$$) {",
        "my($str, $width, $padchar) = @;",
        "return $str . ($padchar x ($width - colwidth($str)));",
        "sub colwidth() {",
        "my($str) = @;",
        "return Unicode::GCString->new($str)->columns;",
        "sub entitle() {",
        "my($str) = @;",
        "$str =~ s{ (?=\\pL)(\\S)     (\\S*) }",
        "{ ucfirst($1) . lc($2)  }xge;",
        "return $str;"
    ],
    "see_also": []
}