{ "content": [ { "type": "text", "text": "# perlunitut (man)\n\n## NAME\n\nperlunitut - Perl Unicode Tutorial\n\n## DESCRIPTION\n\nThe days of just flinging strings around are over. It's well established that modern programs\nneed to be capable of communicating funny accented letters, and things like euro symbols.\nThis means that programmers need new habits. It's easy to program Unicode capable software,\nbut it does require discipline to do it right.\n\n## Sections\n\n- **NAME**\n- **DESCRIPTION** (3 subsections)\n- **SUMMARY** (1 subsections)\n- **ACKNOWLEDGEMENTS**\n- **AUTHOR**\n- **SEE ALSO**\n\nUse structuredContent.sections for detailed options, examples, and full documentation.\n" } ], "structuredContent": { "command": "perlunitut", "section": "", "mode": "man", "summary": "perlunitut - Perl Unicode Tutorial", "synopsis": null, "tldr_summary": null, "tldr_examples": [], "tldr_source": null, "flags": [], "examples": [], "see_also": [], "section_outline": [ { "name": "NAME", "lines": 2, "subsections": [] }, { "name": "DESCRIPTION", "lines": 19, "subsections": [ { "name": "Definitions", "lines": 79 }, { "name": "Your new toolkit", "lines": 8 }, { "name": "I/O flow (the actual 5 minute tutorial)", "lines": 45 } ] }, { "name": "SUMMARY", "lines": 2, "subsections": [ { "name": "Q and A (or FAQ)", "lines": 2 } ] }, { "name": "ACKNOWLEDGEMENTS", "lines": 13, "subsections": [] }, { "name": "AUTHOR", "lines": 2, "subsections": [] }, { "name": "SEE ALSO", "lines": 5, "subsections": [] } ], "sections": { "NAME": { "content": "perlunitut - Perl Unicode Tutorial\n", "subsections": [] }, "DESCRIPTION": { "content": "The days of just flinging strings around are over. It's well established that modern programs\nneed to be capable of communicating funny accented letters, and things like euro symbols.\nThis means that programmers need new habits. It's easy to program Unicode capable software,\nbut it does require discipline to do it right.\n\nThere's a lot to know about character sets, and text encodings. It's probably best to spend a\nfull day learning all this, but the basics can be learned in minutes.\n\nThese are not the very basics, though. It is assumed that you already know the difference\nbetween bytes and characters, and realise (and accept!) that there are many different\ncharacter sets and encodings, and that your program has to be explicit about them.\nRecommended reading is \"The Absolute Minimum Every Software Developer Absolutely, Positively\nMust Know About Unicode and Character Sets (No Excuses!)\" by Joel Spolsky, at\n.\n\nThis tutorial speaks in rather absolute terms, and provides only a limited view of the wealth\nof character string related features that Perl has to offer. For most projects, this\ninformation will probably suffice.\n", "subsections": [ { "name": "Definitions", "content": "It's important to set a few things straight first. This is the most important part of this\ntutorial. This view may conflict with other information that you may have found on the web,\nbut that's mostly because many sources are wrong.\n\nYou may have to re-read this entire section a few times...\n\nUnicode\n\nUnicode is a character set with room for lots of characters. The ordinal value of a character\nis called a code point. (But in practice, the distinction between code point and character\nis blurred, so the terms often are used interchangeably.)\n\nThere are many, many code points, but computers work with bytes, and a byte has room for only\n256 values. Unicode has many more characters than that, so you need a method to make these\naccessible.\n\nUnicode is encoded using several competing encodings, of which UTF-8 is the most used. In a\nUnicode encoding, multiple subsequent bytes can be used to store a single code point, or\nsimply: character.\n\nUTF-8\n\nUTF-8 is a Unicode encoding. Many people think that Unicode and UTF-8 are the same thing, but\nthey're not. There are more Unicode encodings, but much of the world has standardized on\nUTF-8.\n\nUTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. They take only one byte per\ncharacter. All other characters are encoded as two to four bytes using a complex scheme.\nFortunately, Perl handles this for us, so we don't have to worry about this.\n\nText strings (character strings)\n\nText strings, or character strings are made of characters. Bytes are irrelevant here, and so\nare encodings. Each character is just that: the character.\n\nOn a text string, you would do things like:\n\n$text =~ s/foo/bar/;\nif ($string =~ /^\\d+$/) { ... }\n$text = ucfirst $text;\nmy $charactercount = length $text;\n\nThe value of a character (\"ord\", \"chr\") is the corresponding Unicode code point.\n\nBinary strings (byte strings)\n\nBinary strings, or byte strings are made of bytes. Here, you don't have characters, just\nbytes. All communication with the outside world (anything outside of your current Perl\nprocess) is done in binary.\n\nOn a binary string, you would do things like:\n\nmy (@lengthcontent) = unpack \"(V/a)*\", $binary;\n$binary =~ s/\\x00\\x0F/\\xFF\\xF0/; # for the brave :)\nprint {$fh} $binary;\nmy $bytecount = length $binary;\n\nEncoding\n\nEncoding (as a verb) is the conversion from text to binary. To encode, you have to supply the\ntarget encoding, for example \"iso-8859-1\" or \"UTF-8\". Some encodings, like the \"iso-8859\"\n(\"latin\") range, do not support the full Unicode standard; characters that can't be\nrepresented are lost in the conversion.\n\nDecoding\n\nDecoding is the conversion from binary to text. To decode, you have to know what encoding was\nused during the encoding phase. And most of all, it must be something decodable. It doesn't\nmake much sense to decode a PNG image into a text string.\n\nInternal format\n\nPerl has an internal format, an encoding that it uses to encode text strings so it can store\nthem in memory. All text strings are in this internal format. In fact, text strings are\nnever in any other format!\n\nYou shouldn't worry about what this format is, because conversion is automatically done when\nyou decode or encode.\n" }, { "name": "Your new toolkit", "content": "Add to your standard heading the following line:\n\nuse Encode qw(encode decode);\n\nOr, if you're lazy, just:\n\nuse Encode;\n" }, { "name": "I/O flow (the actual 5 minute tutorial)", "content": "The typical input/output flow of a program is:\n\n1. Receive and decode\n2. Process\n3. Encode and output\n\nIf your input is binary, and is supposed to remain binary, you shouldn't decode it to a text\nstring, of course. But in all other cases, you should decode it.\n\nDecoding can't happen reliably if you don't know how the data was encoded. If you get to\nchoose, it's a good idea to standardize on UTF-8.\n\nmy $foo = decode('UTF-8', get 'http://example.com/');\nmy $bar = decode('ISO-8859-1', readline STDIN);\nmy $xyzzy = decode('Windows-1251', $cgi->param('foo'));\n\nProcessing happens as you knew before. The only difference is that you're now using\ncharacters instead of bytes. That's very useful if you use things like \"substr\", or \"length\".\n\nIt's important to realize that there are no bytes in a text string. Of course, Perl has its\ninternal encoding to store the string in memory, but ignore that. If you have to do anything\nwith the number of bytes, it's probably best to move that part to step 3, just after you've\nencoded the string. Then you know exactly how many bytes it will be in the destination\nstring.\n\nThe syntax for encoding text strings to binary strings is as simple as decoding:\n\n$body = encode('UTF-8', $body);\n\nIf you needed to know the length of the string in bytes, now's the perfect time for that.\nBecause $body is now a byte string, \"length\" will report the number of bytes, instead of the\nnumber of characters. The number of characters is no longer known, because characters only\nexist in text strings.\n\nmy $bytecount = length $body;\n\nAnd if the protocol you're using supports a way of letting the recipient know which character\nencoding you used, please help the receiving end by using that feature! For example, E-mail\nand HTTP support MIME headers, so you can use the \"Content-Type\" header. They can also have\n\"Content-Length\" to indicate the number of bytes, which is always a good idea to supply if\nthe number is known.\n\n\"Content-Type: text/plain; charset=UTF-8\",\n\"Content-Length: $bytecount\"\n" } ] }, "SUMMARY": { "content": "Decode everything you receive, encode everything you send out. (If it's text data.)\n", "subsections": [ { "name": "Q and A (or FAQ)", "content": "After reading this document, you ought to read perlunifaq too, then perluniintro.\n" } ] }, "ACKNOWLEDGEMENTS": { "content": "Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants during the Amsterdam Perl\nMongers meetings got me interested and determined to find out how to use character encodings\nin Perl in ways that don't break easily.\n\nThanks to Gerard Goossen from TTY. His presentation \"UTF-8 in the wild\" (Dutch Perl Workshop\n2006) inspired me to publish my thoughts and write this tutorial.\n\nThanks to the people who asked about this kind of stuff in several Perl IRC channels, and\nhave constantly reminded me that a simpler explanation was needed.\n\nThanks to the people who reviewed this document for me, before it went public. They are:\nBenjamin Smith, Jan-Pieter Cornet, Johan Vromans, Lukas Mai, Nathan Gray.\n", "subsections": [] }, "AUTHOR": { "content": "Juerd Waalboer <#####@juerd.nl>\n", "subsections": [] }, "SEE ALSO": { "content": "perlunifaq, perlunicode, perluniintro, Encode\n\n\n\nperl v5.34.0 2025-07-25 PERLUNITUT(1)", "subsections": [] } } } }