{ "content": [ { "type": "text", "text": "# encoding::warnings (perldoc)\n\n## NAME\n\nencoding::warnings - Warn on implicit encoding conversions\n\n## SYNOPSIS\n\nuse encoding::warnings; # or 'FATAL' to raise fatal exceptions\nutf8::encode($a = chr(20000)); # a byte-string (raw bytes)\n$b = chr(20000); # a unicode-string (wide characters)\n# \"Bytes implicitly upgraded into wide characters as iso-8859-1\"\n$c = $a . $b;\n\n## Sections\n\n- **NAME**\n- **VERSION**\n- **NOTICE**\n- **SYNOPSIS**\n- **DESCRIPTION** (3 subsections)\n- **CAVEATS**\n- **SEE ALSO**\n- **AUTHORS**\n- **COPYRIGHT**\n\nUse structuredContent.sections for detailed options, examples, and full documentation.\n" } ], "structuredContent": { "command": "encoding::warnings", "section": "", "mode": "perldoc", "summary": "encoding::warnings - Warn on implicit encoding conversions", "synopsis": "use encoding::warnings; # or 'FATAL' to raise fatal exceptions\nutf8::encode($a = chr(20000)); # a byte-string (raw bytes)\n$b = chr(20000); # a unicode-string (wide characters)\n# \"Bytes implicitly upgraded into wide characters as iso-8859-1\"\n$c = $a . $b;", "tldr_summary": null, "tldr_examples": [], "tldr_source": null, "flags": [], "examples": [], "see_also": [], "section_outline": [ { "name": "NAME", "lines": 2, "subsections": [] }, { "name": "VERSION", "lines": 2, "subsections": [] }, { "name": "NOTICE", "lines": 9, "subsections": [] }, { "name": "SYNOPSIS", "lines": 8, "subsections": [] }, { "name": "DESCRIPTION", "lines": 1, "subsections": [ { "name": "Overview of the problem", "lines": 10 }, { "name": "Detecting the problem", "lines": 15 }, { "name": "Solving the problem", "lines": 59 } ] }, { "name": "CAVEATS", "lines": 5, "subsections": [] }, { "name": "SEE ALSO", "lines": 4, "subsections": [] }, { "name": "AUTHORS", "lines": 2, "subsections": [] }, { "name": "COPYRIGHT", "lines": 7, "subsections": [] } ], "sections": { "NAME": { "content": "encoding::warnings - Warn on implicit encoding conversions\n", "subsections": [] }, "VERSION": { "content": "This document describes version 0.13 of encoding::warnings, released June 20, 2016.\n", "subsections": [] }, "NOTICE": { "content": "As of Perl 5.26.0, this module has no effect. The internal Perl feature that was used to\nimplement this module has been removed. In recent years, much work has been done on the Perl\ncore to eliminate discrepancies in the treatment of upgraded versus downgraded strings. In\naddition, the encoding pragma, which caused many of the problems, is no longer supported. Thus,\nthe warnings this module produced are no longer necessary.\n\nHence, if you load this module on Perl 5.26.0, you will get one warning that the module is no\nlonger supported; and the module will do nothing thereafter.\n", "subsections": [] }, "SYNOPSIS": { "content": "use encoding::warnings; # or 'FATAL' to raise fatal exceptions\n\nutf8::encode($a = chr(20000)); # a byte-string (raw bytes)\n$b = chr(20000); # a unicode-string (wide characters)\n\n# \"Bytes implicitly upgraded into wide characters as iso-8859-1\"\n$c = $a . $b;\n", "subsections": [] }, "DESCRIPTION": { "content": "", "subsections": [ { "name": "Overview of the problem", "content": "By default, there is a fundamental asymmetry in Perl's unicode model: implicit upgrading from\nbyte-strings to unicode-strings assumes that they were encoded in *ISO 8859-1 (Latin-1)*, but\nunicode-strings are downgraded with UTF-8 encoding. This happens because the first 256\ncodepoints in Unicode happens to agree with Latin-1.\n\nHowever, this silent upgrading can easily cause problems, if you happen to mix unicode strings\nwith non-Latin1 data -- i.e. byte-strings encoded in UTF-8 or other encodings. The error will\nnot manifest until the combined string is written to output, at which time it would be\nimpossible to see where did the silent upgrading occur.\n" }, { "name": "Detecting the problem", "content": "This module simplifies the process of diagnosing such problems. Just put this line on top of\nyour main program:\n\nuse encoding::warnings;\n\nAfterwards, implicit upgrading of high-bit bytes will raise a warning. Ex.: \"Bytes implicitly\nupgraded into wide characters as iso-8859-1 at - line 7\".\n\nHowever, strings composed purely of ASCII code points (0x00..0x7F) will *not* trigger this\nwarning.\n\nYou can also make the warnings fatal by importing this module as:\n\nuse encoding::warnings 'FATAL';\n" }, { "name": "Solving the problem", "content": "Most of the time, this warning occurs when a byte-string is concatenated with a unicode-string.\nThere are a number of ways to solve it:\n\n* Upgrade both sides to unicode-strings\n\nIf your program does not need compatibility for Perl 5.6 and earlier, the recommended\napproach is to apply appropriate IO disciplines, so all data in your program become\nunicode-strings. See encoding, open and \"binmode\" in perlfunc for how.\n\n* Downgrade both sides to byte-strings\n\nThe other way works too, especially if you are sure that all your data are under the same\nencoding, or if compatibility with older versions of Perl is desired.\n\nYou may downgrade strings with \"Encode::encode\" and \"utf8::encode\". See Encode and utf8 for\ndetails.\n\n* Specify the encoding for implicit byte-string upgrading\n\nIf you are confident that all byte-strings will be in a specific encoding like UTF-8, *and*\nneed not support older versions of Perl, use the \"encoding\" pragma:\n\nuse encoding 'utf8';\n\nSimilarly, this will silence warnings from this module, and preserve the default behaviour:\n\nuse encoding 'iso-8859-1';\n\nHowever, note that \"use encoding\" actually had three distinct effects:\n\n* PerlIO layers for STDIN and STDOUT\n\nThis is similar to what open pragma does.\n\n* Literal conversions\n\nThis turns *all* literal string in your program into unicode-strings (equivalent to a\n\"use utf8\"), by decoding them using the specified encoding.\n\n* Implicit upgrading for byte-strings\n\nThis will silence warnings from this module, as shown above.\n\nBecause literal conversions also work on empty strings, it may surprise some people:\n\nuse encoding 'big5';\n\nmy $bytestring = pack(\"C*\", 0xA4, 0x40);\nprint length $a; # 2 here.\n$a .= \"\"; # concatenating with a unicode string...\nprint length $a; # 1 here!\n\nIn other words, do not \"use encoding\" unless you are certain that the program will not deal\nwith any raw, 8-bit binary data at all.\n\nHowever, the \"Filter => 1\" flavor of \"use encoding\" will *not* affect implicit upgrading for\nbyte-strings, and is thus incapable of silencing warnings from this module. See encoding for\nmore details.\n" } ] }, "CAVEATS": { "content": "For Perl 5.9.4 or later, this module's effect is lexical.\n\nFor Perl versions prior to 5.9.4, this module affects the whole script, instead of inside its\nlexical block.\n", "subsections": [] }, "SEE ALSO": { "content": "perlunicode, perluniintro\n\nopen, utf8, encoding, Encode\n", "subsections": [] }, "AUTHORS": { "content": "Audrey Tang\n", "subsections": [] }, "COPYRIGHT": { "content": "Copyright 2004, 2005, 2006, 2007 by Audrey Tang .\n\nThis program is free software; you can redistribute it and/or modify it under the same terms as\nPerl itself.\n\nSee \n", "subsections": [] } } } }