perldoc > Unicode::Normalize

Che Dong

📛 NAME

Unicode::Normalize - Unicode Normalization Forms

🚀 Quick Reference

Use Case	Command	Description
Canonical Decomposition	`NFD($string)`	Normalize to Form D
Canonical Composition	`NFC($string)`	Normalize to Form C
Compatibility Decomposition	`NFKD($string)`	Normalize to Form KD
Compatibility Composition	`NFKC($string)`	Normalize to Form KC
Check Normalization	`checkNFC($string)`	Returns YES/NO/MAYBE
Partial Normalization	`normalize_partial($form, $unproc)`	Normalize incrementally

📖 SYNOPSIS

(1) using function names exported by default:

  use Unicode::Normalize;

  $NFD_string  = NFD($string);  # Normalization Form D
  $NFC_string  = NFC($string);  # Normalization Form C
  $NFKD_string = NFKD($string); # Normalization Form KD
  $NFKC_string = NFKC($string); # Normalization Form KC

(2) using function names exported on request:

  use Unicode::Normalize 'normalize';

  $NFD_string  = normalize('D',  $string);  # Normalization Form D
  $NFC_string  = normalize('C',  $string);  # Normalization Form C
  $NFKD_string = normalize('KD', $string);  # Normalization Form KD
  $NFKC_string = normalize('KC', $string);  # Normalization Form KC

📝 DESCRIPTION

Parameters:

$string is used as a string under character semantics (see perlunicode).
$code_point should be an unsigned integer representing a Unicode code point.

Note: Between XSUB and pure Perl, there is an incompatibility about the interpretation of $code_point as a decimal number. XSUB converts $code_point to an unsigned integer, but pure Perl does not. Do not use a floating point nor a negative sign in $code_point.

🔄 Normalization Forms

$NFD_string = NFD($string) — It returns the Normalization Form D (canonical decomposition).
$NFC_string = NFC($string) — It returns the Normalization Form C (canonical decomposition followed by canonical composition).
$NFKD_string = NFKD($string) — It returns the Normalization Form KD (compatibility decomposition).
$NFKC_string = NFKC($string) — It returns the Normalization Form KC (compatibility decomposition followed by canonical composition).
$FCD_string = FCD($string) — If the string is in FCD ("Fast C or D"), returns it unchanged; otherwise returns an FCD string. Note: FCD is not always unique.
$FCC_string = FCC($string) — It returns the FCC form ("Fast C Contiguous"). FCC is unique.
$normalized_string = normalize($form_name, $string) — Returns the normalization form of $form_name. Valid names: 'C', 'NFC', 'D', 'NFD', 'KC', 'NFKC', 'KD', 'NFKD', 'FCD', 'FCC'.

⚙️ Decomposition and Composition

$decomposed_string = decompose($string [, $useCompatMapping]) — Returns concatenation of decomposition of each character. If second parameter omitted or false, canonical decomposition; if true, compatibility decomposition. Reordering may be required.

 $NFD_string  = reorder(decompose($string));       # eq. to NFD()
 $NFKD_string = reorder(decompose($string, TRUE)); # eq. to NFKD()

$reordered_string = reorder($string) — Reorders combining characters according to Canonical Ordering Behavior.

 $concat_NFD  = reorder(join '', @NFD_strings);
 $concat_NFKD = reorder(join '', @NFKD_strings);

$composed_string = compose($string) — Returns canonical composition without decomposition.

 $NFC_string  = compose($NFD_string);
 $NFKC_string = compose($NFKD_string);

($processed, $unprocessed) = splitOnLastStarter($normalized) — Splits at the last starter. Useful for incremental normalization.

 # Wrong: $concat = $normalized . normalize($form, $unnormalized);
 # Correct:
 ($processed, $unprocessed) = splitOnLastStarter($normalized);
 $concat = $processed . normalize($form,$unprocessed.$unnormalized);

 # Incremental normalization of array:
 my $result = "";
 my $unproc = "";
 foreach my $str (@string) {
     $unproc .= $str;
     my $n = normalize($form, $unproc);
     my($p, $u) = splitOnLastStarter($n);
     $result .= $p;
     $unproc  = $u;
 }
 $result .= $unproc;

$processed = normalize_partial($form, $unprocessed) — Wrapper for normalize + splitOnLastStarter. Modifies $unprocessed as side-effect.

 my $result = "";
 my $unproc = "";
 foreach my $str (@string) {
     $unproc .= $str;
     $result .= normalize_partial($form, $unproc);
 }
 $result .= $unproc;

$processed = NFD_partial($unprocessed) — Like normalize_partial('NFD', $unprocessed). Modifies $unprocessed.
$processed = NFC_partial($unprocessed) — Like normalize_partial('NFC', $unprocessed). Modifies $unprocessed.
$processed = NFKD_partial($unprocessed) — Like normalize_partial('NFKD', $unprocessed). Modifies $unprocessed.
$processed = NFKC_partial($unprocessed) — Like normalize_partial('NFKC', $unprocessed). Modifies $unprocessed.

✅ Quick Check

(see Annex 8, UAX #15; and DerivedNormalizationProps.txt)

The following functions check whether the string is in that normalization form. The result is one of:
YES — The string is in that normalization form.
NO — The string is not in that normalization form.
MAYBE — Dubious.

$result = checkNFD($string) — Returns true (1) if YES; false (empty string) if NO.
$result = checkNFC($string) — Returns true (1) if YES; false (empty string) if NO; undef if MAYBE.
$result = checkNFKD($string) — Returns true (1) if YES; false (empty string) if NO.
$result = checkNFKC($string) — Returns true (1) if YES; false (empty string) if NO; undef if MAYBE.
$result = checkFCD($string) — Returns true (1) if YES; false (empty string) if NO.
$result = checkFCC($string) — Returns true (1) if YES; false (empty string) if NO; undef if MAYBE.
$result = check($form_name, $string) — Returns true (1) if YES; false (empty string) if NO; undef if MAYBE. Valid form names: same as normalize.

Note: In NFD, NFKD, FCD, answer is always YES or NO. In NFC, NFKC, FCC, MAYBE may be returned. A MAYBE string contains at least one combining character. For example, "A\N{COMBINING ACUTE ACCENT}" returns MAYBE with checkNFC. To check exactly, compare with NFC:

if ($string eq NFC($string)) {
    # $string is exactly normalized in NFC;
} else {
    # $string is not normalized in NFC;
}

📊 Character Data

These functions are interface of character data used internally. You don't need to call them for normalization.

$canonical_decomposition = getCanon($code_point) — Returns full canonical decomposition as string, or undef if not decomposable.
$compatibility_decomposition = getCompat($code_point) — Returns full compatibility decomposition, or undef.
$code_point_composite = getComposite($code_point_here, $code_point_next) — Returns composite code point if composable, else undef.
$combining_class = getCombinClass($code_point) — Returns combining class as integer.
$may_be_composed_with_prev_char = isComp2nd($code_point) — Boolean: may be composed with previous character.
$is_exclusion = isExclusion($code_point) — Boolean: composition exclusion.
$is_singleton = isSingleton($code_point) — Boolean: singleton.
$is_non_starter_decomposition = isNonStDecomp($code_point) — Boolean: Non-Starter Decomposition.
$is_Full_Composition_Exclusion = isComp_Ex($code_point) — Boolean: derived property Comp_Ex.
$NFD_is_NO = isNFD_NO($code_point) — Boolean: NFD_Quick_Check=No.
$NFC_is_NO = isNFC_NO($code_point) — Boolean: NFC_Quick_Check=No.
$NFC_is_MAYBE = isNFC_MAYBE($code_point) — Boolean: NFC_Quick_Check=Maybe.
$NFKD_is_NO = isNFKD_NO($code_point) — Boolean: NFKD_Quick_Check=No.
$NFKC_is_NO = isNFKC_NO($code_point) — Boolean: NFKC_Quick_Check=No.
$NFKC_is_MAYBE = isNFKC_MAYBE($code_point) — Boolean: NFKC_Quick_Check=Maybe.

📦 EXPORT

NFC, NFD, NFKC, NFKD: by default.
normalize and other some functions: on request.

⚠️ CAVEATS

Perl's version vs. Unicode version

This module refers to perl core's Unicode database. The Unicode version of normalization depends on what has been compiled into your perl. The following table lists the default Unicode version that comes with various perl versions.

Perl version	Unicode version
5.6.1	3.0.1
5.7.2	3.1.0
5.7.3	3.1.1 (normalization same as 3.1.0)
5.8.0	3.2.0
5.8.1-5.8.3	4.0.0
5.8.4-5.8.6	4.0.1 (normalization same as 4.0.0)
5.8.7-5.8.8	4.1.0
5.10.0	5.0.0
5.8.9, 5.10.1	5.1.0
5.12.x	5.2.0
5.14.x	6.0.0
5.16.x	6.1.0
5.18.x	6.2.0
5.20.x	6.3.0
5.22.x	7.0.0

Correction of decomposition mapping

In older Unicode versions, a small number of characters (CJK compatibility ideographs) may have erroneous decomposition mappings. This module does not refer to NormalizationCorrections.txt, so it may use erroneous mappings on older perl.

Revised definition of canonical composition

In Unicode 4.1.0, the definition D2 of canonical composition (affecting NFC and NFKC) has been changed. This module has used the newer definition since version 0.07 (Oct 31, 2001). It will not support the older definition.

👤 AUTHOR

SADAHIRO Tomoyuki <SADAHIRO AT cpan.org>

Currently maintained by <perl5-porters AT perl.org>

📜 LICENSE

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

🔗 SEE ALSO

Generated by phpman v4.9.26-1-g511901d · Markdown · JSON · MCP Author: Che Dong Under GNU General Public License
2026-07-31 14:20 @216.73.217.152
CrawledBy Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Enhanced by LLM: deepseek-v4-flash / taotoken.net / www.chedong.com - original format