man > Lingua::Stem(3pm)

man perldoc info pydoc3 ri search

📛 NAME

Lingua::Stem - Stemming of words

🚀 Quick Reference

Use Case	Command	Description
Stem a list of words	stem(@words)	Returns array ref of stemmed words
Create OO stemmer	Lingua::Stem->new( -locale => 'en-uk' )	Instantiate with optional locale
Set locale	$stemmer->set_locale('fr')	Change locale for instance
Enable caching (level 2)	$stemmer->stem_caching({ -level => 2 })	Cache indefinitely for speed
Add exceptions	$stemmer->add_exceptions({ 'driven' => 'driven' })	Override stemming for specific words
Stem in‑place (faster)	stem_in_place(@words)	Modifies list directly; ~60% faster than stem
Clear cache	$stemmer->clear_stem_cache	Empty the stemming cache

📝 SYNOPSIS

use Lingua::Stem qw(stem);
my $stemmmed_words_anon_array   = stem(@words);

or for the OO inclined,

use Lingua::Stem;
my $stemmer = Lingua::Stem->new(-locale => 'EN-UK');
$stemmer->stem_caching({ -level => 2 });
my $stemmmed_words_anon_array   = $stemmer->stem(@words);

📖 DESCRIPTION

This routine applies stemming algorithms to its parameters, returning the stemmed words as appropriate to the selected locale.

You can import some or all of the class methods.

use Lingua::Stem qw (stem clear_stem_cache stem_caching
                     add_exceptions delete_exceptions
                     get_exceptions set_locale get_locale
                     :all :locale :exceptions :stem :caching);

• :all – imports stem, add_exceptions, delete_exceptions, get_exceptions, set_locale, get_locale
• :stem – imports stem
• :caching – imports stem_caching, clear_stem_cache
• :locale – imports set_locale, get_locale
• :exceptions – imports add_exceptions, delete_exceptions, get_exceptions

🌐 Supported Locales

• 🇩🇰 DA – Danish
• 🇩🇪 DE – German
• 🇬🇧 EN – English (also EN-US and EN-UK)
• 🇫🇷 FR – French
• 🇪🇸 GL – Galician
• 🇮🇹 IT – Italian
• 🇳🇴 NO – Norwegian
• 🇵🇹 PT – Portuguese
• 🇷🇺 RU – Russian (also RU-RU and RU-RU.KOI8-R)
• 🇸🇪 SV – Swedish

If you have the memory and lots of stemming to do, I strongly suggest using cache level 2 and processing lists in 'big chunks' (long lists) for best performance.

⚖️ Comparison with Lingua::Stem::Snowball

It functions fairly similarly to the Lingua::Stem::Snowball suite of stemmers, with the most significant differences being

1. Lingua::Stem is a 'pure perl' (no compiled XS code is needed) suite. Lingua::Stem::Snowball is XS based (must be compiled).
2. Lingua::Stem works with Perl 5.6 or later. Lingua::Stem::Snowball works with Perl 5.8 or later.
3. Lingua::Stem has an 'exceptions' system allowing you to override stemming on a 'case by case' basis. Lingua::Stem::Snowball does not have an 'exceptions' system.
4. A somewhat different set of supported languages:

Language	ISO code	Lingua::Stem	Lingua::Stem::Snowball
Danish	da	✅	✅
Dutch	nl	❌	✅
English	en	✅	✅
Finnish	fi	❌	✅
French	fr	✅	✅
Galacian	gl	✅	❌
German	de	✅	✅
Italian	it	✅	✅
Norwegian	no	✅	✅
Portuguese	pt	✅	✅
Russian	ru	✅	✅
Spanish	es	❌	✅
Swedish	sv	✅	✅

5. Lingua::Stem is faster for 'stem' (circa 30% faster than Lingua::Stem::Snowball)
6. Lingua::Stem::Snowball is faster for 'stem_in_place' (circa 30% faster than Lingua::Stem)
7. Lingua::Stem::Snowball is more consistent with regard to character set issues.
8. Lingua::Stem::Snowball is under active development. Lingua::Stem is currently fairly static.

Some benchmarks using Lingua::Stem 0.82 and Lingua::Stem::Snowball 0.94 gives an idea of how various options impact performance. The dataset was The Works of Edgar Allen Poe, volumes 1-5 from the Gutenberg Project processed 10 times in a row as single batch of words (processing a long text one word at a time is very inefficient and drops the performance of Lingua::Stem by about 90%: So "Don't Do That" ;) )

The benchmarks were run on a 3.06 Ghz P4 with HT on Fedora Core 5 Linux using Perl 5.8.8.

source: collected_works_poe.txt	words: 454691	unique words: 22802
module	config	avg secs	words/sec
Lingua::Stem 0.82	no cache	1.922	236560
Lingua::Stem 0.82	cache level 2	1.235	368292
Lingua::Stem 0.82	cachelv2, sip	0.798	569494
Lingua::Stem::Snowball 0.94	stem	1.622	280276
Lingua::Stem::Snowball 0.94	stem_in_place	0.627	725129

The script for the benchmark is included in the examples/ directory of this distribution as benchmark_stemmers.plx.

📅 CHANGES

• 🆕 2.30 2020.06.20 – Version renumber for module consistency
• 0.84 2010.04.29 – Documentation fixes to the En stemmer and removal of the accidentally included lib/Lingua/test.pl file. Thanks to Aaron Naiman for the documentation error and Alexandr Ciornii & 'kmx' for the test.pl issue.
• 0.83 2007.06.23 – Disabled Italian locale build tests due to changes in Lingua::Stem::It breaking the tests.
• 0.82 2006.07.23 – Added 'stem_in_place' to base package. Tweaks to documentation and build tests.
• 0.81 2004.07.26 – Minor documentation tweak. No functional change.
• 0.80 2004.07.25 – Added 'RU', 'RU_RU', 'RU_RU.KOI-8' locale. Added support for Lingua::Stem::Ru to Makefile.PL and autoloader. Added documentation stressing caching and batches. Added '_' separator support. Added benchmark script. Expanded copyright.
• 0.70 2004.04.26 – Added FR locale and documentation fixes to Lingua::Stem::Gl
• 0.61 2003.09.28 – Documentation fixes. No functional changes.
• 0.60 2003.04.05 – Added more locales by wrapping various stemming implementations. Documented supported locales.
• 0.50 2000.09.14 – Fixed major implementation error: rulesets 2,3,4 missing from Porter's algorithm. Stemming results were poor. Thanks <csyap AT netfision.com>. If you need identically broken output, use locale 'en-broken'.
• 0.40 2000.08.25 – Added stem caching support. Default off for compatibility.
• 0.30 1999.06.24 – Replaced 'En' stemmer core with code from Jim Richardson <jimr AT maths.au>. Aliased 'en-us' and 'en-uk' to 'en'. Fixed SYNOPSIS return type.
• 0.20 1999.06.15 – Changed to '.pm' module, moved into Lingua:: namespace, added OO interface, export optionalization, named parameters, stemming exceptions, autoloaded locales, isolated case flattening. UTF8 assumption.

🔧 METHODS

• new(...); – Returns a new instance of a Lingua::Stem object and, optionally, selection of the locale to be used for stemming.

  # By default the locale is en
  $us_stemmer = Lingua::Stem->new;
  
  # Turn on the cache
  $us_stemmer->stem_caching({ -level => 2 });
  
  # Overriding the default for a specific instance
  $uk_stemmer = Lingua::Stem->new({ -locale => 'en-uk' });
  
  # Overriding the default and changing the default
  $uk_stemmer = Lingua::Stem->new({ -default_locale => 'en-uk' });
  

• set_locale($locale); – Sets the locale to one of the recognized locales. Locale identifiers are converted to lowercase. Called as a class method, it changes the default locale for all subsequently generated object instances. Called as an instance method, it only changes the locale for that particular instance. 'croaks' if passed an unknown locale.

  # Change default locale
  Lingua::Stem::set_locale('en-uk'); # UK's spellings
  
  # Change instance locale
  $self->set_locale('en-us');  # US's spellings
  

• get_locale; – Called as a class method, returns the current default locale. Called as an instance method, returns the locale for the instance.

  $default_locale = Lingua::Stem::get_locale;
  $instance_locale = $stemmer->get_locale;
  

• add_exceptions($exceptions_hash_ref); – Exceptions allow overriding the stemming algorithm on a case by case basis. It is done on an exact match and substitution basis: If a passed word is identical to the exception it will be replaced by the specified value. No case adjustments are performed. Called as a class method, adds exceptions to the default exceptions list used for subsequently instantiated objects. Called as an instance method, adds exceptions only to the specific instance.

  # adding default exceptions
  Lingua::Stem::add_exceptions({ 'emily' => 'emily',
                                 'driven' => 'driven',
                             });
  
  # adding instance exceptions
  $stemmer->add_exceptions({ 'steely' => 'steely' });
  

  The exceptions shortcut the normal stemming – if an exception matches no further stemming is performed after the substitution. Adding an exception with the same key replaces the pre-existing exception.
• delete_exceptions(@exceptions_list); – The mirror of add_exceptions, this allows the removal of exceptions from either the defaults for the class or from the instance.

  # Deletion of exceptions from class default exceptions
  Lingua::Stem::delete_exceptions('aragorn','frodo','samwise');
  
  # Deletion of exceptions from instance
  $stemmer->delete_exceptions('smaug','sauron','gollum');
  
  # Deletion of all class default exceptions
  delete_exceptions;
  
  # Deletion of all exceptions from instance
  $stemmer->delete_exceptions;
  

• get_exceptions; – As a class method with no parameters it returns all the default exceptions as an anonymous hash of 'exception' => 'replace with' pairs. As a class method with parameters, it returns the default exceptions listed in the parameters. If a parameter specifies an undefined 'exception', the value is set to undef. As an instance method, with no parameters it returns the currently active exceptions for the instance. As an instance method with parameters, it returns the instance exceptions listed.

  # Returns all class default exceptions
  $exceptions = Lingua::Stem::get_exceptions;
  
  # Returns class default exceptions for 'emily' and 'george'
  $exceptions = Lingua::Stem::get_exceptions('emily','george');
  
  # Returns all instance exceptions
  $exceptions = $stemmer->get_exceptions;
  
  # Returns instance exceptions for 'lisa' and 'bart'
  $exceptions = $stemmer->get_exceptions('lisa','bart');
  

• stem(@list); – Called as a class method, it applies the default settings and stems the list of passed words, returning an anonymous array with the stemmed words in the same order. Called as an instance method, it applies the instance's settings. The stemmer performs best when handed long lists of words rather than one word at a time. The cache also provides a huge speed up.

  # Default settings applied
  my $anon_array_of_stemmed_words = Lingua::Stem::stem(@words);
  
  # Instance's settings applied
  my $stemmed_words = $stemmer->stem(@words);
  

• stem_in_place(@list); – ⚠️ Stems the passed list of words 'in place'. It returns a reference to the modified list. This is about 60% faster than the 'stem' method but modifies the original list. This currently only works for the English locales.

  my @words = ( 'a', 'list', 'of', 'words' );
  my $stemmed_list_of_words = stem_in_place(@words);
  
  # '$stemmed_list_of_words' refers to the @words list
  # after 'stem_in_place' has executed
  

  DO NOT use this method if you need to keep the original list of words. Its performance gain derives entirely from the fact it does not make a copy. If you copy the list first, you wipe out the speed gain – use the regular stem method instead.
• clear_stem_cache; – Clears the stemming cache for the current locale. Can be called as either a class method or an instance method.

  $stemmer->clear_stem_cache;
  clear_stem_cache;
  

• stem_caching({ -level => 0|1|2 }); – Sets stemming cache level for the current locale. Can be called as either a class method or an instance method. Default is -level => 0 (no caching) for maximum compatibility. Level definitions:
  • 0 – no caching (default)
  • 1 – cache per run: caches stemming results during each call to stem
  • 2 – cache indefinitely: caches until process exit or clear_stem_cache is called
  Stem caching is global to the locale. If you turn it on for one instance, all instances using the same locale will have it on as well. I STRONGLY suggest turning caching on if you have enough memory and are processing a lot of data.

  $stemmer->stem_caching({ -level => 1 });
  stem_caching({ -level => 1 });
  

📌 VERSION

2.30 2020.06.20

📝 NOTES

It started with the 'Text::Stem' module which has been adapted into a more general framework and moved into the more language oriented 'Lingua' namespace and re-organized to support an OOP interface as well as switch core 'En' locale stemmers.

Version 0.40 added a cache for stemmed words. This can provide up to a several fold performance improvement.

Organization is such that extending this module to any number of languages should be direct and simple.

Case flattening is a function of the language, so the 'exceptions' methods have to be used appropriately to the language. For 'En' family stemming, use lower case words, only, for exceptions.

👥 AUTHORS

Jerilyn Franz <cpan AT jerilyn.info>
Jim Richardson <imr AT maths.au>

🙏 CREDITS

Jim Richardson <imr AT maths.au>
Ulrich Pfeifer <pfeifer AT ls6.de>
Aldo Calpini <dada AT perl.it>
xern <xern AT cpan.org>
Ask Solem Hoel <ask AT unixmonks.net>
Dennis Haney <davh AT davh.dk>
Sebastien Darribere-Pleyt <sebastien.darribere AT lefute.com>
Aleksandr Guidrevitch <pillgrim AT mail.ru>

📚 SEE ALSO

📄 Lingua::Stem::En 📄 Lingua::Stem::En 📄 Lingua::Stem::Da 📄 Lingua::Stem::De 📄 Lingua::Stem::Gl 📄 Lingua::Stem::No 📄 Lingua::Stem::Pt 📄 Lingua::Stem::Sv 📄 Lingua::Stem::It 📄 Lingua::Stem::Fr 📄 Lingua::Stem::Ru 📄 Text::German 📄 Lingua::PT::Stemmer 📄 Lingua::GL::Stemmer 📄 Lingua::Stem::Snowball::No 📄 Lingua::Stem::Snowball::Se 📄 Lingua::Stem::Snowball::Da 📄 Lingua::Stem::Snowball::Sv 📄 Lingua::Stemmer::GL 📄 Lingua::Stem::Snowball 📄 http://snowball.tartarus.org

©️ COPYRIGHT

Copyright 1999-2004

Freerun Technologies, Inc (Freerun), Jim Richardson, University of Sydney <imr AT maths.au> and Jerilyn Franz <cpan AT jerilyn.info>. All rights reserved.

Text::German was written and is copyrighted by Ulrich Pfeifer.

Lingua::Stem::Snowball::Da was written and is copyrighted by Dennis Haney and Ask Solem Hoel.

Lingua::Stem::It was written and is copyrighted by Aldo Calpini.

Lingua::Stem::Snowball::No, Lingua::Stem::Snowball::Se, Lingua::Stem::Snowball::Sv were written and are copyrighted by Ask Solem Hoel.

Lingua::Stemmer::GL and Lingua::PT::Stemmer were written and are copyrighted by Xern.

Lingua::Stem::Fr was written and is copyrighted by Aldo Calpini and Sebastien Darribere-Pley.

Lingua::Stem::Ru was written and is copyrighted by Aleksandr Guidrevitch.

This software may be freely copied and distributed under the same terms and conditions as Perl.

🐛 BUGS

🐛 None known.

🔮 TODO

Add more languages. Extend regression tests. Add support for the Lingua::Stem::Snowball family of stemmers as an alternative core stemming engine. Extend 'stem_in_place' functionality to non‑English stemmers.

📛 NAME 🚀 Quick Reference 📝 SYNOPSIS 📖 DESCRIPTION 📅 CHANGES 🔧 METHODS 📌 VERSION 📝 NOTES 👥 AUTHORS 🙏 CREDITS 📚 SEE ALSO ©️ COPYRIGHT 🐛 BUGS 🔮 TODO