{ "content": [ { "type": "text", "text": "# Lingua::Stem (perldoc)\n\n## NAME\n\nLingua::Stem - Stemming of words\n\n## SYNOPSIS\n\nuse Lingua::Stem qw(stem);\nmy $stemmmedwordsanonarray = stem(@words);\nor for the OO inclined,\nuse Lingua::Stem;\nmy $stemmer = Lingua::Stem->new(-locale => 'EN-UK');\n$stemmer->stemcaching({ -level => 2 });\nmy $stemmmedwordsanonarray = $stemmer->stem(@words);\n\n## DESCRIPTION\n\nThis routine applies stemming algorithms to its parameters, returning the stemmed words as\nappropriate to the selected locale.\n\n## Sections\n\n- **NAME**\n- **SYNOPSIS**\n- **DESCRIPTION** (1 subsections)\n- **CHANGES**\n- **METHODS** (6 subsections)\n- **VERSION**\n- **NOTES**\n- **AUTHORS**\n- **CREDITS**\n- **SEE ALSO**\n- **COPYRIGHT**\n- **BUGS**\n- **TODO**\n\nUse structuredContent.sections for detailed options, examples, and full documentation.\n" } ], "structuredContent": { "command": "Lingua::Stem", "section": "", "mode": "perldoc", "summary": "Lingua::Stem - Stemming of words", "synopsis": "use Lingua::Stem qw(stem);\nmy $stemmmedwordsanonarray = stem(@words);\nor for the OO inclined,\nuse Lingua::Stem;\nmy $stemmer = Lingua::Stem->new(-locale => 'EN-UK');\n$stemmer->stemcaching({ -level => 2 });\nmy $stemmmedwordsanonarray = $stemmer->stem(@words);", "tldr_summary": null, "tldr_examples": [], "tldr_source": null, "flags": [], "examples": [], "see_also": [], "section_outline": [ { "name": "NAME", "lines": 2, "subsections": [] }, { "name": "SYNOPSIS", "lines": 10, "subsections": [] }, { "name": "DESCRIPTION", "lines": 31, "subsections": [ { "name": "Comparision with Lingua::Stem::Snowball", "lines": 62 } ] }, { "name": "CHANGES", "lines": 70, "subsections": [] }, { "name": "METHODS", "lines": 1, "subsections": [ { "name": "new", "lines": 17 }, { "name": "set_locale", "lines": 29 }, { "name": "add_exceptions", "lines": 25 }, { "name": "delete_exceptions", "lines": 44 }, { "name": "stem", "lines": 19 }, { "name": "stem_in_place", "lines": 60 } ] }, { "name": "VERSION", "lines": 2, "subsections": [] }, { "name": "NOTES", "lines": 14, "subsections": [] }, { "name": "AUTHORS", "lines": 3, "subsections": [] }, { "name": "CREDITS", "lines": 9, "subsections": [] }, { "name": "SEE ALSO", "lines": 10, "subsections": [] }, { "name": "COPYRIGHT", "lines": 22, "subsections": [] }, { "name": "BUGS", "lines": 2, "subsections": [] }, { "name": "TODO", "lines": 4, "subsections": [] } ], "sections": { "NAME": { "content": "Lingua::Stem - Stemming of words\n", "subsections": [] }, "SYNOPSIS": { "content": "use Lingua::Stem qw(stem);\nmy $stemmmedwordsanonarray = stem(@words);\n\nor for the OO inclined,\n\nuse Lingua::Stem;\nmy $stemmer = Lingua::Stem->new(-locale => 'EN-UK');\n$stemmer->stemcaching({ -level => 2 });\nmy $stemmmedwordsanonarray = $stemmer->stem(@words);\n", "subsections": [] }, "DESCRIPTION": { "content": "This routine applies stemming algorithms to its parameters, returning the stemmed words as\nappropriate to the selected locale.\n\nYou can import some or all of the class methods.\n\nuse Lingua::Stem qw (stem clearstemcache stemcaching addexceptions deleteexceptions\ngetexceptions setlocale getlocale :all :locale :exceptions :stem :caching);\n\n:all - imports stem addexceptions deleteexceptions getexceptions\nsetlocale getlocale\n:stem - imports stem\n:caching - imports stemcaching clearstemcache\n:locale - imports setlocale getlocale\n:exceptions - imports addexceptions deleteexceptions getexceptions\n\nCurrently supported locales are:\n\nDA - Danish\nDE - German\nEN - English (also EN-US and EN-UK)\nFR - French\nGL - Galician\nIT - Italian\nNO - Norwegian\nPT - Portuguese\nRU - Russian (also RU-RU and RU-RU.KOI8-R)\nSV - Swedish\n\nIf you have the memory and lots of stemming to do, I strongly suggest using cache level 2 and\nprocessing lists in 'big chunks' (long lists) for best performance.\n", "subsections": [ { "name": "Comparision with Lingua::Stem::Snowball", "content": "It functions fairly similarly to the Lingua::Stem::Snowball suite of stemmers, with the most\nsignificant differences being\n\n1) Lingua::Stem is a 'pure perl' (no compiled XS code is needed) suite. Lingua::Stem::Snowball\nis XS based (must be compiled).\n\n2) Lingua::Stem works with Perl 5.6 or later Lingua::Stem::Snowball works with Perl 5.8 or later\n\n3) Lingua::Stem has an 'exceptions' system allowing you to override stemming on a 'case by case'\nbasis. Lingua::Stem::Snowball does not have an 'exceptions' system.\n\n4) A somewhat different set of supported languages:\n\n+---------------------------------------------------------------+\n| Language | ISO code | Lingua::Stem | Lingua::Stem::Snowball |\n|---------------------------------------------------------------|\n| Danish | da | yes | yes |\n| Dutch | nl | no | yes |\n| English | en | yes | yes |\n| Finnish | fi | no | yes |\n| French | fr | yes | yes |\n| Galacian | gl | yes | no |\n| German | de | yes | yes |\n| Italian | it | yes | yes |\n| Norwegian | no | yes | yes |\n| Portuguese | pt | yes | yes |\n| Russian | ru | yes | yes |\n| Spanish | es | no | yes |\n| Swedish | sv | yes | yes |\n+---------------------------------------------------------------+\n\n5) Lingua::Stem is faster for 'stem' (circa 30% faster than Lingua::Stem::Snowball)\n\n6) Lingua::Stem::Snowball is faster for 'steminplace' (circa 30% faster than Lingua::Stem)\n\n7) Lingua::Stem::Snowball is more consistent with regard to character set issues.\n\n8) Lingua::Stem::Snowball is under active development. Lingua::Stem is currently fairly static.\n\nSome benchmarks using Lingua::Stem 0.82 and Lingua::Stem::Snowball 0.94 gives an idea of how\nvarious options impact performance. The dataset was The Works of Edgar Allen Poe, volumes 1-5\nfrom the Gutenberg Project processed 10 times in a row as single batch of words (processing a\nlong text one word at a time is very inefficient and drops the performance of Lingua::Stem by\nabout 90%: So \"Don't Do That\" ;) )\n\nThe benchmarks were run on a 3.06 Ghz P4 with HT on Fedora Core 5 Linux using Perl 5.8.8.\n\n+------------------------------------------------------------------------+\n| source: collectedworkspoe.txt | words: 454691 | unique words: 22802 |\n|------------------------------------------------------------------------|\n| module | config | avg secs | words/sec |\n|------------------------------------------------------------------------|\n| Lingua::Stem 0.82 | no cache | 1.922 | 236560 |\n| Lingua::Stem 0.82 | cache level 2 | 1.235 | 368292 |\n| Lingua::Stem 0.82 | cachelv2, sip | 0.798 | 569494 |\n| Lingua::Stem::Snowball 0.94 | stem | 1.622 | 280276 |\n| Lingua::Stem::Snowball 0.94 | steminplace | 0.627 | 725129 |\n+------------------------------------------------------------------------+\n\nThe script for the benchmark is included in the examples/ directory of this distribution as\nbenchmarkstemmers.plx.\n" } ] }, "CHANGES": { "content": "2.30 2020.06.20 - Version renumber for module consistency\n\n0.84 2010.04.29 - Documentation fixes to the En stemmer and removal\nof the accidentally included lib/Lingua/test.pl file\nThanks goes to Aaron Naiman for bringing the\ndocumentation error to my attention and to\nAlexandr Ciornii and 'kmx' for the pointing out\nthe problem with the test.pl file.\n\n0.83 2007.06.23 - Disabled Italian locale build tests due to\nchanges in Lingua::Stem::It breaking the tests.\n\n0.82 2006.07.23 - Added 'steminplace' to base package.\nTweaks to documentation and build tests.\n\n0.81 2004.07.26 - Minor documentation tweak. No functional change.\n\n0.80 2004.07.25 - Added 'RU', 'RURU', 'RURU.KOI-8' locale.\nAdded support for Lingua::Stem::Ru to\nMakefile.PL and autoloader.\n\nAdded documentation stressing use of caching\nand batches for performance. Added support\nfor '' as a seperator in the locale strings.\nAdded example benchmark script. Expanded copyright\ncredits.\n\n0.70 2004.04.26 - Added FR locale and documentation fixes\nto Lingua::Stem::Gl\n\n0.61 2003.09.28 - Documentation fixes. No functional changes.\n\n0.60 2003.04.05 - Added more locales by wrappering various stemming\nimplementations. Documented currently supported\nlist of locales.\n\n0.50 2000.09.14 - Fixed major implementation error. Starting with\nversion 0.30 I forgot to include rulesets 2,3 and 4\nfor Porter's algorithm. The resulting stemming results\nwere very poor. Thanks go to \nfor bringing the problem to my attention.\n\nUnfortunately, the fix inherently generates *different*\nstemming results than 0.30 and 0.40 did. If you\nneed identically broken output - use locale 'en-broken'.\n\n0.40 2000.08.25 - Added stem caching support as an option. This\ncan provide a large speedup to the operation\nof the stemmer. Caching is default turned off\nto maximize compatibility with previous versions.\n\n0.30 1999.06.24 - Replaced core of 'En' stemmers with code from\nJim Richardson \nAliased 'en-us' and 'en-uk' to 'en'\nFixed 'SYNOPSIS' to correct return value\ntype for stemmed words (SYNOPIS error spotted\nby )\n\n0.20 1999.06.15 - Changed to '.pm' module, moved into Lingua:: namespace,\nadded OO interface, optionalized the export of routines\ninto the caller's namespace, added named parameter\ninitialization, stemming exceptions, autoloaded\nlocale support and isolated case flattening to\nlocalized stemmers prevent i18n problems later.\n\nInput and output text are assumed to be in UTF8\nencoding (no operational impact right now, but\nwill be important when extending the module to\nnon-English).\n", "subsections": [] }, "METHODS": { "content": "", "subsections": [ { "name": "new", "content": "Returns a new instance of a Lingua::Stem object and, optionally, selection of the locale to\nbe used for stemming.\n\nExamples:\n\n# By default the locale is en\n$usstemmer = Lingua::Stem->new;\n\n# Turn on the cache\n$usstemmer->stemcaching({ -level => 2 });\n\n# Overriding the default for a specific instance\n$ukstemmer = Lingua::Stem->new({ -locale => 'en-uk' });\n\n# Overriding the default for a specific instance and changing the default\n$ukstemmer = Lingua::Stem->new({ -defaultlocale => 'en-uk' });\n" }, { "name": "set_locale", "content": "Sets the locale to one of the recognized locales. locale identifiers are converted to\nlowercase.\n\nCalled as a class method, it changes the default locale for all subseqently generated object\ninstances.\n\nCalled as an instance method, it only changes the locale for that particular instance.\n\n'croaks' if passed an unknown locale.\n\nExamples:\n\n# Change default locale\nLingua::Stem::setlocale('en-uk'); # UK's spellings\n\n# Change instance locale\n$self->setlocale('en-us'); # US's spellings\n\ngetlocale;\nCalled as a class method, returns the current default locale.\n\nExample:\n\n$defaultlocale = Lingua::Stem::getlocale;\n\nCalled as an instance method, returns the locale for the instance\n\n$instancelocale = $stemmer->getlocale;\n" }, { "name": "add_exceptions", "content": "Exceptions allow overriding the stemming algorithm on a case by case basis. It is done on an\nexact match and substitution basis: If a passed word is identical to the exception it will\nbe replaced by the specified value. No case adjustments are performed.\n\nCalled as a class method, adds exceptions to the default exceptions list used for\nsubsequently instantations of Lingua::Stem objects.\n\nExample:\n\n# adding default exceptions\nLingua::Stem::addexceptions({ 'emily' => 'emily',\n'driven' => 'driven',\n});\n\nCalled as an instance method, adds exceptions only to the specific instance.\n\n# adding instance exceptions\n$stemmer->addexceptions({ 'steely' => 'steely' });\n\nThe exceptions shortcut the normal stemming - if an exception matches no further stemming is\nperformed after the substitution.\n\nAdding an exception with the same key value as an already defined exception replaces the\npre-existing exception with the new value.\n" }, { "name": "delete_exceptions", "content": "The mirror of addexceptions, this allows the removal of exceptions from either the\ndefaults for the class or from the instance.\n\n# Deletion of exceptions from class default exceptions\nLingua::Stem::deleteexceptions('aragorn','frodo','samwise');\n\n# Deletion of exceptions from instance\n$stemmer->deleteexceptions('smaug','sauron','gollum');\n\n# Deletion of all class default exceptions\ndeleteexceptions;\n\n# Deletion of all exceptions from instance\n$stemmer->deleteexceptions;\n\ngetexceptions;\nAs a class method with no parameters it returns all the default exceptions as an anonymous\nhash of 'exception' => 'replace with' pairs.\n\nExample:\n\n# Returns all class default exceptions\n$exceptions = Lingua::Stem::getexceptions;\n\nAs a class method with parameters, it returns the default exceptions listed in the\nparameters as an anonymous hash of 'exception' => 'replace with' pairs. If a parameter\nspecifies an undefined 'exception', the value is set to undef.\n\n# Returns class default exceptions for 'emily' and 'george'\n$exceptions = Lingua::Stem::getexceptions('emily','george');\n\nAs an instance method, with no parameters it returns the currently active exceptions for the\ninstance.\n\n# Returns all instance exceptions\n$exceptions = $stemmer->getexceptions;\n\nAs an instance method with parameters, it returns the instance exceptions listed in the\nparameters as an anonymous hash of 'exception' => 'replace with' pairs. If a parameter\nspecifies an undefined 'exception', the value is set to undef.\n\n# Returns instance exceptions for 'lisa' and 'bart'\n$exceptions = $stemmer->getexceptions('lisa','bart');\n" }, { "name": "stem", "content": "Called as a class method, it applies the default settings and stems the list of passed\nwords, returning an anonymous array with the stemmed words in the same order as the passed\nlist of words.\n\nExample:\n\n# Default settings applied\nmy $anonarrayofstemmedwords = Lingua::Stem::stem(@words);\n\nCalled as an instance method, it applies the instance's settings and stems the list of\npassed words, returning an anonymous array with the stemmed words in the same order as the\npassed list of words.\n\n# Instance's settings applied\nmy $stemmedwords = $stemmer->stem(@words);\n\nThe stemmer performs best when handed long lists of words rather than one word at a time.\nThe cache also provides a huge speed up if you are processing lots of text.\n" }, { "name": "stem_in_place", "content": "Stems the passed list of words 'in place'. It returns a reference to the modified list. This\nis about 60% faster than the 'stem' method but modifies the original list. This currently\nonly works for the English locales.\n\nExample:\n\nmy @words = ( 'a', 'list', 'of', 'words' );\nmy $stemmedlistofwords = steminplace(@words);\n\n# '$stemmedlistofwords' refers to the @words list\n# after 'steminplace' has executed\n\nDO NOT use this method of stemming if you need to keep the original list of words. Its\nperformance gain derives entirely from the fact it does not make a copy the original list\nbut instead overwrites the original list.\n\nIf you try something like\n\nmy @wordsforstemming = @words;\nmy $stemmedlistofwords = steminplace(@wordsforstemming);\n\nthinking you will get a speed boost while keeping the original list, you won't: You wipe out\nthe speed gain completely with your copying of the original list. You should just use the\n'stem' method instead on the original list of words if you need to keep the original list.\n\nclearstemcache;\nClears the stemming cache for the current locale. Can be called as either a class method or\nan instance method.\n\n$stemmer->clearstemcache;\n\nclearstemcache;\n\nstemcaching ({ -level => 0|1|2 });\nSets stemming cache level for the current locale. Can be called as either a class method or\nan instance method.\n\n$stemmer->stemcaching({ -level => 1 });\n\nstemcaching({ -level => 1 });\n\nFor the sake of maximum compatibility with previous versions, stem caching is set to '-level\n=> 0' by default.\n\n'-level' definitions\n\n'0' means 'no caching'. This is the default level.\n\n'1' means 'cache per run'. This caches stemming results during each\ncall to 'stem'.\n\n'2' means 'cache indefinitely'. This caches stemming results until\neither the process exits or the 'clearstemcache' method is called.\n\nstem caching is global to the locale. If you turn on stem caching for one instance of a\nlocale stemmer, all instances using the same locale will have it turned on as well.\n\nI STRONGLY suggest turning caching on if you have enough memory and are processing a lot of\ndata.\n" } ] }, "VERSION": { "content": "2.30 2020.06.20\n", "subsections": [] }, "NOTES": { "content": "It started with the 'Text::Stem' module which has been adapted into a more general framework and\nmoved into the more language oriented 'Lingua' namespace and re-organized to support a OOP\ninterface as well as switch core 'En' locale stemmers.\n\nVersion 0.40 added a cache for stemmed words. This can provide up to a several fold performance\nimprovement.\n\nOrganization is such that extending this module to any number of languages should be direct and\nsimple.\n\nCase flattening is a function of the language, so the 'exceptions' methods have to be used\nappropriately to the language. For 'En' family stemming, use lower case words, only, for\nexceptions.\n", "subsections": [] }, "AUTHORS": { "content": "Jerilyn Franz \nJim Richardson \n", "subsections": [] }, "CREDITS": { "content": "Jim Richardson \nUlrich Pfeifer \nAldo Calpini \nxern \nAsk Solem Hoel \nDennis Haney \nSébastien Darribere-Pleyt \nAleksandr Guidrevitch \n", "subsections": [] }, "SEE ALSO": { "content": "Lingua::Stem::En Lingua::Stem::En Lingua::Stem::Da\nLingua::Stem::De Lingua::Stem::Gl Lingua::Stem::No\nLingua::Stem::Pt Lingua::Stem::Sv Lingua::Stem::It\nLingua::Stem::Fr Lingua::Stem::Ru Text::German\nLingua::PT::Stemmer Lingua::GL::Stemmer Lingua::Stem::Snowball::No\nLingua::Stem::Snowball::Se Lingua::Stem::Snowball::Da Lingua::Stem::Snowball::Sv\nLingua::Stemmer::GL Lingua::Stem::Snowball\n\nhttp://snowball.tartarus.org\n", "subsections": [] }, "COPYRIGHT": { "content": "Copyright 1999-2004\n\nFreerun Technologies, Inc (Freerun), Jim Richardson, University of Sydney\n and Jerilyn Franz . All rights reserved.\n\nText::German was written and is copyrighted by Ulrich Pfeifer.\n\nLingua::Stem::Snowball::Da was written and is copyrighted by Dennis Haney and Ask Solem Hoel.\n\nLingua::Stem::It was written and is copyrighted by Aldo Calpini.\n\nLingua::Stem::Snowball::No, Lingua::Stem::Snowball::Se, Lingua::Stem::Snowball::Sv were written\nand are copyrighted by Ask Solem Hoel.\n\nLingua::Stemmer::GL and Lingua::PT::Stemmer were written and are copyrighted by Xern.\n\nLingua::Stem::Fr was written and is copyrighted by Aldo Calpini and Sébastien Darribere-Pley.\n\nLingua::Stem::Ru was written and is copyrighted by Aleksandr Guidrevitch.\n\nThis software may be freely copied and distributed under the same terms and conditions as Perl.\n", "subsections": [] }, "BUGS": { "content": "None known.\n", "subsections": [] }, "TODO": { "content": "Add more languages. Extend regression tests. Add support for the Lingua::Stem::Snowball family\nof stemmers as an alternative core stemming engine. Extend 'steminplace' functionality to\nnon-English stemmers.\n", "subsections": [] } } } }