stopwords: the R package
R package providing โone-stop shoppingโ (or should that be โone-shop stoppingโ?) for stopword lists in R, for multiple languages and sources. No longer should text analysis or NLP packages bake in their own stopword lists or functions, since this package can accommodate them all, and is easily extended.
Created by David Muhr, and extended in cooperation with Kenneth Benoit and Kohei Watanabe.
Installation
# from CRAN
install.packages("stopwords")
# Or get the development version from GitHub:
# install.packages("devtools")
devtools::install_github("quanteda/stopwords")Usage
head(stopwords::stopwords("de", source = "snowball"), 20)
## [1] "aber" "alle" "allem" "allen" "aller" "alles" "als"
## [8] "also" "am" "an" "ander" "andere" "anderem" "anderen"
## [15] "anderer" "anderes" "anderm" "andern" "anderr" "anders"
head(stopwords::stopwords("ja", source = "marimo"), 20)
## [1] "็ง" "ๅ" "่ชๅ" "่ช่บซ" "ๆใ
" "็ง้"
## [7] "ใใชใ" "ๅฝผ" "ๅฝผๅฅณ" "ๅฝผใ" "ๅฝผๅฅณใ" "ใใ"
## [13] "ใใ" "ใใ" "ใใใ" "ใใใใฎ" "ใใใ" "ใใใใฎ"
## [19] "ใใใ" "ใใใใฎ"For compatibility with the former quanteda::stopwords():
head(stopwords::stopwords("german"), 20)
## [1] "aber" "alle" "allem" "allen" "aller" "alles" "als"
## [8] "also" "am" "an" "ander" "andere" "anderem" "anderen"
## [15] "anderer" "anderes" "anderm" "andern" "anderr" "anders"Explore sources and languages:
# list all sources
stopwords::stopwords_getsources()
## [1] "snowball" "stopwords-iso" "misc" "smart"
## [5] "marimo" "ancient" "nltk" "perseus"
# list languages for a specific source
stopwords::stopwords_getlanguages("snowball")
## [1] "da" "de" "en" "es" "fi" "fr" "hu" "ir" "it" "nl" "no" "pt" "ro" "ru" "sv"Languages available
The following coverage of languages is currently available, by source. Note that the inclusiveness of the stopword lists will vary by source, and the number of languages covered by a stopword list does not necessarily mean that the source is better than one with more limited coverage. (There may be many reasons to prefer the default โsnowballโ source over the โstopwords-isoโ source, for instance.)
The following languages are currently available:
| Language | Code | snowball | marimo | nltk | stopwords-iso | Other |
|---|---|---|---|---|---|---|
| Afrikaans | af | โ | ||||
| Arabic | ar | โ | โ | โ | misc | |
| Armenian | hy | โ | ||||
| Azerbaijani | az | โ | ||||
| Basque | eu | โ | ||||
| Bengali | bn | โ | ||||
| Breton | br | โ | ||||
| Bulgarian | bg | โ | ||||
| Catalan | ca | โ | misc | |||
| Chinese | zh | โ | โ | misc | ||
| Croatian | hr | โ | ||||
| Czech | cs | โ | ||||
| Danish | da | โ | โ | โ | ||
| Dutch | nl | โ | โ | โ | ||
| English | en | โ | โ | โ | โ | smart |
| Esperanto | eo | โ | ||||
| Estonian | et | โ | ||||
| Finnish | fi | โ | โ | โ | ||
| French | fr | โ | โ | โ | ||
| Galician | gl | โ | ||||
| German | de | โ | โ | โ | โ | |
| Greek | el | โ | misc | |||
| Greek (ancient) | grc | ancient, perseus | ||||
| Gujarati | gu | misc | ||||
| Hausa | ha | โ | ||||
| Hebrew | he | โ | โ | |||
| Hindi | hi | โ | ||||
| Hungarian | hu | โ | โ | โ | ||
| Indonesian | id | โ | โ | |||
| Irish | ga | โ | ||||
| Italian | it | โ | โ | โ | ||
| Japanese | ja | โ | โ | |||
| Kazakh | kk | โ | ||||
| Korean | ko | โ | โ | |||
| Kurdish | ku | โ | ||||
| Latin | la | โ | ancient, perseus | |||
| Lithuanian | lt | โ | ||||
| Latvian | lv | โ | ||||
| Malay | ms | โ | ||||
| Marathi | mr | โ | ||||
| Nepali | mr | โ | ||||
| Norwegian | no | โ | โ | โ | ||
| Persian | fa | โ | ||||
| Polish | pl | โ | ||||
| Portuguese | pt | โ | โ | โ | ||
| Romanian | ro | โ | โ | โ | ||
| Russian | ru | โ | โ | โ | โ | |
| Slovak | sk | โ | ||||
| Slovenian | sl | โ | โ | |||
| Somali | so | โ | ||||
| Southern Sotho | st | โ | ||||
| Spanish | es | โ | โ | โ | ||
| Swahili | sw | โ | ||||
| Swedish | sv | โ | โ | โ | ||
| Thai | th | โ | ||||
| Tagalog | tl | โ | ||||
| Tajik | tg | โ | ||||
| Turkish | tr | โ | โ | |||
| Ukrainian | uk | โ | ||||
| Urdu | ur | โ | ||||
| Vietnamese | vi | โ | ||||
| Yoruba | yo | โ | ||||
| Zulu | zu | โ |
Basic usage
head(stopwords::stopwords("de", source = "snowball"), 20)
## [1] "aber" "alle" "allem" "allen" "aller" "alles" "als"
## [8] "also" "am" "an" "ander" "andere" "anderem" "anderen"
## [15] "anderer" "anderes" "anderm" "andern" "anderr" "anders"
head(stopwords::stopwords("de", source = "stopwords-iso"), 20)
## [1] "a" "ab" "aber" "ach" "acht"
## [6] "achte" "achten" "achter" "achtes" "ag"
## [11] "alle" "allein" "allem" "allen" "aller"
## [16] "allerdings" "alles" "allgemeinen" "als" "also"For compatibility with the former quanteda::stopwords():
head(stopwords::stopwords("german"), 20)
## [1] "aber" "alle" "allem" "allen" "aller" "alles" "als"
## [8] "also" "am" "an" "ander" "andere" "anderem" "anderen"
## [15] "anderer" "anderes" "anderm" "andern" "anderr" "anders"Explore sources and languages:
# list all sources
stopwords::stopwords_getsources()
## [1] "snowball" "stopwords-iso" "misc" "smart"
## [5] "marimo" "ancient" "nltk" "perseus"
# list languages for a specific source
stopwords::stopwords_getlanguages("snowball")
## [1] "da" "de" "en" "es" "fi" "fr" "hu" "ir" "it" "nl" "no" "pt" "ro" "ru" "sv"Modifying stopword lists
It is now possible to edit your own stopword lists, using the interactive editor, with functions from the quanteda package (>= v2.02). For instance to edit the English stopword list for the Snowball source:
# edit the English stopwords
my_stopwords <- quanteda::char_edit(stopwords("en", source = "snowball"))To edit stopwords whose underlying structure is a list, such as the
โmarimoโ source, we can use the list_edit() function:
# edit the English stopwords
my_stopwordlist <- quanteda::list_edit(stopwords("en", source = "marimo", simplify = FALSE))Finally, itโs possible to remove stopwords using pattern matching. The default is the easy-to-use โglobโ style matching, which is equivalent to fixed matching when no wildcard characters are used. So to remove personal pronouns from the English Snowball word list, for instance, this would work:
library("quanteda", warn.conflicts = FALSE)
## Package version: 3.2.0
## Unicode version: 13.0
## ICU version: 69.1
## Parallel computing: 8 of 8 threads used.
## See https://quanteda.io for tutorials and examples.
posspronouns <- stopwords::data_stopwords_marimo$en$pronoun$possessive
posspronouns
## [1] "my" "our" "your" "his" "her" "its" "their"
stopwords("en", source = "snowball") %>%
head(n = 10)
## [1] "i" "me" "my" "myself" "we" "our"
## [7] "ours" "ourselves" "you" "your"See the difference when we remove them โ โmyโ, โoursโ, and โyourโ are gone:
stopwords("en", source = "snowball") %>%
head(n = 10) %>%
char_remove(pattern = posspronouns)
## [1] "i" "me" "myself" "we" "ours" "ourselves"
## [7] "you"There is no char_add(), since itโs just as easy to use c() for this,
but there is a char_keep() for positive selection rather than removal.
Adding stopwords to your own package
In v2.2, weโve removed the function use_stopwords() because the
dependency on usethis added too many downstream package
dependencies, and stopwords is meant to be a lightweight package.
However it is very easy to add a re-export for stopwords() to your
package by adding this file as stopwords.R:
#' Stopwords
#'
#' @description
#' Return a character vector of stopwords.
#' See \code{stopwords::\link[stopwords:stopwords]{stopwords()}} for details.
#' @usage stopwords(language = "en", source = "snowball")
#' @name stopwords
#' @importFrom stopwords stopwords
#' @export
NULLand add stopwords to the list of Imports: in your DESCRIPTION
file.
Contributing
Additional sources can be defined and contributed by adding new data objects, as follows:
-
Data object. Create a named list of characters, in UTF-8 format, consisting of the stopwords for each language. The ISO-639-1 language code will form the name of the list element, and the values of each element will be the character vector of stopwords for literal matches. The data object should follow the package naming convention, and be called
data_stopwords_newsource, wherenewsourceis replaced by the name of the new source. -
Documentation. The new source should be clearly documented, especially the source from which was taken.
License
This package as well as the source repositories are licensed under MIT.