Understanding Tokenization, Stemming, and Lemmatization in NLP | by Ravjot Singh | Jun, 2024

Natural Language Processing (NLP) involves various techniques to handle and analyze human language data. In this blog, we will explore three essential techniques: tokenization, stemming, and lemmatization. These techniques are foundational for many NLP applications, such as text preprocessing, sentiment analysis, and machine translation. Let’s delve into each technique, understand its purpose, pros and cons, and see how they can be implemented using Python’s NLTK library.

What is Tokenization?

Tokenization is the process of splitting a text into individual units, called tokens. These tokens can be words, sentences, or subwords. Tokenization helps break down complex text into manageable pieces for further processing and analysis.

Why is Tokenization Used?

Tokenization is the first step in text preprocessing. It transforms raw text into a format that can be analyzed. This process is essential for tasks such as text mining, information retrieval, and text classification.

Pros and Cons of Tokenization

Pros:

• Simplifies text processing by breaking text into smaller units.
• Facilitates further text analysis and NLP tasks.

Cons:

• Can be complex for languages without clear word boundaries.
• May not handle special characters and punctuation well.

Code Implementation

Here is an example of tokenization using the NLTK library:

# Install NLTK library
!pip install nltk

Explanation:

• !pip install nltk: This command installs the NLTK library, which is a powerful toolkit for NLP in Python.

# Sample text
tweet = "Sometimes to understand a word's meaning you need more than a definition. you need to see the word used in a sentence."

Explanation:

• tweet: This is a sample text we will use for tokenization. It contains multiple sentences and words.

# Importing required modules
import nltk
nltk.download('punkt')

Explanation:

• import nltk: This imports the NLTK library.
• nltk.download('punkt'): This downloads the ‘punkt’ tokenizer models, which are necessary for tokenization.

from nltk.tokenize import word_tokenize, sent_tokenize

Explanation:

• from nltk.tokenize import word_tokenize, sent_tokenize: This imports the word_tokenize and sent_tokenize functions from the NLTK library for word and sentence tokenization, respectively.

# Word Tokenization
text = "Hello! how are you?"
word_tok = word_tokenize(text)
print(word_tok)

Explanation:

• text: This is a simple sentence we will tokenize into words.
• word_tok = word_tokenize(text): This tokenizes the text into individual words.
• print(word_tok): This prints the list of word tokens. Output: ['Hello', '!', 'how', 'are', 'you', '?']

# Sentence Tokenization
sent_tok = sent_tokenize(tweet)
print(sent_tok)

Explanation:

• sent_tok = sent_tokenize(tweet): This tokenizes the tweet into individual sentences.
• print(sent_tok): This prints the list of sentence tokens. Output: ['Sometimes to understand a word's meaning you need more than a definition.', 'you need to see the word used in a sentence.']

What is Stemming?

Stemming is the process of reducing a word to its base or root form. It involves removing suffixes and prefixes from words to derive the stem.

Why is Stemming Used?

Stemming helps in normalizing words to their root form, which is useful in text mining and search engines. It reduces inflectional forms and derivationally related forms of a word to a common base form.

Pros and Cons of Stemming

Pros:

• Reduces the complexity of text by normalizing words.
• Improves the performance of search engines and information retrieval systems.

Cons:

• Can lead to incorrect base forms (e.g., ‘running’ to ‘run’, but ‘flying’ to ‘fli’).
• Different stemming algorithms may produce different results.

Code Implementation

Let’s see how to perform stemming using different algorithms:

Porter Stemmer:

from nltk.stem import PorterStemmer
stemming = PorterStemmer()
word = 'danced'
print(stemming.stem(word))

Explanation:

• from nltk.stem import PorterStemmer: This imports the PorterStemmer class from NLTK.
• stemming = PorterStemmer(): This creates an instance of the PorterStemmer.
• word = 'danced': This is the word we want to stem.
• print(stemming.stem(word)): This prints the stemmed form of the word ‘danced’. Output: danc

word = 'replacement'
print(stemming.stem(word))

Explanation:

• word = 'replacement': This is another word we want to stem.
• print(stemming.stem(word)): This prints the stemmed form of the word ‘replacement’. Output: replac

word = 'happiness'
print(stemming.stem(word))

Explanation:

• word = 'happiness': This is another word we want to stem.
• print(stemming.stem(word)): This prints the stemmed form of the word ‘happiness’. Output: happi

Lancaster Stemmer:

from nltk.stem import LancasterStemmer
stemming1 = LancasterStemmer()
word = 'happily'
print(stemming1.stem(word))

Explanation:

• from nltk.stem import LancasterStemmer: This imports the LancasterStemmer class from NLTK.
• stemming1 = LancasterStemmer(): This creates an instance of the LancasterStemmer.
• word = 'happily': This is the word we want to stem.
• print(stemming1.stem(word)): This prints the stemmed form of the word ‘happily’. Output: happy

Regular Expression Stemmer:

from nltk.stem import RegexpStemmer
stemming2 = RegexpStemmer('ing$|s$|e$|able$|ness$', min=3)
word = 'raining'
print(stemming2.stem(word))

Explanation:

• from nltk.stem import RegexpStemmer: This imports the RegexpStemmer class from NLTK.
• stemming2 = RegexpStemmer('ing$|s$|e$|able$|ness$', min=3): This creates an instance of the RegexpStemmer with a regular expression pattern to match suffixes and a minimum stem length of 3 characters.
• word = 'raining': This is the word we want to stem.
• print(stemming2.stem(word)): This prints the stemmed form of the word ‘raining’. Output: rain

word = 'flying'
print(stemming2.stem(word))

Explanation:

• word = 'flying': This is another word we want to stem.
• print(stemming2.stem(word)): This prints the stemmed form of the word ‘flying’. Output: fly

word = 'happiness'
print(stemming2.stem(word))

Explanation:

• word = 'happiness': This is another word we want to stem.
• print(stemming2.stem(word)): This prints the stemmed form of the word ‘happiness’. Output: happy

Snowball Stemmer:

nltk.download("snowball_data")
from nltk.stem import SnowballStemmer
stemming3 = SnowballStemmer("english")
word = 'happiness'
print(stemming3.stem(word))

Explanation:

• nltk.download("snowball_data"): This downloads the Snowball stemmer data.
• from nltk.stem import SnowballStemmer: This imports the SnowballStemmer class from NLTK.
• stemming3 = SnowballStemmer("english"): This creates an instance of the SnowballStemmer for the English language.
• word = 'happiness': This is the word we want to stem.
• print(stemming3.stem(word)): This prints the stemmed form of the word ‘happiness’. Output: happy

stemming3 = SnowballStemmer("arabic")
word = 'تحلق'
print(stemming3.stem(word))

Explanation:

• stemming3 = SnowballStemmer("arabic"): This creates an instance of the SnowballStemmer for the Arabic language.
• word = 'تحلق': This is an Arabic word we want to stem.
• print(stemming3.stem(word)): This prints the stemmed form of the word ‘تحلق’. Output: تحل

What is Lemmatization?

Lemmatization is the process of reducing a word to its base or dictionary form, known as a lemma. Unlike stemming, lemmatization considers the context and converts the word to its meaningful base form.

Why is Lemmatization Used?

Lemmatization provides more accurate base forms compared to stemming. It is widely used in text analysis, chatbots, and NLP applications where understanding the context of words is essential.

Pros and Cons of Lemmatization

Pros:

• Produces more accurate base forms by considering the context.
• Useful for tasks requiring semantic understanding.

Cons:

• Requires more computational resources compared to stemming.
• Dependent on language-specific dictionaries.

Code Implementation

Here is how to perform lemmatization using the NLTK library:

# Download necessary data
nltk.download('wordnet')

Explanation:

• nltk.download('wordnet'): This command downloads the WordNet corpus, which is used by the WordNetLemmatizer for finding the lemmas of words.

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()

Explanation:

• from nltk.stem import WordNetLemmatizer: This imports the WordNetLemmatizer class from NLTK.
• lemmatizer = WordNetLemmatizer(): This creates an instance of the WordNetLemmatizer.

print(lemmatizer.lemmatize('going', pos='v'))

Explanation:

• lemmatizer.lemmatize('going', pos='v'): This lemmatizes the word ‘going’ with the part of speech (POS) tag ‘v’ (verb). Output: go

# Lemmatizing a list of words with their respective POS tags
words = [("eating", 'v'), ("playing", 'v')]
for word, pos in words:
print(lemmatizer.lemmatize(word, pos=pos))

Explanation:

• words = [("eating", 'v'), ("playing", 'v')]: This is a list of tuples where each tuple contains a word and its corresponding POS tag.
• for word, pos in words: This iterates through each tuple in the list.
• print(lemmatizer.lemmatize(word, pos=pos)): This prints the lemmatized form of each word based on its POS tag. Outputs: eat, play

• Tokenization is used in text preprocessing, sentiment analysis, and language modeling.
• Stemming is useful for search engines, information retrieval, and text mining.
• Lemmatization is essential for chatbots, text classification, and semantic analysis.

Tokenization, stemming, and lemmatization are crucial techniques in NLP. They transform the raw text into a format suitable for analysis and help in understanding the structure and meaning of the text. By applying these techniques, we can enhance the performance of various NLP applications.

Feel free to experiment with the provided code snippets and explore these techniques further. Happy coding!