Python - The Natural Language Toolkit (NLTK)

The Natural Language Toolkit (NLTK) in Python

Introduction to NLTK

The Natural Language Toolkit (NLTK) is a powerful Python library for working with human language data (text). It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.

Designed initially for teaching and research, NLTK is also used in commercial and industrial NLP applications. It serves as a foundational toolkit for natural language processing and computational linguistics.

Installation and Setup

Installing NLTK

pip install nltk

Downloading NLTK Data

import nltk
nltk.download('all')

Downloading 'all' ensures you have access to all corpora, tokenizers, taggers, and other tools required by the examples and real-world projects.

Text Preprocessing with NLTK

Tokenization

Tokenization is the process of breaking a text into words or sentences.

from nltk.tokenize import word_tokenize, sent_tokenize

text = "NLTK is a leading platform for building Python programs to work with human language data."

word_tokens = word_tokenize(text)
sent_tokens = sent_tokenize(text)

print("Word Tokens:", word_tokens)
print("Sentence Tokens:", sent_tokens)

Stop Word Removal

from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

stop_words = set(stopwords.words('english'))
text = "This is an example showing off stop word filtration."

words = word_tokenize(text)
filtered = [w for w in words if w.lower() not in stop_words]

print("Filtered Words:", filtered)

Stemming

Stemming reduces words to their root form.

from nltk.stem import PorterStemmer

ps = PorterStemmer()
words = ["python", "pythoner", "pythoning", "pythonly"]

for w in words:
    print(w, "->", ps.stem(w))

Lemmatization

Lemmatization considers the context and converts a word to its base form using vocabulary and morphological analysis.

from nltk.stem import WordNetLemmatizer
import nltk
nltk.download('wordnet')

lemmatizer = WordNetLemmatizer()

words = ["running", "flies", "better"]
for w in words:
    print(w, "->", lemmatizer.lemmatize(w, pos='v'))

Part-of-Speech Tagging

import nltk
nltk.download('averaged_perceptron_tagger')

sentence = "The quick brown fox jumps over the lazy dog"
tokens = nltk.word_tokenize(sentence)
tags = nltk.pos_tag(tokens)

print(tags)

Named Entity Recognition (NER)

import nltk
nltk.download('maxent_ne_chunker')
nltk.download('words')

sentence = "Barack Obama was born in Hawaii."
tokens = nltk.word_tokenize(sentence)
tags = nltk.pos_tag(tokens)
tree = nltk.ne_chunk(tags)

print(tree)

Visualizing Named Entities

tree.draw()

Chunking

import nltk

sentence = [("the", "DT"), ("little", "JJ"), ("yellow", "JJ"), ("dog", "NN")]
grammar = "NP: {
?*}"

cp = nltk.RegexpParser(grammar)
result = cp.parse(sentence)

print(result)
result.draw()

Corpora and Lexical Resources

Accessing NLTK Corpora

from nltk.corpus import gutenberg
nltk.download('gutenberg')

print(gutenberg.fileids())
text = gutenberg.raw('austen-emma.txt')
print(text[:1000])

Using WordNet

from nltk.corpus import wordnet

syns = wordnet.synsets("computer")

print("Name:", syns[0].name())
print("Definition:", syns[0].definition())
print("Examples:", syns[0].examples())

Finding Synonyms and Antonyms

synonyms = []
antonyms = []

for syn in wordnet.synsets("good"):
    for lemma in syn.lemmas():
        synonyms.append(lemma.name())
        if lemma.antonyms():
            antonyms.append(lemma.antonyms()[0].name())

print("Synonyms:", set(synonyms))
print("Antonyms:", set(antonyms))

Language Modeling

N-grams

from nltk import ngrams

text = "I love natural language processing".split()
bigrams = list(ngrams(text, 2))

print("Bigrams:", bigrams)

Frequency Distributions

from nltk import FreqDist
from nltk.corpus import brown
nltk.download('brown')

words = brown.words(categories='news')
fdist = FreqDist(words)

print(fdist.most_common(10))
fdist.plot(10)

Text Classification

Building a Simple Classifier

from nltk.classify import NaiveBayesClassifier

def word_features(word):
    return {'last_letter': word[-1]}

labeled_words = [('happiness', 'positive'), ('anger', 'negative'),
                 ('joy', 'positive'), ('fear', 'negative')]

featuresets = [(word_features(n), label) for (n, label) in labeled_words]
train_set = featuresets

classifier = NaiveBayesClassifier.train(train_set)

print(classifier.classify(word_features('excitement')))
print(classifier.show_most_informative_features(5))

Movie Reviews Sentiment Analysis

from nltk.corpus import movie_reviews
import random
nltk.download('movie_reviews')

documents = [(list(movie_reviews.words(fileid)), category)
             for category in movie_reviews.categories()
             for fileid in movie_reviews.fileids(category)]

random.shuffle(documents)

all_words = nltk.FreqDist(w.lower() for w in movie_reviews.words())
word_features = list(all_words)[:2000]

def document_features(document):
    words = set(document)
    features = {}
    for word in word_features:
        features[word] = (word in words)
    return features

featuresets = [(document_features(d), c) for (d, c) in documents]
train_set = featuresets[:1900]
test_set = featuresets[1900:]

classifier = nltk.NaiveBayesClassifier.train(train_set)
print("Accuracy:", nltk.classify.accuracy(classifier, test_set))
classifier.show_most_informative_features(10)

Parsing and Syntax Trees

Using CFG Grammar

import nltk

grammar = nltk.CFG.fromstring("""
S -> NP VP
NP -> Det N
VP -> V NP
Det -> 'the'
N -> 'dog' | 'cat'
V -> 'chased' | 'sat'
""")

parser = nltk.ChartParser(grammar)

sentence = ['the', 'dog', 'chased', 'the', 'cat']
for tree in parser.parse(sentence):
    print(tree)
    tree.draw()

Translating and Working with Multilingual Corpora

from nltk.corpus import udhr
nltk.download('udhr')

languages = ['English-Latin1', 'French_Francais-Latin1', 'German_Deutsch-Latin1']
for lang in languages:
    print(lang, "Sample:\n", udhr.raw(lang)[:100])

Advanced Tasks and Integration

Topic Modeling using NLTK and Gensim

from gensim import corpora, models
import gensim
from nltk.tokenize import word_tokenize

documents = ["Human machine interface for computer applications",
             "A survey of user opinion of computer system response time",
             "The EPS user interface management system",
             "System and human system engineering testing of EPS",
             "Relation of user perceived response time to error measurement"]

texts = [[word for word in word_tokenize(doc.lower()) if word.isalpha()]
         for doc in documents]

dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]

lda_model = gensim.models.ldamodel.LdaModel(corpus, num_topics=2, id2word=dictionary, passes=15)

topics = lda_model.print_topics(num_words=4)
for topic in topics:
    print(topic)

NLTK is a foundational library in Python's NLP ecosystem. It offers a broad collection of tools, corpora, and resources for language processing, making it ideal for education, prototyping, and research. Though libraries like SpaCy have emerged for industrial-scale processing, NLTK still holds significant value due to its vast dataset access, flexibility, and educational utility.

From tokenization and stemming to full-blown classification and parsing, NLTK allows you to build end-to-end NLP pipelines. With continued exploration and integration, it remains a key tool for anyone working with natural language data in Python.