Python - Natural Language Processing (NLP) with NLTK and SpaCy

Natural Language Processing (NLP) with NLTK and SpaCy in Python 

Introduction to NLP

Natural Language Processing (NLP) enables computers to understand, interpret, and generate human language. Python has become a dominant language in NLP due to libraries like NLTK and SpaCy, which provide robust tools for text processing, statistical analysis, and linguistic annotation.

In this document, we'll cover the theory and usage of both NLTK and SpaCy, walk through extensive code examples, and highlight best practices. By the end, you’ll be equipped with practical NLP skills using two of the most popular Python libraries.

Overview of NLTK and SpaCy

NLTK (Natural Language Toolkit)

NLTK is one of the oldest and most widely used libraries in NLP. It provides a suite of tools for tokenization, tagging, parsing, semantic reasoning, and more. Its educational and experimental focus makes it ideal for learning and research.

• Extensive collection of corpora and lexical resources (e.g., WordNet)
• Flexible and modular architecture
• Great for learning, prototyping, and academic research

SpaCy

SpaCy is a modern, high-performance NLP library designed for production. It excels in speed and usability, offering efficient pre-trained pipelines for tokenization, part-of-speech tagging, named entity recognition, dependency parsing, and word vectors.

• Blazing-fast Cython backend
• Effortless integration with machine learning workflows
• Extensible models and neural-network-powered components

Installation and Setup

Installing NLTK

pip install nltk

After installation, download required data packages:

import nltk
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('wordnet')
nltk.download('stopwords')

Installing SpaCy

pip install spacy
python -m spacy download en_core_web_sm

Text Preprocessing

Tokenization

import nltk

text = "Hello world! This is an example text for NLP."
tokens = nltk.word_tokenize(text)
print(tokens)

With SpaCy:

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp(text)
tokens = [token.text for token in doc]
print(tokens)

Sentence Segmentation

sentences = nltk.sent_tokenize(text)
print(sentences)

Using SpaCy:

sentences = list(doc.sents)
print([sent.text for sent in sentences])

Lowercasing and Stopword Removal

from nltk.corpus import stopwords

stop_words = set(stopwords.words('english'))
filtered = [w.lower() for w in tokens if w.lower() not in stop_words]
print(filtered)

SpaCy version:

filtered = [token.text.lower() for token in doc if not token.is_stop]
print(filtered)

Normalization with Lemmatization and Stemming

from nltk.stem import PorterStemmer, WordNetLemmatizer

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

stems = [stemmer.stem(w) for w in filtered]
lemmas = [lemmatizer.lemmatize(w) for w in filtered]

print(stems)
print(lemmas)

Using SpaCy’s lemmatizer:

lemmas = [token.lemma_ for token in doc if not token.is_stop]
print(lemmas)

Part‑of‑Speech Tagging

nltk_pos = nltk.pos_tag(tokens)
print(nltk_pos)

With SpaCy:

pos_tags = [(token.text, token.pos_, token.tag_) for token in doc]
print(pos_tags)

Named Entity Recognition (NER)

import nltk
nltk.download('maxent_ne_chunker')
nltk.download('words')

tree = nltk.ne_chunk(nltk_pos)
print(tree)

With SpaCy’s pipeline:

entities = [(ent.text, ent.label_) for ent in doc.ents]
print(entities)

Dependency Parsing

SpaCy supports accurate dependency parsing:

for token in doc:
    print(token.text, token.dep_, token.head.text)

This is not supported in classic NLTK.

Chunking and Shallow Parsing (NLTK)

grammar = "NP: {
?*}"
cp = nltk.RegexpParser(grammar)
result = cp.parse(nltk_pos)
print(result)

WordNet Lexical Database (NLTK)

from nltk.corpus import wordnet as wn

syn = wn.synsets('dog')
print(syn[0].definition())

lemmas = wn.lemmas('dog')
print([l.name() for l in lemmas])

Similarity Measures

syn1 = wn.synsets('car')[0]
syn2 = wn.synsets('automobile')[0]
print(syn1.path_similarity(syn2))

Topic Modeling with Gensim and NLTK

from gensim import corpora, models
from nltk.tokenize import word_tokenize

docs = [
    "Cats are small animals.",
    "Dogs are loyal and friendly.",
    "Cats and dogs are popular pets."
]

texts = [[w.lower() for w in word_tokenize(doc) if w.isalpha()] for doc in docs]
dictionary = corpora.Dictionary(texts)
corpus = [dictionary.doc2bow(text) for text in texts]

lda = models.LdaModel(corpus, id2word=dictionary, num_topics=2)
print(lda.print_topics())

Text Vectorization

Bag‑of‑Words with CountVectorizer (sklearn)

from sklearn.feature_extraction.text import CountVectorizer

cv = CountVectorizer(stop_words='english')
X = cv.fit_transform(docs)
print(cv.get_feature_names_out())
print(X.toarray())

TF‑IDF with TfidfVectorizer

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words='english')
X_tfidf = tfidf.fit_transform(docs)
print(tfidf.get_feature_names_out())
print(X_tfidf.toarray())

Word Embeddings with SpaCy

nlp = spacy.load('en_core_web_md')
doc = nlp("SpaCy provides word embeddings.")

vec = doc[2].vector
print(len(vec))

Similarity with Embeddings

doc1 = nlp("I like cats.")
doc2 = nlp("I like dogs.")
print(doc1.similarity(doc2))

Text Classification Pipeline

from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

docs = [
    "I love programming.",
    "This movie is terrible.",
    "The food was amazing!",
    "I hate this weather."
]
labels = [1, 0, 1, 0]

X_train, X_test, y_train, y_test = train_test_split(
    docs, labels, test_size=0.5, random_state=42)

pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(stop_words='english')),
    ('clf', MultinomialNB())
])

pipeline.fit(X_train, y_train)
preds = pipeline.predict(X_test)

print(classification_report(y_test, preds))

Named Entity Classification with SpaCy

text = "Apple is looking at buying U.K. startup for $1 billion"
doc = nlp(text)

for ent in doc.ents:
    print(ent.text, ent.label_)

Custom Named Entity Training (SpaCy)

from spacy.training import Example
from spacy.util import minibatch

nlp = spacy.load("en_core_web_sm")
ner = nlp.get_pipe("ner")
ner.add_label("ANIMAL")

TRAIN_DATA = [
    ("Cats are animals", {"entities": [(0, 4, "ANIMAL")]}),
    ("Dogs are animals too", {"entities": [(0, 4, "ANIMAL")]})
]

optimizer = nlp.begin_training()

for itn in range(20):
    for text, annotations in TRAIN_DATA:
        doc = nlp.make_doc(text)
        example = Example.from_dict(doc, annotations)
        nlp.update([example], sgd=optimizer)

doc = nlp("I have a pet dog")
print([(ent.text, ent.label_) for ent in doc.ents])

Evaluation Metrics for NLP

• Accuracy, useful for classification
• Precision, Recall, and F1‑score
• BLEU Score for translation quality
• ROUGE for summarization overlap

from sklearn.metrics import precision_score, recall_score, f1_score

y_true = [1, 0, 1, 1]
y_pred = [1, 0, 0, 1]

print(precision_score(y_true, y_pred))
print(recall_score(y_true, y_pred))
print(f1_score(y_true, y_pred))

Advanced Topics and Extensions

Sentiment Analysis

Leverage NLTK’s VADER or train your own classifier for sentiment tasks. SpaCy offers extensible pipelines too.

Text Summarization

Use extractive techniques (e.g., Gensim’s summarize) or abstractive methods with deep learning.

Machine Translation

Use SpaCy for preprocessing and integrate with transformer models via Hugging Face.

Topic Modeling

Combine SpaCy for preprocessing and Gensim for LDA topic modeling.

Best Practices in NLP

• Always preprocess text: clean, tokenize, normalize.
• Use pipelines to organize workflows.
• Choose the right tool for the job (SpaCy for speed, NLTK for flexibility).
• Evaluate using suitable metrics and cross‑validation.
• Document and experiment with preprocessing variations.

Summary

This comprehensive guide introduces NLP with both NLTK and SpaCy. We covered tokenization, tagging, parsing, normalization, embeddings, classification, and more advanced topics. NLTK provides a flexible platform for experimentation and learning, while SpaCy focuses on speed and production readiness.

By combining both libraries along with tools like Scikit‑learn and Gensim, you're well poised to build effective NLP pipelines for real‑world text mining, classification, summarization, translation, and information extraction tasks.