Generative AI - Transfer Learning with BERT Model

Transfer Learning with BERT Model

For many natural language processing (NLP) tasks, transfer learning with BERT (Bidirectional Encoder Representations from Transformers) can work very well. We will show you how to use the Hugging Face Transformers library to fine-tune BERT for a text classification job.

Step 1: Setup and Import Necessary Libraries

First, we need to get the packages we need and install them.

# Install necessary libraries
!pip install transformers torch sklearn

# Import libraries
import torch
from transformers import BertTokenizer, BertForSequenceClassification, Trainer, TrainingArguments
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from datasets import load_dataset

We use pip to set up the Hugging Face Transformers library, PyTorch, and scikit-learn.

We bring in the tools we need from scikit-learn, transformers, and torch.

To load the dataset, we use the load_dataset function from the datasets library.

Step 2: Load and Prepare the Dataset

We'll use the IMDb movie reviews information to sort text into two groups.

# Load the dataset
dataset = load_dataset('imdb')

# Split the dataset into training and testing sets
train_dataset, test_dataset = train_test_split(dataset['train'], test_size=0.2, random_state=42)

We use the load_dataset function to get the IMDb dataset.

An 80-20 split was used to divide the information into training and testing sets.

Step 3: Preprocess the Data

For the BERT model to work, we need to tokenize the word data.

# Load the BERT tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Tokenize the data
def tokenize_function(examples):
    return tokenizer(examples['text'], padding='max_length', truncation=True)

train_dataset = train_dataset.map(tokenize_function, batched=True)
test_dataset = test_dataset.map(tokenize_function, batched=True)

# Convert labels to tensor format
train_dataset = train_dataset.map(lambda examples: {'labels': examples['label']}, batched=True)
test_dataset = test_dataset.map(lambda examples: {'labels': examples['label']}, batched=True)

The "bert-base-uncased" model has already been trained, so we load the BERT tokenizer from that.

To tokenize the written input, we create a method called tokenize_function.

On both the training and testing datasets, we use the tokenization method.

For both types of data, we change the names to tensor format.

Step 4: Initialize the BERT Model

We set up the BERT model to classify sequences.

# Load the BERT model
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

We use the pre-trained BERT model to sort sequences into two groups, which is called binary classification.

Step 5: Define Training Arguments and Trainer

We specify the training arguments and start the trainer.

# Define training arguments
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./logs',
    logging_steps=10,
)

# Define the compute metrics function
def compute_metrics(p):
    preds = np.argmax(p.predictions, axis=1)
    precision, recall, f1, _ = precision_recall_fscore_support(p.label_ids, preds, average='binary')
    acc = accuracy_score(p.label_ids, preds)
    return {'accuracy': acc, 'f1': f1, 'precision': precision, 'recall': recall}

# Initialize the Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
    compute_metrics=compute_metrics,
)

We provide the training parameters, including the output directory, number of epochs, batch sizes, warmup steps, weight decay, and logging settings.

We create the compute_metrics function to compute accuracy, precision, recall, and F1 score.

We provide the Trainer with the model, training arguments, training dataset, evaluation dataset, and metrics function.

Step 6: Train and Evaluate the Model

We will use the Trainer to train and assess the model.

# Train the model
trainer.train()

# Evaluate the model
results = trainer.evaluate()
print(results)

We use the Trainer's train approach to fine-tune the BERT model using the training data.

We use the evaluate method to run the model on the test data and publish the results.