Generative AI -Implementing a Transformer for Text generation

Application: Implementing a Transformer for Text Generation

Step 1: Setup and Import Necessary Libraries

​​To begin, we need to download and add the required files. We'll use PyTorch and the Hugging Face Transformers library to make our code work.

# Install necessary libraries
!pip install torch transformers

# Import libraries
import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

Pip is used to set up the torch and transformers.

To do tensor operations, we bring in torch.

We use a GPT-2 model that has already been trained to create text by importing GPT2LMHeadModel and GPT2Tokenizer from the Hugging Face Transformers package.

Step 2: Load Pre-trained Model and Tokenizer

We'll load a GPT-2 model that has already been trained and the tokenizer that goes with it.

# Load pre-trained model and tokenizer
model_name = 'gpt2'
model = GPT2LMHeadModel.from_pretrained(model_name)
tokenizer = GPT2Tokenizer.from_pretrained(model_name)

We choose the pre-trained model we want to use by setting model_name to "gpt2."

We use GPT2LMHeadModel.from_pretrained to load the model that has already been trained.

We use GPT2Tokenizer.from_pretrained to load the right tokenizer.

Step 3: Prepare the Input Text

We have to get the text input ready that we want the model to use to make text.

# Define the input text
input_text = "Hey there!"
input_ids = tokenizer.encode(input_text, return_tensors='pt')

We say that "Once upon a time" is a starting line for input_text.

The tokenizer takes the input text and turns it into input IDs. This turns the text into a tensor format that the model can understand.

Step 4: Generate Text

Based on the text that is given to it, we use the model to make new text.

# Generate text
output = model.generate(input_ids, max_length=50, num_return_sequences=1)
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)

To make writing, we call the model's create method.

The decoded text that was entered is input_ids.

The max_length parameter tells the generator how long the text can be.

num_return_sequences tells the program how many sequences to make.

The tokenizer turns the created data back into text that can be read.

We print the words that we made.

Full Code

# Install necessary libraries
!pip install torch transformers

# Import libraries
import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

# Load pre-trained model and tokenizer
model_name = 'gpt2'
model = GPT2LMHeadModel.from_pretrained(model_name)
tokenizer = GPT2Tokenizer.from_pretrained(model_name)

# Define the input text
input_text = "Hey there!"
input_ids = tokenizer.encode(input_text, return_tensors='pt')

# Generate text
output = model.generate(input_ids, max_length=50, num_return_sequences=1)
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)