LLM stands for Large Language Model. It refers to a highly advanced computer that has been trained to understand and generate human-like text. These models are trained on vast amounts of data to closely mimic human language.
NLP, or Natural Language Processing, is the technology that enables LLMs to understand human language. It allows computers to interpret and work with human language.
Hugging Face is a company that provides tools (APIs and libraries) to make working with AI and machine learning easier. These tools are available for free and are open-source.
In AI, transformers are tools available through Hugging Face. They are used for various language tasks like translation and text generation.
A pipeline is like a pathway between your computer and the Hugging Face tools. It processes raw data using the brains of the LLM and returns the output you need.
OpenAI provides a more robust and exclusive experience compared to Hugging Face. It offers advanced capabilities, though it is not free. Unlike Hugging Face, OpenAI often requires cloud-based resources to perform tasks.
AI prefers to process information in smaller, manageable pieces. This is where tokenization comes in, breaking down text into smaller units for easier processing. All LLMs require tokenization to function. Here are two methods for text segmentation:
NLTK (Natural Language Toolkit) breaks text into smaller sentences.
import nltk
nltk.download('punkt')
from nltk.tokenize import sent_tokenize
text = """Little Robo was a small, kind-hearted robot who lived in a bustling city full of tall buildings and bright lights."""
sentences = sent_tokenize(text)
print("Tokenized Sentences:")
for i, sentence in enumerate(sentences):
print(f"{i+1}: {sentence}")
chunk_size = 3
chunks = [' '.join(sentences[i:i + chunk_size]) for i in range(0, len(sentences), chunk_size)]
print("\\nChunks:")
for i, chunk in enumerate(chunks):
print(f"Chunk {i+1}: {chunk}")
SpaCy also performs text segmentation similar to NLTK.