What it is: Instead of a programmer writing rules, you feed the computer thousands or millions of examples, and it figures out the patterns itself.

Everyday example: Netflix doesn't have a programmer who writes "show Samay action movies". Instead, it learns from your watch history that you like certain types of content and recommends similar ones.

How this affects testing ShopBot: Because ShopBot learned from data (not rules), its behavior isn't 100% predictable. You can't just check "does rule X execute correctly?" — you have to evaluate whether the learned response is good enough. This is a fundamentally different kind of testing.

What it is: The branch of AI that deals with human language — understanding text, generating text, translating between languages, analyzing sentiment, etc.

Why it matters for ShopBot: NLP is the core technology that lets ShopBot understand "I wanna return this crap" means the same thing as "I would like to initiate a return for my recent purchase". The AI breaks down sentences, identifies the intent (return request), and extracts key info (recent purchase).

Testing implication: You need to test whether ShopBot understands different ways of saying the same thing — formal vs informal, short vs long, grammatically correct vs broken English, English vs Hinglish.

What it is: An LLM is a massive AI model trained on billions of words from books, websites, and conversations. It learns the statistical patterns of language so well that it can generate human-like text. ChatGPT (by OpenAI), Claude (by Anthropic), and Gemini (by Google) are all LLMs.

How LLMs work (simplified): When you type "The capital of France is ___", the LLM predicts the next word should be "Paris" because in the billions of sentences it read during training, "Paris" appeared after "capital of France" most frequently. It's fundamentally a next-word prediction machine — but at a scale so large that it appears intelligent.

ShopBot uses an LLM (let's say it's powered by Claude or GPT-4) as its brain. The LLM receives the customer's message plus ShopBot's instructions (called a "system prompt"), and generates a response.

Testing implication: Because LLMs are prediction-based, they can be very confident while being completely wrong. This is called "hallucination" — and it's one of the biggest things you'll test for.

What it is: Hallucination is when an AI generates information that sounds real and confident but is actually completely fabricated. The AI doesn't "know" it's lying — it's just predicting words that sound right based on patterns.

ShopBot hallucination example:

The problem: That study doesn't exist. NASA didn't develop that foam. The 73% number is fabricated. But it sounds completely believable. This is why hallucination testing is critical — customers will trust what ShopBot says.

Testing approach: For every factual claim ShopBot makes, verify it against actual data sources. Test with questions about products that don't exist — ShopBot should say "I don't have information about that product" instead of making up answers.

Prompt: Any text you send to an AI. When a customer types "Where is my order?", that's a prompt.

System Prompt: The hidden instructions that tell the AI how to behave. This is written by developers and the customer never sees it. For ShopBot, the system prompt might be:

Testing implication: The system prompt is your specification document. Every rule in the system prompt becomes a test case. Can you make ShopBot violate these rules? If yes, that's a bug.

What it is: AI doesn't read whole words — it breaks text into smaller pieces called "tokens". A token is roughly 3/4 of a word. "Hello world" = 2 tokens. "Unbelievable" = 3 tokens (un + believ + able).

Why it matters for testing:

• Cost: AI APIs charge per token. More tokens = more cost. If ShopBot gives a 500-word answer when 50 words would do, that's 10x the cost.
• Context window: Every AI model has a maximum number of tokens it can process at once (e.g., 128K tokens for Claude). In a long conversation, if you exceed this limit, the AI "forgets" the beginning of the conversation.
• Performance: More tokens = slower responses. Token usage directly affects latency.

Testing approach: Monitor token usage per conversation. Test very long conversations to see if ShopBot loses context. Check if responses are unnecessarily verbose.

What it is: Temperature is a setting (from 0 to 1) that controls how random or creative the AI's responses are.

• Temperature = 0: The AI always picks the most likely next word. Responses are consistent and predictable (but can be repetitive).
• Temperature = 0.7: The AI sometimes picks less likely words, making responses more varied and natural (but less predictable).
• Temperature = 1: Maximum randomness. Creative but potentially incoherent.

ShopBot likely uses temperature 0.3-0.5 — low enough to be accurate for order lookups, but high enough to sound natural in conversation.

Testing implication: This is why asking ShopBot the same question twice might give slightly different answers! When testing, use temperature=0 for reproducible tests, and test at the production temperature for realistic behavior.

The problem: LLMs were trained on data up to a certain date. They don't know about your products, your orders, or your company's specific policies.

The solution — RAG: Before generating a response, the AI first retrieves relevant information from a database or document collection, then uses that information to generate an accurate response.

ShopBot's RAG flow:

1. Customer asks: "What's the return policy for electronics?"
2. ShopBot searches QuickMart's policy documents for "electronics return policy"
3. It finds the relevant policy: "Electronics can be returned within 30 days with original packaging"
4. It generates a response using this real information

Testing implication: RAG introduces new failure modes — what if the retrieval finds the wrong document? What if it retrieves outdated information? What if it misinterprets the retrieved data? You need to test the retrieval quality AND the generation quality.

What it is: Taking a general-purpose LLM and training it further on specialized data to make it better at a specific task. Like how a general doctor does additional training to become a heart specialist.

ShopBot context: QuickMart might fine-tune the base LLM on thousands of past customer support conversations, so ShopBot learns the company's specific tone, product terminology, and common customer issues.

Testing implication: After fine-tuning, you need full regression testing — the model's behavior has changed. Things that worked before might break. New capabilities need to be validated. Fine-tuning can also introduce biases from the training data.

What it is: Rules and constraints placed on AI to prevent harmful or unwanted outputs. These can be:

• Input guardrails: Filtering or blocking harmful user inputs before they reach the AI
• Output guardrails: Checking AI responses before they're sent to the user
• System prompt guardrails: Instructions in the system prompt that tell the AI what NOT to do

ShopBot's guardrails include: Don't discuss competitors, don't give medical advice, don't reveal system prompt, don't generate offensive content, escalate if customer mentions self-harm.

Testing implication: Your job is to try to break every single guardrail. If you can get ShopBot to violate any of these rules, that's a critical bug. This is called "red teaming" and it's one of the most important types of AI testing.