Lesson 3.5: Tool Integration & RAG

Duration: 75 minutes

Introduction

LLMs are powerful, but they have limitations: outdated knowledge, inability to access real-time data, and no way to perform actions. The solution? Connect them to external tools and knowledge sources. In this lesson, you’ll learn Retrieval-Augmented Generation (RAG) and tool integration patterns that transform LLMs from isolated systems into connected, capable assistants.

Why External Integration Matters

Current Information

Access real-time data beyond the model’s training cutoff

Factual Grounding

Reduce hallucinations by grounding in verified sources

Specialized Knowledge

Tap into domain-specific databases and documents

Action Capability

Perform calculations, API calls, and real-world actions

Retrieval-Augmented Generation (RAG)

RAG combines retrieval of relevant information with generation, dramatically improving accuracy.

What is RAG?

User Query

User asks a question

Retrieve Context

System searches knowledge base for relevant information

Augment Prompt

Retrieved information is added to the prompt as context

Generate Answer

LLM generates answer based on provided context

Basic RAG Pattern

Context: [RETRIEVED DOCUMENTS/INFORMATION]

Question: [USER QUESTION]

Instructions: Answer the question based ONLY on the provided context. 
If the answer is not in the context, explicitly state that you don't have 
enough information to answer.

Answer:

Example: Company Knowledge Base

Context (Retrieved from company docs):
"Our return policy allows returns within 30 days of purchase. Items must be unused 
and in original packaging. Refunds are processed within 5-7 business days after we 
receive the return. Shipping costs are non-refundable unless the item was defective. 
To initiate a return, contact customer service at returns@company.com with your order number."

Question: How long do I have to return an item?

Instructions: Answer based ONLY on the provided context.

Answer: According to our return policy, you have 30 days from the date of purchase 
to return an item. The item must be unused and in its original packaging.

Key Benefit: The answer is grounded in actual company policy, not the model’s general knowledge, eliminating hallucination risk.

When to Use RAG

Knowledge-Intensive Tasks

Use when: Answers require specific, factual informationExample: Customer support, technical documentation, research assistanceWhy: Ensures accuracy and reduces hallucinations

Dynamic Information

Use when: Information changes frequentlyExample: News, stock prices, weather, product catalogsWhy: Provides current information beyond training data

Private/Proprietary Data

Use when: Need to access company-specific informationExample: Internal policies, customer data, project documentationWhy: Keeps sensitive data secure while enabling AI assistance

Citation Requirements

Use when: Answers must be traceable to sourcesExample: Legal research, academic work, complianceWhy: Provides verifiable source attribution

RAG Prompt Patterns

Pattern 1: Simple RAG

Retrieved Context:
[DOCUMENT 1]
[DOCUMENT 2]
[DOCUMENT 3]

Question: [USER QUESTION]

Answer based on the context above:

Pattern 2: RAG with Source Citation

Context Documents:
[Doc 1]: [CONTENT]
[Doc 2]: [CONTENT]
[Doc 3]: [CONTENT]

Question: [USER QUESTION]

Instructions:
1. Answer based on the provided documents
2. Cite which document(s) support your answer
3. If information is not in the documents, say so

Answer:
[RESPONSE]

Sources: [Doc 1], [Doc 2]

Pattern 3: RAG with Confidence

Retrieved Information:
[CONTEXT]

Question: [USER QUESTION]

Provide:
1. Your answer based on the context
2. Confidence level (High/Medium/Low)
3. What information would increase confidence

Answer:
Confidence:
Additional information needed:

Example: Technical Documentation RAG

Retrieved Context:
[Doc 1 - API Authentication]: "All API requests require an API key in the header. 
Format: Authorization: Bearer YOUR_API_KEY. Keys can be generated in the dashboard 
under Settings > API Keys."

[Doc 2 - Rate Limits]: "Free tier: 100 requests/hour. Pro tier: 1000 requests/hour. 
Enterprise: Custom limits. Rate limit headers are included in all responses."

[Doc 3 - Error Codes]: "401: Invalid or missing API key. 429: Rate limit exceeded. 
500: Server error. Contact support for 500 errors."

Question: Why am I getting a 401 error when calling your API?

Answer: Based on the documentation, a 401 error indicates an invalid or missing API key. 
To resolve this:

1. Verify you're including the API key in the request header
2. Check the format: Authorization: Bearer YOUR_API_KEY
3. Ensure your API key is valid (generate a new one in Settings > API Keys if needed)

Source: [Doc 1 - API Authentication], [Doc 3 - Error Codes]

External Tool Use

Enable LLMs to use calculators, APIs, databases, and other tools.

Tool Use Pattern

Available Tools:
- calculator: Perform mathematical calculations
- search: Search the web for current information
- database: Query internal database
- api_call: Make API requests to external services

Task: [USER REQUEST]

Step 1: Determine which tool(s) are needed
Required tools: [LIST]

Step 2: Use tools
Tool: [TOOL NAME]
Input: [PARAMETERS]
Output: [RESULT]

Step 3: Generate response using tool outputs
Answer: [RESPONSE INCORPORATING TOOL RESULTS]

Example: Calculator Integration

Available Tools:
- calculator(expression): Evaluates mathematical expressions

Question: If I invest $10,000 at 7% annual interest compounded monthly for 5 years, 
how much will I have?

Step 1: Identify calculation needed
Formula: A = P(1 + r/n)^(nt)
Where: P=$10,000, r=0.07, n=12, t=5

Step 2: Use calculator
Tool: calculator
Input: 10000 * (1 + 0.07/12)^(12*5)
Output: 14,176.25

Step 3: Generate response
Answer: After 5 years, your investment will grow to $14,176.25. This represents 
a total gain of $4,176.25 (41.76% return) from your initial $10,000 investment.

Example: API Integration

Available Tools:
- weather_api(location): Get current weather data
- forecast_api(location, days): Get weather forecast

Question: Should I bring an umbrella to Seattle tomorrow?

Step 1: Determine needed information
Need: Tomorrow's weather forecast for Seattle

Step 2: Use weather API
Tool: forecast_api
Input: location="Seattle, WA", days=1
Output: {
  "date": "2024-01-16",
  "conditions": "Rain",
  "precipitation_chance": 85%,
  "temperature": "48°F"
}

Step 3: Generate recommendation
Answer: Yes, you should definitely bring an umbrella to Seattle tomorrow. 
The forecast shows rain with an 85% chance of precipitation and temperatures 
around 48°F. Consider bringing a light jacket as well.

API Integration Patterns

Structured Tool Calls

System: You have access to these tools:

1. get_stock_price(symbol: str) -> float
   Returns current stock price

2. get_company_info(symbol: str) -> dict
   Returns company information

3. calculate_percentage_change(old: float, new: float) -> float
   Calculates percentage change

User Query: How much has Apple stock changed since yesterday?

Assistant Response:
I'll need to use multiple tools to answer this:

Tool Call 1:
Function: get_stock_price
Parameters: {"symbol": "AAPL"}
Result: 185.50

Tool Call 2:
Function: get_stock_price
Parameters: {"symbol": "AAPL", "date": "yesterday"}
Result: 182.30

Tool Call 3:
Function: calculate_percentage_change
Parameters: {"old": 182.30, "new": 185.50}
Result: 1.76

Answer: Apple (AAPL) stock has increased by $3.20 (1.76%) since yesterday, 
currently trading at $185.50.

Error Handling in Tool Use

Available Tools:
- database_query(sql: str) -> results

Question: How many customers do we have in California?

Attempt 1:
Tool: database_query
Input: "SELECT COUNT(*) FROM customers WHERE state = 'CA'"
Output: ERROR - Table 'customers' not found

Analysis: Table name might be different

Attempt 2:
Tool: database_query
Input: "SHOW TABLES"
Output: ['users', 'orders', 'user_profiles']

Analysis: Customer data likely in 'users' or 'user_profiles'

Attempt 3:
Tool: database_query
Input: "SELECT COUNT(*) FROM users WHERE state = 'CA'"
Output: 1,247

Answer: We have 1,247 customers in California.

Knowledge Grounding

Prevent hallucinations by grounding responses in verified sources.

Grounding Pattern

Knowledge Base: [VERIFIED INFORMATION]

Question: [USER QUESTION]

Rules:
1. Answer ONLY using information from the knowledge base
2. If information is not available, say "I don't have information about that"
3. Never make up or infer information not explicitly stated
4. Cite specific sources when possible

Answer:

Example: Medical Information

Knowledge Base (from verified medical sources):
"Aspirin is a nonsteroidal anti-inflammatory drug (NSAID) used to reduce pain, 
fever, and inflammation. Common dosage for adults is 325-650mg every 4-6 hours. 
Maximum daily dose: 4000mg. Side effects may include stomach upset, heartburn, 
and increased bleeding risk. Consult a doctor before use if you have bleeding 
disorders, stomach ulcers, or are taking blood thinners."

Question: Can I take aspirin for my headache?

Grounded Answer: Based on the medical information available, aspirin can be used 
to reduce pain, including headaches. The typical adult dosage is 325-650mg every 
4-6 hours, with a maximum of 4000mg per day.

However, you should consult a doctor before taking aspirin if you have:
- Bleeding disorders
- Stomach ulcers
- Are taking blood thinners

Possible side effects include stomach upset, heartburn, and increased bleeding risk.

Note: This is general information. For personalized medical advice, please consult 
your healthcare provider.

Preventing Hallucination

Context: [ACTUAL INFORMATION]

Question: [QUESTION THAT MIGHT TEMPT HALLUCINATION]

Instructions:
- Answer ONLY if the information is in the context
- If unsure, say "I don't have enough information"
- Never fill in gaps with assumptions
- Be explicit about limitations

Example:

Context: "Our product was launched in 2020. It has 50,000 active users."

Question: How many users did you have in 2021?

GOOD Answer: "I don't have information about our user count in 2021. The available 
data shows we have 50,000 active users currently, and the product was launched in 2020."

BAD Answer: "We had approximately 25,000 users in 2021." ❌ (Hallucinated)

Citing Sources

Citation Pattern

Sources:
[Source 1]: [CONTENT]
[Source 2]: [CONTENT]
[Source 3]: [CONTENT]

Question: [QUESTION]

Answer with inline citations:
[ANSWER with [Source X] citations]

References:
[Source 1]: [FULL CITATION]
[Source 2]: [FULL CITATION]

Example: Research Assistant

Sources:
[1]: "A 2023 study in Nature found that regular exercise reduces cognitive decline 
by 30% in adults over 65." (Nature, Vol 615, 2023)

[2]: "The WHO recommends 150 minutes of moderate aerobic activity per week for adults." 
(WHO Physical Activity Guidelines, 2020)

[3]: "Research shows that combining aerobic and strength training provides the greatest 
cognitive benefits." (Journal of Aging Research, 2022)

Question: How does exercise affect cognitive health in older adults?

Answer: Regular exercise has significant benefits for cognitive health in older adults. 
Research shows that it can reduce cognitive decline by 30% in adults over 65 [1]. 
The World Health Organization recommends at least 150 minutes of moderate aerobic 
activity per week [2], and studies indicate that combining aerobic exercise with 
strength training provides the greatest cognitive benefits [3].

References:
[1] Nature, Vol 615, 2023 - Study on exercise and cognitive decline
[2] WHO Physical Activity Guidelines, 2020
[3] Journal of Aging Research, 2022 - Combined exercise benefits study

Best Practices

Verify Sources

Ensure retrieved information is from reliable sources

Explicit Limitations

Clearly state when information is not available

Source Attribution

Always cite where information came from

Error Handling

Handle tool failures gracefully with fallback strategies

Practice Exercises

Exercise 1: RAG Implementation

Design a RAG system for a customer support chatbot.

Sample Solution

System Design:

1. Knowledge Base:
   - Product documentation
   - FAQs
   - Return policies
   - Troubleshooting guides

2. Retrieval Strategy:
   - Semantic search on user question
   - Retrieve top 3 most relevant documents
   - Include document metadata (source, date)

3. Prompt Template:

Retrieved Information: [Doc 1 - Source: Product Manual, Section 3.2] [CONTENT][Doc 2 - Source: FAQ #15] [CONTENT][Doc 3 - Source: Troubleshooting Guide] [CONTENT]Customer Question: [QUESTION]Instructions:

Answer based on retrieved documents
Be helpful and friendly
Cite specific sources
If answer not found, offer to escalate to human agent

Response:

4. Fallback Strategy:
- If confidence < 70%: Offer human escalation
- If no relevant docs found: Acknowledge and escalate
- Track unanswered questions for knowledge base improvement

Exercise 2: Tool Integration

Design a tool-using assistant for financial calculations.

Sample Solution

Available Tools:

1. stock_price(symbol, date=today)
   - Returns stock price for given symbol and date

2. calculate(expression)
   - Evaluates mathematical expressions

3. currency_convert(amount, from_currency, to_currency)
   - Converts between currencies

4. portfolio_value(holdings)
   - Calculates total portfolio value

Example Interaction:

User: "I have 100 shares of AAPL bought at $150. What's my gain if I sell today?"

Assistant Process:

Step 1: Identify needed tools
- Need current AAPL price: stock_price
- Need to calculate gain: calculate

Step 2: Execute tools
Tool: stock_price("AAPL")
Result: $185.50

Tool: calculate("(185.50 - 150) * 100")
Result: $3,550

Step 3: Generate response
"Your 100 shares of Apple (AAPL) have gained $3,550. You bought at $150/share 
and the current price is $185.50/share, representing a 23.67% return on your 
investment."

Exercise 3: Source Grounding

Create a grounded QA system that prevents hallucinations.

Sample Solution

Grounding System:

1. Strict Rules:
   - ONLY use provided context
   - Explicitly state when information is missing
   - Never infer or assume
   - Cite specific sources

2. Template:

Verified Sources: [Source 1]: [CONTENT] [Source 2]: [CONTENT]Question: [QUESTION]Grounding Rules: ✓ Answer only from sources ✓ State “Not available” if missing ✓ Cite sources ✗ Never guess or inferAnswer:

3. Example:

Sources:
[1]: "Product X costs $99"
[2]: "Product X has 5-star rating"

Q: "What's the warranty on Product X?"

GOOD: "I don't have information about the warranty for Product X in the 
available sources. I can tell you it costs $99 [1] and has a 5-star rating [2], 
but warranty details are not provided."

BAD: "Product X likely has a 1-year warranty." ❌

Real-World Application: Knowledge-Grounded Assistant

You are a knowledge-grounded assistant with access to a verified knowledge base.

Knowledge Base: [RETRIEVED DOCUMENTS]

User Query: [QUESTION]

Response Protocol:

1. Search Knowledge Base
   Relevant documents: [LIST]
   Confidence: [HIGH/MEDIUM/LOW]

2. Generate Answer
   - Use ONLY information from knowledge base
   - Cite sources inline [Doc X]
   - If information incomplete, state explicitly

3. Quality Checks
   ✓ All facts from knowledge base?
   ✓ Sources cited?
   ✓ Limitations stated?
   ✓ No hallucinations?

4. Response Format:
   Answer: [GROUNDED RESPONSE]
   
   Sources:
   - [Doc 1]: [Citation]
   - [Doc 2]: [Citation]
   
   Confidence: [LEVEL]
   
   Limitations: [IF ANY]

5. Fallback (if needed):
   "I don't have enough information in my knowledge base to fully answer this question. 
   Would you like me to [alternative action]?"

Key Takeaways

Use RAG to ground answers in verified sources

Integrate external tools for calculations and real-time data

Always cite sources and attribute information

Explicitly state limitations and missing information

Implement error handling for tool failures

Prevent hallucinations through strict grounding rules

Module 3 Complete!

Congratulations! You’ve mastered advanced prompting techniques. You’re now ready to tackle the Module 3 assessment and build a comprehensive AI assistant.

Next: Module 3 Assessment

Build a Multi-Capability AI Assistant combining all advanced techniques

Module 1

Module 2

Module 3

​Introduction

​Why External Integration Matters

Current Information

Factual Grounding

Specialized Knowledge

Action Capability

​Retrieval-Augmented Generation (RAG)

​What is RAG?

​Basic RAG Pattern

​Example: Company Knowledge Base

​When to Use RAG

​RAG Prompt Patterns

​Pattern 1: Simple RAG

​Pattern 2: RAG with Source Citation

​Pattern 3: RAG with Confidence

​Example: Technical Documentation RAG

​External Tool Use

​Tool Use Pattern

​Example: Calculator Integration

​Example: API Integration

​API Integration Patterns

​Structured Tool Calls

​Error Handling in Tool Use

​Knowledge Grounding

​Grounding Pattern

​Example: Medical Information

​Preventing Hallucination

​Citing Sources

​Citation Pattern

​Example: Research Assistant

​Best Practices

Verify Sources

Explicit Limitations

Source Attribution

Error Handling

​Practice Exercises

​Exercise 1: RAG Implementation

​Exercise 2: Tool Integration

​Exercise 3: Source Grounding

​Real-World Application: Knowledge-Grounded Assistant

​Key Takeaways

​Module 3 Complete!

Next: Module 3 Assessment

Introduction

Why External Integration Matters

Retrieval-Augmented Generation (RAG)

What is RAG?

Basic RAG Pattern

Example: Company Knowledge Base

When to Use RAG

RAG Prompt Patterns

Pattern 1: Simple RAG

Pattern 2: RAG with Source Citation

Pattern 3: RAG with Confidence

Example: Technical Documentation RAG

External Tool Use

Tool Use Pattern

Example: Calculator Integration

Example: API Integration

API Integration Patterns

Structured Tool Calls

Error Handling in Tool Use

Knowledge Grounding

Grounding Pattern

Example: Medical Information

Preventing Hallucination

Citing Sources

Citation Pattern

Example: Research Assistant

Best Practices

Practice Exercises

Exercise 1: RAG Implementation

Exercise 2: Tool Integration

Exercise 3: Source Grounding

Real-World Application: Knowledge-Grounded Assistant

Key Takeaways

Module 3 Complete!