Token Economics and Semantic Search with Embeddings
- 6 hours ago
- 6 min read
Course: LLM Foundational Course
Level: Medium
Type: Individual Assignment
Duration: 5–7 days
Total Marks: 100
Objective
The objective of this assignment is to help you:
Understand tokenization and how it affects API costs
Implement token counting and cost calculation functions
Build a vector database from scratch using embeddings
Perform semantic search to retrieve relevant documents
Create a simple RAG system that answers questions using retrieved context
Think practically about token efficiency and cost optimization
Problem Statement
You are building a knowledge base search system for a small documentation website. The system should:
Store documents with their embeddings
Search for relevant documents based on meaning (not keywords)
Answer user questions using only the retrieved information
Track token usage and costs
Your task is to:
Implement token counting and cost calculation utilities
Build a simple vector database
Create semantic search functionality
Combine it all into a RAG system
Dataset Description
You must create a knowledge base with at least 10 documents about a topic of your choice:
Suggested topics:
Programming concepts (e.g., Python basics, data structures)
Technology documentation (e.g., Git, APIs, databases)
General knowledge (e.g., history, science, geography)
Course-related content (e.g., LLM concepts from this course)
Document requirements:
Each document: 100–300 words
Documents should cover different aspects of the topic
Write in clear, informative language
You may use AI to help generate documents (but must acknowledge it)
Example knowledge base structure:
If topic = "Python Programming":
doc1: "What are variables in Python"
doc2: "Understanding Python lists and tuples"
doc3: "Functions and how to use them"
doc4: "File handling in Python"... (at least 10 total)
Tasks & Requirements
Task 1: Token Counting and Cost Calculation (15 Marks)
Objective: Implement utility functions for token management.
Requirements:
Implement the following functions:
count_tokens(text) → int - Count tokens in a text string using
tiktoken visualize_tokens(text) → List[str] - Show how text is broken into
tokens calculate_cost(input_text, output_text) → dict - Return: {'input_tokens',
'output_tokens', 'cost_usd'} - Use Claude Sonnet pricing: $3 per 1M input, $15 per 1M output
Test with examples:
Count tokens in your knowledge base documents
Show the total token count for all documents
Calculate the cost to embed all documents if we were using Claude's API
Deliverable:
Code implementing all 3 functions
Test outputs showing token counts for your documents
Example output:
Document 1: "What are variables..." → 87 tokens
Document 2: "Understanding lists..." → 112 tokens...
Total tokens: 1,234
Average per doc: 123.4 tokens
Task 2: Building a Vector Database (20 Marks)
Objective: Create a simple vector database for storing and searching documents.
Requirements:
Implement a `SimpleVectorDatabase` class with:
init() - Initialize empty lists for documents and embeddings
add_document(doc_id, text) - Generate embedding and store document
search(query, top_k=3) - Return top_k most similar documents with scores
get_stats() - Return: total documents, total embedding dimensions
Use sentence-transformers:
Model: all-MiniLM-L6-v2
Embedding dimensions: 384
Calculate cosine similarity for search
Add all your knowledge base documents to the database
Implement proper storage:
Store document ID, text, and embedding
Keep embeddings as numpy arrays
Deliverable:
Complete SimpleVectorDatabase class implementation
Code showing all 10+ documents being added
Output showing database stats (total docs, dimensions, etc.)
Example output:
Added document: doc1
Added document: doc2...
Knowledge base created with 10 documents
Embedding dimensions: 384
Task 3: Semantic Search Implementation (20 Marks)
Objective: Test and analyze semantic search capabilities.
Requirements:
Test semantic search with at least 5 different queries:
Some queries that exactly match document content
Some queries that are semantically similar (different words, same meaning)
Some queries that have no relevant results
For each query, show:
Top 3 results
Similarity scores
Actual document text
Experiment with `top_k`:
Try retrieving 1, 3, and 5 documents
Which is better for different query types?
Deliverable:
5 test queries with results
Example output:
Query: "How do I store data in Python?"
Result 1 (similarity: 0.752): "Variables in Python are used to store data..."
Result 2 (similarity: 0.601): "Lists and tuples are data structures..."
Result 3 (similarity: 0.421): "File handling allows you to save data..."
Task 4: Building a RAG System (20 Marks)
Objective: Combine semantic search with Claude API to answer questions.
Requirements:
Implement a `rag_query()` function:
def rag_query(query, knowledge_base, top_k=2):
Retrieve relevant documents
Format context
Create prompt with context
Get Claude response
Return answer with sources
Prompt engineering:
System message should instruct Claude to use ONLY provided context
Format context clearly (bullet points or numbered list)
Ask Claude to cite which information it used
Return format:
{ 'answer': "...", 'sources': [...], 'similarities': [...] }
Test with at least 5 questions:
Questions that CAN be answered from your knowledge base
Questions that CANNOT be answered (test if Claude admits this)
Follow-up questions to previous queries
Deliverable:
Complete rag_query() implementation
5+ test queries with full output (answer + sources + scores)
Example output:
Question: "How do functions work in Python?
"Answer:Functions in Python are reusable blocks of code that performspecific tasks. They are defined using the 'def' keyword...Sources used: [0.821] Functions and how to use them in Python [0.542] Understanding Python variables and scope
Task 5: Cost Tracking and Optimization (15 Marks)
Objective: Track and analyze the cost of your RAG system.
Requirements:
Track costs for:
Each RAG query (input tokens + output tokens)
Total cost for all test queries
Cost per query (average)
Create a cost summary:
{ 'total_queries': N, 'total_input_tokens': X, 'total_output_tokens': Y, 'total_cost_usd': Z, 'average_cost_per_query': A }
Projection:
If this system serves 100 users/day with 5 queries each
What would be the monthly cost?
Deliverable:
Cost tracking implementation
Summary table of all costs
Monthly projection calculation
Example output:
Total queries: 10
Total input tokens: 2,456
Total output tokens: 1,234
Total cost: $0.0258
Average per query: $0.00258
Queries possible for $1: ~387
Task 6: Final Integration & Demo (10 Marks)
Objective: Create a complete, working demonstration.
Requirements:
Create a simple demo:
Ask 3 questions about your knowledge base
Show the full workflow:
Question → Search → Context → Answer → Sources
Display costs for each query
Handle edge cases:
Empty query
No relevant documents found
Very long query
Clean presentation:
Clear output formatting
Proper labels and headers
Summarize performance
Deliverable:
End-to-end demo conversation (3+ exchanges)
Edge case handling examples
Final performance summary
Deliverables
You must submit:
1. Code (Required)
Jupyter Notebook (.ipynb)
Well-commented code
Clear section headers for each task
All output visible (don't clear cell outputs)
Executable from top to bottom without errors
2. Knowledge Base (Required)
Either:
Separate text files for each document (in a folder), OR
Code cells that define all documents
3. Report (Required)
Short report (2–4 pages) including:
Introduction (what you built)
Approach for each task
Key findings and analysis
Challenges faced and solutions
Conclusions and learnings
Format: PDF or DOCX
4. Output Samples (Required)
Include in your notebook:
Sample token visualizations
Sample semantic search results
Sample RAG answers with sources
Cost summary tables
Submission Guidelines
Submit via: Your LMS (Moodle / Google Classroom as specified by instructor)
File Naming Convention: <YourName>_LLM_Assignment1.zip
Inside the ZIP: /notebook.ipynb/knowledge_base/ (optional, if using separate files)/report.pdf
Deadline: Submit within 7 days from assignment release
Late Submission Policy:
Up to 24 hours late → 10% penalty
24–48 hours late → 20% penalty
Beyond 48 hours → Not accepted
Important Instructions
Do NOT copy code from external sources without understanding
Explain your logic clearly in comments
Use libraries as taught in the course (tiktoken, sentence-transformers, anthropic)
Test thoroughly before submission
Include all outputs in your notebook
Plagiarism will result in zero marks and disciplinary action.
Call to Action
Ready to transform your business with AI-powered intelligence that accelerates insights, enhances decision-making, and unlocks the full value of your data?
Codersarts is here to help you turn complex data workflows into efficient, scalable, and evidence-driven AI systems that empower teams to make smarter, faster, and more confident decisions.
Whether you’re a startup looking to build AI-driven products, an enterprise aiming to optimize operations through data science, or a research organization advancing innovation with intelligent data solutions, we bring the expertise and experience needed to design, develop, and deploy impactful AI systems that drive measurable business outcomes.
Get Started Today
Schedule an AI & Data Science Consultation:
Book a 30-minute discovery call with our AI strategists and data science experts to discuss your challenges, identify high-impact opportunities, and explore how intelligent AI solutions can transform your workflows and performance.
Request a Custom AI Demo:
Experience AI in action with a personalized demonstration built around your business use cases, datasets, operational environment, and decision workflows — showcasing practical value and real-world impact.
Email: contact@codersarts.com
Transform your organization from data accumulation to intelligent decision enablement — accelerating insight generation, improving operational efficiency, and strengthening competitive advantage.
Partner with Codersarts to build scalable AI solutions including RAG systems, predictive analytics platforms, intelligent automation tools, recommendation engines, and custom machine learning models that empower your teams to deliver exceptional results.
Contact us today and take the first step toward next-generation AI and data science capabilities that grow with your business ambitions.
Comments