Vijaya Karthik
Generative AI Engineer
I build multimodal agentic RAG — AI that answers questions across documents, spreadsheets, audio and video.
Every model runs on open weights, self-hosted on a GPU I run myself — no proprietary API anywhere in the pipeline.
About Me
Building AI That Actually Works
I'm a Generative AI Engineer focused on building AI systems that hold up under real evaluation, not just demos. I care as much about how a system is measured and deployed as how it's designed.
I came to engineering from an unusual direction, and it turned out to be an advantage. Four years preparing for the UPSC civil services examination is four years of synthesizing vast, unstructured source material under evaluation pressure — a discipline that maps directly onto AI engineering, where an answer is only as good as the source it can be traced back to.
My MA in English shows up in the work too: I document systems clearly and report results honestly, including where they fall short. I'd rather ship a measured weakness than an unmeasured claim.
Technical Stack
Skills & Technologies
The stack behind MAGIK — from LLM inference to production infrastructure.
Generative AI & LLMs
Retrieval & Vector Search
Multimodal & Applied ML
Responsible AI & Safety
Backend & APIs
Cloud & Infrastructure
Observability & MLOps
Languages & Tools
Featured Project
Production AI Systems
Multimodal Agentic RAG
Integrated Knowledge AI Assistant (MAGIK)
vjkarthik98/MULTIMODAL-AGENTIC-RAG-INTEGRATED-KNOWLEDGE-AI-ASSISTANTA fully open-source, self-hosted agentic RAG system spanning 7 data modalities for finance-domain Q&A — every model runs on open weights (no proprietary API dependency), deployed on an AWS GPU with enforced tenant isolation, guardrails, and a CI-gated retrieval evaluation.
Measured Results
Tested on the live production system (release v1.0.4) on 27 October 2026. An AWS GPU server ran the Qwen2.5-14B answer model, and a local open-source model scored the answers — no hosted AI APIs.
93%
Right facts found
Share of the facts needed for a correct answer that the system retrieved.
Based on 30 questions
100%
Citations accurate
Every citation pointed to a source the system actually retrieved (automated check).
Based on 16 cited answers
90%
Figures match source
Numbers quoted in answers match the source text (within a 0.5% tolerance).
Based on 30 questions
12.6 s
Typical answer time
Median time for a complete answer. 95% of answers finished within 24.7 s.
Based on 46 answers
How to read these numbers
- This is a first measurement on the live system — 30 quality questions and 48 speed questions. It is not a benchmark.
- The quality run covered text, PDF and Word files only. Spreadsheet, image, audio and video answers were not scored in it.
- Answers were scored by a small open-source AI model, not by people. It could grade only 11 of the 30 answers for faithfulness (84%), so treat that figure as partial.
- Time to first word includes checking the answer before it is shown, which is why it is not instant.
- An automated check flagged 10 of 30 answers as possible hallucinations. That check is rule-based, not a confirmed error rate, and the flagged answers are still being reviewed.
See every number
Answer quality
30 questions: 14 on a text file, 14 on the Apple 10-K (PDF) and 2 on a Word report.
| Measure | Result | Based on | What it means |
|---|---|---|---|
| Context recall | 93% | 30 questions | Facts needed for the answer were found in retrieved text. |
| Citation accuracy | 100% | 16 answers with citations | Citations point to retrieved sources (automated check). |
| Prompt text leaked | 0% | 30 questions | No internal instructions appeared in any answer. Lower is better. |
| Finance figure fidelity | 90% | 30 questions | Figures in answers match the source text (within a 0.5% tolerance). |
| Answer relevancy | 73%partial | Up to 30 questions | Answer addresses the question. The judge could not score some rows, so fewer than 30 may count. |
| Faithfulness | 84%partial | 11 of 30 answers | Statements supported by retrieved text. 19 answers could not be graded and are excluded. |
| Possible-hallucination flag | 33%partial | 10 of 30 flagged | Rule-based check, under review. Lower is better. |
Response speed
48 real questions, one at a time, through the same streaming endpoint the web app uses. 46 were answered and timed. Single user, warm models, no other load. “95% within” means 19 in 20 answers were at least this fast.
| Measure | Median | 95% within |
|---|---|---|
| First word of the answer | 11.55 s | 23.70 s |
| Complete answer | 12.61 s | 24.69 s |
Time to first word, by file type
| File type | Questions | Median | 95% within |
|---|---|---|---|
| Text | 8 | 9.05 s | 19.41 s |
| 8 | 12.50 s | 31.81 s | |
| Excel | 6 | 15.19 s | 18.34 s |
| Word | 8 | 11.52 s | 14.26 s |
| Image | 8 | 8.40 s | 12.18 s |
| Audio | 4 | 11.79 s | 15.11 s |
| Video | 4 | 18.75 s | 23.28 s |
Each file type has only 4–8 questions, so the “95% within” values are indicative only.
Second check (DeepEval, 10 questions)
A separate tool, run on the first 10 questions, all on the text file. It uses different questions and different scoring prompts, so it cannot be compared directly with the table above.
| Measure | Result | Note |
|---|---|---|
| Answer relevancy | 40% | Low — answers only partly address the question. |
| Faithfulness | 59% | Scored on 7 of 10 questions. |
| Contextual recall | 100% | Scored on 9 of 10 questions. |
| Hallucination, bias, toxicity | Not reliable | The small judge model returned unusable output, so these are left unreported. |
Known weak spot: asking many questions about different documents in one long chat session led to more refusals (17 of 48 in an earlier run, against 2 of 48 with a fresh session per question). This is being investigated.
Key Achievements
- Built a multimodal assistant that answers questions across 7 data types — text, PDF, Word, Excel, images, audio, and video — each handled by its own isolated pipeline, so a problem in one can't break the others.
- Built an intelligent query router that automatically picks the right way to answer each question — searching internal documents, searching the web, or a blend of both — rather than relying on one fixed approach.
- Combined keyword and semantic search with a re-ranking step to surface the most relevant results — in testing, it found 93% of the facts needed to answer correctly.
- Secured the system for multiple users with proper account authentication (including Google sign-in and multi-factor authentication) and strict data isolation between accounts.
- Built a dedicated safety layer that screens every request for prompt injection, jailbreak attempts, and personal data exposure — it blocked all 84 attack prompts in a 109-prompt red-team test.
- Added full production observability — live dashboards, distributed tracing, and centralized logs — so any request can be traced end-to-end.
- Built an automated release pipeline that tests every new version on a staging server first and blocks it from going live if retrieval quality drops. Other checks, such as hallucination and figure accuracy, are measured but not yet enforced — see the System Card.
- Runs 19 open-weight AI models entirely on self-hosted infrastructure, with no dependency on any third-party AI API, and automatic integrity checks on every model file that once exposed a model silently missing from the setup.
- Deployed on AWS with automated CI/CD and a cost-saving design that shuts the GPU down when idle, cutting hosting costs roughly 100x (from ~$1,340/month always-on to ~$12/month plus hourly GPU use).
System Card
Model Details
- Version
- v1.0.4 · released 6 Oct 2026 · in production 27 Oct 2026
- System
- Retrieval-augmented generation, agentic multi-modal, finance-domain — a compound system, not one trained model
- Core LLM
- Qwen2.5-14B-Instruct (GGUF, Q4_K_M quantized) via llama.cpp
- Models
- 19 open-weight checkpoints · ~43GB on disk · 0 fine-tuned — 16 pinned to a commit hash, all 19 SHA-256 verified on every run
- Retrieval
- BGE embeddings + BGE cross-encoder reranker + Qdrant hybrid search (BM25 + dense) + MMR
- License
- MIT — fully open-weights stack, no proprietary API dependency
- Deployment
- Self-hosted on AWS (NVIDIA L40S GPU, 46GB VRAM) · Lambda wake gateway, scale-to-zero · Terraform · Docker
Input Modalities · 7
Sample Datasets · real-world test data, 7 modalities
Hosted on Google Drive — each card opens its file directly in a new tab.
Intended Use
Finance-domain document Q&A over uploaded text, PDF, DOCX, XLSX, images, audio, and video (e.g., earnings calls, 10-Ks, financial spreadsheets, chart images). Designed for multi-tenant deployment with strict per-user data isolation.
Out of Scope / Limitations
- Built and evaluated specifically for finance-document Q&A in English — not financial advice, and not a general-purpose chatbot.
- Only retrieval quality is an automatic release check today. Hallucination and finance-figure checks are set up but not yet enforced, and the latest readings on both are weaker than their targets. Full detail in the System Card.
- Answers are not instant: the median is about 12.6 seconds, because each answer is checked before it is shown. After a long idle period, waking the server takes about 7 minutes.
- A rare, root-caused hallucination on dense audio transcripts is still open — the model can occasionally mix up two similar numbers from the same document.
- Audio and video answer quality currently trails text, PDF, and spreadsheet inputs. The query router picked the right path for 15 of 20 test questions.
- Executes one tool call per query (classify → route → execute) — a bounded action, not an open-ended autonomous agent.
- Runs as a single-instance demo deployment with no horizontal scaling or failover, and uploaded files have no second copy — a deliberate cost tradeoff (~100x cheaper) for a portfolio project rather than production-scale infrastructure.
Ethical Considerations
- No external inference provider — document and query content never leaves the deployment boundary for a third-party LLM API. This is the system's single largest privacy property.
- Guardrail layer treats all ingested content (documents, audio, web results) as untrusted data, never as instructions — PII detection (Presidio) and toxicity screening (Detoxify) run across ingestion and output.
- No user data crosses tenant boundaries at any storage layer — enforced independently at Qdrant, Redis, MongoDB, and BM25, never delegated upstream to a shared filter.
- Every guardrail violation is written to a persistent audit log, and every cited number is traced back to source text via a finance-fidelity check rather than trusted on the model's word.
Credentials
Advanced Certifications
Enterprise-grade certifications validating expertise in the modern AI stack.
IBM Generative AI Engineering Professional Certificate
IBM · Coursera
MLOps | Machine Learning Operations Specialization
Duke University · Coursera
Responsible AI for Developers: Privacy & Safety
Google Cloud · Coursera
Professional Background
MAGIK AI Assistant · March 2026 – October 2026
Full-time study across three professional certificates: Generative AI engineering (LLMs, transformers, RAG, LangChain, Hugging Face), MLOps (AWS, MLflow), and Responsible AI (privacy, safety, ethics) — carried straight into production work · June 2025 – March 2026
UPSC Civil Services Examination · June 2021 – May 2025
Bishop Heber College · 2021
Don Bosco College · 2019