Building Reliable LLM Applications: Lessons from Ishwar (JPMorgan Chase, ex-Meta)
Notes from my conversation with Ishwar, Engineering Lead at JPMorgan Chase, on reliability, cost, security and the future of AI-assisted engineering.
Getting an AI feature to work in a demo takes an afternoon. Getting customers, engineers and stakeholders to trust it in production is a different engineering problem.
Most teams find this out the hard way. The prototype impresses everyone, then the real problems show up: token bills that grow every month, answers nobody can verify, an AI provider outage that takes the whole product down, and security reviews that stop the launch.
That gap between demo and production was the focus of our latest Codersarts Podcast episode. My guest was Ishwar, Engineering Lead at JPMorgan Chase, previously a Senior Software Engineer at Meta and a software engineer at ASML. He has worked across India, the Netherlands and the UK, and today leads a live risk platform that streams intraday trading data into real-time monitoring dashboards, where reliability is not optional.
Below are the ten lessons I took from the conversation. Each one ends with what it means if you are building an AI product yourself.
1. Building AI is easy. Building what you think you built is not.
Ishwar's view is blunt: the build is no longer the hard part. Anyone can wire an LLM API into an app in a day. The hard part is the gap between what you think you are shipping and what actually runs in production.
"What you think you are building and what you are actually building has a big difference if you don't put your mind through it."
What it means for your product: A working prototype is the start of the project, not the end. Before launch, plan time for edge cases, evaluation, cost limits, monitoring and failure handling. That work usually takes longer than the first build.
2. Reliability is a business metric, not a model metric
An accurate AI tool that nobody adopts is worth nothing. Ishwar ties reliability back to things stakeholders care about:
Retention: do users keep coming back?
Time saved: is the workflow measurably faster?
Error reduction: are fewer mistakes reaching production or customers?
For internal developer tools, he tracks how much work the AI completes on its own versus how much hand-holding engineers give it, how many tokens it uses per task, and how much faster the overall workflow becomes. If a tool doesn't remove friction from the pipeline, he says, it shouldn't be built at all.
What it means for your product: Agree on success metrics before writing code. "The AI works" is not a metric. "Support tickets resolved without a human went from 0% to 40%" is.
3. AI developer tools are not chatbots
A chatbot is stateless and passive: you ask, it answers, it forgets. A real AI tool lives inside the workflow. It understands the codebase, the history and the current errors, suggests the next action, and automates steps that don't need a human.
The biggest value, according to Ishwar, is removing context switching, the thing engineers hate most.
What it means for your product: If your AI feature is "a chat box in the corner", users will treat it as optional. Put AI where the work already happens: inside the form, the dashboard, the ticket, the editor.
4. Treat every LLM call like a database query
This was the most practical lesson in the episode. Companies are already capping how many tokens their developers can use, because AI usage quietly grows until the bill gets attention.
Ishwar's cost playbook:
Set strict token budgets per feature, user or request
Cache aggressively: many users ask nearly identical questions about the same problem
Summarise repeated prompts instead of resending long context every time
Add a semantic deduplication layer, so near-duplicate queries (found with nearest-neighbour search) hit the cache instead of the model
"The goal is treating every AI call like a database query, not an LLM call."
What it means for your product: LLM cost is an architecture decision, not a billing surprise. Caching, prompt design and model routing can cut API spend a lot without hurting output quality.
5. Don't start with microservices, or with ten agents
Ishwar's advice for a new AI platform: start with a monolith, ship fast and validate. Move to microservices only when real scaling bottlenecks appear. Async workflows and failures that are hard to trace carry a heavy debugging cost.
"Maybe you can solve it by just having a bigger machine. A bigger machine is not that costly at this time."
The same rule applies to multi-agent systems. It's not over-engineering if the problem demands it, but many teams build complex agent orchestration in anticipation rather than in response to need.
What it means for your product: Most AI MVPs need one well-designed pipeline, not a swarm of agents. Add complexity when your data and users prove you need it.
6. Zero trust for every model
Security is the reason many enterprises restrict which AI models their teams can use. Ishwar's answer is a zero-trust architecture, where no model, provider or service gets implicit trust:
Authenticate and authorise every service and every request
Encrypt data at rest and in transit
Give the LLM only the data it needs for the current query, nothing more
Audit-log every prompt, response and tool call
Use a small internal model to review those logs and flag privacy or security issues
What it means for your product: If you sell to businesses, security questions will come up in the first sales call. Building least-privilege data access and audit logging from day one is much cheaper than retrofitting it before an enterprise deal.
7. Design for two kinds of failure
Your AI product can fail in two ways: your system fails, or your AI provider fails. Ishwar keeps them separate:
Use a circuit breaker so an AI outage doesn't take down the whole product
Keep AI error handling separate from your core application errors
If you depend heavily on LLMs, run two or three providers
Keep a small self-hosted model as a fallback
Avoid any single point of failure
What it means for your product: Every major AI provider has outages. A product that degrades gracefully, for example by falling back to a smaller model or a non-AI path, keeps its users. One that shows a blank screen loses them.
8. The real story: test coverage from 35% to 65% with two AI agents
On a previous team, Ishwar was asked to raise test coverage that sat around 35%. Engineers didn't want to write more tests, and a single AI agent produced tests so weak that reviewing them took longer than writing them by hand.
The fix was separating generation from review:
One agent writes the tests
A second, independent agent reviews them
The review feeds back into the first agent in a loop
Coverage climbed to roughly 65%, effectively the ceiling once config and HTML files were excluded.
What it means for your product: If one AI agent gives unreliable output, a second agent with a different job, checking and critiquing, often fixes it. This generator-reviewer pattern works for code, content, data extraction and document processing.
9. The prompting habit that saves the most time
Ishwar shared one tip he says changes how you use AI:
Give the model a role, such as a senior software architect
Ask it to look at the problem holistically and list what it needs to cover
Plan first, then execute
Run a code-critique pass at the end
Check the edge cases it proposes, because AI tends to over-engineer for safety
"If you invest five more minutes into the prompt itself, it's going to save you half an hour of review."
What it means for your product: The same discipline applies to the prompts inside your app. Structured system prompts, planning steps and self-review loops are part of the product, not an afterthought.
10. AI assists engineers, it doesn't replace them
Ishwar writes around 90% of his code with AI, but reviews every line: plan first, milestones second, code third. He believes the productivity gains are real, and he expects companies to recalibrate how much they spend on AI.
The biggest misconception, in his view, sits with leadership: the belief that AI can do everything by itself.
"There has to be a human in the workflow, for the review and even before the review."
What it means for your product: The best AI products keep humans in control of decisions that matter, with AI doing the heavy lifting around them. Design that review step into the workflow from the start.
Your production-readiness checklist
If you are building or scaling an AI product, here is the episode condensed into one checklist:
Area | Question to ask | Production-ready answer |
Business value | How do we measure success? | Clear metrics: time saved, errors reduced, retention |
Cost | What happens when usage grows 10x? | Token budgets, caching, semantic deduplication |
Architecture | Is it simple enough to debug? | Monolith first, scale when bottlenecks appear |
Agents | Does each agent earn its place? | Agents added for real needs, not anticipation |
Security | What data can the model see? | Zero trust, least privilege, full audit logs |
Failures | What if the AI provider goes down? | Circuit breakers, multiple providers, fallback model |
Quality | Who checks the AI's output? | Generator-reviewer loops and human review |
If several rows don't have a clear answer yet, your AI product is still a prototype, however good the demo looks.
Build a production-ready AI app with Codersarts
At Codersarts, we help founders, startups and enterprise teams take AI products from prototype to production. That includes LLM applications, AI agents, RAG systems and AI-powered developer tools that hold up with real users and real data.
We can help you:
Build a new AI app or MVP with production-ready architecture from day one
Harden an existing AI prototype: cost optimisation, caching, failure handling and monitoring
Design secure AI systems with least-privilege data access and audit logging
Build AI agents and multi-agent workflows, including generator-reviewer patterns
Add expert AI engineers to your team to work alongside your developers
Have an AI product that needs to work in production, not just in a demo?
Tell us what you're building: https://www.codersarts.com/contact
Explore our AI services: https://ai.codersarts.com
Frequently asked questions
What makes an LLM application reliable in production?
A reliable LLM application meets clear business metrics (time saved, errors reduced, user retention), controls costs with token budgets and caching, handles AI provider failures with circuit breakers and fallback models, and keeps humans in the loop for review.
How can I reduce LLM API costs?
Set token budgets per feature or user, cache responses aggressively, summarise repeated context instead of resending it, and add a semantic deduplication layer so near-identical queries are served from cache instead of calling the model again.
Should an AI app start with microservices?
Usually not. Start with a monolith, ship fast and validate the product. Move to microservices only when you hit real scaling bottlenecks, because distributed systems add significant debugging and operational complexity.
When do you need multiple AI agents?
When a single agent's output is unreliable or a task naturally splits into different roles. A common, effective pattern is one agent generating output and a second, independent agent reviewing it in a feedback loop.
How do you secure an AI application?
Use a zero-trust approach: authenticate and authorise every request, encrypt data at rest and in transit, give the model only the data it needs, and audit-log every prompt, response and tool call.
Can Codersarts build or fix my AI application?
Yes. Codersarts builds new AI apps and MVPs and helps teams turn AI prototypes into production-ready systems. Share your requirements at https://www.codersarts.com/contact.
Watch the full conversation with Ishwar on the Codersarts Podcast on YouTube. Want to be a guest? Email jitendra@codersarts.com with your name, role and the topic you'd like to discuss.
Comments