06-08, 11:00–11:45 (Europe/London), Doddington Forum
LLM outputs are non-deterministic, making it difficult to ensure reliability in production, especially in high-risk applications. In this talk, we’ll walk through a structured approach to making LLMs production-ready. We’ll cover setting up tests during experimentation, implementing real-time guardrails before responses reach users, and monitoring live performance for critical issues. Finally, we’ll discuss post-deployment log analysis to drive continuous improvements and build trust with stakeholders.
LLMs are transforming how we build applications, but their non-deterministic outputs and potential for hallucination create barriers for adoption in high-risk industries. In this talk will discuss a systematic approach to LLM application development that covers pre-production and experimentation phase, real-time guardrails for output validation and post analysis for identifying areas for improvement.
We’ll talk about:
- Creating comprehensive test sets with edge case coverage
- Unit tests for LLMs and establishing baseline metrics for reliability assessment
- Structured experimentation approaches for prompt optimization
- Real-time guardrails for output validation
- Live monitoring and alert systems
- Log analysis for pattern identification
We'll demonstrate practical implementations using Python libraries and monitoring tools, with real-world examples from production systems. The session will provide actionable insights for software developers, AI engineers and product managers looking to deploy LLM applications responsibly and gain stakeholder trust.
Attendees will leave with:
- A structured framework for LLM application development
- Practical code examples for implementing guardrails
- Strategies for continuous monitoring and improvement
This talk is suitable for intermediate practitioners who work with LLMs and need to ensure their reliable deployment in production environments.
Previous knowledge expected
Lena Shakurova is the founder of ParsLabs (https://parslabs.org), a Conversational AI agency, and Chatbotly (https://chatbotly.co), a no-code platform for building AI assistants trained on custom data.
At ParsLabs, she leads a team blending AI, user research and conversation science to design and develop high quality AI Conversations that sound human. She has background in NLP and Artificial intelligence and 7+ years of experience and 80+ successful projects building production-ready chatbots and voice assistants.
Lena focuses on ethical, user-first AI, leveraging her expertise in Linguistics & AI to create responsible, high-quality AI solutions. She shares insights on AI innovation and human-centered design through her blog (https://shakurova.io/blog) and LinkedIn (https://www.linkedin.com/in/lena-shakurova/).