Building Reliable Autonomous Agentic AI Systems

Over the past few years, CTOs have been building LLM-based systems using a DAG workflow approach. Autonomous agentic systems are a different sport. Reliability is quickly becoming the dividing line between agentic AI that demos well and agentic AI you can trust in production.

As soon as an LLM can take actions, teams run into the same hard questions: How do we test it to make sure it’s producing reliable results? How do we keep tool calling safe? How do we prevent loops and runaway costs? How do we orchestrate long-running work with retries and recoverability? How do we evaluate agent behavior so we can improve it with confidence, not vibes?

This two-hour online mini-conference, open to guests, covered:

• A closed-loop evaluation walkthrough: how to build “plan, act, judge, revise” harnesses, curate ground truth, calibrate automated judges, and catch regressions before they hit production.
• The core orchestration patterns that make agents dependable: durable state, resumability, retries, and deterministic controls around nondeterministic models.
• How to evaluate agentic systems with evidence: task suites, ground truth, judges, and a repeatable iteration loop that improves reliability over time.
• The guardrails high-performing teams use to prevent loops, contain cost, and stop safely when the agent is uncertain or stuck.
• A moderated discussion among presenters and attendees on concrete design questions and tradeoffs from real deployments.

Speakers

• Sinan Ozdemir, founder of Crucible and author of Building Agentic AI
• Jared Zoneraich, founder and CEO of PromptLayer
• Ray Myers, legacy code expert and host of the Craft vs Cruft channel
• Ryan Doan, VP of Engineering at OutRival
• Tony Karrer (moderator), founder of TechEmpower and cofounder of the LA CTO Forum