Exploring the LangChain Agent Evals 101 Webinar Repository

The LangChain Agent Evals 101 webinar repository provides a hands-on learning environment featuring two small AI agents designed to help developers understand evaluation techniques using LangSmith. The repository includes code, configuration templates, and evaluation scripts intended for the free LangSmith Developer plan.

Repository Overview and Setup

To use the repository, developers configure environment variables using a provided template file. Essential configuration parameters include:

• LANGSMITH_API_KEY: Required to send traces, datasets, and experiments to LangSmith.
• OPENAI_API_KEY: Used to run the underlying agents and language model judges.
• Regional Endpoints: Specific configuration lines for accounts in EU or APAC regions.
• Dataset and Judge Settings: Variables defining dataset names and model choices.

Key Components and Exercises

The repository is structured around two primary applications, each containing intentional bugs to be discovered and resolved through tracing and evaluation.

1. Dinner Bot

Dinner Bot is an agent that suggests recipes based on available kitchen ingredients.

• The Bug: An expanded staples list mistakenly includes items like butter, garlic, and onions alongside standard staples such as salt, pepper, oil, and water.
• Evaluation Process: Developers inspect traces in LangSmith, upload saved traces, create a dataset, and run code-based and LLM-based evaluators to identify and fix the issue.

2. Recipe Remixer

Recipe Remixer modifies existing recipes to fit specific dietary requirements, such as vegetarian or dairy-free alternatives.

• The Bug: A logic error prevents correct ingredient swaps from reaching the text-rewriting function.
• Evaluation Process: Traces are uploaded and analyzed using custom evaluators and LLM judges to isolate where the data flow breaks down.

Practical Implications and Verification

These exercises demonstrate how developers can structure systematic evaluations for AI agents by combining code assertions and LLM-as-a-judge patterns. To verify the original setup and instructions, users can review the official repository on GitHub at `langchain-samples/agent-evals-101-webinar` and ensure they utilize appropriate API credentials securely on their local machines.