journal

Over the last couple of weeks, I took some time off for the holidays. During the down time I started working on a new project: prototyping a tool for evaluating generative AI systems. This experience had me thinking about how generative AI is changing software development, introducing entirely new workflows that don't quite fit into the conventional categories of development or testing.

A New Category of Work

As software professionals, we're familiar with the traditional boundaries between development and testing. Development typically involves writing code to implement features, while testing focuses on identifying bugs and ensuring requirements are met. However, working with generative AI systems introduces a bunch of new work that doesn't really fit with either of these labels.

This new work isn't really concerned with debugging code or refining feature requirements. Instead, it centers around a different set of challenges that are unique to LLM-powered systems:

• Defining precise measurement methodologies for system performance
• Establishing meaningful benchmarks
• Creating baseline measurements for system capabilities
• Developing nuanced acceptance criteria
• Managing iterative improvement cycles

The Shift from Code to Text

One of the most striking aspects of working with generative AI systems is how much of the development process revolves around text manipulation rather than code manipulation. While traditional software development primarily focuses on writing and refactoring code, development of these systems often involves:

• Crafting and refining prompts
• Analyzing system outputs
• Adjusting input parameters
• Fine-tuning language patterns
• Documenting edge cases and behavioral patterns

This shift in focus reminds me a lot of the work Continuous Improvement functions do in manufacturing settings. Just as manufacturing teams constantly measure, analyze, and optimize production processes, developers of generative AI systems must be continuously evaluating and refining their systems' outputs and behaviors.

The Need for Purpose-Built Evaluation Tools

As I discovered while building my prototype, effective evaluation of generative AI systems requires specialized tooling. While simple evaluation tools might suffice for basic prompt-response systems, the complexity increases dramatically when dealing with:

1. Multiple-shot prompting scenarios
2. Systems that utilize external tools
3. Complex agent architectures
4. Chain-of-thought reasoning patterns
5. Multi-step decision processes

Generic evaluation tools often fall short because they can't capture the nuanced behaviors and specific requirements of these more sophisticated systems. Each generative AI system may need its own specialized evaluation framework that understands its unique architecture and objectives.

Building and Evaluating

Perhaps the most important lesson from this project is that building generative AI systems and evaluating them are two sides of the same coin. You simply cannot do one effectively without the other. The science of evaluation is proving to be just as fascinating and complex as the development of the systems themselves.

This relationship is a classic feedback loop:

• Better evaluation tools lead to more precise understanding of system behavior
• Deeper understanding enables more targeted improvements
• More sophisticated systems drive the need for more advanced evaluation methods
• Enhanced evaluation techniques reveal new opportunities for system advancement

Looking Forward

As the field of generative AI continues to evolve, I expect we'll see the emergence of more specialized roles and tools focused specifically on this kind of system evaluation. We most likely will see the development of standardized evaluation frameworks and methodologies, similar to how software testing practices have evolved over time.

The challenge ahead lies in developing these evaluation practices while the technology itself is rapidly advancing. It's an exciting time to be working in this space, where we're not just building new technologies, but also creating new ways to understand and measure their capabilities.

The next frontier in generative AI development might not be just about building more powerful models, but about developing more sophisticated ways to evaluate and understand the systems we're creating. After all, you can't improve what you can't measure, and measuring generative AI systems effectively is proving to be an art and science in itself.