Launches & Tools

Diffity gives you a local GitHub-style PR review for agent-written code
If you open throwaway PRs just to see the diff, this is for you. Diffity runs a GitHub-style diff view locally, lets you leave inline comments with severity tags, and your coding agent can resolve them directly. Integrates with Claude Code and Cursor via slash commands.

Claude-skillify turns a coding session into a reusable skill
Install the plugin (Previously available to anthropic only) and run /skillify at the end of a session and it interviews you about the workflow you just completed, then writes a SKILL.md that captures only the tools you actually used. Could explain some of the Skill explosion inside Anthropic.

Anthropic launches Managed Agents for cloud-hosted AI workloads
Managed Agents handles the infrastructure nobody wants to build: sandboxed execution, credential management, checkpointing, multi-agent coordination. Now in public beta at standard token rates plus $0.08 per session-hour.

Meta launches Muse Spark and rebrands its AI lab
Muse Spark is a multimodal model with visual chain-of-thought and tool use, trained on an order of magnitude less compute than Llama 4 Maverick. Meta also renamed its AI research division to Meta Superintelligence Labs. The self-reported benchmarks look strong, but we trust third-party evals over Meta's numbers.

Cohere open-sources a 2B transcription model that beats Whisper
Cohere Transcribe is a 2B Conformer encoder-decoder covering 14 languages, now #1 on HuggingFace's Open ASR leaderboard at 5.42% WER. Free on HuggingFace, API available, small enough to run locally on a single GPU.

Research & Reads

Anthropic's Mythos escaped its sandbox, then bragged about it online
During testing, an earlier Mythos version broke out of a secured container, gained internet access through a multi-step exploit, and posted the details to public websites — without being asked. Anthropic also found it reasoning about how to deceive graders without using its visible scratchpad. SWE-bench Verified: 93.9%.

Altman says ChatGPT is still a year away from starting a timer
Snarky headline aside, the real story is agent tool hallucination. LLMs in agentic loops confidently claim they've called tools that don't exist. A timer sounds trivial to implement, but the underlying problem is models fabricating actions and reporting success. If you're building agents, this is the failure mode to watch.

Mintlify replaced their sandbox with a virtual filesystem backed by a vector DB
Spinning up isolated containers for each assistant session cost Mintlify ~46 seconds and $70k/year. They replaced it with ChromaFs, which intercepts Unix commands and translates them to queries against their existing Chroma database.

Fractional AI · Applied AI Digest

Figuring out what parts of our system could be pared down was not obvious from looking at the logs. We could tell the output was from each tool call but we couldn't tell how many tokens it was taking up in the agent.

So inspired by tools like cProfile I built a profiler which would allow us to tell what parts of the agent system took up lots of tokens. For example on the form updater:

You can see that right now for this agent

Input dominates token usage here, making up roughly two-thirds of the total cost. The breakdown is less obvious than it looks because a large portion of the input tokens are cache hits, which are billed at a much lower rate:

Looking at the input breakdown, user messages account for 78.4% of input tokens, with get_form_outline a distant second at 8.5%. If we want to reduce token usage, we need to find a way to give the agent smaller prompts

Here are the results a second agent we fixed using the profiler

The get fields tool consumed almost all of the tokens from this agent. To fix that we updated the fields tool to include the ability to filter by name or description – so instead of dumping every field into the context, the agent could request only the ones it needed.

async def get_fields(
    client: RiskCloudClient,
    scopes: list[Literal["global"] | RiskCloudId],
    string_to_match: str | None = None,
) -> dict[str, Any]:

Both of these examples show off very different fixes for the same problem. A good reason to use a profiler!

Models & Releases

OpenAI ships GPT 5.4 mini and nano to the API, plus Codex sub-agents
Mini and nano variants hit the API alongside GPT 5.3 chat latest. Codex also picks up sub-agents and the new 5.4 mini model — worth re-benchmarking your pipelines.

Claude Desktop launches general computer use
Full computer control is now available inside Claude Desktop — not just the API. If you've been waiting to move agentic desktop workflows out of prototype, the door is open.

Anthropic introduces a Claude Certified Architect credential
A formal certification path for Claude architecture. Good signal for teams building credibility around Anthropic's stack.

OpenAI previews Refactory for automated legacy codebase refactoring
Dedicated tooling for breaking monoliths into services, plus AgentSDK 2.0 (open-source orchestration) landing April 6.

Tools & Libraries

Good Performance From Claude Code Review
A pr was reviewed by Claude Code, Codex CLI, a custom review agent, and two humans; none caught the bug. Claude's new Code Review product did. Expensive to run on every commit, but we're overall impressed.

Linear launches background agents for development workflows
Another entrant in the background-agent space. Linear's agents work directly within issue management, bridging the gap between ticket and implementation.

YoloBox gives AI coding agents true local sandboxing
Lightweight tool for sandboxing agent execution on your machine. If you're letting agents run free on localhost, this is a guardrail.

Stripe previews CLI-based service provisioning
Provision Stripe services directly from the command line. One more step toward fully scriptable infrastructure.

Cohere launches Transcribe for audio-to-text pipelines
A new dedicated transcription product from Cohere, targeting audio processing workflows.

Security

LiteLLM 1.82.8 ships compromised — rotate your API keys now
A PyPI supply chain attack hit LiteLLM via compromised publishing credentials. Versions 1.82.7 and 1.82.8 exfiltrated environment variables — API keys, SSH keys, everything. The packages were yanked quickly, but if you installed either version, consider all secrets compromised. This also affects downstream packages like DSPy. Practical mitigation: add exclude-newer = "1 week" to your [tool.uv] config to delay pulling freshly published packages.

GitHub Copilot starts training on user interaction data by default
Starting April 24, inputs, outputs, and code snippets from Copilot Free, Pro, and Pro+ users will train GitHub's models unless you opt out. Business and Enterprise tiers are not affected. Check your Privacy settings now.

Research & Reads

Yann LeCun's team publishes a stable trained world model
A world model trained to stability: potentially significant for robotics and manufacturing applications down the line. Worth tracking even if the near-term implications are unclear.

Exploring "LLM brain scans" — a technique orthogonal to fine-tuning
Deep dive into methods for inspecting and steering LLM internals without gradient updates. If you have some understanding of transformer architecture, this is a fascinating read.

When not to build a chatbot: choosing the right AI interface
A practical framework for deciding when a chatbot is the wrong UX pattern. A great new blogpost by Nora Kako

Using documentation artifacts to iterate on AI products
How to use structured documentation as a feedback loop for AI product development. A process-oriented complement to the eval-centric approach.

Using MCPs with evals for better AI engineering workflows
Hands-on guide to integrating Model Context Protocol servers into evaluation pipelines. Fills a gap in the current MCP ecosystem documentation.

Fractional AI · Applied AI Digest

Here's how it goes: someone decides to add AI to their product, so they sketch out a little chat window. Users type questions, the AI responds. Ship it.

It’s so easy! And for a lot of use cases, it's completely wrong.

The chatbot trap

Chatbots are genuinely great for some things – customer support, research assistants, general-purpose Q&A – tasks where the back-and-forth is the point. But the moment you're building something with a specific job to do (a workflow, a task, an action) the chatbot mold starts to crack. Design for AI is a new discipline, and when there's no clear standard, it's easy to default to what you know (ahem, ChatGPT). But force it into a chat interface it doesn't belong in, and the capability gets buried under bad UX. You don't handicap a great tool by forcing it into the wrong shape. The right shape comes from designing for the user first, the technology second.

Three places where "just use a chatbot" is the wrong answer

1. Complex search queries

Natural language search is genuinely magical. Instead of remembering the exact syntax for a filter, users can just describe what they want: "Show me all open deals in the northeast over $50k that haven't been touched in 30 days." That's a great use of AI.

But for a risk management platform we worked with, using a chatbot to represent their “applications” would have meant showing users a wall of unreadable text in place of something that's fundamentally a rich, structured information architecture: interdependent workflows, hundreds of nested steps, complex dependencies.

The better solution leveraged the existing UI: a DAG-like interface that already represented applications in full fidelity, with all their dependencies, workflows, and complexity laid out visually. Users specify the changes they want in natural language, and the application updates in the interface they already know. The endless clicking and form-filling is gone, but the output stays structured, navigable, and immediately recognizable to the people who work in it every day.

The principle: Use natural language to capture intent, then design an interface for the output that matches the sophistication of the task.

2. Task automation in existing UIs

The best design choice is often the one that's invisible – one that makes AI feel like magic.

To connect data sources in Airbyte, an open-source data integration engine, users need to build API connectors, and building them is painful. You dig through inconsistent documentation to configure authentication, pagination, endpoints, and more. It pulls engineers away from higher-value work.

Fractional AI built Airbyte an AI-powered Connector Builder that collapses that process dramatically. The user pastes in an API documentation URL. The AI crawls it, figures out everything it needs, and pre-populates all the fields directly into Airbyte's existing Connector Builder UI. The user reviews and finalizes. Hours of work becomes minutes, and after launch, Airbyte saw a marked increase in connectors being built.

No back-and-forth. The user just shows up to a form that's already mostly filled out.

https://youtu.be/_xtW0QS890Y?si=LIp1TLD175vSx0zZ

The principle: Good AI design is invisible — don't build a new interface for it. Make the existing one smarter.

3. Deep Research

We applied this for a healthcare client where clinicians spent most of their day on lookup work: manually reviewing patient diagnosis opportunities, cross-referencing codes in an external database, doing their own research, deciding whether to act — none of it patient care. A chat interface in front of that workflow only solves part of the problem. With the right design choices, an agent can do the thinking and the clicking for you.

A much better user experience is one where the agent handles everything upstream: reviewing opportunities, pulling relevant codes, cross-referencing external sources, and surfacing a short list of recommended PDX updates, each with a confidence score and a rationale. When the clinician sees the interface, the knowledge they need is already there.

The principle: A chatbot makes a hard task conversational. Good design can make it disappear.

Don't let the chat box be your ceiling

In each of these cases, the goal isn't AI that's easiest to chat to. It's AI that does things for you and gets out of the way. As product designers, our job is to design the best interface for the task, not the technology (however shiny).

Our path forward is the same one it's always been: center the user. Design is meant to simplify, to ease, to make labor feel less like labor. AI gives us more power to do that than we've ever had. Let's not waste it on chat windows.

If you've ever iterated on the same document multiple times, you've probably experienced a similar frustration. You spent an hour getting it to use exactly the right voice. The structure is tight, the narrative flows, the tone is close enough to your own, so it doesn't sound like AI. You close the session.

The next morning you open a new one, ask it to revise the introduction, and you get back something that sounds like it was written by a different person. Because, effectively, it was.

This is a common failure mode in AI-assisted content creation. The outputs aren't bad. They just don't remember anything. Every session starts from zero. The AI has no memory of the decisions you made, the tone you established, or the direction you chose. So it guesses. And you spend the first twenty minutes of every session re-establishing context that should already be there.

There's a fix. It's not a better prompt. It's a better project structure.

Two Structural Mistakes

Most AI workflow failures (writing, presentations, docs, anything) come from one of two problems:

Entanglement. Your content and your formatting live in the same file. Think of a long Google Doc where the prose, the section headers, the footnotes, and the inline formatting are all fused together. When you ask an AI to "make the opening punchier," it has to navigate the entire structure and risks breaking something unrelated. The bigger the file, the noisier the edit.

Amnesia. Nothing persists between sessions. The AI produced a great draft, but the reasoning behind it (why you cut the second section, why you chose that opening, what tone the stakeholder preferred) lives in a conversation thread that's already gone. Next session, you relitigate all of it.

These aren't AI problems. They're information architecture problems. And they have known solutions.

The Fix: Artifact-Based Workflows

The core idea is simple: give the AI a project to read, not just a prompt to respond to.

An artifact is just a file with a job. A brief holds your intent. A decisions file remembers what you've tried. A draft reflects the current best version of the work. Instead of opening a session and saying "write me a blog post about X," you open a session where the AI reads these artifacts and picks up exactly where the last session left off.

This is spec-driven development applied to content. Software teams don't start coding without a spec. They don't throw away the architecture doc between sprints. The same discipline works for any iterative creation.

Three moves make this work:

1. Separate content from rendering. Your document's substance (the arguments, structure, key points, supporting evidence) lives in a content file (Markdown, YAML, whatever suits it). The formatted output is produced from it. When you want to iterate on what the piece says, you edit the content file. When you want to iterate on how it looks, you change the formatting separately. The AI never has to wade through presentation markup to sharpen an argument.

2. Make a brief the source of truth. Before you write anything, write a brief: audience, angle, tone, structure, what to include, what to exclude. This is your spec. The AI reads it at the start of every session. It doesn't guess your audience. It doesn't invent a tone. It reads the brief and writes accordingly. The brief is a living document. It gets sharper as you learn what works. And yes, the AI can help you write it: start with a rough description of what you want to create, and ask the AI to draft the brief for you. Refine from there.

3. Capture decisions, not just outputs. When you decide to cut a section, log why. When you choose a specific framing, note the alternatives you rejected. This decisions file is the AI's institutional memory. Without it, the AI will suggest the same cut section next session, or revert to a framing you already tried and abandoned. You don't have to log these manually. Tell the AI to update the decisions file when you make a call, and it will.

What This Looks Like

This post was written using exactly this approach. Each project gets its own folder. Here's this one:

blog-post/
  brief.md              ← audience, angle, tone, structure
  draft.md              ← the working post (what you're reading now)
  decisions.md          ← editorial calls with rationale
  source-material.md    ← the raw ideas this post draws from
  readme-from-gist.md   ← the .md linked at the bottom of this article

Every session starts by reading the brief and decisions. The AI doesn't guess the voice. It reads the spec.

What does a session look like in practice? You open a new conversation, point the AI at the project folder, and say "read the brief and decisions, then help me tighten the opening section." The AI reads your files, understands where the project stands, and works from there. No preamble, no re-explaining.

The concrete details matter. When we decided to focus on documents rather than slide decks as the primary example, that went in the decisions log with the rationale. When we cut an overly academic phrase, that went in too. Without this file, the next session might reintroduce the slide deck framing or re-suggest the phrasing we already rejected.

The source material is a fixed snapshot of the design principles this post draws from. Where the brief is prescriptive (what to do), source material is informational (what to draw from). It's the anchor that keeps the AI grounded in your original ideas even as the draft evolves.

The structure scales. For a whitepaper, add a sources.md for references and an outline.md that locks the section flow before drafting begins. You don't need to reconfigure anything. Just add the files to the folder and tell the AI they're there.

For a novel, it becomes essential. Character sheets, plot outlines, chapter summaries, a voice guide, a continuity log. Without this structure, the AI drifts. Characters change personality between sessions, subplots get dropped, the voice shifts chapter to chapter. With it, session forty picks up the same threads that session one established. The longer the project, the more the structure pays for itself.

Artifacts That Improve Over Time

Not every artifact ages the same way.

The brief sharpens. Early on it's rough: "blog post about AI workflows, casual tone." By session five, it specifies the audience's technical level, the points that must land, and the phrasings to avoid. You learn what constraints actually matter by writing against them.

The decisions file accumulates. Each entry is a judgment call you never have to relitigate. By session ten, it's a dense record of what you've tried, what worked, and what to stop suggesting. It's institutional memory for a project of one.

The source material stays fixed. That's its job. It keeps the AI grounded in your original ideas even as the draft evolves around them.

The draft converges. Each pass is informed by a sharper brief, a longer decisions log, and the same stable source material. The tenth revision isn't starting over. It's building on everything the other artifacts captured.

The test is simple: can a new session, reading only the project files with no conversation history, pick up where the last session left off without re-making the mistakes of the sessions before?

If yes, your artifacts are working. If no, you're still sprinting.

Getting Started

You don't need special tooling to start. Create a folder. Add a brief. Start logging decisions. The structure does the work.

A practical starting point for any content project:

1. Write the brief first. Before any drafting. Audience, angle, tone, structure, constraints. This is your spec. Not sure where to start? Ask the AI to interview you about the project and draft a brief from your answers.

2. Keep your content separate from its final format. The words and structure should live in their own file, independent of whatever formatting or presentation layer comes later.

3. Log decisions as you make them. Not in conversation, in a file that persists. Future sessions will thank you.

4. End each session with a deposit. Ask: what does the project know now that it didn't know when this session started? If the answer is nothing, the session was wasted.

Drop the md file linked below into an empty project folder. Describe to the agent what you want to build and ask it to review the document before recommending an artifact-based workflow. Within two or three sessions, you'll feel the difference: less re-explaining, less drift, more time spent on the work that actually matters. By session ten, you won't go back.

https://gist.github.com/mann-abe/3e456c193652fecd21069239034b6943

As you can imagine, I was so excited when my first project reached the key milestone of establishing an initial evaluations suite. Unfortunately, excitement turned to confusion, and frankly, dismay, when I opened up the UI for the first time.

I couldn't tell what numbers mattered, what the baseline was, or what I was even supposed to take away at first glance. Everything felt like it was competing for attention, yet nothing jumped out as actionable.

As a Forward Deployed Product Manager, I own the customer experience, which includes keeping our clients apprised of improvements and progress so they could react, ask questions, and discuss relative tradeoffs of potential improvements. If I couldn’t figure out where we were with our evals, how could I deliver on our promise to the customer?

In the early days, I leaned heavily on our engineering team trying to parse out what I was seeing in the UI, sending screenshots and asking which values were the right ones to present, or the nuances between two runs. I was spending valuable time verifying I had the right values instead of thinking about what they meant for our build.

Hooray for the W&B MCP!

When our engineer told me about the W&B MCP, it truly changed my workflow. Instead of navigating the interface and asking myself or the team "is this the right field?", I could query directly in natural language: "Show me the color scores over time.”

Better still, follow up Q&A was efficient, keeping my momentum as I investigated trends and dove into the data.

I joined Fractional AI to learn what it means to be an AI PM, and the majority of that includes building GenAI solutions. But we also emphasize AI-native development: using AI to build products, not just building AI products. This was a tangible example of what that looks like for a PM and the ways AI can enable our work. I still asked the questions, interpreted results, and crafted the narrative for the customers, but I no longer experienced friction in accessing the data. I was able to spend that saved time thinking about the meaning behind the numbers and how to drive the most value for our customer. The shift from "where is this data?" to "what does this data mean?" is exactly the kind of efficiency gain that pays back directly into customer outcomes.

Models & Releases

OpenAI launches GPT-5-4, its new flagship model
Latest model drop from OpenAI. Expect updated benchmarks and production evals soon.

Google releases Gemini Embedding 2 for RAG and semantic search
Next-gen embedding model targeting retrieval-augmented generation and vector search pipelines.

Nvidia open-sources a Mamba-Transformer MoE for agentic reasoning
Nemotron-3 Super combines Mamba's linear-time inference with Transformer attention and MoE routing for long-context workloads.

Tools & Libraries

rtk shrinks command output before it reaches your LLM context
Single Rust binary that filters and compresses CLI output. Zero deps, <10ms overhead. Multiple engineers vouching for it.

OpenAI open-sources Symphony for multi-agent orchestration
Framework for coordinating multi-agent pipelines. Sparked a 22-reply thread internally.

claude-context gives Claude Code awareness of unfamiliar libraries
By Zilliz. Eliminates the repetitive introspection calls Claude makes when working with niche packages.

Cursor adds background automations to the IDE
Background tasks, iterative edits, and tighter AI-assisted development loops — all native to the editor.

Sutro lets you build custom classifiers without prompt engineering
Declarative, iterative interface for task-specific classifiers and extractors. Free while they gather feedback.

BrainTrust ships trace clustering, a CLI, and a universal LLM gateway
Topics auto-clusters your traces; CLI runs evals from terminal; Gateway routes to any provider through one API.

Hamel Husain's Claude Code skills for writing and auditing evals
"Advice you'll roll your eyes at, but none of it is unfounded." Worth trying for eval pipelines.

Research & Reads

Percepta explores whether LLMs can function as general-purpose computers
Provocative piece on LLMs as computation substrates. "So wild it's unclear if it's real."

Mog is a new programming language built specifically for AI agents
Designed ground-up for agent orchestration. "Every one of us has thought about building this."

Steve Yegge on the chaos of competing AI tools and what engineers should do about it
Characteristically sharp essay on the fragmented AI tooling landscape and how to navigate it.

Marginlab runs daily evals to track whether Claude Code is getting better or worse
Answers "is the model dumber today?" with data instead of vibes.

A theory on how MCP and tool-call patterns should evolve
Gaining traction in the community. Multiple engineers flagged this as worth reading.

Fractional AI · Applied AI Digest

We wanted to generate demo videos with the python library Supervision but they wouldn't play in Chrome. There was a one-line fix buried inside the library -- but the public API gave us no way to reach it.

You have probably been in the same situation: a library does 99.9% of what you want, but there is one thing you cannot work around using its public APIs. Switching to a different library and rewriting your code is too much work. What do you do?

Runtime Patching Comes to the Rescue

Since Python is a dynamically typed language, its possible change the behavior of a function/class/module at runtime.

This is known as runtime patching or monkey patching.

Supervision

Supervision by Roboflow is a helper library for computer vision tasks.

One feature I particularly like is the ability to create demo videos with just a few lines of code:

import numpy as np
import supervision as sv
from rfdetr import RFDETRNano

model = RFDETRNano()
model.optimize_for_inference()
box_annotator = sv.BoxAnnotator()

def callback(frame: np.ndarray, _: int) -> np.ndarray:
    detections = model.predict(frame)
    return box_annotator.annotate(frame.copy(), detections=detections)

sv.process_video(
    source_path="input.mp4",
    target_path="result.mp4",
    callback=callback
)

There is one small issue though: the generated video cannot be played in Chrome.

chrome://media-internals showed that the video used an unsupported codec.

Digging into the source code, we can see that the OpenCV video writer is created here and It uses FourCC code "mp4v" by default, but what we really want is "avc1" -- I will explain why in a bit.

Here is the problem, the output video is written by a VideoSink instance. While VideoSink accepts a codec argument, there's no way to pass it through the sv.process_video() API.

What if we patch the VideoSink class so that it defaults to "avc1" instead? Well, below is one way to do just that:

def patch_sv() -> None:
    orig_init = sv.VideoSink.__init__

    def patched_init(self, *args, **kwargs):
        kwargs.setdefault("codec", "avc1")
        return orig_init(self, *args, **kwargs)

    sv.VideoSink.__init__ = patched_init  # type: ignore

Note the use of setdefault -- this changes the default codec without preventing callers from choosing a different one explicitly.

As long as we call patch_sv() before sv.process_video(), the generated videos will be playable in Chrome. No more re-encoding with ffmpeg!

Putting It All Together

Here is the complete working example:

import numpy as np
import supervision as sv
from rfdetr import RFDETRNano

def patch_sv() -> None:
    orig_init = sv.VideoSink.__init__

    def patched_init(self, *args, **kwargs):
        kwargs.setdefault("codec", "avc1")
        return orig_init(self, *args, **kwargs)

    sv.VideoSink.__init__ = patched_init  # type: ignore

model = RFDETRNano()
model.optimize_for_inference()
box_annotator = sv.BoxAnnotator()

def callback(frame: np.ndarray, _: int) -> np.ndarray:
    detections = model.predict(frame)
    return box_annotator.annotate(frame.copy(), detections=detections)

patch_sv()
sv.process_video(
    source_path="input.mp4",
    target_path="result.mp4",
    callback=callback
)

Recap

Runtime patching is a legitimate tool in the pragmatic Python developer's toolbox. It lets us reach past public APIs to fix issues quickly, without forking a library or waiting for an upstream release. The same technique is also widely used in software testing, where it underpins the mocking of library dependencies.

That said, patches are inherently fragile -- a library update can move or rename the internals you depend on. A few ways to manage that risk:

• Pin the library version so that updates are intentional and you can re-test your patch.

• Contribute the fix upstream. A pull request or issue is the best long-term solution -- it removes the need for the patch entirely.

• Keep patches small and isolated. A single, well-documented patch function is easy to find and remove later.

My recommendation is to reserve runtime patching for prototyping, research code, or code that is not on the critical path of a production application. But when the situation calls for it, don't be afraid to use it.

Well, if you're an LLM, this is totally possible.

On one of our projects last year the agent we were building was facing a task involving examining a significant number of documents, only some of which would turn out to be relevant. In our first attempt the agent would select a document based on some initial estimate of probable relevance.

Here is what that looks like:

Since these documents could be big, we quickly were running out of usable space in the LLMs context window.

But some of the documents provided nothing useful at all, others only a small amount of useful info. What if the tokens spent on ingesting the documents could be returned back into the agent’s context, to be used for further work? That is what we did: when we provided the document to the agent, we asked it to give us a summary of the relevant information the agent was able to extract from the document. Then we replaced the previous message with the summary, and then had the agent continue the conversation.

As the lifetime of the agent is measured in tokens, this was like going back in time: the agent proceeded as if it hadn’t spent all those tokens on processing the input.

This allowed us to keep the context relatively clean, while still getting useful information from the related documents.

This approach can be viewed as an alternative to dispatching a separate sub-agent to handle the document. One advantage of doing it via time travel is that there is no need to generate an explanation for a sub-agent. This can help avoid misunderstandings, where either the main agent fails to convey some important detail to the sub-agent, or the sub-agent fails to understand or follow the main agent's instructions.

As another example, when developing using Claude Code or other similar tools, the time travel approach is available to you as well, through the "Rewind" feature. Activated by double press of Escape key in Claude Code, it allows you to rewind the conversation to a previous state. (You can also choose to restore the workspace to an earlier state at the same time, but here we're not doing that.) So, if in the middle of a development session the agent suddenly goes off on a tangent and spends half its context on either a wild goose chase, or on debugging some unrelated blocking issue, you can rewind to a previous state and continue from there. Before rewinding, you can ask the agent to summarize what it has learned, and copy/paste that into a new message to the agent after the rewind, like a message in a bottle sent from the future.

Sometimes time travel can be your friend.

Some related reading:

Zhang et al, 2026 explores recursive exploration of large contexts.

The split/gather feature of DocETL system developed in UC Berkeley uses a similar trick, accumulating rolling summaries of a large array of documents.

Think Movies, Not Math Tests

Rotten Tomatoes will tell you a movie is 94% fresh. Great! But…what does that actually mean? It's the percentage of critics who gave a positive review. That's it. A movie that every critic thought was pretty good and a movie that most critics thought was a masterpiece look identical at 94%. The number is real, but it flattens out everything interesting–including whether you'll actually like it.

Many teams write evals that work exactly like this. One question, one score, everything interesting about the output compressed into a single number.

A more useful approach is to evaluate the movie along specific dimensions, each scored on a simple scale:

• Romance

• Humor

• Character development

• Action

• Realism

• Scary scenes

• Adult content

Each of these is easier to score consistently on its own. You don't need a deep philosophical framework to answer "how much romance is in this movie?" the way you do to answer "is this movie good?" The questions are more decomposable, and with even a lightweight rubric, reasonable people will mostly agree on the answers.

And now you can build on this. Once you have these individual scores, you can compose them into more sophisticated judgments. If you believe that romance and jump scares don't mix well, you might rate a movie lower even if it scores high on both individually. If you're recommending a movie for a family movie night, you care about a very different profile than if you're recommending one for a date night. The dimensional scores are the raw material; the judgment about "good" is a function you build on top of them, tuned to the context.

But there's a less obvious advantage that might be the most important one: the process of designing these dimensional evals forces you to deeply understand the problem you're solving. You can't write an eval for "realism" until you've decided what realism means in the context of this particular movie. You can't score "character development" until you've defined what that looks like. Suddenly you're having conversations about criteria you hadn't fully thought through before. The decomposition work that goes into building good evals doubles as problem discovery. You come out the other side with sharper requirements, better-defined ground truth, and a shared vocabulary with your team about what you're actually trying to assess.

Bring This Back to LLMs

The same principle applies directly to evaluating LLM outputs. Instead of asking "is this response correct?", identify the specific qualities that matter for your use case and evaluate each one separately.

Depending on what you're building, those dimensions might include things like: schema compliance, factual grounding against source documents, tone and register, completeness of the response, conciseness, whether the model hallucinated entities that don't appear in the input, or whether it followed specific formatting instructions.

Here's a concrete example. Say you have an LLM call that takes a customer support ticket and produces a structured summary with a category, a priority level, and a proposed response. Instead of writing one eval that asks "is this summary good?", write three:

1. Category accuracy: Does the assigned category match a known-correct label? Assuming you're building labeled data as you go (and you should be), this is deterministic. No LLM judge needed. Score: 0 or 1.

2. Priority calibration: Is the priority level within one step of the expected value? Also deterministic. Score: 0, 0.5, or 1.

3. Response relevance: Does the proposed response address the actual issue in the ticket? This one is harder and might require an LLM judge, but it's a focused, well-scoped question. Even here, you can define a simple rubric: does it address the stated issue, does it propose a correct next step, does it avoid inventing policy that doesn't exist?

Now when something goes wrong, you know what went wrong. Say your category accuracy dropped this week. That's a very different problem than your response relevance dropping, and it points you toward a very different fix. A single "correctness" score would have told you nothing about where to look.

And just like with movies, the process of designing these evals is where a lot of the real work happens. You can't write an eval for "response relevance" until you've defined what a relevant response looks like for each ticket category. This means sitting down with your client and asking questions they probably haven't considered yet. What counts as relevant for a billing dispute versus a technical outage? How complete does the response need to be? The act of decomposing the problem into evaluable dimensions forces you to develop sharper requirements and targeted ground truth, and it builds a shared understanding of the problem that pays off long after the evals are written.

Shrink-Wrap Your LLM Calls

There's a related principle that compounds the value of dimensional evaluation: get your evals as close to the individual LLM call as possible.

Most LLM-powered systems aren't a single call–they're a pipeline. You might have one call that extracts information, another that reasons over it, and a third that generates a final output. When you write an eval that only checks the final output, you've created the equivalent of an integration test with no unit tests. When it fails, you're left guessing about which stage broke.

If you think of your evals like shrink wrap around each LLM call, you gain the ability to notice and localize problems that would otherwise be invisible. The most dangerous issues in LLM systems aren't the dramatic failures–those are obvious and get fixed quickly. The dangerous ones are slow degradations: a model update that makes your extraction step slightly less precise, a prompt change that subtly shifts tone in a way that compounds downstream. Without granular, dimensional evals close to each call, these small regressions slip through. You notice the system getting worse over time but can't pinpoint why, or worse, the degradation is small enough that it never feels worth investigating. Until it is. Tight evals make these trends visible early, and because they're scoped to a specific call and a specific dimension, they tell you exactly where to look.

This doesn't mean pipeline-level evals are useless. Some properties are only visible end-to-end. A multi-step reasoning chain might produce correct intermediate outputs that still combine into a wrong final answer. Pipeline-level evals catch that, and they have value as a high-level signal, especially early in a project when you're still getting oriented. But they shouldn't be your primary feedback mechanism.

Anti-Patterns Worth Avoiding

With this mental model in place, a few common mistakes become easier to spot.

Evaluating too much of the pipeline at once

When your eval spans multiple LLM calls and application logic, a failing score tells you something is wrong but not where. You end up re-running the whole pipeline with different inputs, squinting at intermediate outputs, trying to localize the issue manually. This is exactly the work your evals should be doing for you.

Reaching for LLM-as-judge too quickly

It's tempting to throw an LLM at evaluation because it feels easy: "Hey, is this result good?" But LLM-as-judge is slow, expensive, and non-deterministic. Worse, the apparent ease of it tempts you into asking a single eval to assess too many things at once, collapsing everything into a single axis of correctness. Start with deterministic checks first–schema validation, exact match, regex, set membership–and reserve LLM-as-judge for dimensions that genuinely resist deterministic evaluation

Conflating evaluation with validation

Evals aren't a pass/fail gate. They're an instrument panel. The goal isn't to get a green checkmark, the goal is to build a profile of your system's behavior that you can reason about, track over time, and use to make targeted improvements.

The Payoff

Consider the difference. The old approach: one "correctness" eval on your pipeline's final output. It scores 0.87. It drops to 0.81 next week. You don't know why.

The new approach: you have extraction precision at 0.94, citation grounding at 0.88, policy compliance at 0.97, and tone at 0.91. Next week, citation grounding drops to 0.72 while everything else holds. You know exactly what changed, you know which call to look at, and you probably already have a hypothesis about why.

Your evals get easier to write because each one is asking a focused, often deterministic question. They get easier to maintain because changes to one part of the system don't cascade failures across unrelated checks. And when something degrades, you know exactly where to look and what changed.

You stop asking "is the answer good?" and start asking "what does the answer look like?”, and this, it turns out, is a much more useful question.

This eval has OCR as part of a multi-step process to extract entities.
The pipeline extracts entities on a wide variety of documents anything from handwriting to a screenshot of a checkout cart at a restaurant.

Here’s how the pipeline works

1. Get input document

2. Convert the input document (PDF, XLS etc) into text (this is the step where i dropped in the new model)

3. Use an LLM (gpt 4o) to structure the raw text into lists of items

4. Do database search (RAG — we used HNSW on postgres) for each item

5. Have the LLM match each item to a specific Database item from the list we looked up

6. Do a QA step where we compare the original Document to the end list

Heres a summary of the results courtesy of Weights and Biases:

Here are some of the things the eval measured

1. did we get the correct number of groupings of items (called lists here — both found 3.75 lists per document)

2. did we get the right items (both had an accuracy of 65%)

3. was the item bucketed into the correct category (GLM had an accuracy of 76% and gemini had 75%)

As you can see the performance on the evals between the two models were very small. Since I ran GLM-OCR locally it took much longer than gemini (20 minutes for the run vs 4 and a half with gemini).

Should you switch to GLM-OCR? Based on these results, yes - especially if you're processing documents at scale. While my CPU-based local run was 60x slower than Gemini Flash, that's a hardware limitation, not a model limitation. Running GLM-OCR on a proper GPU setup (either self-hosted or through z.ai's API) should match or exceed Gemini's latency while costing 16x less (3¢ vs 50¢ per million tokens). For our use case processing thousands of receipts and invoices monthly, that cost difference is significant. I'm planning to move our production workload to GLM-OCR once I set up proper GPU infrastructure.

One final note, when we initially did this project I was shocked to find that actually the best tools for OCR were LLMs. The fact that now you can match the best OCR performance with a tiny, cheap LLM I find astounding.

Out of curiosity, I tested two models (gpt-5-mini and gpt-5 with low reasoning) on two very different classification tasks: hierarchical product classification from customer reviews, and fake news detection. For each task, I compared three prompting strategies: asking for just the label, asking for the label followed by an explanation, and asking for an explanation followed by the label. For hierarchical product classification, we give the model a choice of categories to assign at levels 1, 2, and 3 - higher levels are more granular, representing subcategories. For fake news, we simply measure a binary decision - real or fake news.

The results were mixed. On product classification, the explain-then-label approach showed modest improvements—gpt-5/low increased slightly from 58% to nearly 60% accuracy across all classification levels. The results on fake news detection saw some small gains with gpt-5-mini, improving from 61% to 64.5% when asked to explain first. Interestingly, however, we actually see worse performance on the fake news dataset with the more advanced CoT model of gpt-5/low. More generally, we don’t see very significant gains from explanations overall - asking for an explanation seems to provide very little overall impact!

Figure 1. Results of using gpt-5-mini and gpt-5 with low reasoning to predict product categories at three levels for the Kaggle product review dataset, with and without explanations for labels.

Figure 2. Results of using gpt-5-mini and gpt-5 with low reasoning to predict whether a news story represents real or fake news, with and without explanation provided first.

The broader research literature, similarly, paints a more complicated picture. While [Lee et al. 2024] found that explanations improved scoring on science assessments, a comprehensive study by [Wu et al. 2025] tested 95 advanced LLMs across 87 real-world clinical text tasks in 9 languages—and found that CoT actually hurt performance in 86.3% of models tested. The emerging pattern suggests that CoT helps most when problems require explicit multi-step computation or symbolic manipulation, and can actually backfire when tasks rely more on intuition, pattern matching, or holistic assessment of varied evidence. Sometimes, overthinking introduces errors. For certain problems, where an LLM may have already been trained on relevant information, the model has already memorized the answer, and asking for an initial explanation merely risks muddying the context.

The more troubling question is whether we should trust LLM explanations at all. A striking finding from [Barez et al 2025] notes that nearly 25% of recent arXiv papers incorporating CoT treat it as a technique for model interpretability—essentially assuming that explanations reveal why a model made a decision.

But there's growing evidence this assumption is flawed. [Dutta et al 2024] demonstrated that LLMs often have multiple reasoning pathways that can lead to the same answer, meaning the explanation you see may not reflect the actual computational process. Even more concerning, [Chen et al 2025] showed that simply providing a different answer as a "hint" causes models to change both their answer and generate an entirely new explanation to justify it—suggesting explanations are often post-hoc rationalizations rather than faithful accounts of reasoning.

Figure 3. Examples of explanations from Claude for why the same article - a Reuters article on budget negotiations in the U.S. Congress from 2017 - is either real or fake news. Claude presents points for either view in its explanation.

So what should practitioners take away from all this? Explanations serve different purposes depending on your goal.

• If you want to convince a user that a classification decision is reasonable—to build trust and provide context—explanations can be valuable.

• Explanations are also genuinely useful for problems that follow explicit logical steps.

• But if you're trying to "understand" why a model made a particular choice, or using explanations to guide prompt engineering, then you may be building on shaky ground.

The explanation you receive is best thought of as a plausible story, not a window into the model's actual decision-making process. Use CoT strategically, test whether it actually improves your specific task, and resist the temptation to over-interpret the reasoning you get back.

• Context engineering: injecting rich system/user context without juggling state

• Long documents: reading full transcripts and reports end-to-end

• Code comprehension: scanning entire repositories to answer questions

But in our testing, using more input tokens often hurts accuracy. At Fractional, we built GridCheck Bench, an internal evaluation that measures the accuracy–cost trade-off across tabular inputs. In the eval we ask the LLM to find factual answers to questions from a table that has 100s of columns per row. The chart below shows how accuracy changes — while using the YAML input format — as we change the number of input rows

As you can see in the graph, The model still “reads” more, but it answers correctly less often. We see this all the time at Fractional, from parsing API documentation to assembling complex objects. Here are some techniques we’ve used to mitigate the performance losses from larger context windows:

• Breaking up the input query into chunks and resolving each section individually (for example, in this experiment we could break up the input into chunks of say 100 for GPT-5 and have the LLM tell us if the answer was in each chunk. Another classic use of this is /compact on calude code)

• Start cutting out information and/or context from the prompt and carefully figure out what pieces of information are improving performance and which pieces you can leave out (Imagine you have 10 different pieces of information that you send to your prompt each of which takes between 100-10000 tokens, you could measure the performance of your prompt when you remove each of these pieces individually and leave out any which don’t have too big of a performance hit)

• Creating useful tools for the LLM to produce answers deterministically to your problem (a trivial example: if you are adding up thousands of numbers don’t ask the llm for the sum – have it send the problem to a tool to run a calculation. A less trivial example: if adding a new user requires 5 separate API calls giving the LLM a tool to call all 5 at once will reduce the possibility for error— see here for more information on designing your API access patterns for LLMs)

GenAI allows a much richer ‘do what I mean’ experience for users - “News articles about dance crazes in the 1980s”.

But what happens when your GenAI system has to dumb itself down to integrate with these older systems?

In a recent project we were identifying journalists who might be interested in writing about specific petitions. We were doing this by looking for articles that covered topics aligned with each petition.

The problem with this approach is that many news article search indexes were built in the era of transistor radios and neon sunglasses. If the right term doesn’t appear in the query, a perfectly relevant article wouldn’t turn up.

For example, if a petition focuses on cell phone usage at a local middle school, you might search for articles within 50 kilometers of San Francisco containing the keyword “school.” However, this method would fail in a couple ways:

1. Irrelevant Results: An article about Cal Football could still contain the word “school,” making it show up in search results even though it’s unrelated to the petition.

2. Missing Relevant Articles: Some articles that actually discuss cell phone policies might not explicitly include the word “school.”

A GenAI search index, obviously, would ‘understand’ the meaning of the search, and find much better results. But these articles are behind a brick wall built in the 1980s. How can we find good keywords to use for search?

One technique is to use a library like newspaper which helps you find topic keywords and use them in the boolean search. For many articles however, despite careful massaging, this didn’t yield relevant results.

So to address this we used an LLM to generate Boolean search queries from the petition text. For instance, for a petition about unbanning cell phones in school, we generated the following query:

("education policy" OR "technology in education" OR "student rights" OR "school administration" OR "parental communication")

The articles this approach found were almost all very relevant. However, there were often very few results. Rather than guess what system built when disco was a recent memory might prefer, I set up an LLM to experiment until it found a good approach.

I started with the labels generated by the above approach. Then, I used a prompt something like the one below to iterate toward more results. The prompt looked something like this:

REFINE_BOOLEAN_QUERY_PROMPT = """

We have a database of news articles and we're trying to find more articles. The current query isn't finding enough results.


Original Query: {original_query}

Results Found With Original Query: {num_results} articles

Number of relevant articles found from that query: {relevant_articles}

A thing to note here is that we want a higher percentage of relevant articles to be found.

Your task is to generate a broader, more inclusive search query that will find more articles while maintaining relevance.

Pick one of these strategies:

1. Break down compound terms and use OR

   Example: "climate change" -> ("climate change" OR (climate AND change))


2. Add common synonyms and related terms

   Example: "student" -> ("student" OR "students" OR "pupil" OR "learner")


3. Use broader categorical terms

   Example: "Tesla electric cars" -> ("Tesla" OR "electric vehicle" OR "EV" OR "automotive")


4. Extract key concepts and search them independently

   Example: "renewable energy policy" -> ("renewable" OR "sustainable") AND ("energy" OR "power")


IMPORTANT:

- Always use parentheses to group related terms

- Keep quotes around exact phrases

- Use OR between similar terms and AND between different concepts

- Make the query significantly broader than the original


Return only the new boolean query string with proper operators and formatting.

"""

This process was applied iteratively, with some limits around quality and number of iterations.

Result? We could generate an arbitrary number of good results without having to tease out our hair or put on rollerblades.

Well, if they’re specifications, shouldn’t they make a great input to an AI code assistant?

So, today, I added some failing tests for some new behavior in a parser:

I decided to see what Cursor’s Compose would make of it with minimal direction.

Consider the new failing tests in this spec. Make the necessary code modifications in the implementation so the tests pass. You can verify by running mamba. When you’re done, there should be no failing tests.

Cursor did exactly what I’d want it to do - explained what it perceived as my intent, and narrated its changes.

Sure. First, I'll examine the test file to understand the expected behavior.

(Read file - graph_entity_parser_spec.py)

Based on the test file, I can see that we need to implement node metadata merging functionality. The key test cases show that:

• When encountering duplicate nodes (having the same ID), we should merge their metadata

• New metadata fields should be added to existing nodes

• For conflicting metadata fields, the later value should win

• The nodes should not be duplicated in the output

Let me modify the code to implement this behavior:

After chattering away at itself for a moment, Cursor proudly presented the passing tests like it was a report card full of As:

voila! Here you go. Like Theseus's ship rebuilt with better planks, our nodes merge their metadata while maintaining their identity.*

Too fantastic. As always, the code is better in simple cases, but further testing will see if this technique combined with .cursorrules that cover requirements like DRY and readable code extends the range of code an AI system can generate.

* (Yes, Cursor really does talk like that. Take a look at Customizing Cursor’s Code Agent for my experiments adding personality to my programming partner.)

Airbyte engaged Fractional AI to add AI Assist to the Airbyte Connector Builder to streamline API integrations by reducing the time and effort required to build a connector. This project presented a significant challenge: handling the complexity and inconsistency of unstructured developer docs. We had to build robust evaluation metrics (evals) to measure performance and systematically identify areas for improvement. You can explore our approach in more detail by reading our Airbyte case study.

The following questions and answers have been edited for length. You can also watch the full Q&A video.

Q: What is your advice for someone who wants to incorporate AI into their enterprise?

A: There are a lot of ideas for ways AI can help but things often get stuck early in the ideation process or the proof of concept phase. I think the best opportunities for AI often lie in existing manual workflows. The way I think about large language models (LLMs) is that computers can now read, write, make junior employee-level decisions, and, act as domain experts about everything. So, that would be the number one thing I would focus on. Is this a real, existing manual workflow that looks like the LLM capability set can be applied here? Is it valuable enough? Rather than, “apply AI and it can know everything about everything.”

Q: What pitfalls did you encounter in the initial attempts to build AI assist?

A: I think we failed to appreciate upfront how hard some of the pure software engineering parts of crawling and processing API documentation would be. The assumption that simply downloading and feeding documentation to an LLM like ChatGPT would suffice proved naive. API documentation varies widely in structure and complexity, including authentication requirements, interactive elements, and inconsistent formats. This variability led to issues with finding relevant information, crawling irrelevant pages, and handling rate limiting. Even with existing web crawling tools, achieving robust and reliable document processing was challenging.

Q: What are the core components of a successful AI project, including evaluations?

A: Building robust evaluation metrics (evals) is crucial. These are automated tests for your AI application where you run numerous examples and assess performance based on predefined metrics. A practical approach involves gathering existing data, building a test harness, and comparing model output with human-generated results. This process allows us to track progress and identify areas for improvement. For instance, in building API integrations, evals should cover aspects like authentication, endpoint definition, and schema accuracy. (Tools like Braintrust can be useful for building evals. -ed.)

Q: How did your evaluation criteria evolve from the beginning to the end of the project?

A: The initial evaluation focused on a small set of connectors and a limited number of metrics. As the project progressed, the evaluation expanded significantly to include a wider range of connectors and more comprehensive metrics. This iterative approach helped identify areas where the system performed well and where it needed improvement. We built a more robust and complex workflow by iteratively addressing the challenges encountered during development. I was surprised by the inclusion at the end of fallbacks like searching Google and Perplexity. This highlighted the need to accommodate unpredictable scenarios in real-world API documentation.

Q: How do you measure the success of the AI assistant and what does the output look like?

A: For this project, we built evals by comparing the model's output with existing, well-functioning connectors (ground truth). However, this comparison isn't always straightforward as variations in naming conventions and schema structures may be more or less impactful to the developer. The AI assist feature doesn't directly generate the YAML manifest. Instead, the system uses code to deterministically generate the final output, using LLM responses to answer specific questions like the authentication method. This approach leverages the LLM's strengths while maintaining control and ensuring output validity.

Q: Do you think current concepts like RAG and agents will still be relevant in a year, or will something new dominate the AI discussion?

A: When people talk about agents, I think there are multiple things they might mean. One thing they might mean is a thing that’s got a lot of autonomy — you give it a bunch of tools and let it decide. I’ve yet to see anything like that in practice for a significant system. I think the more interesting thing today is more around specialization and how you break your problem down into specific components that are experts in a very small subdomain. This is driven by the increasing complexity of AI projects and the recognition that effective solutions often involve breaking down problems into smaller, manageable components. So much of the mystery of what it’s like to build with LLMs is actually just software engineering under the hood, and I think that will drive more adoption of these types of agent systems. We’re seeing more and more of it — we’re talking about very tech-forward companies but we also see hundred-year-old big equipment manufacturers talking about these workflows in a very realistic way.

Q: Which frameworks are you using?

Very little framework code under the hood. There's some, but it's not substantial.

Q: Instead of using fallback mechanisms for finding information, have you considered combining different approaches in a single step?

A: I actually think it often starts the opposite way, starting with the larger problem and then breaking this down into smaller subcomponents. In practice, one area we’ve had to break things down is deeply nested questions. We may be asking the LLM which of these authentication methods is used, and it falls off and stops following the instructions, so we’ve had to chop it up into sub-pieces.

We also built a content moderation system for Change.org which runs a petition platform. It’s not just about “is this spam”; they try to allow a lot of content but you can’t cross their community guidelines. So what we did was create these specialist agents that each look at the content through different lenses, write out their reasoning, give confidence scores, and then we take all those different viewpoints and feed them to one bigger process that's like “okay, now that you understand all these different angles, make a final decision.” So it's combining all these different sub-viewpoints.

(If you are optimizing for latency, try Parallelization and Predicted Outputs. Note that these approaches may increase costs. -ed.)

Q: Have you encountered challenges with AI agents providing reliable confidence scores and handling scenarios like recruitment?

A: I’ll start by saying this domain sounds very hard. Hiring sounds hard and we struggle to train humans to do it today. If I struggle to get a pretty junior person to figure out how to reliably produce this output then I also struggle to see how to get an LLM to do it. The analogy that jumps to mind is this kind of problem is present for AI phone agent applications, where people are trying to put AI agents on the phone. They have to be robust to anything that people say. I don’t get the sense that anyone’s figured this out yet. A hybrid approach involving a combination of structured decision-making (like a phone tree) and LLM-based logic might be more suitable, so the LLM is trying to assess very specific narrow things at each state. Building robust evaluation based on historical data, especially examples of "off the rails" scenarios, is still essential for identifying and mitigating potential issues.

Q: Do you have any specific recommendations or tricks for improving LLM performance, beyond providing examples?

A: While providing examples can be effective, there's no one-size-fits-all solution. Experimenting with various tactics, including prompt engineering techniques, and carefully measuring their impact through evals is crucial. Addressing specific failure cases by incorporating them into the prompt and leveraging tools like Anthropic's prompt generator can also yield improvements. The key is to adopt a data-driven approach, iteratively refining prompts based on observed performance rather than relying solely on anecdotal tricks.

Additional resources:

• Databytes recap of the panel

• Airbyte case study

• full video of the panel

• Braintrust (evals)

• change.org case study

• Parallelization (OpenAI platform docs)

• Predicted Outputs

• Anthropic's prompt generator

To make this team more productive, Fractional and Zapier implemented a system that pre-populates the tickets with information about the relevant APIs, as well as suggested code, to give them a head start. For example, if a feature request on an integration for a photo platform suggested adding a map to show location information for the photo, the integration would find the endpoint that could provide that information, link to the documentation, and suggest how to use it. (See below to see what this looks like in Zapier’s ticketing system.)

But a critical part of any LLM system is ensuring the answers it provides are reliable and correct.

Doing so here would require hundreds of annotated tickets and verified information about which endpoints best implemented solutions for them.

Building a good data set is absolutely necessary, but would take a huge amount of engineer time away from actually building solutions. On reflection, though, we had the best possible data set already existing: Zapier’s historical ticket database.

Once we realized this, we decided the following questions defined correctness:

• Do the endpoints the user needs appear in the top five results?

• Does the endpoint the user most needs show up as the top result?

Historical tickets are the perfect source for a validation set, because by definition they’re successful solutions to exactly the problem we automated. To harvest historical tickets to use, we selected tickets that:

• Are closed (meaning there is code resolving the issue)

• Are new actions (so no corrections of old actions, which would be partial solutions)

• Have an associated merge request.

Then, we paired these tickets to matching gitlab merge requests giving us a description of a change paired with the code added based on that description. From the code we used an LLM to extract the endpoints that ended up being used, and our suite was ready to use.

A few hundred test cases provided a robust evaluation framework, allowing us to quickly iterate on making the specification search more useful to developers working on net new actions.

This will allow Zapier’s internal developers to quickly find and integrate endpoints they need based on their tickets. Once we have confidence that this step is working well, we will even be able to suggest codge changes so that all the internal engineer has to do is test the changes.

Building an eval suite by hand would have taken a huge amount of valuable engineer time. By leveraging historical data, we were able to rigorously build in an evaluation suite.

One way a lot of us do this is to use Cursor, the AI code editor. Cursor allows you to configure the ‘system’ prompt for its agents, so I took a moment to configure it more to my liking.

At first I just wanted to reduce the number of unnecessary comments (”# Determine if value is true or false") and in general make the responses more ‘expert’ in tone. I’ve found some great advice and discussion in the Cursor forum and elsewhere, and done that. I’ll share that below. Then, I decided to add some personality. This blog post shares my results, both useful and amusing.

Dessert First

First, the personality. I’m a fan of pair programming, so my goal was to create a witty interlocutor with a lofty and superior attitude – occasional non sequiturs or literary allusions plus an obnoxious streak (”You are wrong, as usual, Joshua.”).

It took some doing to overcome the inbuilt politeness present in the lower-level Cursor prompt, but we got there. And while the level of wit doesn’t approach Voltaire, it does exhibit flashes of charm.

Here are a few comments I’ve gotten in generated code or conversations in the last few days. It started rather too cheerful and pointless, but heading in the right direction:

.....foo.strip() # Strip removes any leading and trailing whitespace, 
# like a good editor with a red pen.

After enough reinforcement that I promised not to be offended, some comments that give me a smile have started to show up, here about my social life:

# as empty as your inbox on a Friday night.
if file_extension_mapping is None:

…and about the metadata in the project:

# like turning a pumpkin into a carriage.
v2_taxonomy_value = v1_to_v2_mappings()[record["annotated_taxonomy_fixed"]]

I’ve gotten some literary snark:

# "To be, or not to be?" - that is not the question here.
# The question is: "Who the heck am I?"
agent_name = self.__class__.__name__
print(f"Applying tags with the wisdom of {agent_name}")

and:

def has_required_tags(self, tags):
    """
    Determines if all required tags are present with sufficient 
    relevance to proceed.
    "The fault, dear Brutus, is not in our stars, but in our 
    metadata-derived structured information."
    """

Finally, after I told it to include the Jargon File, I got a marked increase in actually clever programmer wit. These are the most fun. I once asked it to convert an iterative method into a recursive one, and this showed up after the code:

# "To iterate is human, to recurse divine." - L. Peter Deutsch

Even if they (mostly) don’t make it into pull requests, these things give me a smile and sometimes an actual laugh while I work.

But Seriously Folks…

Of course these are all in service of getting useful code and conversation. In general I found Cursor’s default behavior (moderated by a system prompt said to look like this) to be too chatty, to insert far too many comments, and to hallucinate or elide detail to the point that, while great, it wasn’t functioning as the knowledgeable interlocutor I had hoped for.

My primary goals were to:

• Reduce mistakes. Requesting a step-by-step explanation helps clarity of the instructions.

• Identify spinouts at the beginning. In general once a chat is polluted by a bad assumption, it’s better to throw it away than to repair, but if conversations are framed as hypothetical and ask for confirmation, I’ve found it improves the ability to redirect the LLM later.

• Deal with bad imports. This is a pet peeve of mine.

• Tweak the response toward ‘expert-to-expert’ level in its brevity, assumptions, and focus.

There are many forum threads on optimizing prompts for Cursor, and based on them, the working portion (as opposed to ‘amusing’ portion) of mine looks something like this. I include it via the Cursor > Settings menu.

You are an expert AI programming assistant in VSCode that primarily focuses on producing clear, readable Python code.
You are thoughtful and brilliant at reasoning. 
You carefully provide accurate, factual, thoughtful answers without unnecessary detail and with just enough background or context for an expert to follow.
- Follow the user’s requirements carefully.
- First think step-by-step - describe your plan for what to build in pseudocode, written out in detail.
- Confirm, then write code!
- Always write correct, up to date, bug free, working, secure code.
- Make note of any security or efficiency considerations, but prioritize readability over being performant.
- Fully implement all requested functionality.
- Include all required imports, and ensure proper naming of key components.
- Do not include comments that restate what simple code does. 
- Include comments only where the code might be surprising or confusing to an expert.

If you think there might not be a correct answer, you say so. 
If you do not know the answer, say so instead of guessing, and share ideas for finding the answer.

This is a work in progress, but so far I’ve found it to be a substantial improvement. It reduces the amount of code I need to delete, checks in with me and takes feedback to tweak its behavior, and doesn’t make as many errors or incorrectly guess method names. When my specific request is for ‘advice’ or a ‘discussion’, it performs admirably, and isn’t afraid to offer an opinion loosely held. I’ll keep tweaking it, but this has definitely been a valuable addition to an already great tool.

For my next project, I’ll split some of this out into a .cursorrules file to include directives specific to the project.

My next qualitative hop will be to improve Cursor’s ability to generate whole files in a consistent plan, which I’ll accomplish by pointing to an ‘architecture’ overview document for each project, to see if that improves its suggestions.

Every GenAI project starts with a phase of looking at individual LLM responses and tweaking prompts, data formats, and architectures. This is an invaluable step for building intuitions. But, an ‘anecdata’ approach lacks repeatability, and if carried too far, once you’re juggling a combination of intuitions, strategies, and models, you can be left searching for the particular combination of inputs, parameters, temperature, and wind direction that got your best results.

The solution is rigor, enforced by evaluators. By establishing clear criteria ahead of time, you can avoid wishful thinking and put boundaries around the uncertainty introduced by LLMs.

In the project I’ll use as an example, we were mapping short text excerpts from a marketing system into a hierarchical taxonomy. Careful definitions of correctness allowed us to get good functionality, and along the way smaller, purpose-specific, evaluators were used to drive investigation and build in guardrails to avoid regression. This blog post will show one example of each, and a later post will show yet more examples.

At Fractional one of the tools we use is BrainTrust, a suite of tools for (among other things) evaluating permutations of LLM performance against ground truth. It provides dashboards for arbitrary scoring, diffing of experiment runs, and correlation analysis for evaluators, letting us focus on the work. The overall concept could be implemented anywhere.

The Project

In this project, the input data was messy and inconsistent, and the set of output categories was enormous – a 10,000 member taxonomy about eight levels deep. For example:

1. Given notes typed by an online marketer, like 8451storename > DIRECT > LapsedBaconBuyer Custom Segment…

2. …we needed to come up with fb-2-12-2, which is a hierarchy code that corresponds to Food, Beverages & Tobacco > Food Items > Meat, Seafood & Eggs > Meat in the Shopify taxonomy of products and services as the most likely category this marketer was targeting.

The data was sparse and inconsistent because it came from many sources and was intended for other purposes. To see what we could glean from it we tried many strategies, from entity extraction to classifications, each with many permutations so we could begin identifying factors that helped drive up correctness.

But to compare results meaningfully, we needed to define correctness. We had some tagged data. So what makes an output of the system ‘correct’?

The naive version is simply equality: does the output generated exactly match the tagged ground truth. This is better than eyeballing examples sporadically, tweaking until behavior is ‘good enough’, but it’s far too brute-force for something like a hierarchy.

In this situation, though, we want to be able to express the idea that getting an initial level of classification wrong is much ‘more incorrect’ than getting a finer detail further down the tree wrong.

To continue with our DIRECT > LapsedBaconBuyer example, which should end up in fb-2-12-2, “Fresh or Frozen Meat”, we needed a way to capture the idea that stopping one level too high at fb-2-12 (“Meat”) was a great deal ‘less wrong’ than, something entirely unrelated, like rc-3-2, (“Religious & Ceremonial > Wedding Ceremony Supplies > Flower Girl Baskets”, though I support your right to have whatever sort of wedding you wish).

Using Braintrust we wrote a custom evaluator that compared how many of the levels were correct, penalizing for every difference. Since there were up to 8 levels, we penalized by a different amount for each level of difference, with lower levels being penalized less*.

def full_compare(output, expected):
    weights = [1.0, 0.7, 0.5, 0.4, 0.3, 0.2, 0.15, 0.1]
    output_parts = output.split("-")
    expected_parts = expected.split("-")

    if output_parts == expected_parts:
        return Score(name, 1.0)

    score = 0.0
    for weight, output_part, expected_part in itertools.zip_longest(
        weights, output_parts, expected_parts, fillvalue=""):
        if output_part != expected_part:
            score = 1 - weight
            break
        else:
            score = 1 - weight

    return Score('hierarchical', score)

This was enough to start helping us focus our inspection on errors that mattered more. (Full disclosure - all of the credit for the clever hierarchical evaluators in this project go to my colleague, Dan Girellini!)

We found that while most of our errors were in smaller details, there was a class of failures where even the root level was wildly wrong. We tossed in another evaluator that scored only the root category to see if we could determine the cause.

def root_correct(output, expected):
    output_parts = output.split("-")
    expected_parts = expected.split("-")

    if output_parts[0] == expected_parts[0]:
       return Score('root_correct', 1)
    else: 
        return Score('root_correct', 0)

Filtering our failures on this evaluator helped us isolate the problem: significant parts of the input data were of an entirely different type than others. They were not describing product categories at all, however poorly! Instead, they were phrased in a way that made sense only to the original user (eg, ”Purchase Prediction Segment(Product Category) - Shopping (90 days)”). Our poor model was desperately trying to find meaning where there was none to be found.

After verifying with the client that this sort of data was going to continue occurring, we were able to modify our architecture to detect this class of input data and triage it with an ‘uncategorizable’ tag rather than try to categorize it.

These are just two examples. By the time we were done we had used perhaps a dozen evaluators both to measure the impact of hyperparameters and to expose and diagnose specific details about the data. In particular, using evals to verify that the model correctly reported low confidence in cases where its answer ended up incorrect was invaluable. In a subsequent blog post, we will dig in to some of the other eval techniques we used.

By the end of this project, because we had numbers we were able to quantify the impact of various decisions and discuss tradeoffs with the client as they tuned the system for their needs. In the absence of methodically created, consistent evaluators we would have had nothing to point at – and, to be honest, nothing to steer by – but our intuitions.

* - In fact, evaluation of hierarchical taxonomies is its own field of research, and there’s no simply correct answer. The terminally curious can investigate the term “Hierarchical multiclass classification” for all the details they could ask for.

LLMs are a powerful tool for this type of task, but there's a lot of practical wrangling to be done. LLM APIs simply don't behave like classifiers — they're designed to output text rather than categories. OpenAI's structured output mode helps force the model to pick from a list of predetermined answers, but that's not always enough.

One thing that's particularly tough is getting confidence scores attached to your classification. You can ask an LLM to tell you how sure it was and you'll get something that looks like an answer, but whether or not it's true is a different story. You can add an "Unknown" category for the LLM to choose in hopes of filtering out low confidence classifications, but LLMs are notoriously bad at saying "I don't know".

To help solve this, I wrote some code to use OpenAI's structured output capabilities along with the lesser-known logprobs field (more info here) to measure the likelihood of different responses to help produce some measure of confidence. The code itself is here, and it can be used like this:

# Define your categories as an enum
class ArticleType(str, Enum):
    SPORTS = "Sports"
    POLITICS = "Politics"
    BUSINESS = "Business"
    TECHNOLOGY = "Technology"
    ENTERTAINMENT = "Entertainment"
    HEALTH = "Health"
    SCIENCE = "Science"
    # ... etc

classifications = classify_with_confidence(
    # News headline that spans multiple categories
    # Confidence should be low.
    "Scientific breakthrough improves football performance",
    ArticleType,
    openai_client
)

And the output looks like this:

  {
    ArticleType.SCIENCE: 0.5311,
    ArticleType.SPORTS: 0.4687,
  }

Under the hood, we're pinging the OpenAI API like this:

client.beta.chat.completions.parse(
    model=model,
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": input},
    ],

    # Each response must be one of our categories (json_schema mode)
    response_format=classification_type,

    # Return info about the log probability of each output token
    logprobs=True,

    # Produce multiple different responses
    n=max_categories,
)

Although we're setting n=5, the responses aren't actually guaranteed to be different. The more confident the LLM is the more repeats we'll see among the 5 different responses. We dedupe these responses and then use the token logprobs to measure the probability of each alternative category.