How AI’s Failures Unlock Your Hardest Workflows

Many people within enterprises find themselves caught in the cross-hairs of an agenda which says that going AI-first is existential, and a lived experience of seeing AI fail over and over again at their tasks.

The instinctual reaction is to find a way to apply AI only in the less risky areas. Just use it to search for documents, summarize emails, make code suggestions, gather content together.

The result of this is a lot of AI application but very little movement on margin.

What’s most scary is the idea of putting this tool that has a history of not being able to count how many letters are in a word, inside a workflow that drives sensitive data and huge portions of value for your company.

You’re not crazy for thinking it’s a bad idea to hand the wheel over to this alien that’s just learning your world.

Applying AI into your high value workflows shouldn’t look like this.

Here’s how it should look instead.

A Probabilistic to Deterministic Partnership

The mission of Red Finch is to show how ideas apply to real world use cases, not just talk about it in theory. For this case I’ll use a project I call “The Crate”, specifically its concert, venue, and artist scraping and management pipeline.

While this isn’t the world’s most sensitive data, the accuracy of it is very important to me, and the principles with which I’ve built it can be applied to many different domains.

There are three prereqs in this process:

With just those three things (that take a lot more work than it might appear on the surface) your next job is:

Inevitably the agent will do a bad job.

Don’t despair!

This was the plan all along.

This is the real work. Human in the Loop isn’t about verification, it’s about teaching. Another article on this topic will be up soon - subscribe so you don’t miss it. Your job now is to document where it went wrong and why.

Thus starts the cycle: Provide goal > provide context > provide guidance > agent fails > teach > repeat.

This teaching is the fulcrum of the process, and it’s not an exercise reserved for AI engineers, it boils down to two steps:

redfinch.dev: Train: the two questions, in the code ↗

1. How do I know this is wrong?
   What are the things about this result or the process to arrive at this result that stand out to me. This may be obvious (2+2 does not equal 5) or nuanced (I don’t like the way this UI looks).
2. How can I equip the agent with this information?
   This may be a tool or data set that I need to provide access to, or it may just be guidance on taste or preference. Ask yourself what you have access to that it doesn’t.

The outcome you’re looking for out of this is gradual and granular success of the goal yes, but also of the steps (for longer running workflows).

A Real Example

Let’s go back to The Crate example so you can see this in practice.

The build · redfinch.dev ↗How The Crate's concert pipeline was builtThe goal, the misses, the rules and the gates behind The Crate's concert data, with the code, the numbers and the tools.

Goal: “Let’s create the most comprehensive source of concerts, venue, and artist data in the U.S.”

Context/Guidance: “You can use Chrome MCP to search, and let’s focus on Pittsburgh first since I can verify that first hand”

Target: Initially none. As I built this product I began to acquire a target of known results (based often on literal firsthand experience) of accurate data in Pittsburgh which became my target.

Agent Executes: It got some right! This is exciting, but it also picked up some shows in Fresno and somehow thinks it’s this different venue in Pittsburgh.

Teach:

1. I know this is wrong because I can see in the site that it references an address that’s in Fresno, California.
2. I can do two things - instruct the agent to look for an address on the site. Provide it a location service (like Google Maps) to search places and associate them with addresses.

repeat

Pave: Add a rule to always check for addresses on the sites

Gate: No venue gets added without a confident address

To understand how this process becomes deterministic and reliable with sensitive data, we can use a useful analogy from the physical world: Desire paths - The literal path blazing plan of college campuses which involves just observing where students are walking, and then paving over where the grass is worn down to dirt.
You review the steps an agent takes, where it wanders off the path, you give it tools and guardrails to keep it on the path, then as it establishes over and over again that this is the best way, you pave that path.

After several iterations there is then a process of ‘graduation’ - taking the steps or outcomes that are proven, making them deterministic, and letting humans focus only on the goal.
I no longer need to review the website scraping work for The Crate. It writes the scrapes as ‘recipes’ that are deterministic and dependable based on my reviews and benchmarks.

The result is a process that’s often more code than agent, and can involve a human only when needed.

By outputting rules and code, there’s no concern of an agent hallucinating, costs decrease, and there’s little issue with trust.

The Method in Other Environments

The Crate example is firsthand and illustrates this well, but the process is certainly not limited to that context.
Here are a few examples of what flows could look like in other domains:

---

Finance: Invoice to PO matching

Goal: “Match every incoming invoice to its purchase order and flag anything that doesn’t line up.”

Context/Guidance: “You can read the AP inbox and the PO table in our ERP. Start with our top 20 vendors, since that’s most of the volume. Don’t approve anything, just match and flag.”

Target: Last quarter’s invoices that AP already matched by hand.

Agent Executes: It matched most of the clean ones! But it flagged a pile of partial shipments as overcharges. It also matched two invoices to the same PO, because one vendor reuses PO numbers.

Teach:

1. How do I know this is wrong? Receiving shows only half the order arrived, and AP’s history shows this vendor splits shipments all the time.
2. How can I equip the agent? Give it access to receiving records so it checks quantity received, not just quantity ordered. Add a rule that a PO can’t be fully matched twice.

repeat

Pave: The three-way match (PO, receipt, invoice) becomes a deterministic check for each vendor. Tolerances like “within 2% or $50” become code. The agent only sees invoices that fail the rules, and a person only sees the ones over the approval threshold.

Gate: Nothing gets paid without passing the match or getting human approval.

Healthcare: Claims ingestion

Goal: “Take incoming claims, however messy, and map them into our internal claims schema.”

Context/Guidance: “Here’s our schema and the field definitions. Claims come in as 837s, PDFs, and the occasional fax scan. Start with one source. Never guess a member ID or a code. If it’s not there, mark it missing.”

Target: A batch of claims the ops team already keyed in by hand.

Agent Executes: It got the basics right: patient, provider, dates of service. But it put the rendering provider in the billing provider field, read a date as month-day when the source used day-month, and pulled a diagnosis from a “rule out” note that was never a confirmed diagnosis.

Teach:

1. How do I know this is wrong? The billing NPI belongs to an individual, not the group. The date of service lands after the date we received the claim. The target has a different diagnosis.
2. How can I equip the agent? Give it an NPI lookup so it can check provider type. Add a validation that the date of service can’t be after the received date. Add guidance that codes only come from the claim’s coded fields, never from free-text notes.

repeat

Pave: Each source format gets its own recipe, a deterministic field mapping. The NPI, date and code-validity checks become code. The agent only steps in for a new format or a field the recipe can’t resolve. A person only reviews missing required fields.

Gate: PHI stays inside approved systems, and nothing reaches adjudication without passing schema validation.

Legal: Contract review

Goal: “Review incoming vendor contracts against our playbook and flag anything that deviates.”

Context/Guidance: “Here’s our playbook: positions on liability caps, indemnification, data handling, auto-renewal and governing law. Here are 50 past contracts with legal’s redlines. Don’t redline, just flag and explain why.”

Target: Legal’s past redlines.

Agent Executes: It caught the obvious ones, like uncapped liability and an odd governing-law choice. But it missed an auto-renewal clause buried under “Term and Termination.” It also flagged a liability cap as non-standard when it actually matched our fallback position.

Teach:

1. How do I know this is wrong? Legal redlined that auto-renewal in the target, and the playbook lists that cap as an acceptable fallback.
2. How can I equip the agent? Guidance to look for the concept, not the heading, since renewal language can live anywhere. Add the fallback positions to the context, not just the preferred ones.

repeat

Pave: Detection for common clauses becomes a rules library. Contracts on our own paper that come back unchanged skip review entirely through a simple diff check. The agent handles third-party paper, and a lawyer only sees deviations beyond fallback.

Gate: Nothing gets signed without attorney sign-off on flagged deviations.

Customer support triage

Goal: “Route every incoming ticket to the right queue with the right priority.”

Context/Guidance: “Here are our queues and what each one owns, our priority definitions, and three months of tickets with how they were resolved. Enterprise customers always get flagged.”

Target: Past tickets and where they were finally resolved, not where they were first routed. That’s a sneaky bugaboo, since the first routing is often wrong.

Agent Executes: It routed most billing and password tickets correctly! But it sent “I can’t log in since the update” to account access instead of bugs, and it marked an angry but low-impact ticket as urgent.

Teach:

1. How do I know this is wrong? The target shows those login tickets bounced over to engineering, and there was an open incident that day.
2. How can I equip the agent? Give it access to open incidents and the status page. Add guidance that priority is about impact and customer tier, not tone.

repeat

Pave: Routing for the top ticket types becomes rules. Customer tier comes from the CRM, not the agent’s judgment. The agent handles the ambiguous ones, and a person only sees what it can’t place.

Gate: Anything mentioning legal, security or cancellation always goes to a person.

---

It takes guts and it takes patience to make real change, but start by asking yourself what would make the biggest difference if it worked way better? Usually that’s a product or service working better, not just an admin function.

The way to take on the hardest problems of your enterprise isn’t to just avoid them and only test on easy things, and it’s not to pretend AI is better than it is.

Instead, set a goal, find the path, and pave it.

What path will you pave?