ANNIE DOES AI
WORK / IN PRACTICE FOCUS / RCM + AI

CASE STUDIES

Where the thinking meets the work.

These are real problems I’ve worked on inside healthcare revenue-cycle AI.

The identifying details are intentionally anonymized. The problems, responsibilities, decisions, and lessons are real. Proprietary workflows, internal data, and confidential implementation details stay confidential.

NO FAKE METRICS. NO MADE-UP CASE STUDIES.

SELECTED WORK

CASE / 001 AGENT WORKFLOWS

Teaching an AI Agent to Think Like an RCM Specialist

Translating real revenue-cycle investigation into behavior an AI system could execute, test, and improve.

↓
CASE / 002 AI QA + EVALUATION

Building QA for AI-Driven Revenue Cycle Work

Helping define how another system should determine whether AI-generated RCM work was actually correct.

↓
CASE / 001 RCM AGENT DEVELOPMENT

AGENT WORKFLOWS

Teaching an AI Agent to Think Like an RCM Specialist

The difficult part wasn’t teaching the agent to click through a workflow. It was teaching the system what an experienced revenue-cycle specialist would need to understand before taking the next action.

SETTING Healthcare revenue cycle
MY ROLE RCM domain expert + AI enablement
FOCUS Agent behavior, evaluation + workflow logic
01 / THE PROBLEM

A workflow can be followed perfectly and still produce the wrong result.

Revenue-cycle investigation is rarely just a sequence of predetermined clicks.

A specialist looks at account history, payer responses, documentation, prior actions, current claim state, deadlines, and sometimes conflicting evidence before deciding what should happen next.

An AI agent may be able to execute an action once instructed. The harder problem is determining whether that action is appropriate for the situation in front of it.

02 / WHY IT WAS HARD

Much of the actual expertise was hiding between the steps.

Experienced specialists routinely make decisions that never appear in a formal SOP because those decisions feel obvious once you know the work.

QUESTION / 01

What evidence has to exist before the agent can safely reach this conclusion?

QUESTION / 02

Which details change the workflow even when the denial reason appears identical?

QUESTION / 03

When should the agent proceed, and when should it stop and escalate?

QUESTION / 04

How do we know the action actually improved the account instead of merely completing a task?

03 / MY ROLE

Bring the RCM brain into the system.

My role was to work from the specialist side of the problem: investigate how competent RCM work should happen, compare that standard against agent behavior, and translate the gap into actionable product and workflow feedback.

01

Reproduce real RCM investigations and identify the decisions a specialist makes along the way.

02

Test agent outputs against expected RCM behavior rather than accepting task completion as success.

03

Identify missing context, unsupported conclusions, sequencing problems, and workflow gaps.

04

Provide concrete examples, evidence, screenshots, and specialist reasoning when a workflow failed.

05

Retest revised behavior and validate whether the resulting action actually matched the intended RCM outcome.

04 / THE SHIFT

From “Did the agent finish?” to “Did the agent understand enough to be right?”

That distinction changed what counted as a successful test.

An agent completing every programmed action could still fail if it relied on the wrong evidence, misunderstood the account state, skipped a meaningful exception, or produced an action that did not actually move the revenue-cycle problem toward resolution.

05 / WHAT THE WORK PRODUCED

A clearer standard for agent behavior.

The work helped make previously implicit RCM judgment more visible and testable.

  • Better definition of expected specialist behavior
  • More explicit decision points and exception handling
  • Reusable examples of agent failure modes
  • Stronger distinction between action completion and operational success
  • A tighter feedback loop between domain expertise and system behavior
CASE TAKEAWAY
The agent should inherit the specialist’s reasoning, not just the specialist’s clicks.
CASE / 002 AI QUALITY ASSURANCE

AI QA + EVALUATION

Building QA for AI-Driven Revenue Cycle Work

Once an AI system starts performing revenue-cycle work, another problem appears: how do you reliably determine whether what it produced is actually correct?

SETTING AI-assisted RCM operations
MY ROLE RCM domain expert + AI QA evaluation
FOCUS Correctness, failure detection + evidence
01 / THE PROBLEM

AI QA cannot simply ask whether the output looks reasonable.

Large language models are extremely good at producing outputs that are structured, confident, and plausible.

In revenue cycle, that is not enough.

A QA system has to distinguish between an answer that sounds right and one that is supported by the available account, payer, workflow, and documentation evidence.

02 / THE COMPLICATION

AI-generated work fails differently than traditional human workflow QA.

Traditional QA often checks whether required steps were performed correctly.

AI introduces additional failure modes.

FAILURE / 01

Unsupported inference

The conclusion may be possible, but the evidence does not actually establish it.

FAILURE / 02

Plausible fabrication

Missing information gets silently replaced by something that sounds likely.

FAILURE / 03

Correct answer, wrong reasoning

The output lands on the right result through a path that cannot be safely generalized.

FAILURE / 04

False QA confidence

The checker approves work because it mirrors the same flawed assumptions as the system it is evaluating.

03 / MY ROLE

Help define what the checker should actually care about.

My contribution centered on applying RCM judgment to QA behavior: pressure-testing what the system flagged, what it allowed to pass, and whether those decisions matched how an experienced specialist would evaluate the work.

01

Compare agent output, source evidence, and expected specialist behavior.

02

Identify high-risk errors that should trigger QA review or human escalation.

03

Test whether correct work was incorrectly flagged and whether incorrect work was allowed through.

04

Document examples where technically plausible output failed the underlying RCM standard.

05

Feed recurring QA failures back into the larger agent and workflow development loop.

04 / THE CENTRAL QUESTION

What does “correct” actually mean?

Not:

Did the response look professional?
Did the system complete the expected steps?
Did the answer resemble previous successful outputs?

But: was the conclusion supported, was the action appropriate, and would an experienced RCM specialist defend the result?

05 / WHAT THE WORK PRODUCED

A more domain-aware definition of quality.

The QA work helped surface the difference between generic output quality and revenue-cycle correctness.

  • Clearer criteria for acceptable versus risky outputs
  • Better visibility into false positives and false negatives
  • Reusable examples of domain-specific AI failure
  • Greater emphasis on source evidence and supportability
  • A stronger feedback loop between QA findings and agent development
CASE TAKEAWAY
You cannot automate quality until you can define what “correct” means.

A NOTE ON CONFIDENTIALITY

The interesting parts are the problems, not the PHI.

Healthcare AI work often involves proprietary systems, confidential workflows, internal product decisions, and protected data.

These case studies are intentionally written at the level required to demonstrate the work without exposing the material that made the work confidential in the first place.

Real work. Anonymized context. No patient data.

NEXT / LAB NOTES

The work creates questions.
I write about those too.

Lab Notes are where I pull apart the patterns, edge cases, failure modes, and questions that show up when AI meets actual revenue-cycle operations.

Read Lab Notes