Context
In claims intake the bad day is the one that matters: a file that silently never arrives, a batch that half-exports, a claim the payer rejected that nobody noticed. Each health plan also has its own normalization and validation rules, and adding a plan must not destabilize the others.
Constraints
- Several health plans, each with its own normalization and validation rules.
- Many document types: UB-04 and CMS-1500 claims, CMS-1500 with an EOB, dental claims and prior authorization requests.
- Files arrive by SFTP and email, on the sender's schedule, sometimes twice.
- A claim is not done when it is sent, only when the payer acknowledges it.
- Operators must be able to fix one document without reprocessing a whole batch.
- Batch received from SFTPMoved to S3 with a manifest; duplicate drops are ignored
- Pages prepared and extractedImages cleaned before recognition runs
- Plan rules appliedNormalization and validation for this health plan's module
- 837 sent, TA1 and 999 acceptedEnvelope and syntax acknowledgements matched
- 277 not receivedClaim stays open for follow-up until it arrives
My responsibility
I design and build the pipeline's microservices, from ingest and orchestration through per-plan rules, exports and EDI reconciliation.
Architecture
- Ingest
- Scheduled jobs move paper-intake, email and cover-letter files from SFTP to S3 with idempotency, retries and batch manifests. An SQS listener feeds each new S3 object into the pipeline.
- Orchestration
- One orchestrator owns each batch's lifecycle, from prepare through extraction and post-extraction to export, and consumes the result of every worker. A dedicated API lets operators reject or retry a single batch or document.
- Image preparation
- A Python worker cleans page images (JPG, PNG, TIFF, PDF) before extraction, because recognition quality is decided before the model ever sees the page.
- Per-plan rules
- Normalization and validation rules for each health plan are modules on a shared base, selected by Spring profile.
- Export
- Export workers produce JSON exports, scheduled archive exports to Azure Storage and SFTP, pushes into an external case management system over REST, and reject letters where a document cannot be processed.
- EDI reconciliation
- Outbound X12 837 claims are tracked in PostgreSQL and matched against TA1, 999 and 277 acknowledgements.
Critical decisions
Idempotent, manifest-driven ingest
Every drop is recorded in a batch manifest, so a file that arrives twice is processed once and a file that never arrives is noticed.
One owner for the batch lifecycle
A single orchestrator holds the state of every batch. Workers report to it; nothing advances a batch on its own.
Each plan's rules in a module
Each plan's rules live in a module on a shared base. Onboarding a plan is configuration plus a module, and the other plans are never redeployed with untested changes.
Acknowledgements define done
An 837 counts as done only when its 999 and 277 come back, and the whole system is built around that definition.
The model proposes, an engineer decides
A Jira webhook service drafts normalization rules with an LLM and posts them on the ticket for review. Nothing it writes reaches production without a person.
When things go wrong
| When | What happens |
|---|---|
| The same file is dropped twice | The manifest recognizes it and it is processed once. |
| A page can't be processed | An operator rejects or retries that one document from the UI; a reject letter is produced where needed. |
| An 837 gets no 999 or 277 back | It stays open in reconciliation until it does. |
| A new health plan needs different rules | A new rule module on the shared base; the existing plans are untouched. |
Result
Running in production for multiple US health plans, with every file traceable from SFTP drop to payer acknowledgement.
Claims processed every day
Claim and document types, from UB-04 and CMS-1500 to dental and prior authorization
Stack
Java · Spring Boot · Python · PostgreSQL · AWS S3 & SQS · Azure Storage · SFTP · X12 EDI