Event-Driven Judgment Ingestion

Companion code: 09-event-ingestion in indian-legal-ai-lab.

Previous: Part 8: A Multi-Agent Legal Research Workflow

A new judgment should be normalized, checked for a duplicate identity, chunked, and indexed once, even if the message arrives twice.

One job, one outcome

`IngestionJob` is a job id, a `JudgmentRecord`, and `FailTimes`. `IngestionPipeline.ProcessNext` dequeues one job.

If `_completed` already has that job id, the outcome is `idempotent-skip` and the chunks are left alone. For the first `FailTimes` attempts the loop throws `IOException` with the message `Simulated extraction failure.` After `maxAttempts` the job is enqueued on the dead-letter queue and the status is `dead-letter`.

If the record has no text, the status is `incomplete-document` and the error field is the extraction status, such as `metadata_only_pdf_not_vendored`. If the text is present, `TextChunker.Chunk` replaces any previous chunks for that case id and the status is `completed`.

`DuplicateDetector.Fingerprint` hashes neutral citation, citation, and decision date. `DuplicateCaseIds` returns every id in a group whose fingerprint appears more than once. The test copies one record, changes only the case id, and expects both ids back.

`MetadataNormalizer` trims the court and collapses whitespace in the citation. `PublicBucketBootstrap.DownloadMetadataAsync` accepts a year and a `.json` file name, refuses `..`, and downloads one object from the public bucket. The lab does not call it.

dotnet run --project labs/09-event-ingestion

The Thappar job is created with `FailTimes` 1, so the first attempt throws and the second completes. The lab prints that outcome and a chunk count above zero. `Pipeline_retries_then_dead_letters` uses `FailTimes` 5 and `maxAttempts` 2 and expects one dead letter.

The queue is in memory. It dies with the process. `infrastructure/aws/README.md` describes a later SQS shape and does not apply it.

Disk queue

`FileIngestionQueue` writes each job under `pending/`, then moves it to `done/` or `dead/`. A second enqueue of a job id that already has a done file is ignored. `Durable_queue_skips_a_completed_job_id` covers that. The in-memory `IngestionPipeline` is still what chunks the text.

Next

Part 10: Agent Memory and Durable Execution

Verified in this revision

`POST /ingest` enqueues on `FileIngestionQueue` and then calls `ProcessNext` on the in-memory pipeline. A completed job file is skipped. The queue is the durable path the API uses. There is no message broker.