Systematic Search Upload and Study Creation Flow¶

This document describes the complete data flow for uploading systematic search reference files through the SyRF platform, from the Angular frontend through to the creation of Study records in MongoDB.

Bulk PDF correction: The ordinary ReferenceUpload and ReferenceUpdate flows below are unchanged. For BulkPdfUpload, ADR-015 supersedes the former direct Lambda-to-agent presigned-URL path: the notifier persists the exact object version, Project Management gates release, and a fresh version-pinned URL exists only in an accepted agent-claim response. Completed Bulk PDF versions do not use age-based lifecycle expiry.

Table of Contents¶

Overview
Architecture Diagram
Sequence Diagram
Detailed Flow
Phase 1: Frontend File Selection and Validation
Phase 2: S3 Presigning Deep Dive
Phase 3: Direct S3 Upload
Phase 4: S3 Event Notification (Lambda)
Phase 5: State Machine Orchestration
Phase 6: Reference File Parsing
Phase 7: Study Creation
Phase 8: Systematic Search Creation
Message Contracts
Domain Models
Example Payloads
Error Handling
Troubleshooting
Performance Characteristics
Implementation Critique
Related Documentation

Overview¶

The systematic search upload process is a multi-service, event-driven workflow that:

Uploads reference library files (CSV, TSV, Endnote XML, PubMed XML, Living Search JSON) to S3
Triggers a state machine in the Project Management service
Parses the uploaded files to extract study metadata
Creates Study aggregate roots in MongoDB

Architecture Diagram¶

┌─────────────────────────────────────────────────────────────────────────────┐
│                              ANGULAR FRONTEND                                │
│  ┌──────────────────┐    ┌─────────────────┐    ┌────────────────────────┐  │
│  │ CreateSearch     │───▶│ ProjectDetail   │───▶│ S3FileService          │  │
│  │ Component        │    │ Effects         │    │ (presigned upload)     │  │
│  └──────────────────┘    └─────────────────┘    └──────────┬─────────────┘  │
└──────────────────────────────────────────────────────────────┼──────────────┘
                                                               │
                         ┌─────────────────────────────────────┼──────────────┐
                         │              API SERVICE            │              │
                         │  ┌──────────────────┐               │              │
                         │  │ SearchController │               │              │
                         │  │ GetS3Signature() │──────┐        │              │
                         │  └──────────────────┘      │        │              │
                         └────────────────────────────┼────────┼──────────────┘
                                                      │        │
                                                      ▼        ▼
┌─────────────────────────────┐              ┌────────────────────────────────┐
│        RABBITMQ             │◀────────────▶│          AWS S3                │
│  ISearchUploadStartedEvent  │              │   (Reference File Storage)     │
│  ISearchUploadSavedToS3Event│              └────────────────┬───────────────┘
└──────────────┬──────────────┘                               │
               │                                              │
               │                    ┌─────────────────────────┼───────────────┐
               │                    │     S3 NOTIFIER LAMBDA  │               │
               │                    │  ┌─────────────────────────────────┐    │
               │                    │  │ S3FileReceivedHandler           │    │
               │                    │  │ (S3:ObjectCreated:* trigger)    │────┘
               │                    │  └─────────────────────────────────┘
               │                    └─────────────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│                      PROJECT MANAGEMENT SERVICE                               │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                   SearchImportJobStateMachine                          │  │
│  │  ┌──────────┐    ┌──────────┐    ┌─────────┐    ┌───────────┐         │  │
│  │  │ Initial  │───▶│Uploading │───▶│Uploaded │───▶│  Parsing  │──┬──────┤  │
│  │  └──────────┘    └──────────┘    └─────────┘    └───────────┘  │      │  │
│  │       │                                                        │      │  │
│  │       │ (S3 event first)                                       │      │  │
│  │       └───────────────────────────────────────▶ (to Parsing)   │      │  │
│  │                                 ┌────────────┐  ┌─────────┐   │      │  │
│  │                                 │ Completed  │◀─┤         │   │      │  │
│  │                                 └────────────┘  │  Error  │◀──┘      │  │
│  │                                                 └─────────┘          │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                                                                              │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                   ReferenceFileParseJobConsumer                        │  │
│  │  ┌──────────────────────┐    ┌────────────────────┐                   │  │
│  │  │ ProjectManagement    │───▶│ StudyReferenceFile │───▶ MongoDB       │  │
│  │  │ Service              │    │ Parser             │    (Studies)      │  │
│  │  └──────────────────────┘    └────────────────────┘                   │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────────────────┘

Sequence Diagram¶

sequenceDiagram
    participant User
    participant Angular as Angular Frontend
    participant API as API Service
    participant S3
    participant Lambda as S3 Notifier Lambda
    participant RabbitMQ
    participant PM as Project Management
    participant MongoDB

    User->>Angular: Select reference file
    Angular->>Angular: Validate file format
    Angular->>Angular: Calculate SHA-256 hash
    Angular->>API: POST /api/projects/{id}/searches/getSignature

    Note over API: Generate ReferenceFileId (UUID)
    Note over API: Build S3 metadata headers
    Note over API: Sign with AWS Sig V4

    API->>RabbitMQ: Publish ISearchUploadStartedEvent
    API-->>Angular: Return presigned URL + headers

    RabbitMQ->>PM: Deliver ISearchUploadStartedEvent
    PM->>PM: Create SearchImportJob (Uploading state)
    PM->>PM: Schedule 60-min timeout

    Angular->>S3: PUT file (presigned URL)
    S3-->>Angular: 200 OK

    S3->>Lambda: S3:ObjectCreated:Put event (this flow)
    Lambda->>Lambda: Extract metadata from headers
    Lambda->>RabbitMQ: Publish ISearchUploadSavedToS3Event

    RabbitMQ->>PM: Deliver ISearchUploadSavedToS3Event
    PM->>PM: Transition to Uploaded state
    PM->>PM: Execute StartParseJobsActivity
    PM->>RabbitMQ: Publish IStartParsingReferenceFileCommand

    loop For each reference file
        RabbitMQ->>PM: Deliver parse command
        PM->>S3: Download file
        PM->>PM: Parse records
        PM->>MongoDB: Batch insert Studies (5000/batch)
        PM->>RabbitMQ: Publish IReferenceFileParsingCompletedEvent
    end

    PM->>PM: All files parsed → Completed state
    PM->>MongoDB: Create SystematicSearch
    PM-->>User: Search ready for screening

Detailed Flow¶

Phase 1: Frontend File Selection and Validation¶

Component: CreateSearchComponent (src/services/web/src/app/project/project-overview/create-search/create-search.component.ts)

User selects a reference library file in the Create Search dialog
Frontend validates the file format:
CSV/TSV: Checks for required column headers (Title, Authors, PublicationName, etc.)
XML: Validates Endnote or PubMed XML structure
User optionally configures screening import settings (mapping columns to investigators)
User submits the form, dispatching projectDetailActions.startSearchImport()

Supported File Types (LibraryFileType enum):

Value	Type	Description
`0`	CSV	Comma-separated values
`1`	TSV	Tab-separated values
`2`	EndnoteXml	Endnote XML export
`3`	PubmedXml	PubMed XML format
`4`	LivingSearchJson	Living search JSON format

Required CSV/TSV Columns:

Column	Description
`title`	Study title
`authors`	Author list
`publicationName`	Journal/publication name
`alternateName`	Alternative publication name
`abstract`	Study abstract
`url`	Link to study
`authorAddress`	Author contact information
`year`	Publication year
`doi`	Digital Object Identifier
`referenceType`	Type of reference
`pdfRelativePath`	Path to PDF file
`keywords`	Study keywords
`customId`	User-defined identifier

Phase 2: S3 Presigning Deep Dive¶

The presigning process allows the frontend to upload files directly to S3 without the file passing through our API servers. This is critical for large files and reduces server load.

Effect: project-detail.effects.ts (addSearch$ effect) (src/services/web/src/app/core/services/project/project-detail.effects.ts)

Effect generates SHA-256 hash of the file content
Calls SearchService.getS3RequestSignature() with file metadata

API Endpoint: POST /api/projects/{projectId}/searches/getSignature

Controller: SearchController.cs (lines 92-156) (src/services/api/SyRF.API.Endpoint/Controllers/SearchController.cs)

AWS Signature Version 4 Process¶

The API uses AWS Signature Version 4 (SigV4) to create presigned URLs. This is implemented in S3PostSigner.cs (src/libs/appservices/SyRF.AppServices/Services/S3PostSigner.cs).

Signing Algorithm (AWS4-HMAC-SHA256):

StringToSign = Algorithm + '\n' +
               RequestDateTime + '\n' +
               CredentialScope + '\n' +
               HashedCanonicalRequest

CredentialScope = Date + '/' + Region + '/' + Service + '/aws4_request'

Signature = HMAC-SHA256(SigningKey, StringToSign)

Key Derivation Chain:

kSecret  = "AWS4" + SecretAccessKey
kDate    = HMAC-SHA256(kSecret, Date)
kRegion  = HMAC-SHA256(kDate, Region)      // "eu-west-1"
kService = HMAC-SHA256(kRegion, Service)   // "s3"
kSigning = HMAC-SHA256(kService, "aws4_request")

Canonical Request Format:

CanonicalRequest = HTTPMethod + '\n' +
                   CanonicalURI + '\n' +
                   CanonicalQueryString + '\n' +
                   CanonicalHeaders + '\n' +
                   SignedHeaders + '\n' +
                   HashedPayload

What the API Does¶

Validates the search ID and search name
Reuses the supplied search ID as the single-file import identity
Constructs S3 key: Projects/{projectId}/Imported Search Libraries/SyRF Library - {searchId}.{ext}
Builds flat metadata headers (x-amz-meta-*):
x-amz-meta-projectid: Project GUID
x-amz-meta-searchid: Search GUID
x-amz-meta-searchname: URL-encoded search name
x-amz-meta-description: URL-encoded description
x-amz-meta-originalfileurl: exact stored object URL
x-amz-meta-librarytype: selected library type
x-amz-meta-screeningimportsettings: optional serialized settings
x-amz-meta-uploadkind: ReferenceUpload
Signs the request using AWS SigV4
Publishes ISearchUploadStartedEvent to RabbitMQ (starts state machine)
Returns presigned URL, headers, and metadata to frontend

S3 Configuration¶

Bucket: Configured via AWS_S3_BUCKET_NAME environment variable
Region: eu-west-1
CORS: Must allow PUT from web origins
Presign expiry: Configurable (default implementation)

Phase 3: Direct S3 Upload¶

Service: S3FileService (src/services/web/src/app/core/services/s3-file.service.ts)

Frontend performs HTTP PUT directly to S3 using presigned URL
Includes all metadata headers from signature response
Tracks upload progress via HttpEventType.UploadProgress
S3 stores file with metadata in x-amz-meta-* headers

Phase 4: S3 Event Notification (Lambda)¶

Lambda Function: S3FileReceivedFunction.cs (src/services/s3-notifier/SyRF.S3FileSavedNotifier.Endpoint/S3FileReceivedFunction.cs)

When S3 receives the object, the notifier's S3 notification pattern is ObjectCreated:*. The handler therefore accepts both S3:ObjectCreated:Put for this ordinary presigned-PUT flow and S3:ObjectCreated:CompleteMultipartUpload for a completed Bulk PDF multipart upload. The record's uploadkind metadata, rather than the event name, selects the downstream dispatch. A recognised ordinary record is fully prevalidated before any RabbitMQ side effect: immutable malformed metadata is logged and skipped so it cannot block a later valid record in the same S3 notification. Reserved Bulk PDF key shapes are the exception required by ADR-015: after a successful exact-version metadata read, malformed, missing, or unrecognized metadata creates a non-publishable RejectedPreHold quarantine row before success. It carries no caller-derived job authority and snapshots bounded environment retention. Under the notifier's environment-bound release-state read scope, a dedicated resolver accepts only bucket/key/version, rejects any caller job identity, and resolves only Project Management's server-owned binding. A deletion-independent BulkPdfUploadStorageBinding is persisted at completed-object transition and already blocks Project deletion/pruning; it retires only with objectless absence proof or the exact cleanup-acknowledgement transaction. The read-only resolver therefore cannot expose a pre-fence deletion window. An unowned version is reclaimed only after the row durably records the zero-match digest, exact delete result, and a later authoritative-inventory absence checkpoint; a full-binding conditional delete then retires the row without a fabricated acknowledgement or outbox. One exact Uploaded or authoritative terminal completed-object match repeats version-specific HEAD and idempotently persists/reuses CleanupOnly with no agent work; Uploaded is terminalized failed-cleanup-pending and releases capacity in the same Mongo transaction that persists CleanupOnly, while an already-terminal match keeps its disposition and history. Objectless contradiction, active/retriable claim, mismatch, or ambiguity first registers a PM-owned no-TTL BulkPdfUploadQuarantineFence for every server-owned candidate; only after the immutable fence-set receipt does the capture become QuarantinedHold. Unavailable state remains a retryable active hold. The quarantine retains immutable binding/evidence, retry authority, duplicate suppression, cleanup-client requirement, source event, fence-set receipt, and aggregate debit until ordinary cleanup or a durable non-production quarantine transfer. That transfer uses a stable ID and idempotent PM-prepare, dormant-destination-accept, PM-link, source-tombstone, destination-activate, then PM-finalize handshake; it assumes no cross-service transaction. Candidate fences remain active throughout, every receipt is queryable after crash/lost response, conflicting input fails closed, the shared debit is counted once, and only the linked destination may acknowledge exact cleanup. Pruning/Project deletion remains fenced until that acknowledgement. S3, configuration, and broker failures remain retryable; exhausted invocations enter an encrypted exact-environment on-failure queue. Independently, scheduled exact-prefix version inventory reconstructs missing holds after queue loss/expiry or destination failure, so no second S3 event or Lambda retry window is the durability boundary.

For this ordinary reference-file flow:

S3 triggers S3:ObjectCreated:Put event.
Lambda function syrfAppUploadS3Notifier receives the event and reads the object's x-amz-meta-* headers via GetObjectMetadataAsync.
Dispatch is driven by the uploadkind metadata value, first matched against the exact, case-sensitive declared UploadKind names (so numeric enum values are rejected), then routed through a switch (UploadEventDispatch.cs):
ReferenceUpload → this phase's flat single-file contract (below).
ReferenceUpdate → Submits IStartBulkStudyUpdateJobCommand as a job via MassTransit's SubmitJob (job-service routing — distinct from both Publish and the point-to-point Send used below), after validating projectid/studyupdatefileinfo metadata.
BulkPdfUpload → version-validates and durably records the exact release envelope, then Sends IBulkPdfUploadObjectReadyCommand point-to-point to Project Management's release gate. The command carries stable job/environment/bucket/key/version identity and no URL. PM proves committed Uploaded, performs an exact version-specific HEAD, and sends stable work identity to the PDF agent. The agent's accepted/idempotently accepted claim response mints a fresh version-pinned GET on every retry; the notifier, hold, and queue command never contain a capability. See ADR-015.
Absent or unrecognised uploadkind → a logged no-op, not a throw. This is a deliberate semantics change: before it, any object without uploadkind (or with an uploadkind other than ReferenceUpdate) fell into the reference-upload branch and threw on its missing required metadata, and S3 retried the failing invocation indefinitely. This was live production behaviour for every single-study PDF upload (StudyController.GetPdfUploadSignature, which tags only searchId/studyId/ investigatorId — no uploadkind, no virtualhost).
For ReferenceUpload (this phase's flow), Publishes ISearchUploadSavedToS3Event to RabbitMQ with flat, single-file properties (the multi-file x-amz-meta-referencefiles JSON contract described in old versions of this document, and in Known Issue #8 below, no longer exists — search imports carry exactly one reference file per job):
ProjectId, SearchId, SearchName, Description
OriginalFileUrl, LibraryType, ScreeningImportSettings (flat properties, not a nested file array)
DateTimeEventOccurred

For each recognised S3 record, the RabbitMQ connection is resolved from the RabbitMqHost/ RabbitMqUsername/RabbitMqPassword environment variables plus that object's virtualhost metadata (which vhost — production/staging/per-PR-preview — the command should land on). This resolution is skipped entirely for a logged-no-op record, so an object carrying neither uploadkind nor virtualhost metadata does not require either to be present. It is not resolved once for a whole multi-record event batch.

Phase 5: State Machine Orchestration¶

State Machine: SearchImportJobStateMachine.cs (src/services/project-management/SyRF.ProjectManagement.Endpoint/Sagas/SearchImportJobStateMachine.cs)

The MassTransit state machine manages the import workflow.

States¶

State	Description
`Initial`	State machine not yet started
`Uploading`	API has been notified, waiting for S3 confirmation
`Uploaded`	File successfully stored in S3
`Parsing`	Reference files being parsed into studies
`Completed`	All parsing complete, SystematicSearch created. Saga persists with `CompletedAt` timestamp; cleaned up by TTL index after 7 days.
`Error`	Failure occurred at any stage. Saga persists with `CompletedAt` timestamp; cleaned up by TTL index after 7 days.

SearchImportJobStatus Enum:

Value	Status	Description
`0`	Uploading	Initial upload in progress
`1`	Parsing	File parsing underway
`2`	Complete	Successfully finished
`3`	Error	Failed with error

Dual-Path Entry (Race Condition Handling)¶

The state machine handles a potential race condition where events can arrive in either order:

Path A (Normal flow - API event first):

Initial → [ISearchUploadStartedEvent] → Uploading → [ISearchUploadSavedToS3Event] → Uploaded

Path B (S3 event arrives first):

Initial → [ISearchUploadSavedToS3Event] → Parsing

In Path B, SearchUploadSaved arriving first goes directly through Initially to Parsing (via CreateSearchImportJobActivity, SetFileReceivedActivity, and StartParseJobsActivity). When SearchUploadStarted arrives later, it is handled by During(Uploading, Uploaded, Parsing) which sets metadata but does not re-trigger parsing.

This ensures the workflow completes correctly regardless of which event arrives first.

Events and Transitions¶

                    ISearchUploadStartedEvent
Initial ──────────────────────────────────────────▶ Uploading
    │                                                   │
    │ ISearchUploadSavedToS3Event                       │ ISearchUploadSavedToS3Event
    │ (S3 event arrives first)                          │ (normal flow)
    │                                                   ▼
    │                                               Uploaded
    │                                                   │
    │                                                   │ StartParseJobsActivity
    │                                                   ▼
    └──────────────────────────────────────────────▶ Parsing
                                                        │
                    ┌───────────────────────────────────┼───────────────────────────────┐
                    │                                   │                               │
        IReferenceFileParsingCompletedEvent             │     IReferenceFileParsingFaultedEvent
                    │                                   │                               │
                    ▼                                   ▼                               ▼
               (all done?)                       (still pending)                  (mark fault)
                    │                                                                   │
        ┌───────────┴───────────┐                                                       │
        │                       │                                                       │
   (no faults)             (has faults)                                                 │
        │                       │                                                       │
        ▼                       ▼                                                       │
   Completed ◀─────────────  Error ◀────────────────────────────────────────────────────┘

Terminal states (Completed, Error) persist with CompletedAt timestamp.
MongoDB TTL index (ttl_CompletedAt_7d) cleans up after 7 days.
All 6 events are Ignore()'d in terminal states to handle late duplicates.

Activities Executed¶

Activity	Trigger	Purpose
`CreateSearchImportJobActivity`	`SearchUploadStarted`, `SearchUploadSaved`	Creates `SearchImportJob` aggregate with `ReferenceFileParseJob` entities
`SetFileReceivedActivity`	`SearchUploadSaved`	Sets upload completion timestamp
`StartParseJobsActivity`	Transition to `Parsing`	Publishes `IStartParsingReferenceFileCommand` for each reference file
`CompleteSearchJobActivity`	All parse jobs complete (no faults)	Creates `SystematicSearch` aggregate
`FailSearchJobActivity`	Any parse job faulted	Marks import job with error status

Timeout Handling¶

60-minute timeout scheduled when entering Uploading state
If S3 confirmation not received, transitions to Error state
Publishes ISearchImportJobErrorEvent with timeout message

Phase 6: Reference File Parsing¶

Consumer: ReferenceFileParseJobConsumer.cs (src/services/project-management/SyRF.ProjectManagement.Endpoint/Consumers/ReferenceFileParseJobConsumer.cs)

For each reference file, the consumer:

Receives IStartParsingReferenceFileCommand
Calls ProjectManagementService.ParseReferenceFile()
On success: Publishes IReferenceFileParsingCompletedEvent
On failure: Publishes IReferenceFileParsingFaultedEvent

Configuration:

Job timeout: 5 minutes
Retry policy: 4 incremental retries (1 min + 1 min increments)
Concurrent job limit: 5

Service: ProjectManagementService.cs (lines 129-199) (src/libs/project-management/SyRF.ProjectManagement.Core/Services/ProjectManagementService.cs)

The ParseReferenceFile method:

Retrieves the ReferenceFileParseJob from the project
Sets up progress tracking with throttled updates (every 2 seconds)
Calls StudyReferenceFileParser.ParseStudiesAsync()
Handles progress updates and final completion/error states

Phase 7: Study Creation¶

Parser Router: StudyReferenceFileParser.cs (src/libs/project-management/SyRF.ProjectManagement.Core/Services/StudyReferenceFileParser.cs)

Selects appropriate parser implementation based on LibraryFileType
Creates new SyRF reference file URL for storing parsed output
Invokes parser's ParseStudiesAsync() method

Parser Implementations:

Parser	File Types	Location
`SpreadsheetRecordProcessorParseImplementation`	CSV, TSV	SpreadsheetRecordProcessorParseImplementation.cs
`EndnoteXmlParseImplementation`	Endnote XML	EndnoteXmlParseImplementation.cs
`PubmedXmlParseImplementation`	PubMed XML	PubmedXmlParseImplementation.cs

Record Processors:

Processor	Purpose
`StudySpreadsheetRecordProcessor`	Converts CSV/TSV rows to `Study` objects
`StudyEndnoteRecordProcessor`	Converts Endnote XML records to `Study` objects
`StudyPubmedRecordProcessor`	Converts PubMed XML records to `Study` objects

Each parser:

Downloads the original file from S3
Parses records using the appropriate record processor
Creates Study aggregate roots with metadata:
Title, Authors, Abstract, Year
DOI, URL, Keywords
PublicationName, ReferenceType
Project and Search associations
Batches studies (5000 at a time) for MongoDB insertion
Creates a new SyRF reference file with Study IDs added
Reports progress back to the state machine

Phase 8: Systematic Search Creation¶

Domain Model: SearchImportJob.cs (line 41) (src/libs/project-management/SyRF.ProjectManagement.Core/Model/ProjectAggregate/SearchImportJob.cs)

When all ReferenceFileParseJob entities complete:

public SystematicSearch CompleteSearchImportJob()
{
    Status = SearchImportJobStatus.Complete;
    var search = new SystematicSearch(Id, Name, Description,
        ReferenceFileParseJobs.Select(rfj => rfj.CreateStudyReferenceFile()),
        ProjectId, LivingSearchId);
    return search;
}

Result: A SystematicSearch aggregate is created containing:

Search ID (same as SearchImportJob ID)
Name and Description
Collection of StudyReferenceFile objects
Project association
Optional Living Search association

Message Contracts¶

Events¶

Event	Publisher	Subscribers	Purpose
`ISearchUploadStartedEvent`	API Service	PM State Machine	Notifies upload has begun
`ISearchUploadSavedToS3Event`	S3 Notifier Lambda	PM State Machine	Confirms file stored in S3
`IReferenceFileParsingCompletedEvent`	ReferenceFileParseJobConsumer	PM State Machine	Parse job succeeded
`IReferenceFileParsingFaultedEvent`	ReferenceFileParseJobConsumer	PM State Machine	Parse job failed
`ISearchImportJobErrorEvent`	PM State Machine	Error handlers	Import job failed

Commands¶

Command	Publisher	Consumer	Purpose
`IStartParsingReferenceFileCommand`	StartParseJobsActivity	ReferenceFileParseJobConsumer	Triggers parsing of individual file
`IStartBulkStudyUpdateJobCommand`	S3 Notifier Lambda (`ReferenceUpdate` dispatch case)	PM `StudyUpdateJobFileSavedToS3Comsumer`	Triggers a bulk study update job (not a systematic-search import — a separate upload kind sharing the same Lambda)
`IBulkPdfUploadObjectReadyCommand`	S3 Notifier durable reconciler (`BulkPdfUpload`)	Project Management release gate	Point-to-point, stable exact bucket/key/version binding, never a URL; PM verifies committed job state and version-specific `HEAD`. See ADR-015
`IProcessBulkPdfUploadCommand`	Project Management release gate	PDF agent (`src/services/pdf-agent/`)	Point-to-point stable work identity, never a presigned URL; the agent obtains a fresh version-pinned GET only from each accepted/idempotently accepted claim response

Shared Interfaces¶

ICanStartSearchImportJob (src/libs/kernel/SyRF.SharedKernel/Interfaces/ICanStartSearchImportJob.cs)

Base interface implemented by ISearchUploadStartedEvent and ISearchUploadSavedToS3Event (the two events that can initiate a search import job). Note: IStartSearchImportJobCommand was removed as it had no producer anywhere in the monorepo.

public interface ICanStartSearchImportJob
{
    Guid ProjectId { get; }
    Guid SearchId { get; }
    string SearchName { get; }
    string Description { get; }
    string OriginalFileUrl { get; }
    LibraryFileType LibraryType { get; }
    ScreeningImportSettings? ScreeningImportSettings { get; }
}

Domain Models¶

SearchImportJob¶

Location: SearchImportJob.cs

MongoDB Collection: Projects (embedded in Project aggregate)

Aggregate root tracking the import process:

Property	Type	Description
`Id`	`Guid`	Same as SearchId
`Name`	`string`	User-provided search name
`Description`	`string`	User-provided description
`Status`	`SearchImportJobStatus`	Uploading, Parsing, Complete, Error
`ReferenceFileParseJobs`	`IEnumerable<ReferenceFileParseJob>`	Individual file parsing jobs
`TotalNumberOfStudies`	`int?`	Sum across all parse jobs
`NumberOfParsedStudies`	`int`	Progress counter
`Errors`	`List<string>`	Error messages

ReferenceFileParseJob¶

Location: ReferenceFileParseJob.cs

MongoDB Collection: Embedded in SearchImportJob

Entity tracking individual file parsing:

Property	Type	Description
`Id`	`Guid`	ReferenceFileId
`LibraryType`	`LibraryFileType`	File format
`OriginalFileUrl`	`string`	S3 location of uploaded file
`SyrfReferenceFileUrl`	`string?`	S3 location of parsed file with IDs
`TotalNumberOfStudies`	`int?`	Total studies in file
`NumberOfParsedStudies`	`int`	Parsed count
`ScreeningImportSettings`	`ScreeningImportSettings?`	Column-to-investigator mapping
`IsComplete`	`bool`	Parsed == Total

Study¶

Location: Study.cs

MongoDB Collection: Studies

Aggregate root for individual studies:

Property	Type	Description
`Id`	`Guid`	Unique study identifier
`Title`	`string`	Study title
`Authors`	`IEnumerable<Author>`	Author list
`Abstract`	`string?`	Study abstract
`Year`	`int?`	Publication year
`DOI`	`string?`	Digital Object Identifier
`PublicationName`	`PublicationName`	Journal information
`Keywords`	`List<string>`	Study keywords
`ProjectId`	`Guid`	Owning project
`SystematicSearchId`	`Guid`	Owning search
`ReferenceFileId`	`Guid`	Source file
`ScreeningInfo`	`ScreeningInfo`	Screening decisions
`ExtractionInfo`	`ExtractionInfo`	Data extraction

SystematicSearch¶

Location: SystematicSearch.cs

MongoDB Collection: SystematicSearches

Aggregate root representing a completed search:

Property	Type	Description
`Id`	`Guid`	Search identifier
`Name`	`string`	Search name
`Description`	`string`	Search description
`SyrfReferenceFiles`	`IEnumerable<StudyReferenceFile>`	Parsed file references
`ProjectId`	`Guid?`	Owning project
`NumberOfStudies`	`int`	Total studies across all files

Example Payloads¶

Presigning Request¶

Request: POST /api/projects/{projectId}/searches/getSignature

Search imports carry exactly one reference file (flattened — see ICanStartSearchImportJob), not a file array:

{
  "searchId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "searchName": "PubMed Search 2024",
  "uploadInfo": {
    "description": "Search results from PubMed for cardiac intervention studies",
    "libraryFileType": 0,
    "payloadHash": "abc123def456..."
  },
  "screeningImportSettings": null
}

Presigning Response¶

{
  "uploadUrl": "https://syrf-bucket.s3.eu-west-1.amazonaws.com/Projects/...",
  "headers": {
    "x-amz-meta-projectid": "project-123",
    "x-amz-meta-searchid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "x-amz-meta-searchname": "PubMed%20Search%202024",
    "x-amz-meta-description": "Search%20results%20from%20PubMed...",
    "x-amz-meta-virtualhost": "/",
    "x-amz-meta-originalfileurl": "https://syrf-bucket.s3.eu-west-1.amazonaws.com/Projects/project-123/Imported%20Search%20Libraries/SyRF%20Library%20-%20a1b2c3d4.csv",
    "x-amz-meta-librarytype": "CsvLibrary",
    "x-amz-meta-screeningimportsettings": "",
    "x-amz-meta-uploadkind": "ReferenceUpload",
    "x-amz-meta-content-type": "text/csv",
    "x-amz-content-sha256": "abc123def456...",
    "Authorization": "AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20241207/eu-west-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-meta-content-type;x-amz-meta-description;x-amz-meta-librarytype;x-amz-meta-originalfileurl;x-amz-meta-projectid;x-amz-meta-screeningimportsettings;x-amz-meta-searchid;x-amz-meta-searchname;x-amz-meta-uploadkind;x-amz-meta-virtualhost, Signature=..."
  }
}

HeaderDictionaryBuilder prefixes every supplied field, including content-type, with x-amz-meta-; this endpoint therefore does not return an ordinary content-type upload header. S3PostSigner adds host and x-amz-date and canonicalizes every supplied header, so the complete lower-case SignedHeaders set above is part of the upload contract. Header field names are case-insensitive on the wire; the signature's lower-case ordering is authoritative.

ISearchUploadStartedEvent¶

{
  "messageId": "msg-uuid-123",
  "messageType": ["urn:message:SyRF.API.Messages.Events:ISearchUploadStartedEvent"],
  "message": {
    "projectId": "project-123",
    "searchId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "searchName": "PubMed Search 2024",
    "description": "Search results from PubMed...",
    "originalFileUrl": "https://syrf-bucket.s3.eu-west-1.amazonaws.com/Projects/project-123/Imported%20Search%20Libraries/SyRF%20Library%20-%20a1b2c3d4.csv",
    "libraryType": 0,
    "screeningImportSettings": null,
    "dateTimeEventOccurred": "2024-12-07T12:00:00Z"
  }
}

ISearchUploadSavedToS3Event¶

Published by the Lambda's ReferenceUpload dispatch case (Phase 4) after re-reading the same flat properties back from the object's x-amz-meta-* headers:

{
  "messageId": "msg-uuid-456",
  "messageType": ["urn:message:SyRF.S3FileSavedNotifier.Messages:ISearchUploadSavedToS3Event"],
  "message": {
    "projectId": "project-123",
    "searchId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "searchName": "PubMed Search 2024",
    "description": "Search results from PubMed...",
    "originalFileUrl": "https://syrf-bucket.s3.eu-west-1.amazonaws.com/Projects/project-123/Imported%20Search%20Libraries/SyRF%20Library%20-%20a1b2c3d4.csv",
    "libraryType": 0,
    "screeningImportSettings": null,
    "dateTimeEventOccurred": "2024-12-07T12:00:05Z"
  }
}

Bulk PDF release and processing commands¶

IBulkPdfUploadObjectReadyCommand is sent point-to-point by the durable notifier reconciler to Project Management. Its immutable version enters the hold and command, not a new Project BSON field. After an exact version-specific S3 HEAD, Project Management idempotently persists the same environment/job/stable-message binding, including bucket/key/version/size/hash/metadata and release state, in a deletion-independent BulkPdfUploadReleaseRecord before acknowledging the command. The record is outside Project BSON and no durable surface contains a URL:

{
  "messageId": "msg-uuid-789",
  "messageType": ["urn:message:SyRF.ProjectManagement.Messages.Commands:IBulkPdfUploadObjectReadyCommand"],
  "message": {
    "releaseId": "release-uuid-456",
    "projectId": "project-123",
    "searchId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "uploadId": "50db2b0a-fb30-4e54-be79-0f4f3028d114",
    "bucketName": "syrf-bucket",
    "objectKey": "Projects/_bulk-staging/50db2b0a.zip",
    "objectVersionId": "3HL4kqtJlcpXroDTDmjVBH40Nrjfkd",
    "conflictPolicy": "SkipExisting",
    "expectedSizeBytes": 104857600,
    "expectedSha256": "77a5c9f1a75b59c9782e02f32feb930243657536f1fabda2e36dd5d11652e9",
    "environmentRoot": "staging",
    "dateTimeEventOccurred": "2026-08-18T12:00:05Z"
  }
}

After validation, Project Management sends IProcessBulkPdfUploadCommand with stable work identity but no URL. The agent claims that identity before side effects. Every initial or retry claim reads the durable release record, rechecks the exact job/version binding and version-specific S3 object, and returns a newly minted version-pinned zipDownloadUrl only inside each accepted or idempotently accepted claim response. Capability expiry therefore causes a fresh claim response, not a stored or republished URL, and the agent receives no AWS credentials.

If the authenticated job is terminal before publication, the reconciler emits no processing command. It sends an idempotent IBulkPdfUploadCleanupRegistrationCommand; Project Management validates the live terminal job and exact version-specific HEAD, then persists the same immutable binding as a non-claimable CleanupOnly release record before acknowledging registration. The exact-version reclaimer cannot start before that cleanup authority is durable.

The release record is also the cleanup authority after embedded job history is bounded. Before history pruning, a processing-capable or cleanup-only record must already exist for any job with a completed-object cleanup obligation, and the same Mongo session transaction copies terminal disposition and claim state to it. Objectless cancellation/abandonment instead atomically persists the sweeper's MPU/object absence proof with terminal state and requires no fictional release record or receipt. Both BulkPdfMultipartSweepProcessor and BulkPdfUploadController.CancelUpload hand a completed-object cancellation to the durable cleanup-pending path and retain active upload capacity until Project Management atomically persists/reuses the exact-version CleanupOnly record, terminalizes the job, and releases capacity; neither performs an unversioned delete, creates a delete marker, or records object absence while the exact noncurrent version survives. Whole-Project deletion fails closed for nonterminal work, a completed-object terminal job without a completed cleanup receipt, an objectless terminal job without its persisted absence proof, an active processing/cleanup record, an unresolved BulkPdfUploadStorageBinding, or a BulkPdfUploadQuarantineFence. Bounded history pruning applies the same fence. After exact-version deletion, the idempotent cleanup acknowledgement verifies the exact delete/ack digest and owner and, in one PM transaction, converts the active record to a non-authorizing audit receipt, completes the storage binding, and retires every member of an optional fence set under one immutable final-disposition receipt. Missing/extra/mismatched/partial sets fail closed and replay returns the same receipt. It does not remove the embedded terminal job/report. The receipt satisfies the Project-deletion fence while ordinary last-20-per-search pruning owns history removal. The reconciler then marks that row ReceiptRetirementPending, persists the immutable delete/ack result, and in one transaction creates an immutable HoldRetirementOutbox while converting the stable capture key in place to a replay-fenced CaptureTombstone. Exact-binding duplicate capture succeeds against the tombstone without a new hold; mismatch poisons. Only that outbox calls the dedicated hold-retired-confirmation route under the environment-bound cleanup-write scope; it conditionally stores the immutable acknowledgement ID/time and is deleted only afterward. Receipt policy snapshots P90D staging or P7D preview retention; Project Management purges only after expiry, acknowledged retirement, OutboxDrainedAt from its receipt reconciler's later strongly consistent exact-key DynamoDB absence proof, zero active record, and ordinary history pruning or Project deletion. Retries within retention return the stored result; a safely purged binding returns fail-closed 410 CleanupReceiptRetired. An unlocked preview database remains isolated in protected cleanup custody with the last-approved cleanup-only API/PM path and protected synthetic-harness identity through every snapshotted P7D receipt. The server derives only manifest-owned synthetic Projects, rejects caller-selected/user/persistent targets, invokes ordinary deletion after receipt/hold/outbox fences, then purges receipts, revokes/drains the last identity/path, and deletes the exact database; lock-db continues to preserve it unchanged. Notifier absence requires zero active release records, zero active holds, zero QuarantinedHold items, and zero retirement outboxes (or a finalized replay-fenced transfer for every quarantine); TTL is never absence proof. Transfer uses the durable prepare/accept/link/source-tombstone/destination-activate/finalize handshake above, retaining the PM fence and stable debit through every crash boundary. Capture and transfer tombstones remain through their snapshotted event/DLQ replay deadline, empty visible and in-flight failure queues, an oldest-message watermark newer than each tombstone, and an authoritative inventory checkpoint. A transfer tombstone additionally retains its replay evidence/debit and cannot purge until matching destination exact-version cleanup acknowledgement, PM fence-set final clearance, and authoritative shared-debit release; pending/failed cleanup retains the table and purge record. Redrive is forbidden after conditional tombstone deletion. The exact table, credential-free protected purge owner/schedule, and tombstone-only dynamic record remain through zero tombstones and only then may final notifier/table/record absence be claimed.

Error Handling¶

Upload Timeout¶

60-minute timeout from upload start
State machine transitions to Error state
ISearchImportJobErrorEvent published with "Timed out waiting for file upload"

Parsing Failures¶

Individual file parsing has 5-minute timeout
4 automatic retries with incremental backoff
On final failure, IReferenceFileParsingFaultedEvent published
If any file fails, entire import marked as Error

Rollback on Parse Error¶

If parsing fails mid-stream:

All studies created for that file are deleted
FileParseResult contains error details
ReferenceFileParseJob marked with errors
SearchImportJob status set to Error

Troubleshooting¶

Common Issues¶

Symptom	Likely Cause	Resolution
Upload stuck at 0%	CORS misconfiguration on S3 bucket	Verify S3 CORS allows PUT from web origin
"Timed out waiting for file upload"	Lambda not triggered or RabbitMQ connectivity	Check Lambda CloudWatch logs, verify S3 event notification
Search stuck in "Uploading"	S3 event never reached PM service	Check S3 bucket event notifications, Lambda invocations
Parsing errors	Malformed input file	Validate CSV columns, check XML structure
Studies not appearing	MongoDB connectivity or batch insert failure	Check PM service logs, MongoDB connection

Diagnostic Steps¶

Check RabbitMQ queues: Verify messages are being delivered
search-import-job-state queue for state machine events
start-parsing-reference-file-command queue for parse jobs
Check Lambda CloudWatch logs: /aws/lambda/syrfAppUploadS3Notifier
Look for S3 event processing errors
Verify metadata extraction succeeded
Check PM service logs: Search for SearchImportJobStateMachine
State transitions logged with correlation ID
Activity execution results
MongoDB queries:

// Check SearchImportJob status
db.Projects.find({"SearchImportJobs.Id": UUID("search-id")})

// Count studies for a search
db.Studies.countDocuments({SystematicSearchId: UUID("search-id")})

Debug Mode¶

Enable verbose logging in Project Management service:

{
  "Logging": {
    "LogLevel": {
      "MassTransit": "Debug",
      "SyRF.ProjectManagement": "Debug"
    }
  }
}

Screening Import Integration¶

When ScreeningImportSettings is provided:

CSV/TSV columns are mapped to investigator IDs
Screening decisions are extracted during parsing
Studies are created with pre-populated ScreeningInfo
Allows bulk import of existing screening decisions

ScreeningImportSettings Structure:

public record ScreeningImportSettings(
    Guid StageId,
    Dictionary<string, Guid> UserColumnMap  // column name -> investigator ID
);

Performance Characteristics¶

Operation	Typical Duration	Bottleneck
S3 signature generation	< 100ms	API response
S3 upload	Variable (file size)	Network bandwidth
S3 notification	1-5 seconds	Lambda cold start
Parse job (CSV, 10K studies)	30-60 seconds	MongoDB batch writes
Parse job (XML, 10K studies)	60-120 seconds	XML parsing + MongoDB

Note: Performance numbers are estimates based on typical workloads and may vary.

Optimizations:

Studies batched in groups of 5,000 for MongoDB insertion
Progress updates throttled to every 2 seconds
Up to 5 concurrent parse jobs per instance
Incremental retry with backoff on transient failures

Implementation Critique¶

This section documents known architectural concerns and potential improvements for the systematic search upload flow.

Strengths¶

Robust Race Condition Handling: The dual-path state machine entry handles the case where S3 events can arrive before API events. Both SearchUploadStarted and SearchUploadSaved can initiate the saga via Initially, and late-arriving events are handled by explicit During blocks.
Good Separation of Concerns: The architecture cleanly separates:
API (presigning + event publishing)
Lambda (S3 event notification)
State machine (orchestration)
Consumers (parsing work)
Proper Use of MassTransit Sagas: Using state machines for workflow orchestration is the right pattern for this use case.
Batching Strategy: The 5,000 study batch size for MongoDB inserts is a reasonable trade-off between memory usage and database round-trips.

Known Issues and Technical Debt¶

1. Presigned URL Security Gap¶

Location: SearchController.cs:92-156

The presigning endpoint publishes ISearchUploadStartedEvent before the file is actually uploaded. If a user requests a signature but never uploads:

A SearchImportJob gets created in "Uploading" state
It will time out after 60 minutes and transition to "Error"
This creates orphaned/failed jobs in the database

Recommendation: Consider a two-phase approach where the job isn't created until the S3 event confirms the upload, or implement cleanup for abandoned uploads.

2. Silent Exception Swallowing¶

Location: ReferenceFileParseJobConsumer.cs:29-32

catch (Exception e)
{
    await PublishFaultEvent(context, command);
}

The exception e is caught but never logged. This makes debugging production issues difficult. The fault event doesn't contain the exception details.

Recommendation: Log the exception with correlation ID before publishing the fault event.

3. State Machine Duplicate Event Handling (Resolved)¶

Location: SearchImportJobStateMachine.cs

Previously, duplicate events could create duplicate jobs or trigger duplicate parse commands. This has been resolved with a three-layer defense: scoped UseMessageRetry (only MongoDbConcurrencyException), UseInMemoryOutbox, and Ignore() handlers for all 6 events in terminal states. Saga instances persist in terminal states and are cleaned up by a MongoDB TTL index after 7 days. See SearchImportJob Saga Duplicate Event Handling for full details.

4. Tight Coupling to S3 Path Convention¶

Location: SearchController.cs

The S3 key pattern Projects/{projectId}/Imported Search Libraries/SyRF Library - {searchId}.{ext} is hard-coded. If this convention changes, both the API and Lambda need coordinated updates.

Recommendation: Extract path construction to a shared utility, or store the full URL in metadata rather than reconstructing it.

5. SystematicSearch Schema Migration Complexity¶

Location: SystematicSearch.cs:72-105

The SyrfReferenceFiles property has complex getter/setter logic handling schema version 0 vs newer versions. This backward compatibility logic makes the domain model harder to understand and maintain.

Recommendation: Consider a one-time data migration script to upgrade all v0 documents, then remove the backward compatibility code.

6. Missing Validation in ReferenceFileParseJob.UpdateProgress¶

Location: ReferenceFileParseJob.cs:95-133

The UpdateProgress method silently returns false when HasError || IsComplete. Callers may not check this return value, leading to silent failures where progress updates are ignored.

Recommendation: Either throw an exception or log a warning when progress updates are attempted on completed/errored jobs.

7. No Circuit Breaker for MongoDB Operations¶

The parsing flow does batch inserts of 5,000 studies without circuit breaker protection. If MongoDB is under load:

All 5 concurrent parse jobs will pile up
Retries will compound the problem
No backpressure mechanism exists

Recommendation: Implement a circuit breaker pattern (Polly) for MongoDB operations, and consider adding backpressure through RabbitMQ prefetch limits.

8. Metadata Size Limit Risk (Resolved by the flattened single-file contract)¶

Location: SearchController.cs

S3 object metadata has a 2KB limit. This section previously warned about an x-amz-meta-referencefiles header carrying JSON-serialized info for multiple files, which could exceed the limit. That multi-file contract no longer exists: a search import now carries exactly one reference file, described by flat properties (originalfileurl, librarytype, screeningimportsettings) rather than a nested array — see Phase 4 above. The risk this section warned about does not apply to the current contract.

The BulkPdfUpload kind (Phase 4) stays within the same limit by a different means: it never lists individual files in metadata at all (project/search/upload identity, conflict policy, and the expected whole-object size/SHA-256 only) — the full per-file manifest lives in the ZIP itself, processed by the PDF agent.

Recommendation: None outstanding for the search-import path. If a future upload kind needs to carry a variable-size manifest in metadata again, store minimal identifiers and have the consumer look up full details from a database, as originally recommended here.

9. Inconsistent Error Handling Strategy¶

Some errors delete created studies (rollback)
Some errors just mark the job as failed
The state machine transitions to Error but existing partial data may remain

Recommendation: Define and document a consistent error handling contract - either always roll back partial data or always keep it for manual review.

10. No Dead Letter Queue Strategy¶

If messages permanently fail, they'll exhaust retries and be moved to error queues, but there's no documented strategy for:

Monitoring dead letters
Alerting on accumulated failures
Manual retry or resolution procedures

Recommendation: Add observability and runbooks for DLQ handling.

Minor Improvements¶

Progress throttling (2 seconds) could be configurable
Concurrent job limit (5) should be based on resource constraints, not a magic number
Add correlation IDs to all log statements for distributed tracing
Consider using CloudEvents format for better observability tooling compatibility

Operational Notes¶

Saga State Collection Growth¶

The pmSearchImportJobState MongoDB collection stores saga instances for the SearchImportJobStateMachine. With TTL-based cleanup (replacing immediate SetCompleted deletion), saga instances persist for 7 days after reaching a terminal state (Completed or Error). The TTL index ttl_CompletedAt_7d on the CompletedAt field handles automatic cleanup.

Monitor: Collection size should remain proportional to the volume of search imports over the past 7 days. If the collection grows unexpectedly, check:

TTL index exists: db.pmSearchImportJobState.getIndexes() should show ttl_CompletedAt_7d
CompletedAt is being set: documents in terminal states should have a non-null CompletedAt value
MongoDB TTL monitor thread is running (runs every 60 seconds by default)

Dependency Map - Service dependencies
System Overview - Overall architecture
ADR-003: Cluster Architecture - Deployment architecture

Systematic Search Upload and Study Creation Flow¶

Table of Contents¶

Overview¶

Architecture Diagram¶

Sequence Diagram¶

Detailed Flow¶

Phase 1: Frontend File Selection and Validation¶

Phase 2: S3 Presigning Deep Dive¶

AWS Signature Version 4 Process¶

What the API Does¶

S3 Configuration¶

Phase 3: Direct S3 Upload¶

Phase 4: S3 Event Notification (Lambda)¶

Phase 5: State Machine Orchestration¶

States¶

Dual-Path Entry (Race Condition Handling)¶

Events and Transitions¶

Activities Executed¶

Timeout Handling¶

Phase 6: Reference File Parsing¶

Phase 7: Study Creation¶

Phase 8: Systematic Search Creation¶

Message Contracts¶

Events¶

Commands¶

Shared Interfaces¶

Domain Models¶

SearchImportJob¶

ReferenceFileParseJob¶

Study¶

SystematicSearch¶

Example Payloads¶

Presigning Request¶

Presigning Response¶

ISearchUploadStartedEvent¶

ISearchUploadSavedToS3Event¶

Bulk PDF release and processing commands¶

Error Handling¶

Upload Timeout¶

Parsing Failures¶

Rollback on Parse Error¶

Troubleshooting¶

Common Issues¶

Diagnostic Steps¶

Debug Mode¶

Screening Import Integration¶

Performance Characteristics¶

Implementation Critique¶

Strengths¶

Known Issues and Technical Debt¶

1. Presigned URL Security Gap¶

2. Silent Exception Swallowing¶

3. State Machine Duplicate Event Handling (Resolved)¶

4. Tight Coupling to S3 Path Convention¶

5. SystematicSearch Schema Migration Complexity¶

6. Missing Validation in ReferenceFileParseJob.UpdateProgress¶

7. No Circuit Breaker for MongoDB Operations¶

8. Metadata Size Limit Risk (Resolved by the flattened single-file contract)¶

9. Inconsistent Error Handling Strategy¶

10. No Dead Letter Queue Strategy¶

Minor Improvements¶

Operational Notes¶

Saga State Collection Growth¶

Related Documentation¶