Skip to content

Materialized Project Statistics Programme

Status and authority

This technical plan is the architecture gate for the complete pre-calculated/materialized statistics programme. It records intended architecture and staged delivery only. Approval authorizes completion of the detailed Phase 0 catalogue, mutation-ownership matrix and benchmark specification in a separate review; it does not authorize Phase 1 implementation, staging activation, production activation, data migration or legacy removal.

The programme replaces repeated full-project MongoDB aggregation with a rebuildable read projection while preserving the current authoritative Project and Study aggregates. It covers valid screening and annotation statistics at project, stage, membership/reviewer, question, and domain reconciliation scopes, together with durable per-project history.

The programme deliberately does not revive the broad agreement/kappa implementation from pull request

2534. That implementation can compare screening decisions originating in different stages. Any future

inter-rater agreement measure requires a separately reviewed same-stage cohort and denominator.

Contents

Decision summary

The following decisions are proposed for approval:

  1. Project and Study remain authoritative. Statistics are a disposable, rebuildable projection and must not become a second source of domain truth.
  2. One logical ProjectStatisticsProjection is physically partitioned by metric family and scope. A single ever-growing project document is rejected because it creates write contention, full-payload write amplification, and a MongoDB document-size risk.
  3. Current and historical readers use the same catalogue keys, scope identifiers, authorization rules, provenance fields, and generation model. Pages may adapt that contract to presentation DTOs, but may not create page-specific statistics stores or history formats.
  4. Each source mutation durably captures a stable event ID, aggregate identity/version, affected scope keys and content digest in the source transaction. Source writes never update a shared project manifest. An asynchronous per-project projector orders publication; unsupported multi-document operations first create durable affected-scope fences, then mutate, then rebuild.
  5. A requested statistics bundle is served entirely from one compatible Fresh generation or entirely from the authoritative calculation. Mixed-generation and partly materialized responses are rejected.
  6. History stores immutable, real observations at bounded checkpoints. It never invents pre-enable data, averages observations during compaction, or rewrites old snapshots after a repair.
  7. Materialized serving is disabled by default. A global kill switch, family flags, consumer flags, project allowlist, and shadow parity mode make every cutover independently reversible.
  8. Pull request #2985 supplies screening mechanics and tests, not the final storage or API shape. Its useful work is reconciled in Phase 2 after the shared Phase 1 contract exists.
  9. SignalR and page migration begin only after the projection contract is approved and proven. SignalR publishes generation invalidations; clients fetch an authorized snapshot rather than receiving a recalculated full aggregate on every Study event.
  10. Production activation and legacy removal remain separate explicit user decisions after staging soak.
  11. Bounded state-profile distributions are first-class statistics. Their stable profiles and formulas mirror approved live calculations; clients cannot request arbitrary dimension cross-products. Project profiles preserve valid combinations such as different reviewers recording both include and exclude decisions. Equivalent membership/reviewer profiles are returned only when authorized.

Programme MVP and progressive enhancement

The minimum usable runtime slice is:

  • Phase 1 shared dark foundation;
  • Phase 2 project screening materialization; and
  • the first Phase 5 screening-only API/Project Overview consumer cutover.

That screening-only consumer must request only the Phase 2 family; it must not use the broad FullStats bundle, which would coherently fall back until annotation and membership families also exist. The slice must reduce authoritative aggregation load, preserve current externally visible results, provide truthful history, and remain instantly reversible. Annotation and question families, additional pages, enhanced history UI, automatic repair, and any statistically validated agreement measure remain ordered enhancements rather than being forced into the first runtime cutover.

Problem and current architecture

StudyStatsQuery.GetFullProjectStatsAsync currently derives a broad FullStats response from a MongoDB aggregation. The response contains project screening, stage annotation, membership screening, and membership-stage annotation data. It is invoked by REST controllers, synchronous Project DTO mapping, reviewer-specific queries, and SignalR subscriptions.

The present hot paths have four consequences:

  • full multi-facet aggregation is repeated on ordinary reads;
  • Project DTO mapping blocks on asynchronous I/O through .Result;
  • every qualifying Project or Study change can trigger another full SignalR aggregation; and
  • there is no shared, versioned history or reliable source provenance.

Current main also contains separate live ProjectReviewerStats/StageReviewerStats count queries and a manually refreshed annotation question-answer tally embedded in Project. Those are part of the Phase 0 catalogue; they must not be duplicated under new names.

An annotation unit belongs to an individual Study and reviewer annotation. No demonstrated cross-study aggregate, shared unit definition, consumer contract, or performance need exists. Outcomes likewise belong to the per-Study annotation/export model, with no approved cross-study statistical aggregate or demonstrated materialization need. Unit- and outcome-level statistics and materialization are therefore outside this programme. A future feature may propose either only through a separate approval that identifies a concrete useful aggregate and its authoritative definition.

Operational progress such as import jobs, export jobs, Bulk PDF delivery, or live SignalR presence is not project statistical history. It stays in its existing operational model unless a later independent feature explicitly reclassifies it.

Scope and authoritative metric catalogue

Catalogue rules

Every materialized metric requires the following catalogue fields before implementation:

  • stable metric key and human meaning;
  • aggregation grain and scope identifiers;
  • authoritative source fields and inclusion/exclusion filters;
  • exact numerator, denominator, tally or grouping equation;
  • configuration inputs and compatibility digest;
  • mutation authorities and invalidation triggers;
  • current API, page, SignalR, export, or report consumers;
  • authorization class, including own-row and peer-row rules;
  • whether it is durable, checkpoint-history eligible, derived at query time, or intentionally live; and
  • exact parity representation, including ordering and rounding rules.

Integer counters and tallies require exact equality. A later floating-point metric must define a tolerance and rounding rule in its catalogue entry; no implicit tolerance is allowed.

Catalogue baseline

Family Scope Current metrics or intended definition Phase 0 disposition
Project screening Project Study count plus bounded distributions over approved screening-state profiles: included/excluded reviewer decisions, decision-count/status, sufficiency, started and overscreening states, including valid conflicting include/exclude combinations Approved current authoritative family; Phase 2
Membership screening Project + membership/reviewer Per-reviewer study distributions over approved decision, availability, sufficiency and completion profiles; included, excluded, screened, available, unavailable and total remain compatible views Approved current authoritative family; Phase 4 after project screening
Stage annotation Project + stage + inclusion class Study distributions over approved annotation-session profiles, including no session, incomplete/in-progress and completed states; domain-reconciliation states remain separately identified Approved current authoritative family; Phase 3
Membership-stage annotation Project + membership + stage + inclusion class Per-reviewer study distributions over approved available, unavailable, incomplete/in-progress and completed candidate-session profiles; reconciliation and capacity-derived values remain compatible views Approved current authoritative family; Phase 3
Reviewer screening Project + membership/reviewer Screened, available, unavailable and total from the separate reviewer query Catalogue against membership screening to remove aliases; Phase 4
Reviewer annotation Project + membership + stage Included/unexcluded and excluded in-progress/completed/available values, total and capacity status Catalogue against membership-stage annotation; Phase ¾
Question answers Project + stage + question/version Existing distinct-study and annotation-answer tally, with active/suppressed/version semantics made explicit Replace manual refresh only after catalogue approval; Phase 3
Project/stage derived summaries Catalogue-specific Valid totals or presentation-ready summaries derived from approved blocks Derive without duplicating source counters; Phase 4
Domain reconciliation Project + stage + membership where applicable Candidate/reconciliation session availability, started, in-progress and completed states Included with annotation families; not statistics parity repair
Agreement/kappa Same-stage cohort only No approved formula Excluded until a separate statistical specification is approved
Operational progress/presence Job or live connection Import/export/Bulk PDF progress and active connections Intentionally outside this programme

Question scopes must carry the immutable definition/version identifiers required to interpret an old observation. Renaming or deleting a definition cannot silently change historical meaning.

Bounded study-state profile distributions

A state-profile distribution is a catalogue-defined set of meaningful current Study classifications, not a generic analytics cube. Each profile family specifies stable profile keys, exact formulas, source fields, stage/configuration inputs, exclusivity or permitted overlap, ordering and schema/source version. The dimensions must mirror an approved authoritative live calculation. Adding a dimension or changing a formula is a catalogue/schema change, not a query option.

The initial profile families are:

  • Project screening: counts of studies by meaningful combinations of reviewer include/exclude decisions and screening-decision number/status, plus approved sufficiency, started and overscreening classifications. A Study may validly have include and exclude decisions from different reviewers; those conflicting combinations are preserved and tallied rather than labelled impossible.
  • Membership/reviewer screening: equivalent counts from the authorized individual reviewer's perspective, retaining that reviewer's decision/status and the approved project context without leaking peer decisions.
  • Project/stage annotation: counts of studies by approved candidate-session state combinations, such as no session, incomplete/in-progress and completed, partitioned only by the stage and inclusion class used by the current authoritative calculation.
  • Membership/reviewer-stage annotation: equivalent authorized per-reviewer distributions over that reviewer's availability and incomplete/in-progress/completed session states.

Profiles are omitted as impossible only when an enforced application invariant proves the state cannot exist. Observed combinations that are unusual, internally conflicting between reviewers, or merely absent from a fixture remain representable. Unknown combinations caused by schema drift make the family Incompatible or Stale and force authoritative fallback; they are never silently dropped into an other bucket. The catalogue may retain simple totals as named projections of these distributions, but the materialized contract must not collapse the distributions into totals alone.

Target architecture

Source-of-truth boundary

Project owns configuration, stages, memberships, permissions and question definitions. Study owns screening decisions, annotation sessions, domain reconciliation sessions and annotations. Statistics consume facts from those aggregates but cannot enforce or redefine their invariants.

Source mutations write stable domain facts to a durable statistics source-event outbox in the same transaction rather than returning a statistics-shaped delta from the aggregate. A projector translates versioned source facts into metric blocks. This keeps the domain model independent of one reporting schema and makes replay possible after process failure or disabled projector writes.

Logical data model

flowchart LR
    P[Project / Study source mutation] -->|same transaction| E[Durable source-event outbox]
    P -->|bulk/configuration operation| F[Durable per-scope operation fence]
    E --> X[Asynchronous per-project projector]
    X --> L[Applied-event ledger]
    X --> B[Versioned statistics blocks]
    B --> I[Partitioned generation index]
    I --> G[Immutable generation header]
    G -->|CAS publish| M[Project statistics manifest]
    M --> Q[Authorized current/history query]
    Q -->|Fresh and compatible| C[Materialized response]
    Q -->|Missing, stale, rebuilding, incompatible or disabled| A[Authoritative response]

ProjectStatisticsManifest

One manifest per project contains control metadata only:

  • project ID;
  • metric catalogue, storage schema and projector source versions;
  • current published generation ID;
  • per-family/per-scope lifecycle and projection revision;
  • last published projection sequence and bounded source-aggregate gap metadata;
  • configuration digest by affected family;
  • last rebuild/parity failure metadata; and
  • tombstone/deletion state.

The manifest must not contain the complete metric payload or full history.

ProjectStatisticsBlock

A block is addressed by:

(ProjectId, BlockVersionId, MetricFamily, ScopeKey, ProfileSchemaVersion?).

ScopeKey is a canonical typed tuple containing every dimension in a fixed family-specific order. For example: project; stage + inclusion class; membership + stage + inclusion class; or stage + immutable question version. Its canonical encoding and digest are versioned. A generic single ScopeId is not permitted because it can collide across multidimensional families.

It contains a canonical typed payload, content digest, projection revision and metric-family schema/source versions. Blocks are immutable once a generation references them. Copy-on-write publication creates only changed blocks and reuses unchanged block references.

ProjectStatisticsGeneration and index partitions

A generation is a small immutable header containing:

  • generation ID and monotonic project generation number;
  • source-event high-water mark;
  • observed and calculated times;
  • catalogue/schema/source versions;
  • configuration and content digests;
  • reason/trigger and initiating operation/event ID;
  • affected family/scope set;
  • history checkpoint kind; and
  • backfill, rebuild, parity-repair, import or ordinary-mutation provenance.

Block references live in bounded immutable ProjectStatisticsGenerationPartition documents keyed by project, generation and a family/scope hash range. A delta generation names only changed partition keys and its parent; readers resolve requested partitions through a bounded parent chain. A periodic checkpoint generation materializes all current partitions as separate documents when the chain reaches the configured maximum depth. No generation document contains or rewrites the full project-wide map.

The manifest points to one current generation header. A history checkpoint references an already immutable generation, so current publication and history publication cannot disagree after partial failure.

ProjectStatisticsSourceEvent

The durable source-event outbox has a unique stable event ID and records project, source aggregate identity/version, affected canonical scope keys, event type/version, bounded replay payload or durable payload reference, content digest, operation identity, creation time and Pending/Applied/Failed status. Source data and this event commit atomically. Pending events are indexed by project + affected scope + status and act as a read fence: a requested scope cannot be considered Fresh while a matching event is pending. Payload retention exceeds the maximum replay/rebuild horizon and an event cannot be compacted until application is durably recorded.

ProjectStatisticsAppliedEvent

The applied-event ledger has a unique key of:

(ProjectId, SourceEventId, ProjectorVersion).

It records source aggregate IDs/versions, affected metric scopes, event content digest, assigned projection sequence, applied generation and time. Detailed rows use bounded retention only after the source event is terminal and the manifest/projector checkpoint proves it cannot be replayed as new work.

ProjectStatisticsRebuildLease

Leases are separate documents keyed by project + metric family + canonical scope key. Each records its token, monotonic lease generation, owner, expiry and captured projection/source watermark. Independent scopes can rebuild concurrently without overwriting a single manifest field.

ProjectStatisticsOperationFence

A durable operation fence is keyed by project + operation ID + metric family + canonical scope key. It records the operation kind, affected scope, lifecycle state, restartable batch progress, creation/expiry times and the source snapshot boundary required for publication. Active fences are indexed by scope and are independent of the manifest so a point publisher cannot accidentally clear a bulk, import, configuration or deletion fence.

Lifecycle

Each metric family/scope follows:

Missing/Incompatible -> Stale -> Rebuilding -> Fresh.

  • A supported source mutation commits its Pending source event with the source aggregate. Until the asynchronous projector publishes it and marks it Applied in one projection transaction, the pending scope fence forces authoritative current reads.
  • An unsupported, bulk or configuration mutation advances the fence and marks only affected scopes Stale before source changes can make old data unsafe.
  • Rebuilding data always uses authoritative reads.
  • A rebuild publishes only when its lease, source snapshot boundary and captured source high-water mark still match.
  • A newer source event either causes compare-and-swap publication failure or is replayed before publish.
  • No scope can become Fresh while an active operation fence covers it, even if a supported point event is projected during that operation.
  • Failure returns the scope to Stale and preserves current/history for diagnosis.
  • A stale or incompatible block is never served merely because its values look plausible.

History and provenance

Current versus historical generations

Operational current generations may advance after every supported mutation. User-visible history uses explicit checkpoints so storage growth is bounded and timelines remain meaningful.

The initial policy is:

  • retain each changed daily observation for 90 days;
  • retain one real changed observation per week through one year;
  • retain one real month-end observation through five years;
  • retain at most 256 history generations per project unless measured payload growth requires a lower bound; and
  • keep current blocks independent from the history cap.

Compaction selects an existing observation. It does not average, interpolate, merge incompatible schema versions, or synthesize a value for a day on which nothing was observed.

Reachability and storage reclamation

Operational generations that are neither current nor retained history checkpoints remain eligible for a short recovery grace period only. Garbage collection is reference-aware:

  1. capture a collection epoch and mark the current generation plus every retained history generation as roots;
  2. traverse their bounded parent chains, generation partitions and block references;
  3. retain every reachable header, partition and block, including blocks reused by newer generations;
  4. sweep only unreachable objects older than the safety grace period after verifying the manifest has not advanced past the collection epoch; and
  5. retry from a new epoch rather than delete when the root set changes concurrently.

This applies the 256-checkpoint history bound without deleting blocks reused by a surviving generation. Telemetry records roots, reachable/unreachable bytes, sweep duration and aborted collection races.

Bootstrap and rebuild truthfulness

Bootstrap records exactly one backfill-observed generation at the time the authoritative calculation was made. It includes source high-water mark, definitions/configuration digests, versions, scope and initiating operation. It does not create points for earlier dates.

A rebuild records its captured and published source high-water marks. A parity repair retains the failed audit evidence and publishes a new generation; it does not rewrite the mismatching historical record.

History survives Stale and Rebuilding state transitions. It remains queryable only through the normal current authorization boundary.

Definition changes and deletion

  • Question history preserves immutable definition/version identifiers and display metadata needed for interpretation.
  • Removing a membership prevents the former member from reading current or historical project data. Stored reviewer rows are filtered using current authorization.
  • Question deletion removes that scope from current generations but does not mutate old history.
  • Project deletion immediately makes current/history APIs unavailable, tombstones the manifest and schedules projection/history purge under the approved project-retention policy. This programme does not create an independent right to retain a deleted project.

Event and invalidation contract

The complete implementation matrix is a Phase 0 deliverable and a gate for each family. The baseline matrix is:

Source event or operation Affected families Consistency strategy Historical reason
Screening submit, correction or rescreen Project and membership/reviewer screening Source + Pending event in one transaction; asynchronous idempotent delta publication; stable operation ID screening-submitted / screening-corrected
Screening decision deletion/reset Project and membership/reviewer screening Atomic before/after event when bounded; otherwise fence and rebuild screening-reset
Candidate annotation session start/save/complete/delete Stage and membership-stage annotation profiles; approved question blocks Source + Pending event in one transaction; asynchronous projector classifies before/after state annotation-session-*
Durable slot-reservation allocate, release, disconnect-timeout or expiry Membership-stage annotation availability/capacity and affected annotation-state profiles Source + Pending event in the reservation transaction; live presence alone remains operational and outside this programme annotation-reservation-*
Domain reconciliation start/save/complete/delete Stage and membership-stage reconciliation profiles; approved question blocks Source + Pending event in one transaction; reconciliation is not parity repair domain-reconciliation-*
Annotation content or suppression change Question and affected annotation-state profile blocks Stable question definition/version IDs; atomic event; rebuild when profile classification is unsupported annotation-content-changed
Screening threshold or agreement setting change Screening families Advance configuration fence and mark affected scopes Stale before mutation; rebuild screening-configuration-changed
Stage target, self-reconciliation, selected-question or filter change Annotation and affected screening/derived families Configuration fence; digest mismatch is a second fail-safe stage-configuration-changed
Membership add/update/remove or permission change Membership/reviewer families and authorized response shape Fence affected scopes; current query authorization always re-evaluated membership-changed
Question create, version, reorder, detach or delete Annotation/question families Fence affected stage scopes before whole-Project replacement; retain old history identity definition-changed
Study import or single Study creation All study-population-dependent families Bounded create may publish atomic event; imports use two-phase fence/mutate/rebuild studies-imported
Bulk Study update or deletion Catalogue-derived affected scopes Two-phase operation with durable operation ID and Stale fence studies-bulk-changed
Systematic search deletion or import rollback Population, screening and annotation families Fence before multi-document mutation; rebuild or delete affected blocks search-studies-changed
Project deletion All Tombstone and deny reads, then policy-driven purge No new user-visible checkpoint
Manual legacy question-tally refresh Question family Superseded after projector cutover; before cutover it remains authoritative legacy-question-tally-refresh
Retry, broker redelivery or unknown commit result Event's original scopes Deterministic event/operation ID and sequence; verify ledger before replay No duplicate checkpoint

Exports, reports, DTO mapping, page reads and SignalR are consumers and do not mutate statistics.

Concurrency, retries, and rebuilds

Source ordering without a workflow write hotspot

The source transaction writes only its source aggregate and ProjectStatisticsSourceEvent; it never updates the shared project manifest, generation or a project-wide counter. Each event carries the source aggregate ID and monotonic aggregate version. The asynchronous projector partitions work by project, validates contiguous versions per source aggregate, and assigns a monotonic ProjectionSequence while serializing publication. Independent reviewer writes to different Studies therefore do not conflict on a statistics document.

Events from different source aggregates may be projected in either arrival order because their assigned projection order is recorded. A missing aggregate version, conflicting duplicate or incompatible projector version marks affected scopes Stale rather than guessing. Pending source events remain durable read fences, so publication lag cannot expose an old generation as Fresh.

Detailed event rows may expire only after the maximum replay/recovery horizon plus safety margin. The terminal source-event record, applied ledger and projector checkpoint must together prove that an older aggregate version cannot be accepted as new work before payload compaction.

Transaction and idempotency rules

  • A supported point mutation commits source data and its Pending source event atomically with a stable operation/event ID. It does not publish statistics synchronously.
  • The projector transaction atomically creates changed blocks/partitions/generation, advances the manifest, records the applied-event digest and changes the source event to Applied. A crash cannot acknowledge the event without publishing the corresponding generation.
  • Duplicate event IDs are successful no-ops only when their recorded source aggregate version, projector version and content digest agree. Conflicting duplicates fail closed and trigger parity investigation.
  • Transient source or projector transaction errors retry the complete respective transaction with the same operation/event ID.
  • An unknown source commit is resolved from the source-event row and aggregate version; an unknown projector commit is resolved from the applied-event row and manifest generation before retrying.
  • No handler generates a fresh operation ID during redelivery.
  • Projection publication is optimistic; lost races reload and reclassify or mark Stale.

Bulk and import operations

Multi-document import, bulk update and deletion use a durable two-phase protocol:

  1. allocate one operation/event identity and persist an Active operation fence for every affected scope;
  2. mark the exact affected metric scopes Stale; the Active fence, not a manifest state written by a point publisher, is the authority that prevents those scopes returning to Fresh;
  3. commit source batches with restartable progress and stable child IDs;
  4. calculate an authoritative result from a snapshot-isolated source read tied to a durable source-event boundary;
  5. replay every affected event committed after that snapshot boundary, or reject and restart when the datastore cannot prove the replay boundary;
  6. publish through the normal lease/CAS transaction and clear the operation fences atomically with the generation that incorporates every fenced batch and concurrent point event; and
  7. mark the bulk operation complete only after the publication and fence-clear postconditions are verified.

A supported point event may be projected while the bulk operation runs, but its transaction must observe the Active fence and cannot mark or expose that scope as Fresh. An abandoned or expired operation leaves the scope Stale and requires an explicit resumed operation or rebuild; timeout never clears the fence into Fresh. A failure after the fence therefore causes extra authoritative reads but cannot serve a partial materialized result.

Rebuild leases

Leases are scoped by project, metric family and scope. They contain a random token, monotonic lease generation, owner, start/expiry times and captured source high-water mark. Expired owners cannot publish or release a successor's lease. Lease acquisition never blocks authoritative fallback.

The authoritative rebuild calculation must run in a snapshot-isolated source read whose database snapshot token/cluster time and visible durable source-event boundary are captured together. The projector then replays every affected event committed after that exact boundary and loses the final compare-and-swap if another event arrives before publication. If the datastore or repository path cannot provide and validate that snapshot boundary, the rebuild rejects and restarts whenever any affected source revision advances; it must not replay events over an indeterminate live calculation. It therefore cannot double-apply or silently omit a concurrent source mutation.

Query and authorization contracts

Canonical endpoints

The proposed domain-neutral contract is:

GET /api/projects/{projectId}/statistics/current?metrics={keys}&stageIds={ids}&membershipIds={ids}
GET /api/projects/{projectId}/statistics/history?metrics={keys}&stageIds={ids}&membershipIds={ids}&beforeGeneration={cursor}&limit={n}
GET /api/projects/{projectId}/statistics/history/{generation}?metrics={keys}&stageIds={ids}&membershipIds={ids}
POST /api/projects/{projectId}/statistics/audits
POST /api/projects/{projectId}/statistics/rebuilds

Current and available-history responses carry:

  • requested metric catalogue keys and scopes;
  • lifecycle and read source (Materialized, AuthoritativeFallback or MaterializedCheckpoint);
  • fallback reason;
  • generation and projection revision;
  • source-event high-water mark;
  • catalogue/schema/source versions;
  • configuration and content digests;
  • observed/calculated times and trigger; and
  • authorized typed metric blocks, including stable profile keys and counts for requested profile families.

The history cursor is a monotonic generation number, not a timestamp. Ordering is newest first and stable under retention compaction. The list and exact-generation contracts accept the same membership and stage scope selectors as current reads.

Authorization

  • Project/stage totals require the existing project-view boundary and metric-specific graph permissions.
  • Reviewer/member blocks are keyed by membership identity while carrying investigator/person identity. A reviewer's own row remains available where current behavior permits it; peer rows/leaderboards require the corresponding project permission.
  • Historical responses are filtered under current authorization, not authorization captured when the snapshot was written.
  • Audit/rebuild operations require project-edit or the narrower administrative permission approved in Phase 0.
  • Export/report callers use their existing explicit export permissions and the same query service.
  • The persistence document is a superset and is never serialized directly.

Coherent current fallback and historical availability

A current response may use materialization only when every requested block is present, Fresh, compatible, part of the same published generation and has no matching Pending source-event or Active operation fence. Otherwise the query executes the authoritative current aggregate for the entire requested bundle. This prevents a response combining fresh screening with stale annotation or different configuration versions. The manifest/generation and pending-fence predicate are evaluated in one majority read snapshot; an implementation without that facility must recheck the fence after loading blocks before returning them.

A historical response never uses the current authoritative aggregate as fallback because Project and Study cannot reconstruct an older checkpoint. A retained compatible generation returns MaterializedCheckpoint. A checkpoint that was compacted, is incomplete, or has no compatible historical reader returns a typed HistoryCheckpointUnavailable result (and 410 Gone for an exact-generation request) with generation, observed time when known, and bounded reason. It never returns current values labelled as history or mixes blocks from different checkpoints.

Observability and acceptance metrics

Required telemetry

OpenTelemetry instruments must cover:

  • current/history requests and requested block counts;
  • materialized reads, authoritative current fallbacks and historical-checkpoint unavailability by bounded reason;
  • authoritative aggregation count and duration;
  • materialized read and publication duration;
  • event applied, deduplicated, gap and conflict counts;
  • source-to-projection sequence lag;
  • rebuild requested/acquired/completed/failed/lost-race counts and duration;
  • parity audits, mismatches, invalidations and repairs;
  • history checkpoint, compaction and retained-block counts;
  • generation/block byte sizes and per-family growth;
  • global/family/consumer flag decisions; and
  • SignalR invalidations and subsequent snapshot fetches.

Metrics must not label by project, membership, investigator or question ID. Those IDs may appear in structured, access-controlled diagnostic logs with operation/event correlation.

Histograms must support p50 and p95 reporting for authoritative aggregation, materialized reads, publication, rebuild and end-to-end consumer latency.

Common correctness gates

Every implementation phase requires:

  • 100% exact parity for integer counters/tallies over its representative corpus and pilot observations;
  • zero materialized responses from Missing, Stale, Rebuilding or Incompatible scopes;
  • 100% authoritative fallback success in injected lifecycle, flag and failure cases;
  • mutation/event coverage for every catalogue authority affecting that family;
  • no partial source/current/history state after injected transaction failure;
  • idempotent retry and old redelivery tests;
  • deterministic ordering and retention/compaction tests; and
  • permission parity for current and historical responses.

Common performance gates

Phase 0 records current p50/p95 and query/facet counts using fixed datasets. Before a consumer cutover:

  • materialized response p95 must be at least 20% lower than its authoritative equivalent;
  • that consumer must execute at least 80% fewer authoritative statistics aggregations during a representative shadow/pilot run;
  • supported source-mutation p95 regression must remain below 10%;
  • materialized block/storage growth must remain within the Phase 0 capacity model; and
  • parity or fallback instrumentation must add no unbounded-cardinality series.

If a family fails a performance gate, it remains dark or is revised; correctness gates are never relaxed to obtain the performance result.

The preserved screening candidate measured 1,000 studies over alternating samples with legacy p50/p95 of 75.29/99.06 ms and candidate p50/p95 of 69.63/77.47 ms. This is promising local evidence, not a Phase 2 pass: the generalized implementation must rerun the controlled proof and assert the agreed threshold.

Rollout, fallback, and rollback

Flags and allowlists

Later implementation uses the repository's generated feature-flag catalogue. The proposed controls are:

  • global materialized serving kill switch;
  • global projector/write kill switch;
  • shadow parity mode;
  • per-family write/serve controls for screening, annotation and each later family;
  • independent consumer flags for Project Overview/API, Stage Overview, reviewer pages, SignalR, exports/reports and any history UI; and
  • an explicit project pilot allowlist, with no implicit wildcard outside reviewed configuration.

The exact names are finalized with the generated flag implementation. All default to off. Disabling serving immediately returns reads to the unchanged authoritative path without deleting projection or history data. Disabling writes freezes projection advancement and therefore also prevents serving once a matching Pending source event exists beyond the published projection checkpoint.

Rollback principles

  • Schema additions are additive until Phase 6 legacy retirement.
  • Legacy calculations and wire adapters remain available throughout pilot and soak.
  • A consumer flag rollback restores the legacy response without data migration.
  • Projection schema incompatibility marks affected scopes Stale and falls back.
  • A bad projector release stops writes or advances the source version; it never rewrites compatible old history in place.
  • Rollback does not require deleting projection collections.
  • No production flag activation, data migration or legacy deletion is implicit in merging code.

Staged delivery plan

Phase 0: inventory, catalogue, baselines, and architecture

Outcome: one reviewed source of design truth before runtime changes.

PR boundary and waves

  • This documentation PR defines architecture, safety invariants, phase boundaries and the baseline catalogue. It does not claim the method-level Phase 0 inventory is complete.
  • A separate documentation-only Phase 0 completion PR must name every calculation/consumer, method-level mutation owner, stable profile formula, fixed benchmark dataset and executable benchmark command.
  • Parallel read-only analysis may refine metric equations/consumers, mutation authorities and benchmark datasets. The results join this document or a directly linked catalogue update; they do not create product code.

Migration, backfill and rollback: none.

History and event coverage: approve the canonical history semantics and complete method-level source/mutation ownership for every catalogue family. Unit- and outcome-level materialization cannot enter this programme. Every profile family must document its bounded keys, formulas, authorized scopes, permitted overlaps and the application invariants that make any state impossible.

Acceptance and tests: this architecture PR passes documentation validation and link checks. The separate Phase 0 completion PR must prove every known current calculation and consumer has a catalogue disposition and must name fixed benchmark datasets and executable commands before Phase 1 is authorized.

Staging and production safety: no environment work.

User gate: approval of this PR authorizes only the separate documentation/read-only Phase 0 completion work. Phase 1 planning or implementation requires a second explicit user approval after that completion PR satisfies the catalogue, mutation-ownership and benchmark gates. No later merge, staging activation or production activation is authorized here.

Phase 1: shared versioned projection foundation, dark only

Outcome: reusable lifecycle, storage, history, event, rebuild and query infrastructure with no page cutover.

PR boundary and waves

  1. 1A — contracts and additive schema: manifest/block/generation/generation-partition, source-event/applied-event/rebuild-lease/operation-fence types, indexes, serializers and repository contracts.
  2. After 1A, two non-overlapping PRs may proceed in parallel:
  3. 1B — publication/event foundation: transactional source-event outbox, aggregate-version gap handling, asynchronous projection sequence, idempotency, CAS publication, transaction retry and invalidation registry;
  4. 1C — read/control foundation: domain-neutral query envelopes, current authorization filtering, lifecycle, flags and telemetry.
  5. 1D — lifecycle integration: per-scope rebuild leases, backfill, bounded generation-chain compaction, reachability collection, history query/unavailability and failure/repair evidence, integrating 1B and 1C.

Migration: create additive pmProjectStatisticsManifest, pmProjectStatisticsBlock, pmProjectStatisticsGeneration, pmProjectStatisticsGenerationPartition, pmProjectStatisticsSourceEvent, pmProjectStatisticsAppliedEvent, pmProjectStatisticsRebuildLease and pmProjectStatisticsOperationFence collections/indexes. No Project or Study rewrite and no legacy deletion.

Backfill and rollback: allowlisted dry-run only; reads stay authoritative. Rollback disables writes and ignores the new collections.

History and event coverage: prove durable source-event capture, asynchronous projection ordering, aggregate-version gap handling, generation publication, checkpoint selection and current/history atomicity with synthetic metric blocks before domain payloads arrive.

Acceptance and tests: domain tests, Mongo replica-set source/outbox and publication transaction tests, duplicate/digest/unknown-commit tests, independent concurrent source-writer tests proving no shared manifest write, concurrent per-scope lease tests, generation-partition/chain-bound tests, snapshot-boundary/replay tests, point-publication-during-bulk-fence tests, reachability-retention tests, migration/index tests, authorization tests and failure-injection fallback and historical-unavailability tests.

Staging proof: after separate approval, deploy disabled, verify additive indexes and startup, enable only a synthetic/non-serving allowlisted projection, exercise kill switches and rollback. No page reads may use it.

Production safety and user gate: flags remain off; no production backfill. User reviews every PR and approves completion of Phase 1 before Phase 2 merges.

Phase 2: project screening statistics

Outcome: reconcile the useful pull request #2985 screening work onto the shared foundation.

PR boundary and waves

  1. 2A — authoritative definition and projector: project screening profile payload, classifier/delta, authoritative query and history block. Profiles preserve valid conflicting include/exclude decisions across reviewers and mirror the approved live classifier.
  2. 2B — mutation adapters: screening submit/correct/rescreen/reset, threshold/stage/membership paths, imports, bulk updates and deletion fencing. This can proceed alongside 2A tests only where file ownership is disjoint.
  3. 2C — integration proof: API query-family adapter, parity audit, backfill/rebuild and controlled performance/regression harness. No ordinary page or SignalR cutover.

Migration: no source schema rewrite; create screening blocks/generations through pilot backfill.

Backfill and rollback: project allowlist, restartable and idempotent. A flag rollback returns every read to the authoritative screening facets and preserves history.

History and event coverage: screening checkpoints use the shared policy; current deltas and history publication share the generation transaction; configuration changes preserve old history and rebuild current.

Acceptance and tests: exact profile and total parity across zero/one/many-screening cases, mixed include/exclude decisions from different reviewers, threshold boundaries, rescreening stages, duplicate delivery, concurrent reviewers, transaction rollback, import/bulk/delete fences, authorization, history ordering/retention and the common performance gates. Tests exclude a state only when an enforced domain invariant proves it impossible.

Staging proof: after explicit approval, dark-write an allowlisted corpus, run shadow parity and load, force Stale/Rebuilding/Incompatible and kill-switch paths, and prove legacy response equivalence.

Production safety and user gate: no production serving. User accepts Phase 2 evidence before any consumer or wider-family work is promoted.

Phase 3: annotation, question, and domain reconciliation

Outcome: materialize the catalogue-approved annotation graph without reviving broad kappa work.

PR boundary and waves

  1. 3A — stage candidate/reconciliation statistics: bounded Study distributions over approved inclusion, no-session, incomplete/in-progress, completed and domain-reconciliation state profiles.
  2. 3B — membership/reviewer-stage statistics: authorized per-reviewer distributions over availability, incomplete/in-progress/completed and reconciliation profiles, plus approved capacity-derived metrics.
  3. 3C — question family: version-aware question tallies only.
  4. 3D — mutation/invalidation integration: serialized wiring across shared Study/Project and durable SlotReservation repository/controller mutation paths, followed by the combined parity/load proof.

3A–3C may develop projector payloads, authoritative queries and tests in parallel after Phase 1 contracts. Shared source mutation files remain single-writer and join only in 3D.

Migration: additive blocks only. The manual Project question tally remains until its replacement has passed parity and consumer cutover.

Backfill and rollback: family/project allowlists; each family can rebuild independently. Disabling one family causes coherent authoritative fallback for bundles requesting it.

History and event coverage: candidate and domain reconciliation transitions, annotation content changes, question-definition versions, suppression, membership/stage configuration, imports, correlated persistence, bulk update and deletion.

Acceptance and tests: profile classification truth tables, stage/inclusion splits, membership availability, candidate/reconciliation lifecycle, incomplete/completed session combinations, current-schema and legacy answer cases, reservation allocate/release/disconnect-timeout/expiry, concurrent writers, out-of-order events, definition deletion/version changes, parity, authorization, history and storage growth. Live connection presence remains outside the programme; only its durable reservation side effects are authoritative inputs. Unit- and outcome-level statistics remain outside the programme and are not reconsidered by this phase.

Staging proof: separately approved family-by-family dark backfill, representative annotation projects, controlled concurrent saves/reconciliation/import/delete operations, shadow parity and rollback drills.

Production safety and user gate: all serve flags off. User accepts each family and its metric catalogue before Phase 4 or consumer migration.

Phase 4: remaining valid project, stage, membership and derived summaries

Outcome: complete the approved catalogue without duplicating source counters.

PR boundary and waves

  • 4A — membership/reviewer screening: reconcile reviewer and membership aliases into canonical scoped blocks.
  • 4B — approved project/stage derived summaries: derive from generation-consistent blocks where possible; materialize only when the performance case requires it.
  • 4C — report/export projection adapters: prove the shared contract can satisfy approved reporting statistics without importing job-progress semantics.

These PRs may proceed in parallel when they consume stable Phase 1–3 blocks and do not share mutation adapters.

Migration/backfill/rollback: additive family blocks and independent allowlists. Derived summaries are rebuildable and revert to authoritative calculation on any dependency mismatch.

History and event coverage: a derived historical response references one generation; it cannot join blocks from different checkpoints. Dependency changes invalidate the derived scope.

Acceptance and tests: exact equality with current reviewer queries and approved report/export fixtures, dependency invalidation, authorization, retention and common performance gates.

Staging proof: dark family backfill, parity across admin and ordinary reviewer views, report/export fixture comparison, storage/load measurement and rollback.

Production safety and user gate: no consumer activation. Agreement/kappa remains excluded unless an independent statistical specification has been approved.

Phase 5: page-by-page and consumer migration

Outcome: remove authoritative aggregation from hot consumers one reversible surface at a time.

PR boundary and order

  1. 5.1 — screening-only API and Project Overview screening surface. Request only the Phase 2 family so this first runtime slice can actually use materialization. Keep broad Project DTO/FullStats mapping authoritative until all families it requests are available.
  2. 5.2 — Stage Overview and stage charts. Consume catalogue keys rather than a page-specific cache.
  3. 5.3 — reviewer/stage review surfaces. Preserve own-row/peer-row authorization and availability semantics.
  4. 5.4 — SignalR. Publish generation/scope invalidations; authorized clients fetch a snapshot and discard older revisions.
  5. 5.5 — exports and reports. Use the same query service and generation provenance.
  6. A visible history UI is optional and follows the shared history endpoint; it is not required to prove durable backend history.

Each consumer is its own PR and independently reversible flag. The active-reviewer work delivered by pull request #2467 is now on main; SignalR begins only after the shared statistics contract is approved and integrates against that published boundary.

Migration/backfill/rollback: no destructive schema migration. A consumer may activate only for projects whose requested families are Fresh; its flag rollback immediately restores the unchanged legacy path.

History and event coverage: consumer adapters cannot invent additional counters or historical formats. SignalR carries generation/scope/revision only, not unauthorized raw projection blocks.

Acceptance and tests: API/DTO and ngrx contract tests, page selectors, SignalR ordering/reconnect, authorization, export fixture parity, shadow comparison and common performance gates.

Staging proof: one page/consumer at a time, independently flagged, with before/after p50/p95, authoritative aggregation count, parity, fallback and rollback evidence.

Production safety and user gate: no automatic activation after merge. User approves each consumer's staging evidence and any later production pilot separately.

Phase 6: fleet backfill, soak, audit, and legacy retirement

Outcome: prove the programme under sustained load before removing legacy calculations.

PR boundary and order

  • 6A — bounded fleet backfill/repair operations: admission, rate limits, leases, progress and safe cancellation.
  • 6B — soak/audit/retention evidence: parity sampling, rollback drill, load/cost and storage proof.
  • 6C — legacy deprecation: separate PRs after every consumer has passed and the rollback window is complete.

Migration/backfill/rollback: bounded batches, project/family admission, resume tokens and global stop. Backfill never invents history. Legacy code and indexes remain through the agreed rollback window.

History and event coverage: audit every catalogue family; compaction and schema migration retain interpretability and provenance. Repairs append evidence and a new generation.

Acceptance and tests: at least seven days of staging soak plus controlled 10,000 reads and 1,000 relevant mutations; 100% exact counter parity; zero stale materialized serves; 100% injected fallback success; no unresolved rebuild failures; bounded event lag/storage; and the common performance gates.

Staging proof: full backfill, failure injection, rollback to authoritative reads, restart/resume, retention compaction and consumer rollback.

Production safety and user gate: production pilot, wider production rollout, legacy calculation removal and old collection/index cleanup are four separate explicit approvals. This plan grants none of them.

Reuse and supersession

Pull request #2985: reusable screening foundation

The preserved screening-only implementation contributes:

  • explicit Fresh/Stale/Rebuilding and authoritative fallback;
  • schema/source versions, calculation time, reason, stage scope, content/configuration watermarks;
  • rebuild lease expiry and publication-race rejection;
  • atomic Study + screening statistics mutation, optimistic retry and redelivery handling;
  • pilot allowlist, global kill switch, parity audit and OpenTelemetry instruments;
  • current/history authorization precedent; and
  • domain, repository, API, SignalR and controlled Mongo transaction tests.

It must be revised before reuse:

  • replace screening-specific document/service/controller/flag terminology with shared contracts;
  • split manifest, blocks, generations and applied-event history;
  • rename its projection-local SourceRevision and introduce an authoritative source-event sequence;
  • stop appending a complete payload snapshot after every mutation;
  • replace the bounded in-document recent-operation list with the durable event ledger/high-water mark;
  • add per-family lifecycle and coherent whole-bundle fallback;
  • add shadow mode and independent consumer flags; and
  • replace its final mixed FullStats/SignalR splice with the Phase 5 consumer contract.

The local #2985 branch remains evidence only until Phase 1 is approved. Useful commits may be cherry-picked or reimplemented into Phase 2 after comparison with current main; the stale branch is not rebased or continued as the programme branch.

Pull request #2534: design evidence, not implementation base

Retain:

  • broad metric and consumer inventory;
  • screening/annotation/member/stage intent;
  • original event/invalidation and import/deletion analysis;
  • sparse history and realtime goals; and
  • test/risk/rollout questions.

Supersede:

  • magic current/tracking document identities and mutually exclusive payload/tracking rows;
  • daily snapshot overwrites without authoritative revision/provenance;
  • Legacy/Backfilling/Live serving ambiguity;
  • write-freeze/bootstrap coupling where fail-safe fallback suffices;
  • broad admin rollout UI in the foundation;
  • project-wide pairwise/Fleiss kappa and stage chart claims; and
  • its stale base branch and unrelated large diff.

Once this plan and successor implementation PRs provide traceable links, #2534 can be closed as superseded without deleting its preserved worktree until the history is safely recorded.

Validation strategy

Layer Required evidence
Metric domain Classification/delta truth tables, zero/empty cases, configuration boundaries, exact canonical serialization
Projection domain lifecycle, version compatibility, generation/block references, dedupe, ordering, retention and history provenance
Mongo repository unique/index migration, atomic source/current/history publication, CAS, leases, concurrency, rollback and unknown commit
Authoritative parity fixed representative datasets, admin and reviewer authorization views, exact per-family and whole-bundle comparison
Mutation paths every matrix row, supported atomic deltas, two-phase fences, imports/bulk/delete, retries and redelivery
API current/history/audit/rebuild contracts, cursor ordering, coherent fallback, permissions and no raw-superset disclosure
SignalR generation ordering, duplicate/old notification rejection, reconnect snapshot, fallback and authorization
Web consumers DTO/ngrx/selector/chart parity, independent flags, history adapter and rollback
Export/report approved fixture equality, permissions, provenance and no operational job-stat contamination
Performance fixed data sizes, alternating warm/cold samples, p50/p95, query/facet count, mutation overhead and storage growth
End-to-end controlled local/preview project through screen, annotate, reconcile, import, bulk update, delete, audit, rebuild and rollback

Risks and controls

Risk Consequence Control
Missed mutation path Fresh-labelled stale data Complete catalogue/method matrix, configuration digests, Stale fencing, parity audits and fallback
Projection becomes source of truth Irrecoverable semantic drift Project/Study remain authoritative; rebuildable blocks; no domain invariant depends on projection
One large project document Contention, write amplification, document-size failure Partitioned immutable blocks and copy-on-write generations
Mixed generation response Internally inconsistent page/export Whole-request generation check and coherent authoritative fallback
Duplicate/out-of-order delivery Double counts or regression Stable event ID, per-project sequence, unique ledger, gap detection and high-water mark
Rebuild races with writers Rebuild silently omits changes Captured high-water mark, replay/CAS, expiring generation-fenced lease
False history Misleading research record Observed-at timestamps, immutable definitions, no retroactive synthesis or averaging
Unbounded history/event growth Atlas storage/cost pressure Tiered 90-day/one-year/five-year retention, 256 cap, bounded event details and size telemetry
Permission leakage through history Former member or reviewer sees restricted rows Current authorization on every query; stored superset never serialized directly
Arbitrary profile cross-product Unbounded payload/cardinality and undefined meaning Only versioned catalogue profiles matching approved live formulas; no client-defined dimensions
Valid conflicting reviewer decisions dropped Project profile undercounts real Study states Preserve mixed include/exclude combinations; omit only states forbidden by enforced application invariants
Invalid cross-stage agreement Misleading quality claims Exclude #2534 kappa; require separate same-stage statistical specification
Rollout couples every page Large blast radius and difficult rollback Global/family/consumer flags, project allowlist, shadow parity and one-consumer PRs
Source write latency regression Review workflow slows Supported mutation p95 regression below 10%; otherwise use fenced async projection/rebuild

Review gates

Approval of this document should answer:

  • Are the metric families and exclusions correct?
  • Is a partitioned generation/block model preferred over a single project document?
  • Are the history checkpoints and five-year bound acceptable?
  • Are the exact-parity, 20% read-p95, 80% aggregation-reduction and 10% mutation-regression gates appropriate?
  • Are Phase 1 dark foundation, Phase 2 screening, Phase 3 annotation, Phase 4 remaining families, Phase 5 consumer cutovers and Phase 6 retirement the correct order?
  • Is a separate Phase 0 completion PR required before any Phase 1 implementation approval? (Proposed: yes.)

Until those questions are accepted, broad implementation remains paused.

  • Epic #1831 — pre-calculated statistics programme
  • Pull request #2534 — broad original implementation/design evidence
  • Pull request #2985 — preserved screening-only implementation evidence
  • Issues #1836–#1850 — original screening, annotation, audit and history decomposition
  • FEAT-006 — domain reconciliation workflow
  • FEAT-009 — screening annotations
  • FEAT-013 — exports and reporting consumers
  • StudyStatsQuery and FullStats on current main — authoritative legacy calculation

FEAT-006 and FEAT-009 are catalogue compatibility inputs, not delivery prerequisites: this programme mirrors whatever versions of their live calculations are approved when a metric family enters Phase 0. Accordingly, depends-on remains empty and the catalogue does not claim hard dependency edges.