Skip to content

Materialized Project Statistics

SyRF will provide fast, authorized current and historical project statistics without repeatedly running the complete authoritative MongoDB aggregate on ordinary reads. Materialized values remain disposable, versioned read projections; Project and Study remain the source of truth and every current consumer can fall back safely to the approved live calculation.

The detailed architecture and phased delivery gates are defined in the technical plan.

User outcomes

  • Project users can load approved screening and annotation statistics without the latency of a full project-wide recalculation on every ordinary read.
  • Users can understand meaningful distributions of Study workflow state, rather than seeing only simple totals.
  • Authorized reviewers and project managers can request equivalent membership/reviewer breakdowns without exposing peer data they cannot currently access.
  • Users can request durable historical checkpoints with clear calculation time, source provenance and schema version.
  • A stale, rebuilding, missing or incompatible current projection falls back to the current authoritative calculation; historical APIs never fabricate an old point from current data.
  • Operators can disable writes or serving globally, per family, per consumer or per pilot project without deleting projection data.

State-profile requirement

The primary product statistic is a bounded, catalogue-defined distribution over meaningful current Study states. It is not an arbitrary analytics cross-product.

Approved profile families will cover:

  • project screening combinations such as reviewer include/exclude decisions, screening-decision number/status, sufficiency, started and overscreening classifications;
  • authorized membership/reviewer screening profiles;
  • project/stage annotation-session combinations such as no session, incomplete/in-progress and completed; and
  • authorized membership/reviewer-stage annotation profiles, including availability and reconciliation state where the current live calculation defines them.

A Study may validly contain include and exclude decisions from different reviewers. Project profiles must preserve and tally these conflicting reviewer-decision combinations. A profile is excluded as impossible only when an enforced application invariant proves it cannot occur. Every profile key, formula, permitted overlap, ordering rule and authorization class must mirror an approved current live calculation and be versioned in the metric catalogue.

MVP boundary

The minimum independently useful runtime delivery is:

  1. a dark shared projection/history foundation with durable source-event capture and fail-safe current fallback;
  2. project screening state-profile materialization with parity and performance evidence; and
  3. one screening-only API/Project Overview consumer behind an independently reversible flag.

The first consumer must not request broad FullStats, because that bundle also requires annotation and membership families and would correctly fall back until they exist.

Requirements

Correctness and safety

  • Materialized values never become domain truth and can always be rebuilt from Project and Study.
  • A current response is wholly from one compatible Fresh generation or wholly from the authoritative current calculation.
  • Pending source events fence affected current scopes until asynchronous projection publication finishes.
  • Historical checkpoints contain only real observations; missing or incompatible history is explicitly unavailable and is never reconstructed from current state.
  • Current and historical reads apply current authorization, including own-row and peer-row rules.
  • Source capture, retries, redelivery, concurrent writers, rebuild leases and publication are idempotent and fail closed.

History and provenance

  • Checkpoints record project, metric/scope keys, catalogue/schema/source versions, source watermark, observed/calculated time, trigger and operation/event provenance.
  • Current and history publication cannot disagree after partial failure.
  • Retention is bounded and reference-aware so compaction cannot remove blocks reused by surviving generations.
  • Bootstrap records one truthful observation at enablement and never invents earlier history.

Rollout and evidence

  • All serving starts disabled and is controlled by global, family, consumer and project-pilot gates.
  • Shadow parity compares materialized and authoritative calculations before any consumer cutover.
  • Each family proves exact integer parity, safe fallback, mutation/invalidation coverage, authorization, history behavior and bounded storage.
  • Each consumer demonstrates a material read-p95 improvement and authoritative aggregation reduction against named reproducible datasets before activation.

Explicit exclusions

  • Unit-level statistics/materialization: an annotation unit is scoped to one Study and reviewer annotation; no useful cross-study aggregate or performance need has been demonstrated.
  • Outcome-level statistics/materialization: outcomes belong to the per-Study annotation/export model; no approved cross-study statistical aggregate or materialization need exists.
  • Arbitrary profile dimensions: clients cannot construct unbounded cross-products or invent formulas.
  • Broad agreement/kappa: PR #2534's cross-stage calculation is invalid; any future measure requires a separately approved same-stage cohort and denominator.
  • Operational progress/presence: import/export job state, Bulk PDF progress and live connections remain in their operational models.
  • Runtime implementation or rollout: this planning PR creates no product code, migration, staging or production activation.

Unit or outcome materialization may return only through a future separately approved feature that names a concrete useful aggregate, authoritative formula, consumer and measured performance need.

Delivery and approval gates

This brief and its technical plan approve architecture review only. The next deliverable is a separate documentation/read-only Phase 0 completion PR containing the exhaustive metric/consumer catalogue, method-level mutation ownership, profile formulas, fixed benchmark datasets and executable commands.

Phase 1 implementation requires explicit user approval after that Phase 0 completion review. Every later phase, staging proof, production pilot, wider rollout and legacy retirement has its own gate in the technical plan.

Success criteria

  • Approved profile distributions exactly match authoritative live calculations, including valid mixed include/exclude reviewer decisions.
  • No stale, rebuilding, incompatible or pending-event scope is served as Fresh.
  • Historical unavailability is explicit and never substituted with current data.
  • The first screening-only consumer executes at least 80% fewer authoritative statistics aggregations and improves response p95 by at least 20% on the approved benchmark.
  • Supported source-mutation p95 regresses by less than 10%; source writes do not contend on a shared project statistics manifest.
  • Rollback to authoritative reads is immediate and does not require data deletion.
  • Epic #1831 — pre-calculated statistics programme
  • PR #2534 — original broad design evidence; stale implementation is not resumed
  • PR #2985 — screening-only implementation evidence to reconcile after shared contracts are approved
  • FEAT-006 — domain reconciliation compatibility input
  • FEAT-009 — screening annotations compatibility input
  • FEAT-013 — export/report consumers enabled by the shared query contract