# GCS-DATA-003 - Persistence Contract Spec ID : GCS-DATA-003 Title : Persistence Version : v1.0 Status : Settled Owner : Phil Approver : Phil Settled : 2026-07-08 Depends on : GCS-SYS-001, GCS-NAME-001, GCS-DATA-002, GCS-SEC-001 Supersedes : - ## 1. Current Phase There is no database implementation before Phase 0D. No migration, ORM model, database URL, SQL file, active Postgres service, or persistence package may be introduced until this contract's physical model inventory and checker gate are complete. Phase 0B may introduce only a file-backed point-in-time storage skeleton under: - `app/raw_store/` - `app/normalization/` - `app/sources/fixtures/` - `tests/raw_store/` - `tests/normalization/` - `tests/fixtures/` That skeleton is not a database implementation. It exists only to prove the raw -> normalized point-in-time contract before physical persistence begins. Phase 0C is design-only. Phase 0C may amend this physical model inventory and the checker that proves the inventory is present. It may not create database implementation files, migrations, SQL files, ORM models, active database containers, API routes, queues, workers, schedulers, live source clients, or ingestion code. Phase 0D may implement only the physical persistence surface opened by this contract and `GCS-TEST-001`. Phase 0D may introduce: - `app/persistence/` - `migrations/` Python migration files named `NNNN_.py` - `tests/persistence/` - persistence validation tooling under `tools/` Phase 0D still may not introduce active database containers, Compose files, API routes, queues, workers, schedulers, Redis, RabbitMQ, Celery, live source clients, ingestion code, entity-resolution code, signal code, analytics code, prediction code, or standalone `.sql` files without a later contract amendment. ## 2. Future Persistence Shape When persistence begins, Postgres is the authoritative normalized store. Object storage or filesystem-backed immutable artifacts may hold raw payloads. Required table families: | Family | Required purpose | | --- | --- | | `raw_source_payload` | Immutable source payload, fetch metadata, digest | | `source_record` | Normalized source record with source authority tier | | `entity_node` | Issuer, filer, fund, recipient, government node | | `entity_edge` | Holds, votes, manages, sub-advises, disburses, gates | | `identifier_crosswalk` | Effective-dated CIK/ticker/CUSIP/LEI/UEI mapping | | `independence_audit_result` | Adjusted holder blocs and collapse evidence | | `signal_run` | Versioned analytical run metadata | | `prediction_record` | Point-in-time forecast with outcome and decay clause | ## 2.1 Phase 0C Physical Table Inventory This inventory is binding for future Phase 0D. Phase 0D may implement only these table names, column names, column types, keys, constraints, and indexes unless this contract is amended first. | Table | Columns and types | Natural key | Idempotency key | Primary key | Foreign keys | Unique constraints | Check constraints | Indexes | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `raw_source_payload` | `raw_source_payload_id uuid`; `source_identifier text`; `source_authority_tier source_authority_tier_enum`; `source_clock source_clock_enum`; `as_of_timestamp timestamptz`; `fetch_timestamp timestamptz`; `source_url text`; `raw_payload_digest text`; `payload_byte_size integer`; `payload_storage_uri text`; `first_seen_timestamp timestamptz`; `created_at timestamptz` | `source_identifier`, `as_of_timestamp`, `raw_payload_digest` | `raw_payload_digest` | `pk_raw_source_payload` | None | `uq_raw_source_payload__raw_payload_digest`; `uq_raw_source_payload__source_identifier__as_of_timestamp__raw_payload_digest` | `ck_raw_source_payload__fetch_not_before_as_of`; `ck_raw_source_payload__payload_byte_size_positive`; `ck_raw_source_payload__raw_payload_digest_sha256` | `ix_raw_source_payload__source_identifier__as_of_timestamp`; `ix_raw_source_payload__fetch_timestamp`; `ix_raw_source_payload__source_authority_tier` | | `source_record` | `source_record_id uuid`; `raw_source_payload_id uuid`; `source_record_kind source_record_kind_enum`; `source_identifier text`; `source_authority_tier source_authority_tier_enum`; `as_of_timestamp timestamptz`; `fetch_timestamp timestamptz`; `normalization_version text`; `natural_key text`; `record_payload_json jsonb`; `raw_payload_digest text`; `created_at timestamptz` | `source_record_kind`, `source_identifier`, `as_of_timestamp`, `natural_key`, `normalization_version` | `source_record_kind`, `natural_key`, `normalization_version`, `raw_payload_digest` | `pk_source_record` | `fk_source_record__raw_source_payload_id__raw_source_payload` | `uq_source_record__source_record_kind__natural_key__normalization_version`; `uq_source_record__raw_source_payload_id__natural_key` | `ck_source_record__fetch_not_before_as_of`; `ck_source_record__raw_payload_digest_sha256`; `ck_source_record__normalization_version_present` | `ix_source_record__source_identifier__as_of_timestamp`; `ix_source_record__source_record_kind`; `ix_source_record__raw_payload_digest` | | `entity_node` | `entity_node_id uuid`; `entity_node_type entity_node_type_enum`; `canonical_name text`; `source_record_id uuid`; `source_authority_tier source_authority_tier_enum`; `as_of_timestamp timestamptz`; `valid_from_timestamp timestamptz`; `valid_to_timestamp timestamptz`; `source_identifier text`; `raw_payload_digest text`; `created_at timestamptz` | `entity_node_type`, `source_identifier`, `valid_from_timestamp` | `entity_node_type`, `source_identifier`, `valid_from_timestamp`, `raw_payload_digest` | `pk_entity_node` | `fk_entity_node__source_record_id__source_record` | `uq_entity_node__entity_node_type__source_identifier__valid_from_timestamp` | `ck_entity_node__valid_window_order`; `ck_entity_node__raw_payload_digest_sha256`; `ck_entity_node__canonical_name_present` | `ix_entity_node__entity_node_type__canonical_name`; `ix_entity_node__source_identifier`; `ix_entity_node__as_of_timestamp` | | `entity_edge` | `entity_edge_id uuid`; `entity_edge_type entity_edge_type_enum`; `from_entity_node_id uuid`; `to_entity_node_id uuid`; `source_record_id uuid`; `source_authority_tier source_authority_tier_enum`; `as_of_timestamp timestamptz`; `valid_from_timestamp timestamptz`; `valid_to_timestamp timestamptz`; `edge_weight numeric`; `raw_payload_digest text`; `created_at timestamptz` | `entity_edge_type`, `from_entity_node_id`, `to_entity_node_id`, `valid_from_timestamp`, `source_record_id` | `entity_edge_type`, `from_entity_node_id`, `to_entity_node_id`, `valid_from_timestamp`, `raw_payload_digest` | `pk_entity_edge` | `fk_entity_edge__from_entity_node_id__entity_node`; `fk_entity_edge__to_entity_node_id__entity_node`; `fk_entity_edge__source_record_id__source_record` | `uq_entity_edge__entity_edge_type__from_entity_node_id__to_entity_node_id__valid_from_timestamp__source_record_id` | `ck_entity_edge__valid_window_order`; `ck_entity_edge__raw_payload_digest_sha256`; `ck_entity_edge__no_self_edge` | `ix_entity_edge__from_entity_node_id__entity_edge_type`; `ix_entity_edge__to_entity_node_id__entity_edge_type`; `ix_entity_edge__as_of_timestamp` | | `identifier_crosswalk` | `identifier_crosswalk_id uuid`; `entity_node_id uuid`; `identifier_scheme identifier_scheme_enum`; `identifier_value text`; `valid_from_timestamp timestamptz`; `valid_to_timestamp timestamptz`; `source_record_id uuid`; `source_authority_tier source_authority_tier_enum`; `raw_payload_digest text`; `created_at timestamptz` | `identifier_scheme`, `identifier_value`, `valid_from_timestamp` | `identifier_scheme`, `identifier_value`, `valid_from_timestamp`, `raw_payload_digest` | `pk_identifier_crosswalk` | `fk_identifier_crosswalk__entity_node_id__entity_node`; `fk_identifier_crosswalk__source_record_id__source_record` | `uq_identifier_crosswalk__identifier_scheme__identifier_value__valid_from_timestamp` | `ck_identifier_crosswalk__valid_window_order`; `ck_identifier_crosswalk__identifier_value_present`; `ck_identifier_crosswalk__raw_payload_digest_sha256` | `ix_identifier_crosswalk__entity_node_id`; `ix_identifier_crosswalk__identifier_scheme__identifier_value`; `ix_identifier_crosswalk__valid_from_timestamp` | | `independence_audit_result` | `independence_audit_result_id uuid`; `issuer_entity_node_id uuid`; `audit_as_of_timestamp timestamptz`; `adjusted_holder_count integer`; `adjusted_bloc_size numeric`; `collapse_evidence_json jsonb`; `source_record_id uuid`; `source_authority_tier source_authority_tier_enum`; `raw_payload_digest text`; `created_at timestamptz` | `issuer_entity_node_id`, `audit_as_of_timestamp`, `source_record_id` | `issuer_entity_node_id`, `audit_as_of_timestamp`, `raw_payload_digest` | `pk_independence_audit_result` | `fk_independence_audit_result__issuer_entity_node_id__entity_node`; `fk_independence_audit_result__source_record_id__source_record` | `uq_independence_audit_result__issuer_entity_node_id__audit_as_of_timestamp__source_record_id` | `ck_independence_audit_result__adjusted_holder_count_nonnegative`; `ck_independence_audit_result__adjusted_bloc_size_range`; `ck_independence_audit_result__raw_payload_digest_sha256` | `ix_independence_audit_result__issuer_entity_node_id__audit_as_of_timestamp`; `ix_independence_audit_result__source_record_id` | | `signal_run` | `signal_run_id uuid`; `signal_run_kind signal_run_kind_enum`; `run_as_of_timestamp timestamptz`; `input_record_digest text`; `method_version text`; `result_payload_json jsonb`; `source_authority_tier source_authority_tier_enum`; `raw_payload_digest text`; `created_at timestamptz` | `signal_run_kind`, `run_as_of_timestamp`, `input_record_digest`, `method_version` | `signal_run_kind`, `input_record_digest`, `method_version` | `pk_signal_run` | None | `uq_signal_run__signal_run_kind__run_as_of_timestamp__input_record_digest__method_version` | `ck_signal_run__input_record_digest_sha256`; `ck_signal_run__raw_payload_digest_sha256`; `ck_signal_run__method_version_present` | `ix_signal_run__signal_run_kind__run_as_of_timestamp`; `ix_signal_run__input_record_digest` | | `prediction_record` | `prediction_record_id uuid`; `signal_run_id uuid`; `prediction_status prediction_status_enum`; `prediction_as_of_timestamp timestamptz`; `target_observation_timestamp timestamptz`; `confidence_interval_low numeric`; `confidence_interval_high numeric`; `regime_tag text`; `decay_clause text`; `observed_outcome_json jsonb`; `source_authority_tier source_authority_tier_enum`; `raw_payload_digest text`; `created_at timestamptz` | `signal_run_id`, `prediction_as_of_timestamp`, `target_observation_timestamp`, `regime_tag` | `signal_run_id`, `prediction_as_of_timestamp`, `target_observation_timestamp`, `raw_payload_digest` | `pk_prediction_record` | `fk_prediction_record__signal_run_id__signal_run` | `uq_prediction_record__signal_run_id__prediction_as_of_timestamp__target_observation_timestamp__regime_tag` | `ck_prediction_record__target_after_prediction_as_of`; `ck_prediction_record__confidence_interval_order`; `ck_prediction_record__raw_payload_digest_sha256`; `ck_prediction_record__decay_clause_present` | `ix_prediction_record__signal_run_id`; `ix_prediction_record__prediction_status`; `ix_prediction_record__prediction_as_of_timestamp` | ## 2.2 Closed Enum Inventory Enum storage uses closed `SCREAMING_SNAKE` members. Ad-hoc text enum columns are forbidden. | Enum | Members | | --- | --- | | `source_authority_tier_enum` | `TIER_1`, `TIER_2`, `TIER_3` | | `source_clock_enum` | `FAST_INPUT`, `FAST_BEHAVIOR`, `SLOW_CONFIRMATION`, `GOVERNANCE`, `AMBIENT` | | `source_record_kind_enum` | `MONETARY_FISCAL`, `EXPENDITURE_CAPABILITY`, `CORPORATE_BEHAVIOR`, `OWNERSHIP_CONFIRMATION`, `GOVERNANCE`, `AMBIENT_TIMING` | | `entity_node_type_enum` | `ISSUER`, `FILER_MANAGER`, `FUND`, `RECIPIENT`, `ULTIMATE_PARENT`, `SECTOR`, `GOVERNMENT_NODE` | | `entity_edge_type_enum` | `HOLDS`, `VOTES`, `MANAGES`, `SUB_ADVISES`, `DISBURSES`, `IS_PARENT_OF`, `GATES` | | `identifier_scheme_enum` | `CIK`, `TICKER`, `CUSIP`, `LEI`, `UEI`, `RECIPIENT_ID` | | `signal_run_kind_enum` | `CONCENTRATION`, `FLOW_CAUSALITY`, `ENGAGEMENT_CHANNEL` | | `prediction_status_enum` | `ACTIVE`, `OBSERVED`, `DECAYED`, `REJECTED` | ## 2.3 Migration Ordering And Rollback Policy Phase 0D migration order must be: 1. Create closed enum types. 2. Create `raw_source_payload`. 3. Create `source_record`. 4. Create `entity_node`. 5. Create `entity_edge`. 6. Create `identifier_crosswalk`. 7. Create `independence_audit_result`. 8. Create `signal_run`. 9. Create `prediction_record`. 10. Create all named indexes after table and constraint creation. Rollback policy: - A Phase 0D migration is reversible only if it drops objects in reverse dependency order and cannot orphan raw payload evidence. - Any destructive rollback for source-derived rows must be marked blocked unless a fixture-backed round-trip rebuild proves raw -> normalized reconstruction. - Anonymous constraints, unnamed indexes, and ad-hoc enum text columns are migration blockers. ## 2.4 Point-In-Time Query Rules Point-in-time reads must filter source-derived records with: ```text as_of_timestamp <= query_as_of_timestamp fetch_timestamp <= query_as_of_timestamp ``` Effective-dated entity reads must also satisfy: ```text valid_from_timestamp <= query_as_of_timestamp and (valid_to_timestamp is null or valid_to_timestamp > query_as_of_timestamp) ``` Prediction reads may use only records knowable as of `prediction_as_of_timestamp`. Any query that reads a later `fetch_timestamp`, later `as_of_timestamp`, or open-ended future observation is lookahead leakage. ## 2.5 Phase 0D Fixture-Backed Proof Requirements Future Phase 0D may exit only after these tests exist and pass: Positive tests: - `tests/persistence/test_phase_0d_schema_inventory.py` proves every table, enum, column, constraint, and index exists with the names in this contract. - `tests/persistence/test_phase_0d_raw_normalized_round_trip.py` persists the Phase 0B fixture as one raw payload row and one deterministic source record. - `tests/persistence/test_phase_0d_point_in_time_queries.py` proves a query at `query_as_of_timestamp` cannot read later `as_of_timestamp` or `fetch_timestamp` rows. - `tests/persistence/test_phase_0d_idempotency.py` proves replaying the same fixture does not create duplicate logical rows. Negative tests: - Missing `as_of_timestamp`, `fetch_timestamp`, `source_authority_tier`, or `raw_payload_digest` fails persistence. - Anonymous constraints or indexes fail schema validation. - Wrong enum storage or ad-hoc text enum values fail schema validation. - Duplicate natural keys or idempotency keys fail. - A lookahead query that reads future-known rows fails. - Migration drift from this inventory fails. ## 2.6 Phase 0D Implementation Eligibility Gate Phase 0D is implementation-eligible only for the persistence boundary named in this section. Allowed Phase 0D implementation files: | Path | Purpose | Required proof | | --- | --- | --- | | `app/persistence/` | Postgres-ready persistence adapters, schema inspection helpers, raw/source-record write paths, and point-in-time query helpers | Module headers plus `tests/persistence/` boundary tests | | `migrations/NNNN_.py` | Ordered Python migration files that create the exact enums, tables, constraints, and indexes in Sections 2.1 and 2.2 | Schema inventory tests and checker migration-name validation | | `tests/persistence/` | Fixture-backed positive and negative tests for schema, round trip, point-in-time queries, and idempotency | `python3 -m unittest discover -s tests/persistence -v` | | `tools/check_persistence_schema.py` | Optional static or database-backed persistence schema validator | Contract header plus invocation from persistence tests or CI | Disallowed Phase 0D implementation files: | Path or token | Reason | | --- | --- | | `api/`, `app/api/` | API transport remains unopened | | `app/sources/` live clients outside `app/sources/fixtures/` | Phase 0D persists fixture-backed records only | | `app/entity_resolution/`, `app/signals/`, `app/prediction/` | Later phases own graph, signal, and prediction behavior | | active Compose files or database containers | GCS-DEPLOY-002 must be amended before container activation | | queue, worker, scheduler, Redis, RabbitMQ, Celery tokens | GCS-RUNTIME-001 still forbids async queues | | standalone `.sql` files | Phase 0D migrations are Python files unless this contract is amended | Phase 0D input: - Phase 0B fixed local payload fixture. - Phase 0B normalized metadata record. - Contract-approved database connection supplied by the test environment or future documented `GCS_DATABASE_URL`. Phase 0D output: - One immutable `raw_source_payload` row with the contracted digest and source metadata. - One deterministic `source_record` row linked to that raw payload. - Query helpers that return only records knowable as of `query_as_of_timestamp`. - Deterministic schema objects whose names match Sections 2.1, 2.2, and 3. Phase 0D positive verification: - `tests/persistence/test_phase_0d_schema_inventory.py` proves exact schema inventory. - `tests/persistence/test_phase_0d_raw_normalized_round_trip.py` proves raw plus source-record persistence from the Phase 0B fixture. - `tests/persistence/test_phase_0d_point_in_time_queries.py` proves no lookahead reads. - `tests/persistence/test_phase_0d_idempotency.py` proves replay creates no duplicate logical rows. Phase 0D negative verification: - Missing `as_of_timestamp`, `fetch_timestamp`, `source_authority_tier`, or `raw_payload_digest` fails. - Anonymous constraints, unnamed indexes, ad-hoc text enums, wrong migration order, duplicate natural keys, duplicate idempotency keys, lookahead reads, active containers, API routes, queues, workers, schedulers, live clients, and standalone SQL files fail. ## 3. Physical Naming Database names use `snake_case`. Deterministic constraint/index names are required: | Object | Pattern | | --- | --- | | Primary key | `pk_` | | Foreign key | `fk_
____` | | Unique constraint | `uq_
__` | | Check constraint | `ck_
__` | | Index | `ix_
__` | Anonymous constraints and ad-hoc enum text columns are forbidden. ## 4. Point-In-Time Rule Every persisted source-derived record carries: - `as_of_timestamp` - `fetch_timestamp` - `source_authority_tier` - `source_url` or source identifier - `raw_payload_digest` Backtests and predictions may read only records knowable as of the prediction timestamp. ## 4.1 Phase 0B File-Backed Storage Contract Phase 0B input: - Fixed local payload fixture. - Source metadata. Phase 0B output: - Immutable raw payload digest. - Normalized metadata record containing: - `as_of_timestamp` - `fetch_timestamp` - `source_authority_tier` - source identifier - normalization version Positive verification: - Same fixture ingested twice produces one raw payload identity. - Same fixture produces deterministic normalized metadata. - No network calls occur. Negative verification: - Missing `as_of_timestamp` fails. - Missing source authority tier fails. - Changed payload under the same digest fails. - Future-known timestamp fails. - Any API, DB migration, queue, worker, scheduler, container, or live source client fails the contract gate. ## 5. Validation Positive verification: Phase 0A/0B passes `python3 -m tools.check_contracts --check` only when implementation paths stay inside the file-backed skeleton allowlist. Negative verification: adding `migrations/`, model files, `GCS_DATABASE_URL` usage in app code, SQL files, live HTTP source clients, or persistence packages without amending this contract and validator inventory fails the gate.