High Value Data Progress Review
Run high_value_data_progress_review_20260720T061235Z | generated UTC 2026-07-20T06:12:35.5733466Z | scheduled next run 07/20/2026 17:00:00
Pause Snapshot
Scheduled taskDisabled
Daily pull pauseTrue
Last task result267014
Active workers0
Active tasks0
Active provider leases0
Model rows ready0/24
Pause reasonControlled high-value data backfill, model, and report work; keep daily scheduled pulls paused until operator resumes.
Approach Recommendation
The broad data foundation is now good enough to stop measuring progress by raw pull volume alone. The next highest-value phase is narrower: paper-trading evidence, stop-loss labels, semantic relationship quality, and promotion gates.
- Pause broad historical pulls unless a specific model gate needs more rows.
- Put the next effort into paper-trading readiness: monitor, account snapshots, fills, slippage, and stop-loss case labels.
- Use the five positive component candidates and twenty stack-validation rows as the next model-training queue.
- Treat relationship graph growth as a semantic-quality problem now, not just an edge-count problem.
- Keep live/real trading disabled until paper fills, risk limits, kill switch, and operator review are all green.
High Value Workstreams
| Workstream | State | Evidence | Remaining Risk | Best Next Move | Report |
|---|---|---|---|---|---|
| Intraday market data foundation | Strong local foundation | 1,374 symbols, 52,517,125 bar/trade/quote rows, cutoff 2026-07-17, zero orders in latest readiness report. | Historical, bounded WebSocket, and supervised read-only service foundations are present. Market-hours soak evidence and downstream fill/risk gates remain. | Use targeted intraday refreshes around top opportunities and stop-loss cases instead of broad pulls unless a coverage gate asks for it. | Alpaca_Market_Data_Readiness.html |
| FMP bulk and fundamentals bridge | Bulk bridge current; daily delta guarded | 18/18 datasets available or succeeded; latest rows published 3,617,716; latest observed 2026-07-19T23:54:52.870635+00:00. Daily delta bulk_first_with_recent_fallback_review: bridge 2026-07-17, latest-date symbol fallback rows 0, latest-date empty rows 0, bridge lag 0, report provider calls 0. | Latest bulk date is bridged without broad symbol-level fallback; older fallback or empty coverage remains a targeted review queue until bulk-first sequencing is fully promoted. | Latest bulk coverage is clean; recent older fallback or empty rows should stay in a small review queue rather than becoming daily baseline work. | FMP_Daily_Delta_Readiness.html |
| Broader news and event history | Production event-impact risk artifact; prospective evidence collecting | Latest run has 1,876,996 article rows, 1,683,534 event evidence rows, 2,408 snapshots, and 16,105 component signals over 3,650 days. Governed impact model 20260720T053747Z: 168,700 matured event days across 2,322 symbols/3,414 dates; 63,808 matured historical shadow predictions; holdout RMSE lift vs constant 0.14003974209876024, incremental news lift vs price-only 0.0009431655694244201, rank correlation 0.5864598752118867, quintile spread 0.08334469006648253; strict prospective outcomes 0/0 dates; status production_model_collecting_strict_prospective_paper_evidence. | The model artifact is production-pinned for advisory paper risk, but 500 matured strictly post-assignment predictions across 20 dates remain mandatory before any decision influence. | Run daily outcome maturity and scoring once per completed news/price publication; do not retrain repeatedly or grant sizing/order authority while prospective gates are open. | News_Event_Impact_Model.html |
| Live market data readiness | Supervised read-only service controls ready; market-hours soak pending | Status live_stop_research_only: bounded stream feeds 2, test/prod verification 1/1, stream data events/orders 2/0; supervisor feeds/controlled 2/2, test/prod/market-hours/kill runs 2/1/0/1, reconnect runtime runs 0, connections/events/orders 2/2/0; stop gates blocked/passed 2/4, signal dates 17/126, paper fill samples 2/100, ready quote symbols 0/50, orders submitted 0, submission enabled False; expected-cost paper shadow collecting_strictly_post_cutoff_paper_shadow, dates 0, remaining 20. | The supervisor implementation now has heartbeat, reconnect, bounded-buffer, stale-feed, backpressure, and kill-switch controls. A real market-hours soak, account-risk limits, and enough paper fill/slippage samples remain before real trading. | Run a supervised SIP soak during market hours, keep order submission disabled, and promote only monitoring/reconciliation/reporting until fill and account-risk gates clear. | Alpaca_Live_Trading_Readiness.html |
| Execution and fill data readiness | Thin, gated; paper collection queue visible | 3 fill feature rows, 2/100 candidate training samples, 3 quote matches, 3 trade matches, 3 open gates. Collection queue 4: 0 paper-fill candidate, 1 proxy-review intent, 1 blocked intents, 2 no-fill-needed rows; paper execution paper_plan_ready, plan-only True, orders 0, provider calls 0; collection stage proxy_review_required_before_paper_collection, allowed now False, possible when authorized False; block reasons sample_depth_gap_98; wide_spread_review_rows_1; slippage_training_guardrail_blocked; paper_order_submission_switch_disabled; no_clean_collection_candidate; proxy_review_intents_1. Immutable expected-cost model 20260719T172030Z is pinned to paper_shadow; historical promotion ready True, prospective shadow dates 0, remaining 20, status collecting_strictly_post_cutoff_paper_shadow. | Expected-cost pretraining is promoted to paper shadow, but 20 future shadow dates and actual clean broker-fill calibration remain required before production sizing review. | No clean paper-fill candidates are available yet; refresh/review market-hours intraday proxy context for 1 eligible intent(s), then collect 98 additional linked, fresh-quote, tight-spread paper fills before slippage-sensitive training. | Alpaca_Fill_Slippage_Features.html |
| SEC filing text depth | Section-aware semantic change foundation working | 19,740 filing evidence rows, 2,405 reviewed symbols, 3 unknowns, 23,293 SEC component-signal rows. Section layer: 18,876 documents, 2,378 symbols, 57,312/73,909 sections extracted (0.7754), 46,936 prior-filing comparisons, high/change review 2,161/16,798, filings 2000-06-29-2026-07-01, status section_change_features_ready_for_research. | Review issuer-format extraction gaps and calibrate semantic changes against matured post-filing outcomes before model weighting. | Join section-change candidates to matured returns and ontology concepts; keep pattern topics as research flags, not allegations. | SEC_Filing_Section_Change_Foundation.html |
| SEC post-filing downside-risk model | Production paper-risk model; prospective evidence collecting | Version 20260719T191251Z: 12,901 matured events over 1,315 dates and 2,174 symbols; historical holdout 3,721, RMSE/MAE lift 0.029217989839482428/0.12403258859423949, rank correlation 0.3168720849870035, risk-quintile spread 0.08077541190613204, 80% envelope coverage 0.8414404729911314; semantic residual active false after validation lift 0.0025292181601336283; prospective rows/dates/matured 0/0/0; orders 0. | Collect 100 strictly prospective matured predictions and integrate gap/fill behavior before this risk envelope can influence paper stop placement. | Run only daily label maturity and scoring; keep semantic residuals disabled until they prove incremental validation value, and keep order authority at zero. | SEC_Filing_Downside_Risk_Model.html |
| Comtrade and global trade foundation | Research feature foundation ready | Comtrade has 2,719 observations, 12 reporters, 2,719 commodity-code observations, 5,320 projected graph edges, years 2015-2025; TiVA has 109 observations, 4 measures, and 109 graph edges. Direct YoY cohort: 1,563 eligible rows, 950 shock labels, 613 reference rows, 12 reporters/10 years; 238 confirmed absences retained as context and 1,363 unresolved gaps excluded; status trade_shock_training_foundation_ready. | The direct-observation training cohort is usable, but unresolved historical gaps stay excluded and the lake remains intentionally priority-industry shaped rather than globally exhaustive. | Run sector and horizon ablations on the eligible direct-year cohort; use further Comtrade pulls only for a named energy, metals, semiconductor, shipping, agriculture, or supply-chain model gap. | Global_Trade_Data_Readiness.html |
| Entity and relationship graph | Large graph; research_feature_ready_quality_reviews_governed | 2,138,296 relationships across 192 grouped rows, 36 source families, and 81 relationship types; concept/meaning coverage 1.0/1.0; model usability quality_reviews_governed_candidate_research_allowed, training allowed False, ready horizons 2, next horizon/date gap 1/0, quality review rows 74 covering 519,045 relationships; manifest research_feature_ready_quality_reviews_governed, policy governed_candidate_relationship_features_allowed_for_research_training_only, candidate/review relationship rows 1,619,211/519,045, research training True, challenger promotion False, governed/ungoverned families 141/0, latest ablation rejected_at_selection_gate with 0/216 passing trials, provider calls 0, orders 0. | All populated relationship families now have explicit full, reduced, or zero-weight dispositions. The latest 216-trial H1 quality ablation produced zero passing trials, so challenger promotion remains rejected even though governed research training is allowed. | Keep the current H1 baseline and governed research weights. Do not repeat the rejected same-block ablation; reopen promotion only after a new temporal block, materially new relationship features, or source improvements satisfy the documented family conditions. | Entity_Relationship_Readiness.html |
| Options data planning and feature plumbing | Historical OPRA training foundation ready; secondary features maturing | Snapshots: 1,038 underlyings, 639,362 rows, latest 2026-07-19T20:22:19.722102+00:00, readiness stale_option_chain_seed: 1,122. Historical OPRA: 5 underlyings, 768,670 bars/trades, 204 contracts, max 135 market dates, latest 2026-07-19T20:43:47.736444+00:00, readiness collecting_historical_depth: 4; historical_option_training_foundation_ready: 1. Daily features: 200 rows across 5 underlyings/135 dates through 2026-07-17. Components production-ready 0/6, max evaluated/remaining dates 3/18. | Historical depth now supports secondary feature research, but option components still need independent evaluated dates and ablation evidence before promotion. | Mature existing option component outcomes and validate them as secondary stock features; avoid broad new pulls unless a coverage gate asks for them. | Alpaca_Options_Data_Readiness.html |
| Model promotion and stop-loss research | Research actions visible; production blocked | 0/24 model rows ready for review; component triage: inspect_inverse_or_risk_orientation: 11; train_next_positive_candidate: 13; validate_in_stack: 37. Critical path: 3 actionable research rows, 11 blocked gates, 1 review gates; direct challengers 1, inverse/risk tests 1, stack ablations 17, optimizer-date gap 109, paper-fill gap 98, top-15 execution-proxy gap 15. Training preflight: 13 groups, 14 missing-policy queue rows, 22 tested-not-weighted rows, 2 row-depth wait groups, largest row gap 1,552. Latest queue ablation: 1 rows, 1 evaluated, 1 positive-review rows, best MSE lift 0.046523983079177884, best directional delta 0.011938202247190999. Stop selector: 22 priority rows, 7 coverage-backfill rows, 1 positive clean / 3 positive-with-friction / 9 friction-rejection rows, 145 training examples, launch gates blocked/review/passed 7/0/2, selector signal-date gap 109, execution-proxy ready top-15/top-50 0/0, wide-spread review 12; paper/live/artifact allowed 0/0/0. Intraday stop proxy: 41/50 symbols covered, 4 full-depth, 27 partial-depth, latest proxy date 2026-07-17, evidence rows shown 25; proxy evidence is not paper-fill/slippage evidence. | Production promotion remains blocked by zero ready rows, stop-loss selector date/temporal gates, paper-fill sample depth, and execution-proxy review; intraday stop proxy evidence helps research but does not replace actual fill/slippage evidence. | Use bounded queue ablations to separate candidates worth review from neutral/no-lift candidates, let h20/h63 rows mature, and keep selector training, artifact writes, live trading, and order submission disabled until hard gates clear. | Model_Promotion_Readiness.html |
| Component outcome maturity diagnostics | Diagnostic surface working | Latest diagnostic component_outcome_sync_diagnostics_20260718T004510Z: inspected 47,358 pending rows, wrote 0 matured evaluations, left 47,149 session-immature rows, and found 209 stale-price rows across 15 symbols; missing-asset/unreadable/invalid rows 0/0/0; stale actions: review_provider_activity_flag_conflict: 15; triage stages: manual_identity_review_after_local_bar_evidence: 2; manual_identity_review_after_local_bridge: 13; no-repeat provider-call symbols 15; local repair candidates 0; training-label excluded symbols 15. | Some queued model inputs are true exact-session maturity waits, while stale-price rows need price-history repair or provider/source review before training; the repair queue and Instrument Source Review now separate no-repeat provider-call symbols, local repair candidates, and rows excluded from training labels. | Do not spend more price-provider calls on symbols marked no-repeat until identity/activity review resolves them; repair only local-repair candidates, rerun diagnostics, then train candidates whose evaluated rows and date gates actually improve. | Component_Outcome_Sync_Diagnostics.html |
Key Public Surfaces
| Report | Exists | Age Hours | Bytes |
|---|---|---|---|
Pipeline_Health.html | true | 1.1 | 40,863 |
High_Value_Data_Progress_Review.html | true | 1.2 | 21,128 |
FMP_Daily_Delta_Readiness.html | true | 5.6 | 17,540 |
FMP_Bulk_Bridge_Readiness.html | true | 5.6 | 17,176 |
Global_Trade_Data_Readiness.html | true | 1.3 | 42,213 |
Alpaca_Market_Data_Readiness.html | true | 8.1 | 29,428 |
Alpaca_Execution_Training_Foundation.html | true | 13.2 | 5,879 |
Alpaca_Execution_Cost_Model.html | true | 12.9 | 3,553 |
Alpaca_Live_Trading_Readiness.html | true | 12.3 | 43,460 |
Alpaca_Fill_Slippage_Features.html | true | 32.2 | 31,780 |
Alpaca_Options_Data_Readiness.html | true | 9.8 | 167,480 |
News_Event_Training_Coverage.html | true | 0.4 | 32,845 |
News_Event_Impact_Model.html | true | 0.6 | 5,799 |
SEC_Filing_Intelligence.html | true | 33.0 | 29,142 |
SEC_Filing_Section_Change_Foundation.html | true | 1.9 | 31,319 |
SEC_Filing_Downside_Risk_Model.html | true | 4.3 | 5,566 |
Market_Relationship_Weight_Training.html | true | 7.3 | 68,363 |
Entity_Relationship_Readiness.html | true | 6.7 | 51,685 |
Model_Promotion_Readiness.html | true | 0.0 | 1,088,213 |
Component_Queue_Ablation.html | true | 14.8 | 51,477 |
Component_Outcome_Sync_Diagnostics.html | true | 53.5 | 32,477 |
Instrument_Source_Review.html | true | 53.5 | 22,980 |
Paper_Strategy_Optimizer.html | true | 66.4 | 375,820 |