Where does the data come from?
Nine distinct sources feed the system. Each lands in a specific table or file, on a specific cadence, in a specific door's lane. Nothing is shared between domains at the source level.
Pulse cadence indicates update frequency: webhooks fast · live workouts · daily polls medium · manual imports slow.
The nine sources, one by one
1 · Live market data — yfinance + FRED + earnings calendar
Three sub-pipes inside the TradingView app's app/market_data/ module:
- OHLCV bars via yfinance — daily refresh for every watchlist ticker (240+ bars). Lands in
ohlcv_bars(composite PK on symbol+interval+ts). - Macro series via yfinance + FRED API — daily 01:00 UTC, gated by
MACRO_ENABLED. Lands inmacro_series. - Earnings calendar — rolling 150-row universe (NASDAQ + Street Tier-1+2 + 8-K Item 2.02 confirm), 90-day TTL. Lands in
earnings_calendar_rows. - IV percentile + earnings dates per ticker upserted into
ticker_market_data.
Door consumer: Lakshmi only.
2 · TradingView webhooks (Pine Script alerts)
The legacy unversioned POST /webhook endpoint receives Pine Script alerts as text. These are notification-semantic — no dedup, fast lane.
- Lands in
alertstable - Fans out to
tv_context_itemswith kind=webhook(7-day TTL, deduped by SHA256 within a rolling window).
Door consumer: Lakshmi only.
3 · Operator chart pastes (separate pipe!)
This is not the webhook pipe. When the operator pastes a
chart screenshot into the app's TV Context route
(POST /v1/tv-context/screenshot), a vision pipeline kicks in:
- Tesseract OCR extracts tickers
- Claude vision summarises the chart
- Qwen2-VL extracts structured chart references
- Sidecar markdown written to
~/Documents/knowledge-vault/Sources/.../<ticker>_<HMS>_<id>.md - Unknown tickers flow into
ticker_review_queue
Door consumer: Lakshmi only. (Per CLAUDE.md demo-discipline: "webhooks and screenshots are two pipes — document them as such.")
4 · YouTube videos (channel-polled, vision-extracted)
The vault-indexer's youtube_channel.py ingester polls every channel listed under Videos/<author>/_channel.yaml. Cadence: hourly, gated by VIDEO_INGEST_ENABLED.
For each new video, a 3-stage pipeline runs:
- Frame detection — sample candidate keyframes
- Tesseract OCR — pull text overlays
- Qwen2-VL (MLX on Apple Silicon) — caption frames + extract structured chart YAML
The output is a markdown at ~/Documents/knowledge-vault/Videos/<author>/<week>-<slug>.md. Unknown tickers seen in vision frames go into ticker_review_queue.
Door consumer: any door whose vault scope includes Videos/<subdomain>.
5 · SEC Edgar filings
The vault-indexer's ingest_edgar.py polls Edgar daily for watchlist tickers, gated by EDGAR_INGEST_ENABLED. Writes per-filing markdown to ~/Documents/knowledge-vault/Filings/<ticker>/<accession>.md. Idempotent on accession number.
Door consumer: Lakshmi only (finance vault scope).
6 · Smart-money snapshots (The Street/)
An aggregator pipeline that imports tier-1/tier-2 conviction lists, politician buys (House STOCK Act, Senate EFD), insider buys (Form-4), and options flow.
Each snapshot is dated: The Street/snapshots/YYYY-MM-DD/. Files: tier-1-conviction.md, tier-2-conviction.md, politicians-buys.md, insiders-top.md, options-bullish.md, plus _index.md and methodology.md.
Chunking unit: per-ticker H2 section (~600 tokens by design).
Status: all 6 data sources currently enabled: false in _source.yaml; manual snapshots exist (last: 2026-05-08).
7 · Verocity workout logs (Supabase, live)
Every workout the operator does in the Lovable verocity app writes one row to Supabase workout_logs. Fields include planned vs actual sets, RPE, total_seconds, avg_hr_bpm.
Derived signals are computed by a SQL view v_drift_signals with seven components (pace, block_lag, gap_breach, load_drop, rpe_drift, recovery, conditioning).
Door consumer: Zeus.
8 · Manual measurements YAML (weekly)
Append-only file at Zeus/03_execution/measurements.yaml. Operator writes bodyweight + waist circumference weekly.
A stdlib helper body_recomp_snapshot.py reads this and emits a body_recomp_sub JSON signal that becomes the 10% second-layer weight on the fitness drift composite (D-022).
Door consumer: Zeus only.
9 · Learning state (markdown only)
Ganesh has no DB. Signals come from heterogeneous markdown via a single helper:
Ganesh/02_library/active_sprints.md— current slot deadlinesGanesh/_inbox/— unprocessed drops (7-day rotation)Ganesh/03_execution/learning_log.md— append-only session logGanesh/02_library/notes/— processed notes withapplied_in:graduation tracking
The helper ganesh_state_snapshot.py reads the markdown and emits a 5-component drift signal JSON.
Door consumer: Ganesh only.
Cadences at a glance
| Source | Cadence | Destination |
|---|---|---|
| OHLCV bars | daily | TradingV ohlcv_bars |
| Macro series | daily 01:00 UTC | TradingV macro_series |
| Earnings calendar | daily | TradingV earnings_calendar_rows |
| TV webhooks | real-time push | TradingV alerts + tv_context_items |
| Chart pastes | operator-driven | TradingV tv_context_items + vault sidecar |
| YouTube | hourly poll | vault Videos/<author>/ |
| Edgar | daily poll | vault Filings/<ticker>/ |
| Smart-money | operator import | vault The Street/snapshots/ |
| Workouts | per-session live | Supabase workout_logs |
| Measurements | weekly manual | file Zeus/03_execution/measurements.yaml |
| Learning state | operator-driven | Ganesh markdown tree |