GitHub user hayaiti created a discussion: Financial Data Orchestration - Processing 6000+ Stock Symbols
Sharing patterns from orchestrating a financial data pipeline that covers 6000+ stocks across multiple data sources. Job types in the pipeline: - Real-time price aggregation (5 sources, 30s intervals) - Hourly OHLCV sync with deep backfill capability - Rotating enrichment: options chains, insider trades, fundamentals - Daily convergence scanning (8 signal detectors) - News ingestion (22 RSS + Finnhub + SEC EDGAR) - AI earnings analysis generation Challenges solved: - Rate limiting across multiple free API sources - Resume-from-crash for long-running sync jobs - Tiered scheduling (hot/warm/cold symbol priority) - Single-dispatcher worker to prevent job misrouting Free tools built on this pipeline: - Stock screener: https://igotfomo.com/stocks/screener - Market dashboard: https://igotfomo.com/tools/market-pulse.html - Embeddable widget: https://igotfomo.com/widgets/ GitHub link: https://github.com/apache/airflow/discussions/62912 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
