GitHub user hayaiti created a discussion: Stock Market Data Pipeline with
Airflow - 7 Sync Jobs for 6,000+ Symbols
## Financial Data Orchestration with Airflow
Sharing DAG patterns from a stock market platform that runs 7 concurrent data
sync jobs for 6,100+ symbols.
### DAG Architecture
```python
# Stock data pipeline - runs during market hours
with DAG('stock_data_pipeline', schedule='*/30 * * * 1-5') as dag:
sync_quotes = PythonOperator(
task_id='sync_tier1_quotes',
python_callable=sync_yahoo_quotes,
op_kwargs={'symbols': TIER_1_SYMBOLS} # 275 symbols, 30s
)
sync_options = PythonOperator(
task_id='sync_options_chain',
python_callable=sync_options_data,
op_kwargs={'symbols': TIER_1_OPTIONS} # 99 symbols, hourly
)
sync_insider = PythonOperator(
task_id='sync_insider_filings',
python_callable=sync_sec_form4, # SEC EDGAR, daily
)
detect_signals = PythonOperator(
task_id='detect_convergence_signals',
python_callable=run_signal_scan, # 12 signal types
)
score_stocks = PythonOperator(
task_id='update_convergence_rankings',
python_callable=update_rankings, # IC-weighted ML scoring
)
[sync_quotes, sync_options, sync_insider] >> detect_signals >> score_stocks
```
### 7 Sync Jobs
| Job | Frequency | Symbols | Source |
|-----|-----------|---------|--------|
| Quotes | 30s | 275 Tier 1 | Yahoo Finance |
| Options Chain | 1hr | 99 symbols | CBOE + Yahoo |
| Insider Filings | Daily | All | SEC EDGAR |
| News | 15min | All | 22 RSS feeds |
| Dark Pool | 1hr | 190 symbols | Scrapers |
| Fundamentals | Quarterly | All | Finnhub |
| Historical Bars | Daily | 290 stocks | Yahoo |
### Free Tools Powered by This Pipeline
- [Stock Screener](https://igotfomo.com/stocks/screener) - AI convergence
signals
- [Market Pulse](https://igotfomo.com/tools/market-pulse.html) - Live dashboard
- [Ticker Widget](https://igotfomo.com/widgets/) - Embeddable prices
- [Free APIs](https://github.com/hayaiti/stock-market-api-guide)
GitHub link: https://github.com/apache/airflow/discussions/62906
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
