GitHub user OpenPj created a discussion: Proposal & Feedback: Native Integration with OpenCrawling for High-Throughput RAG & AI Vector Ingestion
Hey StormCrawler Community! 👋 We are huge fans of Apache StormCrawler and its stream-based, low-latency distributed crawling capabilities on Apache Storm. As part of the OpenCrawling project a secure, decoupled Java 25 reference implementation of the Open Ingestion Standard (OIS) https://github.com/opencrawling/open-ingestion-standard for RAG pipelines and enterprise vector databases, we are proposing a native integration between Apache StormCrawler and OpenCrawling. ### 🌟 Why this Integration? By connecting StormCrawler with OpenCrawling, users can stream millions of web pages at scale, perform document layout analysis and chunking, generate vector embeddings (via Ollama, OpenAI, HuggingFace), and index them directly into vector stores (pgvector, Qdrant, Milvus, Vespa, OpenSearch), all with native ACL security metadata and MCP (Model Context Protocol) support for LLMs. ### 🏗️ Proposed Architecture Highlights 1. OpenCrawlingBolt (`opencrawling-stormcrawler-bolt`): A dedicated Apache Storm Bolt for StormCrawler topologies that normalizes crawled web documents directly into OIS-compliant binary/REST streams. 2. `oc-stormcrawler-repository-connector`: A native repository connector in OpenCrawling to dynamically deploy, monitor, and manage StormCrawler topologies via the Apache Storm Nimbus REST API. ────── ### 💬 We’d Love Your Feedback! We have opened a detailed proposal on our GitHub repository and would love to collaborate with the StormCrawler community to get your thoughts on the design, configuration ergonomics, and Bolt architecture: 👉 GitHub Issue: Integration with Apache StormCrawler for High-Throughput Distributed Web Crawling & RAG Ingestion https://github.com/opencrawling/opencrawling/issues/94 Whether you have ideas on Bolt design, metrics correlation, topology lifecycle management, or are interested in co-contributing, please drop your thoughts in the issue or right here! Looking forward to building this bridge together! 🚀 GitHub link: https://github.com/apache/stormcrawler/discussions/2073 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
