Hi Ozone Community,
I am writing to share an RFC and gather feedback on how we are integrating
Apache Ozone as a high-performance object store in OpenCrawling, the
reference Java 25 & Spring implementation of the Open Ingestion Standard
(OIS).
In enterprise data ingestion pipelines for RAG and vector search, large
binary files (PDFs, Office docs, CAD files) extracted from repository
connectors (e.g. Alfresco, S3, Filesystem) cannot pass directly through
streaming brokers (Apache Kafka) due to message size limitations and broker
memory overhead.
To handle heavy payloads at scale, OpenCrawling implements the claim check
pattern using Apache Ozone:
1. Binary document streams are offloaded directly to Apache Ozone Object
Storage (via Ozone's S3 Gateway / S3 API).
2. Lightweight JSON payload claims (containing document metadata, security
ACLs, and the Ozone S3 URI reference 's3://claims/{claim-id}') are
published to Kafka topics.
3. Downstream consumer microservices (Ingestion Consumer, Embedding
Service, Vector Store Writer) resolve binary content directly from Apache
Ozone on demand.
We would love to hear feedback from the Ozone community on a few
architectural points:
1. S3 Gateway (s3g) vs. native Ozone Client: we currently use the AWS S3
SDK connecting through Ozone's S3 Gateway (s3g). For ultra-high throughput
streaming ingestion, does the community recommend switching to or offering
a native OzoneClient implementation alongside S3G?
2. Bucket lifecycle & garbage collection: claim check payloads are often
temporary (needed only until downstream consumers extract text and generate
vector embeddings). Are there recommended Ozone lifecycle / expiration
policies or chunking settings optimized for high-volume short-lived object
ingestion?
3. Benchmarking & collaboration: we have included an out-of-the-box Apache
Ozone 2.2.0 integration test suite (SCM, OM, Datanode, S3G) in our
decoupled distribution ('test-ozone-
decoupled.sh'). We welcome any Ozone developers interested in testing,
benchmarking or contributing to join us!
Links & Resources:
- OpenCrawling repository: https://github.com/opencrawling/opencrawling
- Open Ingestion Standard (OIS):
https://github.com/opencrawling/open-ingestion-standard
- Apache Ozone claim check Implementation:
https://github.com/opencrawling/opencrawling/tree/main/oc-core
We appreciate any thoughts, architectural advice, or contributions from the
Ozone community!
Best regards,
Piergiorgio Lucidi
Member at The ASF, Apache ManifoldCF PMC Chair and OpenCrawling Founder
--
Piergiorgio
