This is great news! 1. S3 Gateway (s3g) vs. native Ozone Client: we currently use the AWS S3 > SDK connecting through Ozone's S3 Gateway (s3g). For ultra-high throughput > streaming ingestion, does the community recommend switching to or offering > a native OzoneClient implementation alongside S3G?
The native Ozone client (ofs) has much higher performance than s3. But S3 support is still a first-class citizen because of its versatility. 2. Bucket lifecycle & garbage collection: claim check payloads are often > temporary (needed only until downstream consumers extract text and generate > vector embeddings). Are there recommended Ozone lifecycle / expiration > policies or chunking settings optimized for high-volume short-lived object > ingestion? This is a new feature just about to be merged into the master branch. It's not even included in the Ozone 2.2 release. So, I'm afraid there won't be many operational best practices for this feature yet. On Tue, Jul 21, 2026 at 8:24 AM Piergiorgio Lucidi <[email protected]> wrote: > Hi Ozone Community, > > I am writing to share an RFC and gather feedback on how we are integrating > Apache Ozone as a high-performance object store in OpenCrawling, the > reference Java 25 & Spring implementation of the Open Ingestion Standard > (OIS). > > In enterprise data ingestion pipelines for RAG and vector search, large > binary files (PDFs, Office docs, CAD files) extracted from repository > connectors (e.g. Alfresco, S3, Filesystem) cannot pass directly through > streaming brokers (Apache Kafka) due to message size limitations and broker > memory overhead. > > To handle heavy payloads at scale, OpenCrawling implements the claim check > pattern using Apache Ozone: > 1. Binary document streams are offloaded directly to Apache Ozone Object > Storage (via Ozone's S3 Gateway / S3 API). > 2. Lightweight JSON payload claims (containing document metadata, security > ACLs, and the Ozone S3 URI reference 's3://claims/{claim-id}') are > published to Kafka topics. > 3. Downstream consumer microservices (Ingestion Consumer, Embedding > Service, Vector Store Writer) resolve binary content directly from Apache > Ozone on demand. > > We would love to hear feedback from the Ozone community on a few > architectural points: > > 1. S3 Gateway (s3g) vs. native Ozone Client: we currently use the AWS S3 > SDK connecting through Ozone's S3 Gateway (s3g). For ultra-high throughput > streaming ingestion, does the community recommend switching to or offering > a native OzoneClient implementation alongside S3G? > > 2. Bucket lifecycle & garbage collection: claim check payloads are often > temporary (needed only until downstream consumers extract text and generate > vector embeddings). Are there recommended Ozone lifecycle / expiration > policies or chunking settings optimized for high-volume short-lived object > ingestion? > > 3. Benchmarking & collaboration: we have included an out-of-the-box Apache > Ozone 2.2.0 integration test suite (SCM, OM, Datanode, S3G) in our > decoupled distribution ('test-ozone- > decoupled.sh'). We welcome any Ozone developers interested in testing, > benchmarking or contributing to join us! > > Links & Resources: > - OpenCrawling repository: https://github.com/opencrawling/opencrawling > - Open Ingestion Standard (OIS): > https://github.com/opencrawling/open-ingestion-standard > - Apache Ozone claim check Implementation: > https://github.com/opencrawling/opencrawling/tree/main/oc-core > > We appreciate any thoughts, architectural advice, or contributions from the > Ozone community! > > Best regards, > > Piergiorgio Lucidi > Member at The ASF, Apache ManifoldCF PMC Chair and OpenCrawling Founder > > -- > Piergiorgio >
