Hi Wei-Chiu, Thank you so much for the clear and fast feedback! This is incredibly helpful for shaping our implementation of the Open Ingestion Standard (OIS).
1. Dual-Connector Approach (s3g + ofs) That makes total sense. We will design OpenCrawling to support both. We will use S3G as our versatile, baseline connector for standard compatibility and easy developer onboarding. We will implement a dedicated Native Ozone Client (ofs) driver to unlock maximum throughput for enterprise AI streaming pipelines where raw performance is critical. 2. Lifecycle & Claim-Check Garbage Collection It is super exciting to hear that object expiration is about to land in master! Since short-lived claim-check payloads (written during extraction, read during embedding generation, and deleted immediately after) are a core pattern for AI data pipelines, we would love to serve as an early real-world testbed for this feature. We can build our initial Release 1.0 using explicit application-level deletion (post-ingestion ACK), but we would be thrilled to test the master branch's built-in lifecycle expiration against high-volume ephemeral workloads and feed our operational benchmarks back to the community. Thanks again for the great work on Ozone! Cheers, Piergiorgio Il giorno gio 23 lug 2026 alle ore 19:42 Wei-Chiu Chuang <[email protected]> ha scritto: > This is great news! > > 1. S3 Gateway (s3g) vs. native Ozone Client: we currently use the AWS S3 > > SDK connecting through Ozone's S3 Gateway (s3g). For ultra-high > throughput > > streaming ingestion, does the community recommend switching to or > offering > > a native OzoneClient implementation alongside S3G? > > > The native Ozone client (ofs) has much higher performance than s3. > But S3 support is still a first-class citizen because of its versatility. > > 2. Bucket lifecycle & garbage collection: claim check payloads are often > > temporary (needed only until downstream consumers extract text and > generate > > vector embeddings). Are there recommended Ozone lifecycle / expiration > > policies or chunking settings optimized for high-volume short-lived > object > > ingestion? > > This is a new feature just about to be merged into the master branch. It's > not even included in the Ozone 2.2 release. > So, I'm afraid there won't be many operational best practices for this > feature yet. > > > On Tue, Jul 21, 2026 at 8:24 AM Piergiorgio Lucidi <[email protected] > > > wrote: > > > Hi Ozone Community, > > > > I am writing to share an RFC and gather feedback on how we are > integrating > > Apache Ozone as a high-performance object store in OpenCrawling, the > > reference Java 25 & Spring implementation of the Open Ingestion Standard > > (OIS). > > > > In enterprise data ingestion pipelines for RAG and vector search, large > > binary files (PDFs, Office docs, CAD files) extracted from repository > > connectors (e.g. Alfresco, S3, Filesystem) cannot pass directly through > > streaming brokers (Apache Kafka) due to message size limitations and > broker > > memory overhead. > > > > To handle heavy payloads at scale, OpenCrawling implements the claim > check > > pattern using Apache Ozone: > > 1. Binary document streams are offloaded directly to Apache Ozone Object > > Storage (via Ozone's S3 Gateway / S3 API). > > 2. Lightweight JSON payload claims (containing document metadata, > security > > ACLs, and the Ozone S3 URI reference 's3://claims/{claim-id}') are > > published to Kafka topics. > > 3. Downstream consumer microservices (Ingestion Consumer, Embedding > > Service, Vector Store Writer) resolve binary content directly from Apache > > Ozone on demand. > > > > We would love to hear feedback from the Ozone community on a few > > architectural points: > > > > 1. S3 Gateway (s3g) vs. native Ozone Client: we currently use the AWS S3 > > SDK connecting through Ozone's S3 Gateway (s3g). For ultra-high > throughput > > streaming ingestion, does the community recommend switching to or > offering > > a native OzoneClient implementation alongside S3G? > > > > 2. Bucket lifecycle & garbage collection: claim check payloads are often > > temporary (needed only until downstream consumers extract text and > generate > > vector embeddings). Are there recommended Ozone lifecycle / expiration > > policies or chunking settings optimized for high-volume short-lived > object > > ingestion? > > > > 3. Benchmarking & collaboration: we have included an out-of-the-box > Apache > > Ozone 2.2.0 integration test suite (SCM, OM, Datanode, S3G) in our > > decoupled distribution ('test-ozone- > > decoupled.sh'). We welcome any Ozone developers interested in testing, > > benchmarking or contributing to join us! > > > > Links & Resources: > > - OpenCrawling repository: https://github.com/opencrawling/opencrawling > > - Open Ingestion Standard (OIS): > > https://github.com/opencrawling/open-ingestion-standard > > - Apache Ozone claim check Implementation: > > https://github.com/opencrawling/opencrawling/tree/main/oc-core > > > > We appreciate any thoughts, architectural advice, or contributions from > the > > Ozone community! > > > > Best regards, > > > > Piergiorgio Lucidi > > Member at The ASF, Apache ManifoldCF PMC Chair and OpenCrawling Founder > > > > -- > > Piergiorgio > > > -- Piergiorgio Lucidi Mobile: 3395381669
