> what's the plan to have streaming writes enabled automatically for large files?
After merging the Streaming Write Pipeline branch, we could work on a new feature adding a buffer with size equal to chunk size (default 4MB) on the client side. When an application writes data, the data first goes to the buffer. If the application closes the stream before the buffer is full, it will use PutSmallFile to fulfill the request since it only has one chunk. Otherwise, the application continues writing more than one chunk, it will open a stream to write the data. When Streaming is enabled, it will use Streaming to write the data. Otherwise, it will use the async API to write the data. Tsz-Wo On Tue, Dec 6, 2022 at 9:36 AM Arpit Agarwal <a...@apache.org> wrote: > +1 for merging this feature. This is an important performance improvement > for Ozone. > > Nicholas, what's the plan to have streaming writes enabled automatically > for large files? > > ----- Original message ----- > From: Tsz Wo Sze <szets...@gmail.com> > To: dev Ozone <dev@ozone.apache.org> > Subject: [DISCUSS] Proposal to merge HDDS-4454 Ozone Streaming Write > Pipeline into master > Date: Thursday, December 01, 2022 12:24 PM > > Dear Ozone Devs, > > We would like to start this discussion thread for the proposal to merge > https://issues.apache.org/jira/browse/HDDS-4454 Ozone Streaming Write > Pipeline to the master branch. > > Currently, Ozone write pattern is bursty and involves multiple buffers > copies as well multiple ratis log syncs in a block write. The idea of the > Jira (HDDS-4454) is to use a zero buffer copy based Ratis Streaming > (RATIS-979) in Ozone write path for better performance and resource > utilization. > > Note that HDDS-4454 is a new feature such that it is mainly adding new > code. Therefore, the risk is very low to the existing functionalities and > features in Ozone. > > For the sake of master branch feature stability, we provide a configuration > (dfs.container.ratis.datastream.enabled) to enable/disable this feature. By > default, this feature is disabled. > > For more information, please check out Ozone the Streaming Write Pipeline > (HDDS-4454) feature wiki page here: > > > https://cwiki.apache.org/confluence/display/OZONE/Merge+Ozone+Streaming+Write+Pipeline+%28HDDS-4454%29+Checklist > > If there are no objections for the merge, we could start the official vote. > > Thanks, > > Tsz-Wo > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org > For additional commands, e-mail: dev-h...@ozone.apache.org > >
