I have a few thoughts on this, but my knowledge may be out-dated. 1. During putBlock, the DN notices the usage has gone beyond 90%, so it sends a close command to SCM via its heartbeat.
2. SCM closes the container on the SCM side. At this point, SCM will not allocate any more blocks to it but there may be some currently being written, previously allocated. 3. The 5GB container limit is a soft-limit - its ok for a container to go beyond this size. 4. It was my understanding, although I cannot find the code right now, that there is some "grace period" for inflight blocks to complete writing when a container starts to close. If we stop allocating blocks in SCM because the close process has been triggered, then the grace period should allow most inflight blocks to complete writing. Does the grace period still exist, and if so, it is not helping with this problem? On Wed, Sep 7, 2022 at 9:31 AM Kaijie Chen <c...@apache.org> wrote: > Hi Ozone devs, > > Recently our team have noticed a problem in Ozone. When a large number of > clients are writing > data to Ozone simultaneously, they will face ContainerNotOpenException > very often. > > The root cause is, when client allocates a block from OM, the space is not > reserved in the container. > Many clients may got blocks from a same container, which does not have > space to hold all of them. > Finally when the container becomes full, the ContainerNotOpenException > will be thrown. > > Based on my understanding, I have created a proposal to reserve space for > allocated blocks. > > > https://docs.google.com/document/d/1NjhBhtto8cuGXnYhYToOLJyD5kzn65UVG5-scGhKaw0/ > > Please leave comments if I missed some points or if you have a better idea. > > PS: I have another idea which is less complicated but is incompatible with > the current design. > The basic idea is writing blocks to a temp storage and assembling > containers in the background. > > Regards, > Kaijie Chen > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org > For additional commands, e-mail: dev-h...@ozone.apache.org > >
