15 GB sounds excessive; I would first investigate how that can happen and
if we have some sort of path this is not explored fully or perhaps a bug,
in the allocation or the client are moving too fast for us to respond.
If you think the issue is with the clients being able to get leases too
fast, I think that you need a solution combination of tracking and leases.
if we can limit, two things :
1. The maximum times you can renew the lease - It limits the maximum time a
client can force the container to remain open.
2. The maximum number of outstanding leases - Have a policy, for example if
you can say that we will have only 50% of unallocated space at any given
time as leases -- That is the proposal that we were discussing on the other
thread.
Also be aware that this is a soft constraint -- if a large number of your
containers behave and tend to converge to your expected size, overall your
system is stable(r).
Thanks
Anu
On Wed, Sep 28, 2022 at 5:56 AM Kaijie Chen <c...@apache.org> wrote:
> Hi Anu,
>
> Thanks for your suggestions. These are indeed where we can
> improve the code. I have something more to share.
>
> I did more tests today, and I have observed containers over 15 GB,
> which is 15 times of the configured container size limit (1 GB).
> It might be related to the pipeline chosing policy and the container
> close threshold (99%).
>
> Because we have no control of how many block can be allocated
> simultaneously, it seems there is risk we can get abnormally
> large containers. What do you think?
>
> I have also tested the simple delay proposal. It sometimes works well.
> But sometimes still produces fragmented blocks. This is expected.
>
> Kaijie
>
> ---- On Wed, 28 Sep 2022 08:00:38 +0800 anu engineer wrote ---
> > Thank you for the POC, and the numbers from your POC. It looks very
> good.
> > I know this is a private POCproposal, yet I have two minor questions.
> >
> > 1. Should we maintain the client ID in "private final Map<ContainerID,
> > Long> containerLeases" map ? so instead of a long we maintain a Long +
> > Client ID is what I was thinking. Might be useful for debugging.
> > 2. Suppose a client keeps on renewing a container lease, do we want to
> > enforce a maximum limit ? It is not needed per se -- more like a
> question
> > that I am asking myself.
> >
> > Thanks
> > Anu
> >
> >
> >
> >
> > On Mon, Sep 26, 2022 at 2:42 AM Kaijie Chen c...@apache.org> wrote:
> >
> > > Hi everyone,
> > >
> > > I've implemented a container lease POC [1], and the result looks good.
> > >
> > > Here's what's changed in the POC:
> > >
> > > 1. SCM will keep a LeaseExipreAt for each OPEN container. If SCM
> > > receives container close command, it will change the container
> > > state to CLOSING, but it will not send close container command
> > > to DN until the lease expires.
> > > 2. OM will forward the container lease request from Client to SCM.
> > > 3. Client will acquire lease when a block is allocated (to be
> improved),
> > > and it will renew leases for open blocks before its expiration.
> > > Client will ignore any errors with leases, and keep writing chunks
> > > to DN even if lease expires. Because the wrost case is simply
> > > ContainerNotOpenException.
> > >
> > > Despite this POC is not perfect, the result in my tests looks good.
> > >
> > > Cluster: 48 datanodes on 4 machines
> > > Client: Ozone freon ockg
> > > Threads: 100
> > > Key count: 1000
> > > Key size: 1000 MB
> > > ReplicationConfig: EC/RS-10-4-1024K
> > >
> > > We should expect 14000x 100 MB blocks in ideal condition.
> > > I'm only showing the data from 1 of the 4 machines.
> > >
> > >
> > > Before the change (commit 1cf5678224bf00dee580ffdb14ab8b650cc1e2e0):
> > > (The number before each sizes is the count of blocks in that size)
> > >
> > > 15 1.0M 48 2.0M 40 3.0M 48 4.0M 37 5.0M 33 6.0M 48 7.0M 51 8.0M
> > > 30 9.0M 49 10M 40 11M 65 12M 33 13M 18 14M 43 15M 46 16M 38 17M
> > > 20 18M 46 19M 32 20M 5 21M 54 22M 58 23M 33 24M 25 25M 39 26M
> > > 44 27M 48 28M 25 29M 18 30M 34 31M 42 32M 22 33M 23 34M 27 35M
> > > 26 36M 33 37M 27 38M 30 39M 60 40M 25 41M 27 42M 26 43M 20 44M
> > > 13 45M 18 46M 40 47M 27 48M 25 49M 15 50M 40 51M 26 52M 41 53M
> > > 41 54M 9 55M 11 56M 11 57M 19 58M 30 59M 28 60M 44 61M 36 62M
> > > 21 63M 14 64M 19 65M 14 66M 23 67M 33 68M 40 69M 34 70M 17 71M
> > > 10 72M 35 73M 28 74M 24 75M 21 76M 34 77M 26 78M 35 79M 18 80M
> > > 27 81M 26 82M 14 83M 19 84M 23 85M 29 86M 4 87M 23 88M 37 89M
> > > 11 90M 23 91M 38 92M 16 93M 12 94M 18 95M 21 96M 27 97M 19 98M
> > > 35 99M 2099 100M
> > >
> > > Container size before the change:
> > >
> > > $ ./ozone admin container list -c 10000 | grep usedBytes | awk
> '{print
> > > $3}' | sort | xargs echo
> > > 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0,
> > > 0, 0, 0, 1001390080,
> > > 1002438656, 1003487232, 1003487232, 1004535808, 1004535808,
> 1004535808,
> > > 1004535808, 1006632960, 1007681536, 1010827264, 1011875840,
> 1011875840,
> > > 1011875840, 1013972992, 1016070144, 1016070144, 1016070144,
> 1019215872,
> > > 1024458752, 1028653056, 1028653056, 1031798784, 1032847360,
> 1032847360,
> > > 1032847360, 1033895936, 1035993088, 1044381696, 1046478848,
> 1050673152,
> > > 1062207488, 1092616192, 1096810496, 968884224, 968884224,
> 970981376,
> > > 970981376, 972029952, 972029952, 973078528, 973078528, 974127104,
> > > 974127104, 975175680, 976224256, 976224256, 976224256, 976224256,
> > > 976224256, 976224256, 976224256, 976224256, 979369984, 980418560,
> > > 980418560, 980418560, 981467136, 981467136, 983564288, 983564288,
> > > 983564288, 984612864, 984612864, 984612864, 985661440, 985661440,
> > > 985661440, 985661440, 986710016, 986710016, 987758592, 987758592,
> > > 988807168, 988807168, 989855744, 989855744, 989855744, 989855744,
> > > 990904320, 990904320, 990904320, 990904320, 990904320, 990904320,
> > > 991952896, 991952896, 993001472, 994050048, 996147200, 997195776,
> > > 998244352, 998244352,
> > >
> > >
> > > After the change (commit 52c903ccc644aba63bbd5354bae98bc8bbe13675):
> > > (Occasionally, there are a few blocks breaked into smaller ones)
> > >
> > > 3571 100M
> > >
> > > Container sizes after the change:
> > >
> > > **Note: "ozone.scm.container.size" was set to 1G**
> > > **Note: "hdds.datanode.storage.utilization.critical.threshold"
> was set
> > > to 0.99**
> > >
> > > $ ./ozone admin container list -c 10000 | grep usedBytes | awk
> '{print
> > > $3}' | sort | xargs echo
> > > 0, 1258291200, 1258291200, 1363148800, 1468006400, 1782579200,
> > > 1887436800,
> > > 1887436800, 1992294400, 2306867200, 2621440000, 2621440000,
> 2726297600,
> > > 2831155200, 2831155200, 2936012800, 2936012800, 3040870400,
> 3040870400,
> > > 3040870400, 3040870400, 3040870400, 3145728000, 3250585600,
> 3250585600,
> > > 3355443200, 3355443200, 3460300800, 3565158400, 3565158400,
> 3670016000,
> > > 3670016000, 3774873600, 3879731200, 3879731200, 4404019200,
> 4404019200,
> > >
> > > I've also done tests in RATIS/THREE, the results looks similiar.
> > >
> > >
> > > What I've implemented in POC is basically don't let DN close a
> > > container if it is recently written to. And it could be implemented
> > > solely in DN by a lastUpdated timestamp in containers.
> > > So we won't need extra RPCs to achieve this, what do you think?
> > >
> > > Please help verify and give feedbacks and suggestions.
> > >
> > > Thanks,
> > > Kaijie
> > >
> > > ---
> > >
> > > [1]: https://github.com/kaijchen/ozone/tree/container-lease
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org
> > > For additional commands, e-mail: dev-h...@ozone.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org
> For additional commands, e-mail: dev-h...@ozone.apache.org
>
>
