Thank you for the POC, and the numbers from your POC. It looks very good.
I know this is a private POCproposal, yet I have two minor questions.
1. Should we maintain the client ID in "private final Map<ContainerID,
Long> containerLeases" map ? so instead of a long we maintain a Long +
Client ID is what I was thinking. Might be useful for debugging.
2. Suppose a client keeps on renewing a container lease, do we want to
enforce a maximum limit ? It is not needed per se -- more like a question
that I am asking myself.
Thanks
Anu
On Mon, Sep 26, 2022 at 2:42 AM Kaijie Chen <c...@apache.org> wrote:
> Hi everyone,
>
> I've implemented a container lease POC [1], and the result looks good.
>
> Here's what's changed in the POC:
>
> 1. SCM will keep a LeaseExipreAt for each OPEN container. If SCM
> receives container close command, it will change the container
> state to CLOSING, but it will not send close container command
> to DN until the lease expires.
> 2. OM will forward the container lease request from Client to SCM.
> 3. Client will acquire lease when a block is allocated (to be improved),
> and it will renew leases for open blocks before its expiration.
> Client will ignore any errors with leases, and keep writing chunks
> to DN even if lease expires. Because the wrost case is simply
> ContainerNotOpenException.
>
> Despite this POC is not perfect, the result in my tests looks good.
>
> Cluster: 48 datanodes on 4 machines
> Client: Ozone freon ockg
> Threads: 100
> Key count: 1000
> Key size: 1000 MB
> ReplicationConfig: EC/RS-10-4-1024K
>
> We should expect 14000x 100 MB blocks in ideal condition.
> I'm only showing the data from 1 of the 4 machines.
>
>
> Before the change (commit 1cf5678224bf00dee580ffdb14ab8b650cc1e2e0):
> (The number before each sizes is the count of blocks in that size)
>
> 15 1.0M 48 2.0M 40 3.0M 48 4.0M 37 5.0M 33 6.0M 48 7.0M 51 8.0M
> 30 9.0M 49 10M 40 11M 65 12M 33 13M 18 14M 43 15M 46 16M 38 17M
> 20 18M 46 19M 32 20M 5 21M 54 22M 58 23M 33 24M 25 25M 39 26M
> 44 27M 48 28M 25 29M 18 30M 34 31M 42 32M 22 33M 23 34M 27 35M
> 26 36M 33 37M 27 38M 30 39M 60 40M 25 41M 27 42M 26 43M 20 44M
> 13 45M 18 46M 40 47M 27 48M 25 49M 15 50M 40 51M 26 52M 41 53M
> 41 54M 9 55M 11 56M 11 57M 19 58M 30 59M 28 60M 44 61M 36 62M
> 21 63M 14 64M 19 65M 14 66M 23 67M 33 68M 40 69M 34 70M 17 71M
> 10 72M 35 73M 28 74M 24 75M 21 76M 34 77M 26 78M 35 79M 18 80M
> 27 81M 26 82M 14 83M 19 84M 23 85M 29 86M 4 87M 23 88M 37 89M
> 11 90M 23 91M 38 92M 16 93M 12 94M 18 95M 21 96M 27 97M 19 98M
> 35 99M 2099 100M
>
> Container size before the change:
>
> $ ./ozone admin container list -c 10000 | grep usedBytes | awk '{print
> $3}' | sort | xargs echo
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 1001390080,
> 1002438656, 1003487232, 1003487232, 1004535808, 1004535808, 1004535808,
> 1004535808, 1006632960, 1007681536, 1010827264, 1011875840, 1011875840,
> 1011875840, 1013972992, 1016070144, 1016070144, 1016070144, 1019215872,
> 1024458752, 1028653056, 1028653056, 1031798784, 1032847360, 1032847360,
> 1032847360, 1033895936, 1035993088, 1044381696, 1046478848, 1050673152,
> 1062207488, 1092616192, 1096810496, 968884224, 968884224, 970981376,
> 970981376, 972029952, 972029952, 973078528, 973078528, 974127104,
> 974127104, 975175680, 976224256, 976224256, 976224256, 976224256,
> 976224256, 976224256, 976224256, 976224256, 979369984, 980418560,
> 980418560, 980418560, 981467136, 981467136, 983564288, 983564288,
> 983564288, 984612864, 984612864, 984612864, 985661440, 985661440,
> 985661440, 985661440, 986710016, 986710016, 987758592, 987758592,
> 988807168, 988807168, 989855744, 989855744, 989855744, 989855744,
> 990904320, 990904320, 990904320, 990904320, 990904320, 990904320,
> 991952896, 991952896, 993001472, 994050048, 996147200, 997195776,
> 998244352, 998244352,
>
>
> After the change (commit 52c903ccc644aba63bbd5354bae98bc8bbe13675):
> (Occasionally, there are a few blocks breaked into smaller ones)
>
> 3571 100M
>
> Container sizes after the change:
>
> **Note: "ozone.scm.container.size" was set to 1G**
> **Note: "hdds.datanode.storage.utilization.critical.threshold" was set
> to 0.99**
>
> $ ./ozone admin container list -c 10000 | grep usedBytes | awk '{print
> $3}' | sort | xargs echo
> 0, 1258291200, 1258291200, 1363148800, 1468006400, 1782579200,
> 1887436800,
> 1887436800, 1992294400, 2306867200, 2621440000, 2621440000, 2726297600,
> 2831155200, 2831155200, 2936012800, 2936012800, 3040870400, 3040870400,
> 3040870400, 3040870400, 3040870400, 3145728000, 3250585600, 3250585600,
> 3355443200, 3355443200, 3460300800, 3565158400, 3565158400, 3670016000,
> 3670016000, 3774873600, 3879731200, 3879731200, 4404019200, 4404019200,
>
> I've also done tests in RATIS/THREE, the results looks similiar.
>
>
> What I've implemented in POC is basically don't let DN close a
> container if it is recently written to. And it could be implemented
> solely in DN by a lastUpdated timestamp in containers.
> So we won't need extra RPCs to achieve this, what do you think?
>
> Please help verify and give feedbacks and suggestions.
>
> Thanks,
> Kaijie
>
> ---
>
> [1]: https://github.com/kaijchen/ozone/tree/container-lease
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org
> For additional commands, e-mail: dev-h...@ozone.apache.org
>
>
