anuragp010 opened a new pull request, #11180: URL: https://github.com/apache/ozone/pull/11180
## What changes were proposed in this pull request? Any long lived client JVM which is writing EC Keys using `ECKeyOutputStream` leaks native memory. This occurs because `ECKeyOutputStream.close()` never calls `encoder.release()`. When `NativeRSRawEncoder` is in use by `ECKeyOutputStream`, each key creation results in acquiring native memory to store lookup tables used in calculation of Parity Data. This native memory is only freed by calling `release()`. This PR adds `encoder.release` in `ECKeyOutputStream.close()` to free the native memory. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-15916 ## How was this patch tested? 1. Unit Test - Ensures `encoder.release()` is called. 2. Local Docker compose cluster: ### **Env Setup:** 1. Build with `-Phadoop-native-lib`: `mvn clean package -Pdist -Phadoop-native-lib -DskipTests -DskipShade -DskipRecon -DskipDocs` 2. Update `docker-compose.yaml `to include `isa-l` libraries pre starting `S3Gateway`: ```yaml s3g: <<: *common-config environment: OZONE_OPTS: <<: *replication ports: - 9878:9878 - 19878:19878 command: [ "bash","-c","sudo dnf install -y epel-release && sudo dnf install -y isa-l && exec ozone s3g" ] ``` 3. Use `linux/amd64` as the default platform: `export DOCKER_DEFAULT_PLATFORM=linux/amd64` 4. Start the cluster: `docker compose up -d --scale datanode=5` ### **Running the test:** 1. Create an S3 bucket. 2. Create many objects for key put's through the s3 gateway: ```shell mkdir -p /tmp/many && cd /tmp/many # 20,000 tiny objects = 20,000 EC keys for i in $(seq 1 20000); do echo x > f$i; done ``` 3. Put the keys through s3g: `aws s3 $EP cp --recursive --storage-class STANDARD_IA /tmp/many s3://leaktest/r1/` ### **Results:** The RSS (Resident Set Size) for the s3 gateway process was captured after the runs with the fix and without the fix. Commands used: 1. Trigger manual GC: `jcmd $PID GC.run` 2. Grep the Process RSS: `grep VmRSS /proc/$PID/status` #### Post-GC RSS floor of the S3 Gateway JVM | Round | Without fix | Δ/round | With fix | Δ/round | |------:|------------:|--------:|---------:|--------:| | baseline | 564 MB | — | 568 MB | — | | 1 | 789 MB | +225 | 807 MB | +239 | | 2 | 913 MB | **+124** | 830 MB | **+23** | | 3 | 993 MB | **+80** | 845 MB | **+16** | | 4 | 1105 MB | **+112** | 831 MB | **−14** | | **net (r1→r4)** | | **+316 MB** | | **+25 MB, flat** |% Without the fix, post each run, there is a clear upward trend in RSS of the `S3Gateway` process. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
