Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/22559 )
Change subject: IMPALA-11402: Add limit on files fetched by a single getPartialCatalogObject request ...................................................................... Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/22559/9/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/22559/9/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2356 PS9, Line 2356: "the JVM array limit.", full_name_, > See if we can avoid letting java throw this array error and instead throw o It's hard-coded in JVM: https://github.com/openjdk/jdk8u/blob/93122abf52afbd4e3c2de26561a4b3e78918ce14/jdk/src/share/classes/java/io/ByteArrayOutputStream.java#L123 So the only way to avoid it is not allocating long arrays. We use these new checks to avoid Java throwing this array error. BTW, the error is documented in the "Understand the OutOfMemoryError Exception" section of this doc: https://docs.oracle.com/en/java/javase/16/troubleshoot/troubleshoot-memory-leaks.html#GUID-19F6D28E-75A1-4480-9879-D0932B2F305B What about using the full message, i.e. "java.lang.OutOfMemoryError: Requested array size exceeds VM limit"? -- To view, visit http://gerrit.cloudera.org:8080/22559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb13fec20de5a17e7fc33613ca5cdebb9ac1a1e5 Gerrit-Change-Number: 22559 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Comment-Date: Wed, 26 Mar 2025 23:57:20 +0000 Gerrit-HasComments: Yes
