Hello Quanlong Huang, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21367
to look at the new patch set (#4).
Change subject: IMPALA-13020: Change thrift_rpc_max_message_size to int64_t
......................................................................
IMPALA-13020: Change thrift_rpc_max_message_size to int64_t
Thrift 0.16.0 introduced a max message size to protect
receivers against a malicious message allocating large
amounts of memory. That limit is a 32-bit signed integer,
so the max value is 2GB. Impala introduced the
thrift_rpc_max_message_size startup option to set that
for Impala's thrift servers.
There are times when Impala wants to send a message that
is larger than 2GB. In particular, the catalog-update
topic for the statestore can exceed 2GBs when there is
a lot of metadata loaded using the old v1 catalog. When
there is a 2GB max message size, the statestore can create
and send a >2GB message, but the impalads will reject
it. This can lead to impalads having stale metadata.
This switches to a patched Thrift that uses an int64_t
for the max message size for C++ code. It also modifies
thrift_rpc_max_message_size to be an int64_t and increases
the default max message size to 4GB.
The MaxMessageSize error was being swallowed in TAcceptQueueServer.cpp,
so this fixes that location to always print MaxMessageSize
exceptions.
This is only patching the Thrift C++ library. It does not
patch the Thrift Java library. There are a few reasons for
that:
- This specific issue involves C++ to C++ communication and
will be solved by patching the C++ library.
- C++ is easy to patch as it is built via the native-toolchain.
There is no corresponding mechanism for patching our Java
dependencies (though one could be developed).
- Java modifications have implications for other dependencies
like Hive which use Thrift to communicate with HMS.
For the Java code that uses max message size, this converts
the 64-bit value to 32-bit value by capping the value at
Integer.MAX_VALUE.
Testing:
- Added enough tables to produce a >2GB catalog-topic and
restarted an impalad. Without the patch, the catalog-topic
update would be rejected by the impalad. With the patch,
it succeeds.
Change-Id: I681b1849cc565dcb25de8c070c18776ce69cbb87
---
M be/src/common/init.cc
M be/src/rpc/TAcceptQueueServer.cpp
M be/src/rpc/thrift-server-test.cc
M be/src/rpc/thrift-util.cc
M be/src/rpc/thrift-util.h
M be/src/util/backend-gflag-util.cc
M bin/impala-config.sh
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
9 files changed, 30 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/21367/4
--
To view, visit http://gerrit.cloudera.org:8080/21367
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I681b1849cc565dcb25de8c070c18776ce69cbb87
Gerrit-Change-Number: 21367
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
