[
https://issues.apache.org/jira/browse/IMPALA-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951920#comment-17951920
]
ASF subversion and git services commented on IMPALA-14049:
----------------------------------------------------------
Commit 78d1c2cd3a4f2c81a9142bc80b8bba9d2e9ff292 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=78d1c2cd3 ]
IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test
IMPALA-13978 switched HdrHistrogram from using a gscoped_ptr
to unique_ptr. This has been causing TSAN issues during the
teardown for expr-test. gscoped_ptr doesn't null out the
pointer when it gets destructed, but unique_ptr does. This
is a data race with the threads that are still running and
trying to access the metrics.
The full solution would be to have an orderly shutdown of
all the threads before destructing things. That is a large
project that would touch many different components. As a
short-term fix, this avoids the TSAN issue by leaking
the statestore metrics.
We should consider fixing IMPALA-9314 and implementing
orderly shutdown.
Testing:
- Ran expr-test in a loop with TSAN and didn't see this
particular issue. There are other shutdown issues with
much lower frequency that have different symptoms.
Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Reviewed-on: http://gerrit.cloudera.org:8080/22900
Reviewed-by: Riza Suminto <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> expr-test fails in TSAN run due to IMPALA-13978
> -----------------------------------------------
>
> Key: IMPALA-14049
> URL: https://issues.apache.org/jira/browse/IMPALA-14049
> Project: IMPALA
> Issue Type: Task
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> IMPALA-13978 switched HdrHistogram from using a gscoped_array to a
> unique_ptr. expr-test now fails in TSAN when tearing down the global
> structures:
> {noformat}
> WARNING: ThreadSanitizer: data race (pid=1270841)
> Write of size 8 at 0x7b140066d868 by main thread:
> #0 std::unique_ptr<long [], std::default_delete<long []> >::~unique_ptr()
> /opt/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_ptr.h:613:8
> (unifiedbetests+0x7aa7af1)
> #1 impala::HdrHistogram::~HdrHistogram()
> /home/joemcdonnell/upstream/Impala/be/src/util/hdr-histogram.h:70:7
> (unifiedbetests+0x7d5a14d)
> #2 void
> boost::checked_delete<impala::HdrHistogram>(impala::HdrHistogram*)
> /opt/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/core/checked_delete.hpp:36:5
> (unifiedbetests+0x7d5a10e)
> #3 boost::scoped_ptr<impala::HdrHistogram>::~scoped_ptr()
> /opt/Impala-Toolchain/toolchain-packages-gcc10.4.0/boost-1.74.0-p1/include/boost/smart_ptr/scoped_ptr.hpp:88:9
> (unifiedbetests+0x7d5a0e1)
> #4 impala::HistogramMetric::~HistogramMetric()
> /home/joemcdonnell/upstream/Impala/be/src/util/histogram-metric.h:31:7
> (unifiedbetests+0x7d52854)
> ...
> Previous read of size 8 at 0x7b140066d868 by thread T955 (mutexes: write
> M235448423542615688):
> #0 std::__uniq_ptr_impl<long, std::default_delete<long []> >::_M_ptr()
> const
> /opt/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_ptr.h:173:42
> (unifiedbetests+0x7a98944)
> #1 std::unique_ptr<long [], std::default_delete<long []> >::get() const
> /opt/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_ptr.h:666:21
> (unifiedbetests+0x7a98909)
> #2 std::unique_ptr<long [], std::default_delete<long []>
> >::operator[](unsigned long) const
> /opt/Impala-Toolchain/toolchain-packages-gcc10.4.0/gcc-10.4.0/lib/gcc/x86_64-pc-linux-gnu/10.4.0/../../../../include/c++/10.4.0/bits/unique_ptr.h:660:9
> (unifiedbetests+0x7a986fd)
> #3 impala::HdrHistogram::IncrementBy(long, long)
> /home/joemcdonnell/upstream/Impala/be/src/util/hdr-histogram.cc:158:30
> (unifiedbetests+0x8479de6)
> #4 impala::HdrHistogram::Increment(long)
> /home/joemcdonnell/upstream/Impala/be/src/util/hdr-histogram.cc:144:3
> (unifiedbetests+0x8479cd5)
> #5 impala::HistogramMetric::Update(long)
> /home/joemcdonnell/upstream/Impala/be/src/util/histogram-metric.h:46:17
> (unifiedbetests+0x6ca86dd)
> #6 impala::RpcEventHandler::postWrite(void*, char const*, unsigned int)
> /home/joemcdonnell/upstream/Impala/be/src/rpc/rpc-trace.cc:265:40
> (unifiedbetests+0x7cc595e)
> {noformat}
> There is some problem with the order in which things get torn down. expr-test
> is a complete mess in that regard. Anyway, the behavior is different between
> gscoped_array and unique_ptr, and this starts to fail after the switch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
