[
https://issues.apache.org/jira/browse/IMPALA-14160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016124#comment-18016124
]
Michael Smith edited comment on IMPALA-14160 at 8/25/25 9:46 PM:
-----------------------------------------------------------------
{code}I20250825 14:16:19.447407 50666 FileSystemUtil.java:1295]
4243abc54df963b5:571e2c1e00000000] Next file
ofs://localhost:9862/impala/test-warehouse/test_runtime_filters_caea1456.db/runtime_filters/4243abc54df963b5-571e2c1e00000000_1662089718_data.0.txt
is encrypted: false
I20250825 14:16:19.447481 50666 FileSystemUtil.java:1295]
4243abc54df963b5:571e2c1e00000000] Next file
ofs://localhost:9862/impala/test-warehouse/test_runtime_filters_caea1456.db/runtime_filters/5a4e08317decf7f5-3c94458300000000_1971210288_data.0.txt
is encrypted: false
...
I20250825 14:16:20.124722 50900 FileSystemUtil.java:1295] Next file
ofs://localhost:9862/impala/test-warehouse/test_runtime_filters_caea1456.db/runtime_filters/4243abc54df963b5-571e2c1e00000000_1662089718_data.0.txt
is encrypted: true
I20250825 14:16:20.124790 50900 FileSystemUtil.java:1295] Next file
ofs://localhost:9862/impala/test-warehouse/test_runtime_filters_caea1456.db/runtime_filters/5a4e08317decf7f5-3c94458300000000_1971210288_data.0.txt
is encrypted: true{code}
I think I’ve narrowed it down to when we use fs.listStatusIterator vs
fs.listLocatedStatusIterator. When we use located status, which calls
getFileBlockLocations and returns a LocatedFileStatus , it reports that the
file is encrypted. If we just get status, it returns isEncrypted=false. Most of
the time we get located status, but in a few instances we get just status.
I don't think this affects anything other than planning. And this is
long-standing behavior. So in this case I'm just going to omit isEncrypted from
tuple cache key, because it really shouldn't affect planning.
was (Author: JIRAUSER288956):
{code}I20250825 14:16:19.447407 50666 FileSystemUtil.java:1295]
4243abc54df963b5:571e2c1e00000000] Next file
ofs://localhost:9862/impala/test-warehouse/test_runtime_filters_caea1456.db/runtime_filters/4243abc54df963b5-571e2c1e00000000_1662089718_data.0.txt
is encrypted: false
I20250825 14:16:19.447481 50666 FileSystemUtil.java:1295]
4243abc54df963b5:571e2c1e00000000] Next file
ofs://localhost:9862/impala/test-warehouse/test_runtime_filters_caea1456.db/runtime_filters/5a4e08317decf7f5-3c94458300000000_1971210288_data.0.txt
is encrypted: false
...
I20250825 14:16:20.124722 50900 FileSystemUtil.java:1295] Next file
ofs://localhost:9862/impala/test-warehouse/test_runtime_filters_caea1456.db/runtime_filters/4243abc54df963b5-571e2c1e00000000_1662089718_data.0.txt
is encrypted: true
I20250825 14:16:20.124790 50900 FileSystemUtil.java:1295] Next file
ofs://localhost:9862/impala/test-warehouse/test_runtime_filters_caea1456.db/runtime_filters/5a4e08317decf7f5-3c94458300000000_1971210288_data.0.txt
is encrypted: true{code}
I think I’ve narrowed it down to when we use fs.listStatusIterator vs
fs.listLocatedStatusIterator. When we use located status, which calls
getFileBlockLocations and returns a LocatedFileStatus , it reports that the
file is encrypted. If we just get status, it returns isEncrypted=false. Most of
the time we get located status, but in a few instances we get just status.
> TestTupleCacheCluster.test_runtime_filters fails on Ozone
> ---------------------------------------------------------
>
> Key: IMPALA-14160
> URL: https://issues.apache.org/jira/browse/IMPALA-14160
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Michael Smith
> Priority: Blocker
>
> On certain versions of Ozone, TestTupleCacheCluster.test_runtime_filters
> fails with:
> {noformat}
> custom_cluster/test_tuple_cache.py:501: in test_runtime_filters
> assertCounters(rerun_two_files_result.runtime_profile, 1, 0, 0,
> num_matches=8)
> custom_cluster/test_tuple_cache.py:75: in assertCounters
> assertCounter(profile, NUM_HITS, num_hits, num_matches)
> custom_cluster/test_tuple_cache.py:71: in assertCounter
> assert len([v for v in values if v == val]) in num_matches, values
> E AssertionError: [0, 0, 0, 1, 1, 0, ...]
> E assert 2 in [8]
> E + where 2 = len([1, 1]){noformat}
> This is running "invalidate metadata" and expecting the cache keys not to
> change. It seems to have been working until recently, so it is possible there
> is some interaction with the version of Ozone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
