Sergey Shelukhin created HIVE-18938:
---------------------------------------
Summary: LLAP: cache priorities (soft pin, exclusions)
configuration
Key: HIVE-18938
URL: https://issues.apache.org/jira/browse/HIVE-18938
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
We want persistent configuration and not an ad-hoc pin API because the latter
is both unreliable (e.g. node might go down and restart, so someone has to keep
track of global state anyway - might as well have it done explicitly), and also
dangerous because errant pin calls can fill cache with garbage and make it
unusable.
So, the idea is that we can allow to
1) Give some tables or partitions (tags, see HIVE-16992) a priority boost at
cache time.
2) Exclude some tables or partitions from being cached.
Ideally we also want this changeable without LLAP restart (and managed by HS2).
So the choice here is either make this a config setting, like many Hadoop
settings that have user/group/etc lists, and implement ZK-based/HS2-based
configuration changes without restart, that we want to do anyway at some point;
or make this a separate thing, probably managed and stored as policies in
metastore (kind of like workload management) and update via ZK.
It's possible to go with config settings that do require restart for the first
cut.
Separately, we can add a workload management policy that would disable cache
for certain query pools (i.e. users/groups/apps).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
