[
https://issues.apache.org/jira/browse/KUDU-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17400820#comment-17400820
]
ASF subversion and git services commented on KUDU-2671:
-------------------------------------------------------
Commit b5cf9d9a5738d77f692a4dc1a33dd9c1a7a367e2 in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=b5cf9d9 ]
[partition] update naming of related entities
With this patch, the naming for the partition-related entities
in the C++ code has changed:
* renamed HashBucketSchema into HashDimension
* renamed HashBucketSchemas into HashSchema, so HashSchema is now
a typedef for std::vector<HashDimension>
* partitioning schema --> partition schema
* hash bucket schema --> hash schema
* hash partitioning schema --> hash schema
* range partition schema --> range schema
* hash_bucket_schemas_ --> hash_schema_
* removed the PerRangeHashBucketSchemas typedef
The idea is to represent the hash-related component of the partition
schema as a set of hash bucket dimensions. Before, it was called
'hash bucket schemas' to be 1-to-1 mapping with the
PartitionSchemaPB::HashBucketSchemaPB protobuf message defined in
common.proto. I found it a bit awkward calling a single dimension of
hash bucketing 'schema' because it's semantically confusing, since
'schema' usually means the whole set of rules, not just a part of those.
It would be nice to update the proto-related names as well, but doing so
would break backwards compatibility.
I also updated the in-line documentation for the partition-related
entities in the PartitionSchema's class-wide comment and added TODOs
where more updates are required to accommodate the recent changes
introduced in the context of KUDU-2671.
Change-Id: I6a858e97090930b21e9c767dac2f5cc8b9816033
Reviewed-on: http://gerrit.cloudera.org:8080/17775
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Andrew Wong <[email protected]>
Reviewed-by: Mahesh Reddy <[email protected]>
> Change hash number for range partitioning
> -----------------------------------------
>
> Key: KUDU-2671
> URL: https://issues.apache.org/jira/browse/KUDU-2671
> Project: Kudu
> Issue Type: Improvement
> Components: client, java, master, server
> Affects Versions: 1.8.0
> Reporter: yangz
> Assignee: Mahesh Reddy
> Priority: Major
> Labels: feature, roadmap-candidate, scalability
> Attachments: 屏幕快照 2019-01-24 下午12.03.41.png
>
>
> For our usage, the kudu schema design isn't flexible enough.
> We create our table for day range such as dt='20181112' as hive table.
> But our data size change a lot every day, for one day it will be 50G， but for
> some other day it will be 500G. For this case, it be hard to set the hash
> schema. If too big, for most case, it will be too wasteful. But too small,
> there is a performance problem in the case of a large amount of data.
>
> So we suggest a solution we can change the hash number by the history data of
> a table.
> for example
> # we create schema with one estimated value.
> # we collect the data size by day range
> # we create new day range partition by our collected day size.
> We use this feature for half a year, and it work well. We hope this feature
> will be useful for the community. Maybe the solution isn't so complete.
> Please help us make it better.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
