[
https://issues.apache.org/jira/browse/HIVE-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-15571:
------------------------------------
Description:
Add support of inset into operator for druid storage handler.
Implementation -
1) Using DruidOutputFormat, multiple reducers write segments to a staging
directory with the new versions.
2) When doing insert into, before publishing the segments to the metadata
storage, versions for the segments are updated based on the existing segments
in druid metadata storage.
3) After final versions are checked, the segments are moved from the staging
directory to the segment directory.
4) New segment versions are published to the metadata storage.
Reading of existing segments from metadata storage and updating the versions of
newly created segments based on them is done in a single transaction. This will
make sure that segments generated by parallel Insert Into statements being
executed will not overwrite each other. In case of conflict in metadata storage
transaction insert into will fail.
was:
Add support of inset into operator for druid storage handler.
> Support Insert into for druid storage handler
> ---------------------------------------------
>
> Key: HIVE-15571
> URL: https://issues.apache.org/jira/browse/HIVE-15571
> Project: Hive
> Issue Type: New Feature
> Components: Druid integration
> Reporter: slim bouguerra
> Assignee: Nishant Bangarwa
> Attachments: HIVE-15571.01.patch, HIVE-15571.02.patch
>
>
> Add support of inset into operator for druid storage handler.
> Implementation -
> 1) Using DruidOutputFormat, multiple reducers write segments to a staging
> directory with the new versions.
> 2) When doing insert into, before publishing the segments to the metadata
> storage, versions for the segments are updated based on the existing segments
> in druid metadata storage.
> 3) After final versions are checked, the segments are moved from the staging
> directory to the segment directory.
> 4) New segment versions are published to the metadata storage.
> Reading of existing segments from metadata storage and updating the versions
> of newly created segments based on them is done in a single transaction. This
> will make sure that segments generated by parallel Insert Into statements
> being executed will not overwrite each other. In case of conflict in metadata
> storage transaction insert into will fail.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
