Nishant Bangarwa created HIVE-18976:
---------------------------------------
Summary: Add ability to setup Druid Kafka Ingestion from Hive
Key: HIVE-18976
URL: https://issues.apache.org/jira/browse/HIVE-18976
Project: Hive
Issue Type: Bug
Components: Druid integration
Reporter: Nishant Bangarwa
Assignee: Nishant Bangarwa
Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
e.g. Below query can submit a kafka supervisor spec to the druid overlord and
druid can start ingesting events from kafka.
{code:java}
CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language string,
`user` string, added int, deleted int, delta int)
STORED BY
'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
TBLPROPERTIES (
"druid.segment.granularity" = "HOUR",
"druid.query.granularity" = "MINUTE",
"kafka.bootstrap.servers" = "localhost:9092",
"kafka.topic" = "test-topic",
"druid.kafka.ingest.useEarliestOffset" = "true"
);
{code}
Design - This can be done via a DruidKafkaStreamingStorageHandler that extends
existing DruidStorageHandler and adds the additional functionality for
Streaming.
Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster +
Single Node Kafka Broker. The broker can be populated with a test topic that
has some predefined data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
