@Edward can you create upstream Oozie JIRA for that HivePartition using stuff? Maybe providing your bits as patch ;)
@Serega @Artem do you have ideas where Oozie HDFS path handling is way inflexible? Thanks, Andras On Tue, Dec 19, 2017 at 8:42 PM, Edward Capriolo <edlinuxg...@gmail.com> wrote: > The point is valid. Oozie has a good amount of specific HDFS sauce baked > in, and not in a good way. I was contemplating DataFileDependencies that > use HivePartition information. It was not easy to do. > > On Tue, Dec 19, 2017 at 10:38 AM, Serega Sheypak <serega.shey...@gmail.com > > > wrote: > > > Don't push NiFi :) Please don't :) > > > > 2017-12-19 16:16 GMT+01:00 Artem Ervits <artemerv...@gmail.com>: > > > > > you're probably better off using Apache Nifi at that point, Kafka Nifi > is > > > better integrated today than Oozie and Kafka. Probably wrong mailing > list > > > to push Nifi but it's a common use case in that realm. > > > > > > Thanks > > > > > > On Mon, Dec 18, 2017 at 7:04 PM, Serega Sheypak < > > serega.shey...@gmail.com> > > > wrote: > > > > > > > Hi, I know default coordinator functionality, but it's limited > (almost) > > > to > > > > HDFS. > > > > Kafka (any other pub/sub or queue like rabbitMQ, whatever-MQ) makes > > > > integration contract much more flexible. > > > > I could have traceability, debuggability, transparency, throttling, > > > > concurrency of oozie and push coordinator job on demand. And I'm not > > > > limited to strict HDFS path pattern. > > > > > > > > > > > > 2017-12-18 18:14 GMT+01:00 Andras Piros <andras.pi...@cloudera.com>: > > > > > > > > > Hi Serega, > > > > > > > > > > not to my knowledge. Would be interested on your use case, though. > > > > > > > > > > Would start w/ *Coordinator Input Events / Datasets > > > > > <https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec. > > > > > html#a5._Dataset>* > > > > > . > > > > > > > > > > Andras > > > > > > > > > > On Sat, Dec 16, 2017 at 2:54 PM, Serega Sheypak < > > > > serega.shey...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi, did anyone try to integrate oozie coordinator with kafka? > > > > > > use case: > > > > > > > > > > > > System publishes message to kafka topic (sample message) > > > > > > - cluster: hdfs://prod-cluster > > > > > > - path: /my/input/data > > > > > > - format: avro > > > > > > > > > > > > Oozie coordinator listens to kafka topic, consumes message and > > starts > > > > > > workflow. > > > > > > > > > > > > > > > > > > > > >
