Kevin, In the forthcoming 0.9 release, the community is adding a new ingest/export <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767#KIP-26-AddCopycatconnectorframeworkfordataimport/export-HDFSExport> feature to help with importing and exporting data to and from Kafka. There is a Kafka-HDFS connector under active development. If you are interested in knowing more about it, I'd consider pinging the confluent-platf...@googlegroups.com mailing list.
Thanks, Neha On Fri, Nov 6, 2015 at 5:52 AM, Damian Guy <damian....@gmail.com> wrote: > Hi, Assuming you are using Avro then you might want to take a look at: > https://github.com/confluentinc/camus > > If not you can still adopt the pattern used to get data into HDFS. We use > something similar at my place of employment. > > Thanks, > Damian > > On 6 November 2015 at 08:46, Woodbine, Kevin <kevin.woodbi...@centrica.com > > > wrote: > > > Dear Sirs, > > > > I'm working on a development project for my company that is looking at > the > > possibility of live streaming transaction data from our Oracle databases > > into Hadoop HDFS. Doing some research I've been looking into leveraging > > the power of Kafka to form part of the solution and need some a pointer > > which I hope you can help me with. > > > > We use Goldengate to stream data into a Kafka topic and can see this data > > when running to command: > > > > bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test > > --from-beginning > > > > My question is how would I get Kafka to write this data from the topic > > into a HDFS directory? > > > > Any help you can provide will be gratefully received. > > > > Many thanks > > > > Regards > > > > Kevin Woodbine > > * (CUCI) 01784 736039 > > * 07990 563 530 > > kevin.woodbi...@centrica.com<mailto:kevin.woodbi...@centrica.com> > > > > > > _____________________________________________________________________ > > The information contained in or attached to this email is intended only > > for the use of the individual or entity to which it is addressed. If you > > are not the intended recipient, or a person responsible for delivering it > > to the intended recipient, you are not authorised to and must not > disclose, > > copy, distribute, or retain this message or any part of it. It may > contain > > information which is confidential and/or covered by legal professional or > > other privilege (or other rules or laws with similar effect in > > jurisdictions outside England and Wales). > > > > The views expressed in this email are not necessarily the views of > > Centrica plc, and the company, its directors, officers or employees make > no > > representation or accept any liability for its accuracy or completeness > > unless expressly stated to the contrary. > > > > PH Jones is a trading name of British Gas Social Housing Limited. > British > > Gas Social Housing Limited (company no: 01026007), British Gas Trading > > Limited (company no: 03078711), British Gas Services Limited (company no: > > 3141243), British Gas Insurance Limited (company no: 06608316), British > Gas > > New Heating Limited (company no: 06723244), British Gas Services > > (Commercial) Limited (company no: 07385984) and Centrica Energy (Trading) > > Limited (company no: 02877397) are all wholly owned subsidiaries of > > Centrica plc (company no: 3033654). Each company is registered in England > > and Wales with a registered office at Millstream, Maidenhead Road, > Windsor, > > Berkshire SL4 5GD. > > > > British Gas Insurance Limited is authorised by the Prudential Regulation > > Authority and regulated by the Financial Conduct Authority and the > > Prudential Regulation Authority. British Gas Services Limited and > Centrica > > Energy (Trading) Limited are authorised and regulated by the Financial > > Conduct Authority. British Gas Trading Limited is an appointed > > representative of British Gas Services Limited which is authorised and > > regulated by the Financial Conduct Authority. > -- Thanks, Neha
