I am running flume-ng on one host and Cassandra on another host. How i have to connect it. Which host i have to install flume-casandra sink? Do i need to run flume-ng on the host which is having Cassandra?
Please reply me... On Wed, Jan 30, 2013 at 12:54 AM, Ralph Goers <[email protected]>wrote: > We ended up writing our own custom sink for Cassandra so that we could > customize the fields, write to Cassandra indexes and multiple column > families. We primarily use Cassandra to store log records and we insure > they can't be duplicated by generating a guid in the Appender that writes > to Flume (via Avro). The guid is then used as the row key. > > What do you mean by separation? > > We use Log4j 2's Flume Appender with the RFC5424Layout and > StructuredDataMessages (see EventLogger). All the individual items in the > StructuredDataMessage and in the ThreadContextMap (i.e. MDC) are added to > the FlumeEvent data map. The Cassandra Sink then writes these to > individual columns in Cassandra. The Cassandra Sink also writes these > items to the appropriate indexes so that we can perform whatever queries > that perform well. The ipAddress is definitely one of the items we are > interested in and capture in our indexes. > > Ralph > > > > On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote: > > hi Ralph > > I am able to integrate flume with Cassandra. But the problem is ' if i > send a single event to Cassandra to store, it is storing multiple row in > Cassnadra. Thats why i thought of shifting to logsandra. But its very > difficult to to use that logsandra plug in. Any way by your suggestionsNow > i am staring work with Flume-ng. > > Make me clear in one thing, Is there any separation of storing data in > Cassandra based on flume-ng agent(IP address)??? > > Thank you > > > On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers > <[email protected]>wrote: > >> This is absolutely NOT correct. We are using Flume NG to log to >> Cassandra. We fought with Flume OG for months trying to get it to work >> but it was never stable. The architecture of Flume OG had serious >> problems, which is why Flume NG was created. >> >> Ralph >> >> >> >> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote: >> >> Hi , >> Can you give details like your input format and all ... >> I think going with flume 0.9.4 for cassandra is better. >> >> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <[email protected]> wrote: >> >>> What do you mean by "collector"? >>> >>> >>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <[email protected]> wrote: >>> >>>> Thank you very much. But I need a collector in my application, >>>> flume-ng does not have any collector. Thats why i started with Cloudeera >>>> flum. Now i am getting a problem with integrating flume collector with >>>> cassandra. If any body tried it before please help me. >>>> thank in advance. >>>> >>>> >>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <[email protected]>wrote: >>>> >>>>> Hi Sri, >>>>> Cloudera originally created Flume, then contributed it to the Apache >>>>> Software Foundation (ASF), and continues to invest heavily into it under >>>>> the auspices of the ASF. The current generation of Flume is called Flume >>>>> NG. I encourage you to use the latest "NG" generation of Flume (version >>>>> 1.x) over the so-called "OG" generation (version 0.9.x). >>>>> >>>>> Regards, >>>>> Mike >>>>> >>>>> >>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <[email protected]>wrote: >>>>> >>>>>> Thank you very much. >>>>>> Can u tell me whats the difference between cloudera flume and >>>>>> flume-ng . And which is the better option to use. >>>>>> Thank you. >>>>>> >>>>>> >>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> You can use develop your own customized sink which can do >>>>>>> pre-processing in flume and then send the events to cassandra >>>>>>> >>>>>>> Regards, >>>>>>> Som >>>>>>> >>>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <[email protected]> wrote: >>>>>>> >>>>>>>> You may also want to check out >>>>>>>> >>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink >>>>>>>> >>>>>>>> Roy >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: rektide [mailto:[email protected]] >>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM >>>>>>>> To: [email protected] >>>>>>>> Subject: Re: flume-cassandra >>>>>>>> >>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration ? >>>>>>>> How did that go? >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote: >>>>>>>> > hi >>>>>>>> > >>>>>>>> > How to intigrate flume with cassandra by using logsandra >>>>>>>> plugin. >>>>>>>> > >>>>>>>> > thank you. >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> > >
