hi I am unable to talk with cassandra using flume-ng sink. Can you tell me what is the configuration file to give to flume agent.
On Thu, Jan 31, 2013 at 4:26 PM, Sri Ramya <[email protected]> wrote: > thanks for ur support, > i will get back to u for farther help > > > On Thu, Jan 31, 2013 at 12:40 PM, Ralph Goers > <[email protected]>wrote: > >> Running Flume-ng on one host (or cluster of hosts) and Cassandra on its >> own Cluster of hosts is correct. Just as all the other Flume Sinks, the >> Flume Cassandra Sink has to run in Rlume. The Flume Sink writes to >> Cassandra using one of the supported protocols. I believe we are using >> Hector but you could use Astyanax, Thrift or I suppose even CQL. You could >> start with https://github.com/btoddb/flume-ng-cassandra-sink and tailor >> it to meet your needs. >> >> Ralph >> >> >> On Jan 30, 2013, at 8:35 PM, Sri Ramya wrote: >> >> >> I am running flume-ng on one host and Cassandra on another host. How i >> have to connect it. Which host i have to install flume-casandra sink? Do i >> need to run flume-ng on the host which is having Cassandra? >> >> Please reply me... >> >> >> On Wed, Jan 30, 2013 at 12:54 AM, Ralph Goers <[email protected] >> > wrote: >> >>> We ended up writing our own custom sink for Cassandra so that we could >>> customize the fields, write to Cassandra indexes and multiple column >>> families. We primarily use Cassandra to store log records and we insure >>> they can't be duplicated by generating a guid in the Appender that writes >>> to Flume (via Avro). The guid is then used as the row key. >>> >>> What do you mean by separation? >>> >>> We use Log4j 2's Flume Appender with the RFC5424Layout and >>> StructuredDataMessages (see EventLogger). All the individual items in the >>> StructuredDataMessage and in the ThreadContextMap (i.e. MDC) are added to >>> the FlumeEvent data map. The Cassandra Sink then writes these to >>> individual columns in Cassandra. The Cassandra Sink also writes these >>> items to the appropriate indexes so that we can perform whatever queries >>> that perform well. The ipAddress is definitely one of the items we are >>> interested in and capture in our indexes. >>> >>> Ralph >>> >>> >>> >>> On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote: >>> >>> hi Ralph >>> >>> I am able to integrate flume with Cassandra. But the problem is ' if i >>> send a single event to Cassandra to store, it is storing multiple row in >>> Cassnadra. Thats why i thought of shifting to logsandra. But its very >>> difficult to to use that logsandra plug in. Any way by your suggestionsNow >>> i am staring work with Flume-ng. >>> >>> Make me clear in one thing, Is there any separation of storing data in >>> Cassandra based on flume-ng agent(IP address)??? >>> >>> Thank you >>> >>> >>> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers < >>> [email protected]> wrote: >>> >>>> This is absolutely NOT correct. We are using Flume NG to log to >>>> Cassandra. We fought with Flume OG for months trying to get it to work >>>> but it was never stable. The architecture of Flume OG had serious >>>> problems, which is why Flume NG was created. >>>> >>>> Ralph >>>> >>>> >>>> >>>> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote: >>>> >>>> Hi , >>>> Can you give details like your input format and all ... >>>> I think going with flume 0.9.4 for cassandra is better. >>>> >>>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <[email protected]>wrote: >>>> >>>>> What do you mean by "collector"? >>>>> >>>>> >>>>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <[email protected]>wrote: >>>>> >>>>>> Thank you very much. But I need a collector in my application, >>>>>> flume-ng does not have any collector. Thats why i started with Cloudeera >>>>>> flum. Now i am getting a problem with integrating flume collector with >>>>>> cassandra. If any body tried it before please help me. >>>>>> thank in advance. >>>>>> >>>>>> >>>>>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <[email protected]>wrote: >>>>>> >>>>>>> Hi Sri, >>>>>>> Cloudera originally created Flume, then contributed it to the Apache >>>>>>> Software Foundation (ASF), and continues to invest heavily into it under >>>>>>> the auspices of the ASF. The current generation of Flume is called Flume >>>>>>> NG. I encourage you to use the latest "NG" generation of Flume (version >>>>>>> 1.x) over the so-called "OG" generation (version 0.9.x). >>>>>>> >>>>>>> Regards, >>>>>>> Mike >>>>>>> >>>>>>> >>>>>>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <[email protected]>wrote: >>>>>>> >>>>>>>> Thank you very much. >>>>>>>> Can u tell me whats the difference between cloudera flume and >>>>>>>> flume-ng . And which is the better option to use. >>>>>>>> Thank you. >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> You can use develop your own customized sink which can do >>>>>>>>> pre-processing in flume and then send the events to cassandra >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Som >>>>>>>>> >>>>>>>>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <[email protected]>wrote: >>>>>>>>> >>>>>>>>>> You may also want to check out >>>>>>>>>> >>>>>>>>>> https://github.com/btoddb/flume-ng-cassandra-sink >>>>>>>>>> >>>>>>>>>> Roy >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: rektide [mailto:[email protected]] >>>>>>>>>> Sent: Wednesday, January 23, 2013 10:24 AM >>>>>>>>>> To: [email protected] >>>>>>>>>> Subject: Re: flume-cassandra >>>>>>>>>> >>>>>>>>>> Have you tried https://github.com/thobbs/logsandra#configuration? >>>>>>>>>> How did that go? >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote: >>>>>>>>>> > hi >>>>>>>>>> > >>>>>>>>>> > How to intigrate flume with cassandra by using logsandra >>>>>>>>>> plugin. >>>>>>>>>> > >>>>>>>>>> > thank you. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >> >> >
