Re: Flume sink exeption:org.apache.flume.EventDeliveryException: Failed to send events

Hi, Isn't it possible that your spark streaming application is not running when flume tries to send events to it? That might be the reason of flume not being able to connect to the port 31002. According to the Spark streaming + Flume integration guide ( http://spark.apache.org/docs/latest/streamin

Re: Flume Event Ordering

Hi Sreeram, No, Flume doesn't guarantee the order of event delivery. Flume has at least once guarantee (when configured properly), so duplicated events and out-of-order event delivery may occur. As a start I'd really recommend Hari Shreedharan's book about Flume. More info regarding to the guarant

Re: Flume-Kinesis

Hi Joaquin, According to the flume-kinesis and the AWS ARN documentations ( http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-kinesis-streams ) you are right in both cases: the endpoint is valid and the stream name is the last part of the ARN indeed. Best regard

Re: hdfs.idleTime

Hi Justin, Could you please share your config file with us? Thanks, Denes On Thu, Jan 12, 2017, 20:20 Justin Workman wrote: > sorry for cross posting to user and dev. I have recently set up a flume > configuration where we are using the regex_extractor interceptor to parse > the actual event d

Re: Deadlock between roll timer and PollingRunner threads

Hi, Yes, it seems to be a bug, I also bumped into it. It seems that the conf file poller detects change in the config file and tries to stop the components and in the same time HDFS sink tries to roll a file. It should be solved by https://issues.apache.org/jira/browse/FLUME-2973 >From your threa

Re: Ingestion to Solr is very slow

Hi, The Flume config seems OK for me, one minor thing: I'd suggest to try the memory channel, it can speed up the things a little bit. The morphline part might be a bottleneck, could you please share its config as well? Some sample input files might also be useful to be able to help with the debug

Re: morphline interceptor

Hi, I'm not too familiar with morphlines but my first guess is that the value of the outputFields should be an array. See: http://kitesdk.org/docs/1.0.0/morphlines/morphlines-reference-guide.html#split Or you might use "outputField" instead of "outputFields" Regards, Denes On Mon, Feb 6, 2017 at

Re: Alerts when Flume agent fails

Hi Suresh, Sink: - BatchCompleteCount Number of processed "complete" batches where the number of events in the batch reached the configured batch size. - BatchUnderflowCount Number of batches processed where the number of events is less than the configured maximum batch size. This can happen when

Re: Write transformed data to file_roller sink with Morphline

Hi Scott, I think it's not because of the morphlines. If the data appears properly on loggersink then it should be the same with any other sinks. I did some debugging on the file roll sink and apparently it rotates empty files as well: after the configured rollInterval it marks the current file to

Re: A puzzy problem about flume Failover Sink Processor

Hi, If you are using HDFS HA you don't have to do any special configuration on Flume side. One HDFS Sink is enough, just use the logical nameservice id in the hdfs.path property instead of the host name. See: https://community.cloudera.com/t5/Data-Ingestion-Integration/Flume-HDFS-HA/td-p/29141 R

Re: AWS SQS Flume Source

Hi Roberto, No, unfortunately the Flume codebase doesn't contain AWS SQS source. Best, Denes On Thu, Apr 6, 2017 at 5:13 PM Roberto Coluccio wrote: > Hello community, > > I was wondering if is there an "official" AWS SQS Source component to > leverage w/ Flume 1.6 or 1.7. I couldn't find any r

Re: Getting "ClassCastException: FlumeEvent cannot be cast to ..." when using File Channel

Hi Muhammad, File channel converts the events to its internal FlumeEvent class ( https://github.com/apache/flume/blob/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java) during the put operation: https://github.com/apache/flume/blob/trunk/flume-n

Re: Use case for Flume

Hi GIntas, What is the average (or expected maximum) size of the files you'd like to process? In general it is not recommended to transfer large events (i.e. >64MB if you use file channel, as this is a hard limit of the protobuf implementation). If your files fit into this limit then I'd suggest t

[ANNOUNCE] Apache Flume 1.8.0 released

The Apache Flume team is pleased to announce the release of Flume version 1.8.0. Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. This release can be downloaded from the Flume download page at: http://flume.apac

Re: pr to be merged

Hi Filippo, I'm about to merge #155 and #158 and will have a look on the other two as well. Regards, Denes On Wed, Oct 25, 2017 at 1:54 AM Filippo Balicchia wrote: > Hi all, > > I've noticed that there are a lot pull request that improve Flume > feature ready to be merged from a long time for

Re: [ANNOUNCE] Apache Flume 1.9.0 released

Hi Ferenc, Great news, thank you for the hard work you put into this release. Of course thanks to all the contributors too. Best regards, Denes On Tue, Jan 8, 2019 at 3:34 PM Ferenc Szabo wrote: > The Apache Flume team is pleased to announce the release of Flume > version 1.9.0. > > Flume is a

[ANNOUNCE] New Flume PMC member - Ferenc Szabo

Hello Flume community, On behalf of the Apache Flume PMC I am pleased to announce that Ferenc Szabo (szaboferee) has accepted our invitation to become a PMC member on the Apache Flume project. Ferenc has been regularly contributing improvements to Flume including multiple bigger features/enhanceme