No specific reason. I was familiar with Jetty and we already had a dependence
on Jetty for the metrics stuff and I think some avro IPC stuff too, so I
decided to just use the same.
Thanks,
Hari
--
Hari Shreedharan
On Monday, November 5, 2012 at 2:09 PM, Harish Mandala wrote:
> J
Flume.
Hope this helps.
Thanks,
Hari
--
Hari Shreedharan
On Monday, November 5, 2012 at 10:09 AM, Nathaniel Auvil wrote:
> One thing i do not see with this HTTPSource is any way to customize the
> response.
>
>
> On Mon, Nov 5, 2012 at 10:58 AM, Brock Noland (mailto:br.
out if you are hitting some limit on a single agent.
Thanks,
Hari
--
Hari Shreedharan
On Wednesday, November 7, 2012 at 9:37 AM, Brock Noland wrote:
> Hi,
>
> What version of NG are you running? Comment below inline.
> On Tue, Nov 6, 2012 at 8:10 PM, Cameron Gandevia (m
tiple sink runners), or make your sink multithreaded
(spawn multiple threads inside the process method and then wait for all threads
to succeed/fail), so more threads do I/O.
Hari
--
Hari Shreedharan
On Wednesday, November 7, 2012 at 10:48 AM, Nathaniel Auvil wrote:
> in addition to HDFS
conds before calling the process method
again.
Hari
--
Hari Shreedharan
On Wednesday, November 7, 2012 at 11:08 AM, Nathaniel Auvil wrote:
> it is my understanding, perhaps incorrectly, that when you start a
> transaction in a sink, the channel blocks until that transaction is
> c
Rahul,
The interceptor adds headers, not content to the body. Unless you are somehow
writing the headers out, you will not see the headers in the output. The
sequence of numbers you see are generated by the SEQ source - which is what it
does.
Hari
--
Hari Shreedharan
On Wednesday
json support.
Thanks,
Hari
--
Hari Shreedharan
On Thursday, November 8, 2012 at 10:45 AM, Bart Verwilst wrote:
> Hi,
>
> I've been spending quite a few hours trying to push avro data to Flume
> so i can store it on HDFS, this all with Python.
> It seems like somethin
No, I am talking about:
https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=bc1928bc2e23293cb20f4bc2693a3bc262f507b3
This will be in the next release which will be out soon.
Thanks,
Hari
--
Hari Shreedharan
On Thursday, November 8, 2012 at 10:57 AM, Bart Verwilst wrote:
>
the source to use
that handler. Once the data is in Flume, just plug in your own serializer
(which can take the byte array from the event and convert it into the schema
you want) and write it out.
Thanks,
Hari
--
Hari Shreedharan
On Thursday, November 8, 2012 at 1:02 PM, Bart Verwilst wrote
Hari
--
Hari Shreedharan
On Thursday, November 8, 2012 at 1:34 PM, Bart Verwilst wrote:
> Would the sink serializer from
> https://cwiki.apache.org/FLUME/flume-1x-event-serializers.html ( avro_event )
> by the right tool for the job? Probably not since i won't be able to send the
't know if anyone is using net cat in
production - you should probably test using Avro source or the new HTTP
source(for this you would need to build trunk/1.3 branch or wait for 1.3
release).
Thanks
Hari
--
Hari Shreedharan
On Thursday, November 8, 2012 at 3:05 PM, Rahul Ravindran
Dan,
I believe that most major issues have been committed. I will try to commit
others today, so Brock can roll an RC soon.
Thanks,
Hari
--
Hari Shreedharan
On Friday, November 9, 2012 at 10:30 AM, Dan Young wrote:
> One quick followup, does it look like 1.3 RC will be out this w
On behalf of the Apache Flume PMC, I am excited to welcome Patrick Wendell as
a committer on Flume! Patrick has contributed significantly to the project, by
adding new features, fixing bugs and helping users on the Flume users list.
Here is a list of jiras Patrick has worked on:
http://s.apach
You can find the details of the components and how to wire them together here:
http://flume.apache.org/FlumeUserGuide.html
Thanks,
Hari
--
Hari Shreedharan
On Tuesday, November 13, 2012 at 6:26 AM, S Ahmed wrote:
> Hello,
>
> So I downloaded the flume source, and I was hoping so
Which version of Flume are you using? It looks like the transaction was never
rolled back or committed. It is likely that the rollback method too threw some
exception, and the rollback was not successful. Also, what channel are you
using?
Thanks,
Hari
--
Hari Shreedharan
On Wednesday
with Flume, we
expect Hadoop to be available on the machines you are running Flume on. Once
you install Hadoop you should not have any dependency issues. Same is true for
HBase.
Hari
--
Hari Shreedharan
On Monday, November 19, 2012 at 2:33 PM, Mohit Anchlia wrote:
> Easiest way is
directly does not
depend on Hadoop anymore, but still needs them in the class path because we
support migration from the older format to new format.
Thanks,
Hari
--
Hari Shreedharan
On Monday, November 19, 2012 at 4:04 PM, Rahul Ravindran wrote:
> Thanks for the responses.
>
> Goo
other stuff which will not
> be used in any box except the one running the hdfs sink.
>
> Thanks,
> ~Rahul.
> From: Hari Shreedharan
> To: user@flume.apache.org; Rahul Ravindran
> Sent: Monday, November 19, 2012 4:08 PM
> Subject: Re: Hadoop jars
>
> Unfor
ately after a
checkpoint - you can verify this by making sure each data dir has only 2 files
each), since all events have been sent out - so during restart the channel will
be quite empty, with very little to replay).
Hope this helps.
Thanks,
Hari
--
Hari Shreedharan
On Friday, November 23,
pretty good.
Run git branch to see all the branches.
Cheers,
Hari
--
Hari Shreedharan
On Wednesday, November 28, 2012 at 10:26 PM, Mohit Anchlia wrote:
> When I run "git clone git://git.apache.org/flume.git
> (http://git.apache.org/flume.git)" which version am I cloning? How
additional disks on
your Flume agent machine for file channel data directories, that will also
improve performance.
Hari
--
Hari Shreedharan
On Wednesday, December 12, 2012 at 7:36 AM, Brock Noland wrote:
> Hi,
>
> Why not try increasing the batch size on the source and sink
Yep, each sink with a different prefix will work fine too. My suggestion was
just meant to avoid collision - file prefixes are good enough for that.
--
Hari Shreedharan
On Wednesday, December 12, 2012 at 1:13 PM, Bhaskar V. Karambelkar wrote:
> Hari,
> If each sink uses a differen
nk and build it locally
if you need to deploy it urgently.
Hari
--
Hari Shreedharan
On Wednesday, December 12, 2012 at 9:42 PM, Guy Peleg wrote:
> Hi,
>
> From the documentation: "Flume uses a transactional approach to guarantee the
> reliable delivery of the events"
mance too much - which is what I'd recommend you use.
Hari
--
Hari Shreedharan
On Wednesday, December 12, 2012 at 11:34 PM, Guy Peleg wrote:
> Say I have multi-hop flow, and lets say the last one stores its data in HDFS
> using the HDFS sink.
>
> In the last agent, as in
The installation instructions you have referenced here is for Flume OG, not Flume 1.x. Flume 1.x currently does not have a Windows installer or Windows daemons.Hari-- Hari Shreedharan
On Thursday, December 13, 2012 at 8:53 AM
Yep. The disk space calls require an NFS call for each write, and that slows
things down a lot.
--
Hari Shreedharan
On Tuesday, December 18, 2012 at 8:43 AM, Brock Noland wrote:
> We'd need those thread dumps to help confirm but I bet that FLUME-1609
> results in a NFS c
We recently committed https://issues.apache.org/jira/browse/FLUME-1702 to
trunk. This will be available in the next release of Flume. This should
help in the Pig case, not sure about Hive though.
Hari
On Thursday, December 27, 2012, Paul Chavez wrote:
> **
> This is kind of a generic HDFS ques
Looks like your config file contains the configuration for an agent named
"agent" and you are trying to run flume with the agent name "agent1". Run
the flume agent as agent or change the name of your agent in the config
file to "agent1" (simply replace all instances of "agent" with "agent1"
Hari
The Apache Flume team is pleased to announce the release of Flume
version 1.3.1.
Flume is a distributed, reliable, and available service for efficiently
collecting, aggregating, and moving large amounts of log data.
This release can be downloaded from the Flume download page
at:http://flume.apach
at 4:06 PM, Mohammad Tariq wrote:
> Hello sir,
>
> Mirrors are not working for me. Getting 404 when I visit the link
> specified by you. Backup sites are working though.
> Thank you.
>
> Best Regards,
> Tariq
> +91-9741563634
> https://mtariq.jux.com/
>
>
&g
Hi Juhani,
You might also want to try sending an email to legal-discuss@
Hari
--
Hari Shreedharan
On Thursday, January 3, 2013 at 8:28 PM, Juhani Connolly wrote:
> Myself and a colleague will be doing a presentation/usecase on flume at
> "Hadoop Conference Japan 2013 winter&
release yet, but in trunk - so
will be in next release).
Hari
--
Hari Shreedharan
On Wednesday, January 9, 2013 at 2:12 AM, Chhaya Vishwakarma wrote:
> The expected output I pasted is from file only which I can see in file but
> while writing to HDFS its giving some junk value and wh
The LoadBalancing capability similar to the LoadBalancingRpcClient can be
configured for multiple Avro Sinks using a LoadBalancingSinkProcessor, if you
are looking for that functionality.
Hari
--
Hari Shreedharan
On Wednesday, January 9, 2013 at 11:05 PM, Connor Woodson wrote:
> Sh
+1 - using sink groups with load balancing sink processor is the solution.
backoff is optional (only if you want failed sinks to be not tried for a while).
Hari
--
Hari Shreedharan
On Thursday, January 10, 2013 at 12:10 AM, Connor Woodson wrote:
> Forgot about sink processors; yes,
happy to review and commit it.
Thanks,
Hari
--
Hari Shreedharan
On Monday, January 14, 2013 at 9:29 AM, Andrew Otto wrote:
> Hi all,
>
> I'm an Systems Engineer at the Wikimedia Foundation, and we're investigating
> using Flume for our web request log HDFS imports.
It means that the channel can store that many events. If it is full, then the
put() calls (on the source side) will start throwing ChannelException. The put
call will block only for keep-alive number of seconds, after which it will
throw.
Hari
--
Hari Shreedharan
On Wednesday, January
nel's sizes and
see if it is filling up (there will be ChannelExceptions in the logs if it is).
Hari
--
Hari Shreedharan
On Wednesday, January 16, 2013 at 2:34 PM, Brock Noland wrote:
> Good to hear! Take five six thread dumps of it and then them our way.
>
> On Wed, Jan
poll). Without a sink processor,
each sink gets its own sink runner thread.
Hari
--
Hari Shreedharan
On Wednesday, January 16, 2013 at 3:03 PM, Andrew Otto wrote:
> Ok, thanks. Quick Q: Won't each sink consume the same data? Do I need to
> set up the load balancing sink proces
I remember RollingFileSink had some issues with reconfiguration before. I
thought it was fixed, though I am not sure. Restarting the JVM will fix it in
any case.
Thanks
Hari
--
Hari Shreedharan
On Wednesday, January 16, 2013 at 6:18 PM, Brock Noland wrote:
> FLUME-1630 (in flume 1.
I'd recommend using the FileChannel instead of the JDBC Channel. The
FileChannel would give you better performance and the same reliability
guarantees as the JDBC Channel.
To answer your question - you can insert custom headers in an agent using
Interceptors. You can write an interceptor which ins
Alan,
This is the path your event would follow:
log4jappender -> source -> interceptor -> selector (technically this is the
channel processor, selector simply selects the channels) -> channel
Hari
--
Hari Shreedharan
On Wednesday, January 23, 2013 at 12:01 AM, Al
.
Thanks,
Hari
--
Hari Shreedharan
On Wednesday, January 23, 2013 at 1:23 PM, Mike Percy wrote:
> Not sure when or how it broke, as I know of people using it in production.
> There is a way to configure it for different versions of Ganglia, like 3.0,
> 3.1. Might be worth trying both valu
I attached a patch to FLUME-1870. Can you guys give it a try and see if you are
hitting the issue now?
Hari
--
Hari Shreedharan
On Wednesday, January 23, 2013 at 1:38 PM, Hari Shreedharan wrote:
> Looks like this is happening because for certain metrics which cannot be
> con
Have you set up the appender correctly? The log4j appender class is
org.apache.flume.client.log4jappender.Log4jAppender - this is what needs to go
in where you specify the appender in the log4j.properties file.
Hari
--
Hari Shreedharan
On Friday, January 25, 2013 at 9:05 AM, yogi nerella
> Hari,
>>
>> Sorry, I havent provided the zip file correctly.
>>
>> In my log4j.properties I have is
>> "org.apache.flume.clients.log4jappender.Log4jAppender" note client is
>> plural clients.
>>
>> Please see the attached file.
>>
ialization, removing it simply made it invalid avro.
My advice would be to use the RpcClient to read the file, and send the data
such that you send the data in a valid format, by making sure one avro
"container" is in one event.
Hari
--
Hari Shreedharan
On Wednesday, Februar
Here you are: http://flume.apache.org/FlumeDeveloperGuide.html#client
Hari
--
Hari Shreedharan
On Wednesday, February 6, 2013 at 10:20 AM, Alan Miller wrote:
> Thanks Hari,
>
> Are there any links to examples of how to use the RpcClient?
>
> Alan
>
> F
To unsubscribe, please send an email to user-unsubscr...@flume.apache.org
Hari
On Sat, Feb 9, 2013 at 3:17 PM, Parag Hukeri wrote:
> Unsubscribe
be lost. We recommend using a
source which actually does report failure, like Avro/Thrift (available on
trunk, not in any release yet) or HTTP. This will allow you to retry if Flume
reports failure.
Hari
--
Hari Shreedharan
On Tuesday, February 12, 2013 at 11:24 AM, matt.elli
Are you using secure hbase? Flume has not yet made a release with secure HBase
support. You would need to checkout trunk to use secure hbase with Flume.
--
Hari Shreedharan
On Friday, February 15, 2013 at 5:06 AM, Brock Noland wrote:
> Hi,
>
> Check to make sure you spelled your
We recently committed thrift RPc support. If you are willing to try out
some new code, you can checkout trunk and try it out
Hari
On Sunday, February 17, 2013, 周梦想 wrote:
> thank you smth.
> but there isn't windows version of flume-ng. we have to gather logs from
> windows servers.
>
> :)
> Andy
se it by checking out
trunk - though right now, there is no real documentation yet.
Thanks,
Hari
--
Hari Shreedharan
On Sunday, February 17, 2013 at 10:44 PM, Alexander Alten-Lorenz wrote:
> This is answered in the thread I sent, it doesn't exist a valuable NIO
> framework.
lizer).
Hari
--
Hari Shreedharan
On Monday, February 18, 2013 at 7:09 PM, 周梦想 wrote:
> hello,
> I put some data to hdfs via flume 1.3.1,but it changed!
>
> source data:
> [zhouhh@Hadoop47 ~]$ echo "<13>Mon Feb 18 18:25:26 2013 hello world zhh " |
> nc -v hado
See comment below.
--
Hari Shreedharan
On Monday, February 18, 2013 at 7:43 PM, 周梦想 wrote:
> hello,
> I change the conf file like this:
> [zhouhh@Hadoop48 flume1.3.1]$ cat conf/testhdfs.conf
> syslog-agent.sources = Syslog
> syslog-agent.channels = MemoryChannel-1
>
he.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable▒뿱▒5▒_▒rU▒<▒\▒)Mon
> Feb 18 18:25:26 2013 hello world zhh
>
> 2013/2/19 Hari Shreedharan
>
> See comment below.
>
> --
> Hari Shreedharan
>
> On Monday, February 18, 2013 at 7:43 PM, 周梦想 wrote:
&g
or the IPC sources).
Thanks
Hari
--
Hari Shreedharan
On Tuesday, February 19, 2013 at 4:05 PM, Noel Duffy wrote:
> If I disable the agent.sinks line, both my sinks are disabled and nothing
> gets written to HDFS. The status page no longer shows me any sinks.
>
> From: Yogi Ne
No, it does not mean that. To talk to different HDFS clusters you must specify
the hdfs.path as hdfs://namenode:port/. You don't need to specify the
bind etc.
Hope this helps.
Hari
--
Hari Shreedharan
On Tuesday, February 19, 2013 at 8:18 PM, Noel Duffy wrote:
> Hari Shr
Also, as Jeff said, sink-2 has a higher priority (the absolute value of the
priority being higher, that sink is picked up).
--
Hari Shreedharan
On Tuesday, February 19, 2013 at 8:37 PM, Hari Shreedharan wrote:
> No, it does not mean that. To talk to different HDFS clusters you m
Rahul,
Those messages actually just suggest that your channel is replaying. The
channel will complete the replay and the agent will start the sinks once the
channel is ready. It might take a few minutes based on how many events you have
in the channel.
Hari
--
Hari Shreedharan
On
Can you send your full logs? I suspect the channel did a full replay because it
was restarted during a restart. (If it did, the logs would show a
BadCheckpointException).
Hari
--
Hari Shreedharan
On Monday, February 25, 2013 at 1:20 PM, Rahul Ravindran wrote:
> Thanks Hari. I had wai
This is because the memory channel has a default transaction capacity of 100.
Increasing it (or keeping sinks's batchSize < transaction capacity of the
channel will fix the issue). See
http://flume.apache.org/FlumeUserGuide.html#memory-channel for more details.
Hari
--
Hari Shr
certain criteria - and making sure you
have a non-nullable last column is a good check - so if the last column is null
(select * from table where last_row!=null), the row can be ignored - since it
may not have been written out correctly.
Hope this helps.
Hari
--
Hari Shreedharan
On
Can also send the flume agent logs? Did you check the contents of the files?
--
Hari Shreedharan
On Thursday, February 28, 2013 at 2:43 PM, Roshan Naik wrote:
> would you be able to you verify if the same problem can be reproduced by
> using the memory channel instead in a test
You need to increase the transactionCapacity of the channel to at least the
batchSize of the HDFS sink. In your case, it is 1000 for the channel
transaction capacity and your hdfs batch size is 1.
--
Hari Shreedharan
On Thursday, February 28, 2013 at 4:00 PM, Paul Chavez wrote:
>
file at a time. (note
that each HDFS Sink should write to a different directory).
--
Hari Shreedharan
On Thursday, February 28, 2013 at 7:10 PM, 周梦想 wrote:
> Hello,
>
> I wrote a test java file to put local files to hdfs, it put 1.1GB files to
> hdfs spent about 35 seconds.
>From the logs, the logs look OK and the channel is working fine. It seems to
>have been replaying - that is pretty much it.
--
Hari Shreedharan
On Monday, February 25, 2013 at 4:28 PM, Rahul Ravindran wrote:
> I have attached the zipped log file at
> https://issues.apache.org
HDFS and
transaction closed. If pupper does not rewrite the config file, do you see this
issue?
--
Hari Shreedharan
On Monday, March 4, 2013 at 3:06 PM, Sagar Mehta wrote:
> I think we found the issue, not sure if this is the root cause but looks
> highly correlated.
>
>
HI Vivek,
I cannot be sure of why that is happening. Channel.take() gets called even
if there are no events in the channel. If the take() method returns null,
then there are no events in the channel. You can use the Status.BACKOFF
return value to tell the sink poller to not retry immediately. But
on the mailing list). Since
this is pluggable, you can use any logic to figure out when an event is
complete.
Hari
--
Hari Shreedharan
On Sunday, March 10, 2013 at 11:45 PM, Ravi Kiran wrote:
> Hi Hari ,
>We are planning to work on Flume NG to stream all our application l
capacity.
Hari
--
Hari Shreedharan
On Tuesday, March 12, 2013 at 11:03 AM, Vikram Kulkarni wrote:
>
> I have my custom Source and Sink that I have hooked with a memory channel but
> I am noticing that it is not very consistent. Even after sending many events
> to th
That is a known issue. I believe there was some feedback on the jira.
Unfortunately, it has not been incorporated. I will get to it at some point and
fix it :-)
--
Hari Shreedharan
On Tuesday, March 12, 2013 at 8:33 PM, larryzhang wrote:
> Hi,
>I did a simple test about exec
Bigtop does have start and stop scripts for Flume, depending on which
distro of Linux you are using. You can take a look at using Bigtop:
bigtop.apache.org.
Hari
On Sun, Mar 17, 2013 at 10:09 PM, Nitin Pawar wrote:
> if you want to stop the the agent gracefully then just send a SIGINT or
> SIG
). Eventually
the channel will clean up the files which have all events taken (though it will
keep at least 2 files per data directory, just to be safe).
--
Hari Shreedharan
On Tuesday, March 19, 2013 at 10:32 AM, Alexander Alten-Lorenz wrote:
> Hey,
>
> what says debug? Do you c
6 will get deleted, but you will
still will see log-7 and log-8. So what you are seeing is not unexpected.
Hari
--
Hari Shreedharan
On Tuesday, March 19, 2013 at 6:30 PM, Zhiwen Sun wrote:
> Thanks all for your reply.
>
> @Kenison
> I stop my tail -F | nc program and there is no new
> Is that means I can reduce the capacity of file channel to cut down max
> disk space used by file channel?
>
>
> Zhiwen Sun
>
>
>
> On Wed, Mar 20, 2013 at 3:23 PM, Hari Shreedharan <
> hshreedha...@cloudera.com> wrote:
>
>> Hi,
>>
&g
in the format you want.
Hari
--
Hari Shreedharan
On Friday, March 22, 2013 at 4:58 AM, Taimoor Aslam wrote:
> I used static interceptor and avro sink in client configuration to add custom
> information in headers , and in server configuration i used avro source and
> hbase sin
Nikhil,
Flume's HDFS Sink will by default write to HDFS as Sequence Files. If you want
it to write as text or avro, you must use DataStream. Please see the Flume User
Guide.
Thanks,
Hari
--
Hari Shreedharan
On Tuesday, March 26, 2013 at 11:01 PM, Nikhil Shirke wrote:
> Hello,
ew it :-)
Thanks
Hari
--
Hari Shreedharan
On Friday, March 29, 2013 at 9:43 AM, Bertrand Dechoux wrote:
> It seems quite easier than dealing with commons configuration.
>
> I might take a shot at it during the week-end, if I find the time.
>
> About the customisation of t
FLUME-1491 looks like, but if it
is possible to integrate it into this, it might make sense.
Thanks
Hari
--
Hari Shreedharan
On Tuesday, April 2, 2013 at 12:23 PM, Bertrand Dechoux wrote:
> I did a short proof of concept for the YAML part here
> https://github.com/BertrandDechoux/fl
This is due to a dependency on some hadoop classes. You should drop in the
hadoop-common jar file (download the hadoop-common binary and drop this jar
in) into the flume classpath. Though this is no longer used directly, there
is code which is meant to upgrade legacy format of the file channel - so
Yes. We will need to remove the format upgrade code and the old format code
from the file channel as well. I think it should be possible to do it, but I
suspect it is just that no one had the time to look at this as yet.
--
Hari Shreedharan
On Wednesday, April 10, 2013 at 1:38 PM, Roshan
Ah, yes, I remember that patch. It basically copied the Writables
interfaces/classes I think. Yeah, copying those would be a nice workaround (I
think those dont have any implementation anyway).
Hari
--
Hari Shreedharan
On Wednesday, April 10, 2013 at 1:44 PM, Brock Noland wrote:
>
will be deleted
at the next checkpoint. Unless a log-3 is created, both log-1 and log-2 will
not be deleted.
Hari
--
Hari Shreedharan
On Thursday, April 11, 2013 at 12:13 PM, Madhu Gmail wrote:
> Hello,
>
> I have not heard from anyone. so just want make sure I have explained the
Looks like you are hitting Avro IPC timeouts - you should probably increase it,
especially if you are talking over WAN.
--
Hari Shreedharan
On Tuesday, April 16, 2013 at 11:38 AM, Chris Neal wrote:
> I'm seeing the same thing :)
>
> Mine is all on a local LAN though, so the fa
massive number of
connections causing too many threads.
--
Hari Shreedharan
On Tuesday, April 16, 2013 at 12:07 PM, Chris Neal wrote:
> Thanks Hari.
>
> I increased both the connect and request timeouts to 4ms, and I'm testing
> that now. I am talking on a LAN though,
.
Thanks
Hari
--
Hari Shreedharan
On Tuesday, April 30, 2013 at 2:16 PM, Eran Kutner wrote:
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:34)
Did you have an issue with the checkpoint that the entire 6G of data was
replayed (look for BadCheckpointException in the logs to figure out if the
channel was stopped in middle of a checkpoint)?
With the next version of Flume, you should be able to recover even if the
channel stopped while the ch
u are able to see the
metrics fine. Even with these errors in the logs, the components should work
fine. In the logs you can see that the sink has started. These exceptions do
not cause any data loss or components not to function.
Hari
--
Hari Shreedharan
On Wednesday, May 8, 2013 at 12:
What version of Hadoop are you using? Looks like you are getting hit by
https://issues.apache.org/jira/browse/HADOOP-6762.
Hari
--
Hari Shreedharan
On Monday, May 13, 2013 at 6:50 PM, Matt Wise wrote:
> So we've just had this happen twice to two different flume machines... we
Looks like CDH4.2.1 does have that patch:
http://archive.cloudera.com/cdh4/cdh/4/hadoop-2.0.0-cdh4.2.1.CHANGES.txt (but
it was not in CDH4.1.2)
Hari
--
Hari Shreedharan
On Monday, May 13, 2013 at 7:23 PM, Rahul Ravindran wrote:
> We are using cdh 4.1.2 - Hadoop version 2.0.0. Looks l
The patch also made it to Hadoop 2.0.3.
On Monday, May 13, 2013, Hari Shreedharan wrote:
> Looks like CDH4.2.1 does have that patch:
> http://archive.cloudera.com/cdh4/cdh/4/hadoop-2.0.0-cdh4.2.1.CHANGES.txt (but
> it was not in CDH4.1.2)
>
>
> Hari
>
> --
> Hari Shr
You need to use org.apache.flume.interceptor.StaticInterceptor$Builder as the
FQCN.
Cheers,
Hari
On Thursday, June 6, 2013 at 12:01 PM, Allan Feid wrote:
> Hello Everyone,
>
> I've been trying to write my own custom interceptor, but ran into a problem
> when using a FQCN for an intercepto
You should be able to use %{headerName} to replace the escape sequence in the
HDFS sink. For example if you have a header "topic" with value "flume", an HDFS
path that looks like:
hdfs://nameservice/user/flume/%{topic} will drop the event into
user/flume/flume.
Cheers,
Hari
On Monday, Jun
Currently escaping is not supported by File Roll Sink, but I think there is a
jira to enable it - perhaps you want to submit a patch?
Cheers,
Hari
On Friday, June 14, 2013 at 12:04 PM, Lenin Raj wrote:
> Hello,
>
> I tried to use %Y in my sink.directory in the file_roll sink but it resulte
I will see how it is done in HDFS sink.
>
>
> Thanks,
> Lenin
>
>
> On Sat, Jun 15, 2013 at 12:40 AM, Hari Shreedharan (mailto:hshreedha...@cloudera.com)> wrote:
> > Currently escaping is not supported by File Roll Sink, but I think there
> > is a jir
Technically, even the HDFS sink uses only one thread to write to HDFS. The
Async Hbase Sink writes using multiple threads (though they are hidden away
from the sink itself - it is in the underlying API).
Cheers,
Hari
On Wednesday, June 19, 2013 at 11:17 AM, Roshan Naik wrote:
> take a look
Does not look like you are using the http source at all. Your source type needs
to be HTTP
Cheers,
Hari
On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote:
> Hi all,
> I am new to flume and all that logging stuff and probably many things are
> unclear to me despite I read the docs.
I filed https://issues.apache.org/jira/browse/FLUME-2109 to track this. Feel
free to submit a patch!
Cheers,
Hari
On Monday, June 24, 2013 at 8:05 AM, Ashish wrote:
> So am I :) I am just into my 2nd week of reading about Flume. If I were in
> your place, I would have added Jetty SSL example
Hi all,
I am sorry if this is a bit late, but I''d like to invite you all to the Flume
meetup at Hadoop Summit in San Jose, CA. Please see
http://www.meetup.com/Flume-User-Meetup/events/125485602/ for details.
Hari
Looks like the file may have been corrupted. Can you verify if you are out
of disk space or can see something that might have caused the data to be
corrupted?
Hari
On Thu, Jun 27, 2013 at 6:41 AM, Rahul Ravindran wrote:
> Hi,
> We are using CDH flume 1.3 (which ships with 4.2.1). We see this
1 - 100 of 448 matches
Mail list logo
