Hello Flume users,
I'm getting this error when starting the agent. The source is a rabbit mq
that has millions of messages, channel is file and sink is HDFS..
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "RabbitMQ Consumer #0"
Exception in thread "SinkR
Can you please drop your config in a reply or pastebin (omitting any
sensitive info)
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the i
Here it is:
# Name the components on this agent
myagent.sources = r1
myagent.sinks = k1
myagent.channels = c1
myagent.sources.r1.type = com.aweber.flume.source.rabbitmq.RabbitMQSource
myagent.sources.r1.host = xxx.yyy.com
myagent.sources.r1.port = 5671
myagent.sources.r1.username = xxx
myagent.so
Config seems sane, im not familar with the rounding values.
transactionCapacity seems a bit high, getting max 10k events from the
source at a time (i've only ever used 100, perhaps 10k is normal).
How big is each event?
Could you also paste flume-env and the output of sudo jps -v or ps
auxww|grep
The events are only about 5 to 10 kilobytes.
The flume-env.sh has only the below line, I was trying with really high
memory allocations as well,
export JAVA_OPTS="-Xms8000m -Xmx32000m -Dcom.sun.management.jmxremote"
The box has 120GB memory and nothing else is running on it. It is an AWS
EC2.
T
Does the output of JPS or PS show it was started with those parameters?
Maybe the env file is not being sourced in correctly and its starting with
a default heap?
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is
You were right, ps shows -Xmx20m instead of what I defined in env.sh.
Apparently, I was initiating from a different folder and it was not reading
my env.sh.
Fixed it now and works like a charm. Thank you very much!
Suresh.
On Wed, Mar 22, 2017 at 2:17 PM, iain wright wrote:
> Does the out
Nice, glad you were able to sort 👍
--
Iain Wright
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate
