I am asking point 1 , because in some cases I could see a line in logfile
around 2 MB.So i need to know what mamimum event size.How to measure it?
On 16 July 2014 10:18, SaravanaKumar TR wrote:
> Hi Natty,
>
> Please help me to get the answers for the below queries.
>
> 1,In case of exec sou
Hi Natty,
While looking further , i could see memory channal stops if a line comes
with greater than 2 MB.Let me know which parameter helps us to define max
event size of about 3 MB.
On 16 July 2014 12:46, SaravanaKumar TR wrote:
> I am asking point 1 , because in some cases I could see a lin
Hey Saravana,
I'm attempting to reproduce this, but do you happen to know what the Java
heap size is for your Flume agent? This information leads me to believe
that you don't have enough memory allocated to the agent, which you may
need to do with the -Xmx parameter when you start up your agent. T
I guess i am using defaulk values , from running flume i could see these
lines "/cv/jvendor/bin/java -Xmx20m
-Dflume.root.logger=DEBUG,LOGFILE.."
so i guess it takes 20 mb as agent flume memory.
My RAM is 128 GB.So please suggest how much can i assign as heap memory and
where to define it.
Hi Natty,
Thank you so much for letting me to know about jstack command.
Now the problem got solved.
In flume custom source thread execution is stopping at line
"System.out.println() or e.printStackTrace()" the statements which prints
on console.
After knowing this thing I removed all the console
That's definitely your problem. 20MB is way too low for this. Depending on
the other processes you're running with your system, the amount of memory
you'll need will vary, but I'd recommend at least 1GB. You should define it
exactly where it's defined right now, so instead of the current command,
y
Okay thanks , So for 128 GB , I will allocate 1 GB as a heap memory for
flume agent.
But I am surprised why there was no error registered for this memory issues
in log file (flume.log).
Do i need to check in any other logs?
On 16 July 2014 21:55, Jonathan Natkins wrote:
> That's definitely yo
A morphline receives a flume event at a time. What and how much is contained in
the flume event is up to you, but flume isn’t really designed to send large
events such as whole files or parts of files, it’s designed to send small
discrete events, like a log line per event, or similar.
There is
Add -XX:-HeapDumpOnOutOfMemoryError parameter as well, if your process is
OOME, would generate a Heap dump. Allocate Heap based on the number of
events you need to keep in channel. Try with 1 GB, but calculate according
the Channel size as (average event size * number of events), plus object
over h
Thanks Ashish , So I wil go ahead and update the flume-env,sh file with
JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote
-XX:-HeapDumpOnOutOfMemoryError"
On 17 July 2014 11:39, Ashish wrote:
> Add -XX:-HeapDumpOnOutOfMemoryError parameter as well, if your process is
> OOME, would ge
Use these JAVA_OPTS="-Xms1g -Xmx1g -Dcom.sun.management.jmxremote -XX:-
HeapDumpOnOutOfMemoryError"
On Thu, Jul 17, 2014 at 11:55 AM, SaravanaKumar TR
wrote:
> Thanks Ashish , So I wil go ahead and update the flume-env,sh file with
>
> JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote
yes , sorry I missed to update as 1 GB.
But for out of memory error ,do we get notified in flume logs? I haven't
see any exception till now.
On 17 July 2014 11:55, SaravanaKumar TR wrote:
> Thanks Ashish , So I wil go ahead and update the flume-env,sh file with
>
> JAVA_OPTS="-Xms100m -Xmx200m
Nope, a heap dump shall be generated. Please see more options at
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
to specify path use this -XX:HeapDumpPath=./java_pid.hprof
On Thu, Jul 17, 2014 at 12:09 PM, SaravanaKumar TR
wrote:
> yes , sorry I missed to update as
thanks its really helpful.
I guess the default heap dump path is /tmp ?
On 17 July 2014 12:11, Ashish wrote:
> Nope, a heap dump shall be generated. Please see more options at
> http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
>
> to specify path use this -XX:HeapDum
14 matches
Mail list logo
