Out of curiosity to find the exact issue , i started flume with 20 MB memory (default) with the option as /cv/jvendor/bin/java -Xms20m -Xmx20m -Dcom.sun.management.jmxremote -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./java_pid<pid>.hprof
Again memory channel stops processing the event after a while , but heap dump is not generated here.Is the arguments i provided is corrrect ? Tried getting the jstack of running flume agent and seems the process got hung."tail -F <logfile> process is no more running. >From jstack -F <pid> , Deadlock Detection: No deadlocks found. Thread 14788: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2043 (Compiled frame) - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=98, line=1085 (Compiled frame) - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=807 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=744 (Interpreted frame) Thread 14785: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) - java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=398 (Interpreted frame) - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530 (Interpreted frame) - java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame) - sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept() @bci=1, line=52 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop() @bci=55, line=388 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=360 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=744 (Interpreted frame) On Thu, Jul 17, 2014 at 8:21 PM, SaravanaKumar TR <saran0081...@gmail.com> wrote: > Thanks Natty & Ashish. > > I have restarted flume agent with below config.Will monitor it for couple > of days whether it stops randomly. > > JAVA_OPTS="-Xms1g -Xmx1g -Dcom.sun.management.jmxremote > -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./java_pid<pid>.hprof" > > thanks for all again.Hope this will work well. > > > On 17 July 2014 12:24, SaravanaKumar TR <saran0081...@gmail.com> wrote: > >> thanks its really helpful. >> I guess the default heap dump path is /tmp ? >> >> >> On 17 July 2014 12:11, Ashish <paliwalash...@gmail.com> wrote: >> >>> Nope, a heap dump shall be generated. Please see more options at >>> http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html >>> >>> to specify path use this -XX:HeapDumpPath=./java_pid<pid>.hprof >>> >>> >>> On Thu, Jul 17, 2014 at 12:09 PM, SaravanaKumar TR < >>> saran0081...@gmail.com> wrote: >>> >>>> yes , sorry I missed to update as 1 GB. >>>> >>>> But for out of memory error ,do we get notified in flume logs? I >>>> haven't see any exception till now. >>>> >>>> >>>> On 17 July 2014 11:55, SaravanaKumar TR <saran0081...@gmail.com> wrote: >>>> >>>>> Thanks Ashish , So I wil go ahead and update the flume-env,sh file with >>>>> >>>>> JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote >>>>> -XX:-HeapDumpOnOutOfMemoryError" >>>>> >>>>> >>>>> On 17 July 2014 11:39, Ashish <paliwalash...@gmail.com> wrote: >>>>> >>>>>> Add -XX:-HeapDumpOnOutOfMemoryError parameter as well, if your >>>>>> process is OOME, would generate a Heap dump. Allocate Heap based on the >>>>>> number of events you need to keep in channel. Try with 1 GB, but >>>>>> calculate >>>>>> according the Channel size as (average event size * number of events), >>>>>> plus >>>>>> object over heads. >>>>>> >>>>>> Please note, this is just a rough calculation, actual memory usage >>>>>> would be higher. >>>>>> >>>>>> >>>>>> On Thu, Jul 17, 2014 at 11:21 AM, SaravanaKumar TR < >>>>>> saran0081...@gmail.com> wrote: >>>>>> >>>>>>> Okay thanks , So for 128 GB , I will allocate 1 GB as a heap memory >>>>>>> for flume agent. >>>>>>> >>>>>>> But I am surprised why there was no error registered for this memory >>>>>>> issues in log file (flume.log). >>>>>>> >>>>>>> Do i need to check in any other logs? >>>>>>> >>>>>>> >>>>>>> On 16 July 2014 21:55, Jonathan Natkins <na...@streamsets.com> >>>>>>> wrote: >>>>>>> >>>>>>>> That's definitely your problem. 20MB is way too low for this. >>>>>>>> Depending on the other processes you're running with your system, the >>>>>>>> amount of memory you'll need will vary, but I'd recommend at least >>>>>>>> 1GB. You >>>>>>>> should define it exactly where it's defined right now, so instead of >>>>>>>> the >>>>>>>> current command, you can run: >>>>>>>> >>>>>>>> "/cv/jvendor/bin/java -Xmx1g >>>>>>>> -Dflume.root.logger=DEBUG,LOGFILE......" >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 16, 2014 at 3:03 AM, SaravanaKumar TR < >>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>> >>>>>>>>> I guess i am using defaulk values , from running flume i could see >>>>>>>>> these lines "/cv/jvendor/bin/java -Xmx20m >>>>>>>>> -Dflume.root.logger=DEBUG,LOGFILE......" >>>>>>>>> >>>>>>>>> so i guess it takes 20 mb as agent flume memory. >>>>>>>>> My RAM is 128 GB.So please suggest how much can i assign as heap >>>>>>>>> memory and where to define it. >>>>>>>>> >>>>>>>>> >>>>>>>>> On 16 July 2014 15:05, Jonathan Natkins <na...@streamsets.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hey Saravana, >>>>>>>>>> >>>>>>>>>> I'm attempting to reproduce this, but do you happen to know what >>>>>>>>>> the Java heap size is for your Flume agent? This information leads >>>>>>>>>> me to >>>>>>>>>> believe that you don't have enough memory allocated to the agent, >>>>>>>>>> which you >>>>>>>>>> may need to do with the -Xmx parameter when you start up your agent. >>>>>>>>>> That >>>>>>>>>> aside, you can set the byteCapacity parameter on the memory channel >>>>>>>>>> to >>>>>>>>>> specify how much memory it is allowed to use. It should default to >>>>>>>>>> 80% of >>>>>>>>>> the Java heap size, but if your heap is too small, this might be a >>>>>>>>>> cause of >>>>>>>>>> errors. >>>>>>>>>> >>>>>>>>>> Does anything get written to the log when you try to pass in an >>>>>>>>>> event of this size? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Natty >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Jul 16, 2014 at 1:46 AM, SaravanaKumar TR < >>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Natty, >>>>>>>>>>> >>>>>>>>>>> While looking further , i could see memory channal stops if a >>>>>>>>>>> line comes with greater than 2 MB.Let me know which parameter helps >>>>>>>>>>> us to >>>>>>>>>>> define max event size of about 3 MB. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 16 July 2014 12:46, SaravanaKumar TR <saran0081...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I am asking point 1 , because in some cases I could see a line >>>>>>>>>>>> in logfile around 2 MB.So i need to know what mamimum event >>>>>>>>>>>> size.How to >>>>>>>>>>>> measure it? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 16 July 2014 10:18, SaravanaKumar TR <saran0081...@gmail.com >>>>>>>>>>>> > wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Natty, >>>>>>>>>>>>> >>>>>>>>>>>>> Please help me to get the answers for the below queries. >>>>>>>>>>>>> >>>>>>>>>>>>> 1,In case of exec source , (tail -F <logfile>) , is that each >>>>>>>>>>>>> line in file is considered to be a single event ? >>>>>>>>>>>>> If suppose a line is considered to be a event , what is that >>>>>>>>>>>>> maximum size of event supported by flume?I mean maximum >>>>>>>>>>>>> characters in a >>>>>>>>>>>>> line supported? >>>>>>>>>>>>> 2.When event stop processing , I am not seeing "tail -F" >>>>>>>>>>>>> command running in the background. >>>>>>>>>>>>> I have used option like "a1.sources.r1.restart = true >>>>>>>>>>>>> a1.sources.r1.logStdErr = true".. >>>>>>>>>>>>> Does these config will not send any errors to flume.log if any >>>>>>>>>>>>> issues in tail? >>>>>>>>>>>>> Will this config doesnt try to restart the "tail -F" if its >>>>>>>>>>>>> not running in the background. >>>>>>>>>>>>> >>>>>>>>>>>>> 3.Does flume supports all formats of data in logfile or it has >>>>>>>>>>>>> any predefined data formats.. >>>>>>>>>>>>> >>>>>>>>>>>>> Please help me with these to understand better.. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 16 July 2014 00:56, Jonathan Natkins <na...@streamsets.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Saravana, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Everything here looks pretty sane. Do you have a record of >>>>>>>>>>>>>> the events that came in leading up to the agent stopping >>>>>>>>>>>>>> collection? If you >>>>>>>>>>>>>> can provide the last file created by the agent, and ideally >>>>>>>>>>>>>> whatever events >>>>>>>>>>>>>> had come in, but not been written out to your HDFS sink, it >>>>>>>>>>>>>> might be >>>>>>>>>>>>>> possible for me to reproduce this issue. Would it be possible to >>>>>>>>>>>>>> get some >>>>>>>>>>>>>> sample data from you? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Natty >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Jul 15, 2014 at 10:26 AM, SaravanaKumar TR < >>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Natty , >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Just to understand , at present my settings is as >>>>>>>>>>>>>>> "flume.root.logger=INFO,LOGFILE" >>>>>>>>>>>>>>> in log4j.properties , do you want me to change it to >>>>>>>>>>>>>>> "flume.root.logger=DEBUG,LOGFILE" and restart the agent. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> But when I start agent , I am already starting with below >>>>>>>>>>>>>>> command.I guess i am using DEBUG already but not in config file >>>>>>>>>>>>>>> , while >>>>>>>>>>>>>>> starting agent. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ../bin/flume-ng agent -c /d0/flume/conf -f >>>>>>>>>>>>>>> /d0/flume/conf/flume-conf.properties -n a1 >>>>>>>>>>>>>>> -Dflume.root.logger=DEBUG,LOGFILE >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If I do some changes in config "flume-conf.properties" or >>>>>>>>>>>>>>> restart the agent , it works again and starts collecting the >>>>>>>>>>>>>>> data. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> currently all my logs move to flume.log , I dont see any >>>>>>>>>>>>>>> exception . >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cat flume.log | grep "Exception" doesnt show any. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 15 July 2014 22:24, Jonathan Natkins < >>>>>>>>>>>>>>> na...@streamsets.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Saravana, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Our best bet on figuring out what's going on here may be to >>>>>>>>>>>>>>>> turn on the debug logging. What I would recommend is stopping >>>>>>>>>>>>>>>> your agents, >>>>>>>>>>>>>>>> and modifying the log4j properties to turn on DEBUG logging >>>>>>>>>>>>>>>> for the root >>>>>>>>>>>>>>>> logger, and then restart the agents. Once the agent stops >>>>>>>>>>>>>>>> producing new >>>>>>>>>>>>>>>> events, send out the logs and I'll be happy to take a look >>>>>>>>>>>>>>>> over them. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Does the system begin working again if you restart the >>>>>>>>>>>>>>>> agents? Have you noticed any other events correlated with the >>>>>>>>>>>>>>>> agent >>>>>>>>>>>>>>>> stopping collecting events? Maybe a spike in events or >>>>>>>>>>>>>>>> something like that? >>>>>>>>>>>>>>>> And for my own peace of mind, if you run `cat >>>>>>>>>>>>>>>> /var/log/flume-ng/* | grep >>>>>>>>>>>>>>>> "Exception"`, does it bring anything back? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>> Natty >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Jul 15, 2014 at 2:55 AM, SaravanaKumar TR < >>>>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Natty, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This is my entire config file. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # Name the components on this agent >>>>>>>>>>>>>>>>> a1.sources = r1 >>>>>>>>>>>>>>>>> a1.sinks = k1 >>>>>>>>>>>>>>>>> a1.channels = c1 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # Describe/configure the source >>>>>>>>>>>>>>>>> a1.sources.r1.type = exec >>>>>>>>>>>>>>>>> a1.sources.r1.command = tail -F /data/logs/test_log >>>>>>>>>>>>>>>>> a1.sources.r1.restart = true >>>>>>>>>>>>>>>>> a1.sources.r1.logStdErr = true >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> #a1.sources.r1.batchSize = 2 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> a1.sources.r1.interceptors = i1 >>>>>>>>>>>>>>>>> a1.sources.r1.interceptors.i1.type = regex_filter >>>>>>>>>>>>>>>>> a1.sources.r1.interceptors.i1.regex = resuming normal >>>>>>>>>>>>>>>>> operations|Received|Response >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> #a1.sources.r1.interceptors = i2 >>>>>>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.type = timestamp >>>>>>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.preserveExisting = true >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # Describe the sink >>>>>>>>>>>>>>>>> a1.sinks.k1.type = hdfs >>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.path = hdfs:// >>>>>>>>>>>>>>>>> testing.sck.com:9000/running/test.sck/date=%Y-%m-%d >>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.writeFormat = Text >>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.fileType = DataStream >>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.filePrefix = events- >>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollInterval = 600 >>>>>>>>>>>>>>>>> ##need to run hive query randomly to check teh long >>>>>>>>>>>>>>>>> running process , so we need to commit events in hdfs files >>>>>>>>>>>>>>>>> regularly >>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollCount = 0 >>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.batchSize = 10 >>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollSize = 0 >>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.useLocalTimeStamp = true >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # Use a channel which buffers events in memory >>>>>>>>>>>>>>>>> a1.channels.c1.type = memory >>>>>>>>>>>>>>>>> a1.channels.c1.capacity = 10000 >>>>>>>>>>>>>>>>> a1.channels.c1.transactionCapacity = 10000 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # Bind the source and sink to the channel >>>>>>>>>>>>>>>>> a1.sources.r1.channels = c1 >>>>>>>>>>>>>>>>> a1.sinks.k1.channel = c1 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 14 July 2014 22:54, Jonathan Natkins < >>>>>>>>>>>>>>>>> na...@streamsets.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Saravana, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> What does your sink configuration look like? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> Natty >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Jul 11, 2014 at 11:05 PM, SaravanaKumar TR < >>>>>>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Assuming each line in the logfile is considered as a >>>>>>>>>>>>>>>>>>> event for flume , >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 1.Do we have any maximum size of event defined for >>>>>>>>>>>>>>>>>>> memory/file channel.like any maximum no of characters in a >>>>>>>>>>>>>>>>>>> line. >>>>>>>>>>>>>>>>>>> 2.Does flume supports all formats of data to be >>>>>>>>>>>>>>>>>>> processed as events or do we have any limitation. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I am just still trying to understanding why the flume >>>>>>>>>>>>>>>>>>> stops processing events after sometime. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Can someone please help me out here. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> saravana >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 11 July 2014 17:49, SaravanaKumar TR < >>>>>>>>>>>>>>>>>>> saran0081...@gmail.com> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi , >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I am new to flume and using Apache Flume 1.5.0. Quick >>>>>>>>>>>>>>>>>>>> setup explanation here. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Source:exec , tail –F command for a logfile. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Channel: tried with both Memory & file channel >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Sink: HDFS >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> When flume starts , processing events happens properly >>>>>>>>>>>>>>>>>>>> and its moved to hdfs without any issues. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> But after sometime flume suddenly stops sending events >>>>>>>>>>>>>>>>>>>> to HDFS. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I am not seeing any errors in logfile flume.log as >>>>>>>>>>>>>>>>>>>> well.Please let me know if I am missing any configuration >>>>>>>>>>>>>>>>>>>> here. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Below is the channel configuration defined and I left >>>>>>>>>>>>>>>>>>>> the remaining to be default values. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> a1.channels.c1.type = FILE >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> a1.channels.c1.transactionCapacity = 100000 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> a1.channels.c1.capacity = 10000000 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>> Saravana >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> thanks >>>>>> ashish >>>>>> >>>>>> Blog: http://www.ashishpaliwal.com/blog >>>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal >>>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> thanks >>> ashish >>> >>> Blog: http://www.ashishpaliwal.com/blog >>> My Photo Galleries: http://www.pbase.com/ashishpaliwal >>> >> >> >
