Thanks guys.
At this point it is clear that it is an operating system issue. Since it "does" work on windows, it is too bad that linux needs some tweaking to make it work. The windows test was done with a default install. As for Centos: We did in fact put the ulimit command in the activemq script and that did not work: #!/bin/bash ulimit -u unlimited ulimit -n 200000 ulimit -s unlimited ulimit -aH > /limit.txt export SUNJMX="-Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote=true" export JDK_HOME=/usr/java/jdk1.6.0_06 export JAVA_HOME=/usr/java/jdk1.6.0_06 /opt/activemq/apache-activemq-5.2.0/bin/activemq-admin start & Then went by this link for a new install http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html At this point ulimit -a will give core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 2112 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 2112 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [r...@tar ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 2112 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 2112 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited If there is a linux distro that would work straight out of the box it would be great. Arjen van der Meijden wrote: > > I saw you mention limits.conf earlier, when I tried that, it didn't work > for me. But it may work better in Centos/Redhat. The easiest way to > check if it did work is to execute 'ulimit -n' as the user that is > starting activemq. The default 1024 will not be enough to hold your 1000 > threads/sockets. > > We use a recent Debian Linux. For me, the only way to increase the > fd-limit beyond 1024 was to become root, execute 'ulimit -n 102400' and > then start activemq with something like "su activemq -c > '/some/path/bin/activemq'". > > Best regards, > > Arjen > > On 18-5-2009 7:17 Rob Davies wrote: >> Are you sure you upped the max number of fds allowable per process ? >> >> >> On 18 May 2009, at 01:23, DataMover wrote: >> >>> >>> >>> After a week of fighting with it, no matter what was done, we could >>> not get >>> it to work. >>> We created a simple test, creating 1000 producers sending one message to >>> 1000 queues. >>> Each producer in its own thread. >>> This would fail to get connections intermittently. >>> Eventually we decided it was an operating system issue. >>> >>> I tested it on windows with no modifications and there is NO problem. >>> >>> Just wondering what linux version you were using? >>> We use Centos 5.3 . >>> >>> What is the best linux distro to use with activemq? >>> Also, are there ports we should not use for transport? >>> >>> >>> >>> >>> Arjen van der Meijden wrote: >>>> >>>> Well, that is really mostly the default config with some small >>>> differences. >>>> >>>> - I completely commented out the "destinationPolicy"-tag. This also >>>> disables the per-queue/topic size limits. >>>> - I upped the memoryUsage to 200 mb, the storeUsage to 1 gb and the >>>> tempUsage to 1 gb. >>>> - I changed the connector uri's (for stomp and openwire) to contain >>>> "?transport.closeAsync=false". >>>> >>>> These settings aren't really well thought through and only aimed at our >>>> very high connect/send/disconnect rate, they're just changes that >>>> should >>>> disable or enlarge some of the limitations I was running in to. >>>> >>>> And as you could see from the issue-report, I used a different >>>> JAVA_OPTS >>>> to allow for some larger heap and such. >>>> >>>> Best regards, >>>> >>>> Arjen >>>> >>>> On 11-5-2009 9:29, DataMover wrote: >>>>> >>>>> I looked at that issue url you gave and wow, had a lot of great info. >>>>> >>>>> Any chance one could get a copy of the configuration xml file you >>>>> created >>>>> that solved the issue for you. >>>>> Just to get some ideas. >>>>> >>>>> I had upped the memory limits via the etc security limit file and >>>>> that at >>>>> least seemed to increase the load and slow the system down. Have not >>>>> tried >>>>> it again after that. >>>>> >>>>> As far as upping the queue sizes, is there a limit? >>>>> Are there best practices anywhere? >>>>> >>>>> >>>>> >>>>> Arjen van der Meijden wrote: >>>>>> There may be at one or more of these three issues that I ran into: >>>>>> >>>>>> - You actually have a too low setting for the open files. Try >>>>>> increasing >>>>>> it (see man ulimit etc, be careful that normally only root can >>>>>> increase >>>>>> it beyond 1024, but other programs, including su do inherit it). >>>>>> >>>>>> - You're opening and closing connections too fast, this is what we >>>>>> had: >>>>>> http://issues.apache.org/activemq/browse/AMQ-1739 >>>>>> Adding the "?transport.closeAsync=false"-parameter to the url >>>>>> helped us >>>>>> here. >>>>>> >>>>>> - You're queues may be getting larger than the limits. Especially the >>>>>> 5mb per queue limit in the default configuration is easy to hit. >>>>>> Once I >>>>>> raised the global limits and removed the per-queue/topic limits it >>>>>> has >>>>>> worked stable for several months in a row (since feb 19 our single >>>>>> broker has queued and dequeued over 300M tiny messages). >>>>>> >>>>>> 30 and 250 producers isn't that many, so unless they're maxing out >>>>>> your >>>>>> broker system on some other resource than file pointers, my guess >>>>>> is the >>>>>> single machine should be able to handle them. >>>>>> >>>>>> Best regards, >>>>>> >>>>>> Arjen >>>>>> >>>>>> On 10-5-2009 22:03 DataMover wrote: >>>>>>> I have seems several posts on this but I have not been able to solve >>>>>>> our >>>>>>> situation. >>>>>>> >>>>>>> We have 30 clients (producers) working with one activemq server. >>>>>>> All worked amazingly well. >>>>>>> >>>>>>> Then we tried a test with around 250 clients. >>>>>>> >>>>>>> That would get many transport errors. >>>>>>> Increasing the file limits on the os caused the system to come to a >>>>>>> crawl >>>>>>> with no benefit. >>>>>>> >>>>>>> I am assuming the problem can be solved with multiple brokers being >>>>>>> run. >>>>>>> One question is do they have to be on different machines, or can we >>>>>>> have >>>>>>> multiple activemqs running on the same server, each listening on a >>>>>>> different >>>>>>> ip? >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/too-many-open-files-tp23473539p23589365.html >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com. >>> >> >> > > -- View this message in context: http://www.nabble.com/too-many-open-files-tp23473539p23592103.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
