ould the client be specifying "?brokerName=" where
is the actual broker name?
Jim
On Mon, Jan 16, 2017 at 11:23 AM James A. Robinson
wrote:
> Several times now we've witnessed a cluster spinning off hundreds of
> threads and spewing warnings like:
>
> 2017
Several times now we've witnessed a cluster spinning off hundreds of
threads and spewing warnings like:
2017-01-16 11:10:35,376 [p209148636-3304] WARN BrokerRegistry - Broker
localhost not started so using amq-prod-1 instead
A thread dump shows hundreds of threads along these lines:
"ActiveMQ VM
I don't know where you got that link from, it looks like an example from
documentation, where you're expected to fill in some details (the path to
the version you want). Look at the links for Windows or Linux on this page:
http://activemq.apache.org/activemq-591-release.html
Those link to a scri
A packet capture on the machines indicates a perfectly valid
sequence of exchanges between the JMS publisher and the
consumer with the broker when the test is succeeding.
I'll run another test when it is in a failure mode to see what
is changing.
Jim
l/208#issuecomment-157861049
On Thu, Apr 21, 2016 at 9:18 AM James A. Robinson
wrote:
>
> Interesting:
>
> https://github.com/chirino/stomp-benchmark/issues/11
>
>
> On Wed, Apr 20, 2016 at 10:54 AM James A. Robinson
> wrote:
>
>> I can't immediately tell if this is
Interesting:
https://github.com/chirino/stomp-benchmark/issues/11
On Wed, Apr 20, 2016 at 10:54 AM James A. Robinson
wrote:
> I can't immediately tell if this is a problem with the stomp client or
> with activemq, but I suspect the latter given what I was seeing. A
> monitor
I can't immediately tell if this is a problem with the stomp client or with
activemq, but I suspect the latter given what I was seeing. A monitor I
put together to test that messages could be passed on the transports
configured for the broker started failing this morning, throwing EOF
Exceptions w
Folks,
In my /etc/activemq/activemq.xml I've got this defined:
and that's where the data files are written:
$ ls /var/log/activemq/activemq-data/amq-prod-1
db-342.log db-343.log db-344.log db-345.log db-380.log db-381.log db.data
db.redo lock scheduler
So why does the attribute "DataDirectory
Hi,
Are there any queues where there is no movement vs. ones where there is a
steady shift in enqueued/dequeued counts? Or any topic where there are one
or more consumers and any pending messages to be dequeued?
I wouldn't typically expect a memory limit on a queue to cause a consumer
to not be
Hi,
Looking for a queue or topic that had a large number of unconsumed messages
would probably be a good star
t.
If your
server
has the webconsole activated you can look at the queues and topics,
for example if your server had a webconsole on port 8161:
http://:8161//admin/queu
You might see if you can get debugging output similar to what KahaDB offers:
http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html
I had a similar situation and the KahaDB debugging showed that the problem
was
due to un-acked persistent messages sprinkled throughout the old
Hi,
I've written a client to exercise the network of brokers in my setup, the
intent is to use this to help monitor that all members of the network can
pull messages from each other.
Currently it's set up to generate N tests, one for each possible
combination of a broker producer and broker consu
Through the JMX service there are an extensive number of mbeans available
to determine the state of the broker and its queues and topics:
http://activemq.apache.org/jmx.html
It possible to access the JMX MBeans over HTTP, and of course you can use
tools that make direct use of the JMX service. Fo
gh I haven't made any
> effort to trace the code); am I missing something?
>
> Tim
>
> On Thu, Mar 31, 2016 at 1:08 AM, James A. Robinson
> wrote:
>
> > I think what it boils down to is figuring out where you need to put the
> jar
> > to make it availabl
Some people are using it. I wanted to, but decided I wasn't comfortable
relying on it.
On Thu, Mar 31, 2016 at 19:15 Natarajan, Rajeswari <
rajeswari.natara...@sap.com> wrote:
> Is replicated level DB store production ready now.
>
>
> -Original Message-
&g
I'm not aware of any other choice. I initially tried to use the replicated
leveldb system but ran into too many stability issues.
On Thu, Mar 31, 2016 at 3:17 PM Natarajan, Rajeswari <
rajeswari.natara...@sap.com> wrote:
> Hi,
>
> Would like to know if ActiveMQ supports HA with message replicat
I think what it boils down to is figuring out where you need to put the jar
to make it available to the class loader in the java instance that is
running your broker.
So, for example, on my Linux setup I have a directory
/usr/share/activemq/lib
that contains the jars needed to run ActiveMQ and I
ues I was seeing.
Jim
On Wed, Mar 23, 2016 at 7:21 AM James A. Robinson wrote:
> Thank you for the information. Would you be able to tell me if your
> NetApp settings for the lock lease are the standard values? I'm told that
> it's 30 seconds, I was thinking about whether
Thank you for the information. Would you be able to tell me if your NetApp
settings for the lock lease are the standard values? I'm told that it's 30
seconds, I was thinking about whether or not I needed to have that raised
much higher to prevent the particular issue I'm seeing (the tcp layer
tr
To answer my own question, no (at least not NFS 4):
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/ch-nfs.html
On Mon, Mar 21, 2016 at 3:46 PM James A. Robinson
wrote:
> I suppose I should pose the question to folks that use NFS,
I suppose I should pose the question to folks that use NFS, do you use TCP
or UDP for your NFS stack?
On Mon, Mar 21, 2016 at 3:29 PM James A. Robinson
wrote:
> Running a packet trace while the problem was occurring let me see that the
> problem is that the NFS layer isn't even bei
Running a packet trace while the problem was occurring let me see that the
problem is that the NFS layer isn't even being involved for 3 minutes due
to the TCP layer retrying for 3 minute before notifying the caller of the
error.
The client sends SYN packets, doesn't get an ACK, and finally times
cond lease, or I could raise the server side lease time to match the 180
seconds the client will try for.
On Fri, Mar 18, 2016 at 8:26 AM, James A. Robinson
wrote:
> Yes, the combination of settings in place right now could add up to 3
> minutes:
>
> On the client side:
>
> nfsve
Is it common that an activemq broker might give up its NFS v4 lock w/o
logging any sort of message? I've got two brokers that logged this:
broker-a which held the log:
2016-03-17 15:01:51,113 [yMonitor Worker] WARN Transport
- Transport Connection to: tcp://104.232.16.4:62269 failed:
org.apac
Yes, the combination of settings in place right now could add up to 3
minutes:
On the client side:
nfsvers=4,proto=tcp,
hard,timeo=600,retrans=2,ac,acregmin=3,acregmax=60,acdirmin=30,fg,retry=120,
sharecache,lookupcache=all,cto
So right now it's got a 60 second timeo value, and it will retry up
t mode helped them detect lost locks.
Anyone who has an HA configuration using NFS that they know works for
failover care to share exactly what mount settings they are using?
On Fri, Mar 18, 2016 at 8:51 AM, James A. Robinson
wrote:
> Yes indeed there was a problem w/ the underlying NFS con
Based on your numbers it's probably not ephemeral port exhaustion. The
default TIME_WAIT is 4 minutes on Solaris, and to use up 32,767 sockets
within 4 minutes, you'd need you to be sending an average of 136 messages
per second.
Jim
met.com> wrote:
> On Thu, 10 Mar 2016 16:13:32 +, "James A. Robinson"
> wrote:
>
> It happens infrequently and unpredictibly. For example sometimes it
> fails after 25 times and sometimes it works for several 100 before
> going wrong.
>
> Another thin
You don't say how frequently it performs this action? If it is frequent,
and I were debugging it, the first thing I would check for would be that
you haven't run out of socket filehandles. On a unix system running
something like "netstat -nt" can tell you what state the socket filehandles
are in.
Thank you for the input. The way I originally set up the Replicated
LevelDB servers was to create 2 sets of 3 hosts:
1a 1b 1c
2a 2b 2c
So within the "1" and "2" clusters each had one master and two slaves, and
then I networked the "1" and "2" clusters together, My intent was to create
highly re
On Tue, Mar 1, 2016 at 7:41 AM, Tim Bain wrote:
> Another possibility: the paths that each broker uses to reach the lock file
> don't resolve to the same file in NFS.
>
In my case they resolve to the same server IP and export path.
On Tue, Mar 1, 2016 at 7:02 AM, artnaseef wrote:
> So something is very wrong then. NFS should *not* allow two NFS clients to
> obtain the same lock.
>
> Three possible explanations come to mind:
>
> * The lock file is getting incorrectly removed (I've never seen ActiveMQ
> cause this)
> * There
On Mon, Feb 29, 2016 at 7:08 PM, artnaseef wrote:
> Something sounds very wrong there. The NFS lock file should prevent more
> than one broker writing to the store at a time.
>
> Is all of /var/log/activemq/activemq-data/ shared across all of the
> brokers?
>
Hi,
Everything under
/var/log/act
This is interesting. When i use a purposefully slow set of
3 brokers sharing an NFS 4 mount, I found that it's very
very easy to get them into a bad state.
As simple a procedure as starting them in sequence and
then restarting them in the same sequence nets me errors
like:
2016-02-29 19:17:46,
Hi,
Thank you for the reply. Do you happen to know what the mount settings
actually are in your setup?
Yes, we are using nfs4 for this.
Jim
On Thu, Feb 25, 2016 at 20:21 artnaseef wrote:
> I've used it successfully more than once without any specific tuning to
> NFS.
> With that said, systems
Hi folks,
Would any of you folks here with an ActiveMQ NFS master/slave setup that
you think has been very reliable be willing to share your NFS client/server
settings?
Right now I'm playing around wth testing some of the suggestions from
http://www.billharlan.com/papers/NFS_for_clusters.html
Bu
Never mind, it appears the problem was due to how the install was run,
there was another section that was overriding the web.xml using an older
version, not the one distributed with 5.13.0.
On Wed, Feb 10, 2016 at 12:47 PM James A. Robinson
wrote:
> I'm working on upgrading from activem
I'm working on upgrading from activemq 5.11.0 to 5.13.0. When I make an
HTTP GET request against the older version of the service, when the queue
doesn't have any pending messages, the request hangs until one is ready.
On the newer version it returns a fatal status 500 error and throws an
exceptio
Is KahaDB considered the more robust backing store of the two options?
We just ran into a variation of
https://issues.apache.org/jira/browse/AMQ-5459
and I couldn't see any way to recover it.
Jim
So when I was building my system I had wanted to use M/S, but the documentation
had indicated the old M/S was deprecated in favor of the newer replicated
LevelDB store. There are some stability issues with replicated LevelDB (w/
the code handling the zookeeper connection). Do you use an older
conf
Have you configured your machine to allow for JMX connections? You might
see if you can connect via something like jconsole and see what the state
of memory and cpu is, the active threads, etc.
On Thu, Aug 20, 2015 at 8:49 AM Richard Sinek wrote:
> The producer continues to send and message bui
t; if you think there's a more appropriate one) to include the recommendation
> we come to. But I don't want to say on the official documentation that it
> has to be = if >= would work fine.
>
> Tim
>
> On Fri, Jul 24, 2015 at 10:39 AM, James A. Robinson
> wrote:
>
>
Hi,
You're talking about http://rhq-project.github.io/rhq/?
On Fri, Jul 24, 2015 at 11:42 AM wrote:
> RHQ is a great tool to use to monitor brokers and queues/topics.
>
zookeeper connections.
If I'm right, then this indicates the instability I saw can be masked over
by making sure the two agree on the session timeout, but that the
fundamental fragility of the activemq zookeeper client code is still a
potential risk.
Jim
On Wed, Jul 22, 2015 at 6:27 PM Ja
tuff or not
> retrieve values super often. As with everything related to performance,
> YMMV so plan to measure how much throughput you can actually do, but don't
> expect to get all stats for all destinations and clients every 100ms.
> On Jul 23, 2015 1:52 PM, "James A. Robinso
im
>
> On Thu, Jul 23, 2015 at 11:39 AM, James A. Robinson
> wrote:
>
> > What tools do you folks find to be effective for monitoring the health of
> > your brokers?
> >
> > I see several nagios plugins do things like check queue depth. I also
> note
> >
What tools do you folks find to be effective for monitoring the health of
your brokers?
I see several nagios plugins do things like check queue depth. I also note
that most of the plugins I see have no concept of monitoring a failover
capable cluster.
Jim
My understanding has been that the election algorithm requires [(n/2)+1]
(where n is the configured size of the cluster) nodes to reach consensus on
who the master is. So [(2/2)+1 = 2] nodes, meaning you have no way to lose
a node and still run. Unless I'm mistaken, that's typically why you see
Hrm... I'm wondering if this is due to the zookeeper server having a
default session timeout of 40 seconds vs. a lower one I set for the
activemq node...
Anyone using replicated leveldb and seeing a somewhat regular set of errors
about lost connections to zookeeper?
2015-07-20 09:26:33,568 [hWire.org:2181)] INFO ClientCnxn
- Opening socket connection to server
zk2.mydomain.org/xxx.xxx.xxx.xxx:2181. Will not attempt to authenticate
using SASL (
ch think they've got the lock long enough to become
> master. That seems like a very unlikely scenario, but easy enough to guard
> against by doing the read-before-write that you asked about, so I think
> it's worth doing.
>
> Tim
>
> On Fri, Jun 19, 2015 at 10:19
On Mon, Jun 15, 2015 at 7:08 AM Tim Bain wrote:
>
> It seems pretty clear that the assumption that acquiring a single file
lock
> without doing any further checks will provide thread-safety in all cases
is
> not an accurate one.
>
> As I see it, here are the failures of the existing approach:
>
>
All the issues you've seen with NFS matches up well with everything I've
seen dealing with NFS storage over the past 18 years. I had always thought
"maybe we're just doing it wrong," but I think everything you outlined
looks familiar to me.
When we decided to try using ActiveMQ a few months ago t
I've had a dev cluster running for a little while now and twice I've seen
interruptions where the cluster didn't recover, didn't select a new master.
I had hoped AMQ-5082 fixed that issue but it looks like there might be
additional problems. How many of you folks are running replicated leveldb,
a
Looking at
https://issues.apache.org/jira/browse/AMQ-5517
I would guess there are some additional dependencies missing from the
pom.xml, but I'm afraid I don't know what those are.
On Wed, May 13, 2015 at 5:43 AM James A. Robinson wrote:
> There should be a few lines below the o
There should be a few lines below the one you pasted that indicate the
actual missing symbol, e.g., "\nsymbol: class WebSocket\nlocation: package
org.eclipse.jetty.websocket". Are there those details in your build error?
Jim
On Wed, May 13, 2015 at 5:29 AM lukas83 wrote:
> Since I need jetty9
)
> between the two, or if you don't want to have to specify all of the other
> clusters so you'd use a hub-and-spoke topology so you only have to specify
> the hub node in your config.
>
> On Fri, Mar 13, 2015 at 12:21 AM, James A. Robinson <
> jim.robin...@gmail.c
On Tue, Mar 31, 2015 at 1:32 PM, wonderkind wrote:
> What version of ActiveMQ are you running with? I am working with 5.11.1..
I'm running 5.11.1 plus the patch in ticket AMQ-5082.
Jim
On Tue, Mar 31, 2015 at 12:08 PM, wonderkind wrote:
>
> Is your broker running pretty clean when you send messages through the
fabric of network of replicated master/slaves?
Yes, though I'm only running the examples/openwire/swissarmy
producer/consumer example using the defaults. On the 2nd
cl
On Mon, Mar 30, 2015 at 4:57 PM, wonderkind wrote:
> Do you have a sample activemq.xml configuration file you could share for
your 3 x 3 cluster? I am trying to do the same, with everything running
on my Mac initially.
On the amq-prod-1 cluster my network connector url is
masterslave:(tcp://am
On Mon, Mar 30, 2015 at 2:09 PM, wonderkind wrote:
> How big is your network? Do you see any significant degradation with a
> network of replicated levelDB brokers?
I haven't started stress testing the configuration yet, the current plan
is to have network two clusters of three. So six machine
On Fri, Mar 27, 2015 at 11:48 AM, wonderkind wrote:
> Has anyone ever created a network of brokers, with each broker been a
> Master/Slave replicated LevelDB store?
>
My understanding has been that, for replicated LevelDB, you need
a set of three brokers, one master two slave, in order to have p
What OS are you running this under? Are you seeing a lot of
threads in your JVM? Are you seeing a lot of GC activity?
On Mon, Mar 23, 2015 at 9:43 AM, phill.tomlinson
wrote:
> Thanks Tim. I tried without message persistence and although I did get
> higher
> throughput it still didn't scale too
On Wed, Mar 18, 2015 at 10:49 AM, mtod wrote:
> INFO | Created default XPathFactory
> com.sun.org.apache.xpath.internal.jaxp.XPathFactoryImpl@343985bd
> INFO | Creating new XPath expression in pool. Namespaces on XPath
> expression: [me: {system ->
> http://camel.apache.org/xml/variables/system-
On Wed, Mar 18, 2015 at 8:17 AM, mtod wrote:
> I tried the XPath attributes you recommended but no luck I get the same
> error.
>
> Maybe I should have used a different subject line :)
So for the 1st recommendation I had:
/bogus/path/just/debugging/here/thanks
the intent was to get the system t
I suspect it's because the question looks related more
to Apache Camel than to ActiveMQ and people here
might not be using the former.
Are you using the same Camel library on both
revisions of the ActiveMQ broker? if you set
/bogus/path/just/debugging/here/thanks
what does that show, does it li
On Wed, Mar 18, 2015 at 1:10 AM, Fred Moore wrote:
>
> Questions:
>
> Q1\ Is this approach a viable one?
>
> Q2\ Are there better ways of obtaining a similar QoS (better in the sense
> of ...more simple and/or more robust and/or more standard)
>
> Q3\ Are there any specific recommendations on whic
Hi folks,
I joined this list because we're starting to look into setting up
ActiveMQ for our developers. My preference at this time is to set
up a highly reliable system that doesn't have single points of
failure.
Due to the concern about single points of failure I didn't want to
introduce a sha
On Thu, Mar 12, 2015 at 9:26 PM, Tim Bain wrote:
> This doesn't really work for topics; you need to get the new messages to
> collect on b2 while the old consumer drains b1, and only then have the old
> consumer disconnect and reconnect on b2; you can't have the consumer
> simultaneously connecte
On Thu, Mar 12, 2015 at 8:58 AM, Tim Bain wrote:
> And there would need to be a change
> to allow a non-durable subscriber to disconnect from the old broker without
> unsubscribing the consumer and to resume that subscription once the
> connection is made to the new broker (and obviously this wou
On Wed, Mar 11, 2015 at 3:28 PM, Gary Tully wrote:
> I think you are correct here. The rebuild should work so long as the
> session has not expired.
The nodes in the zookeeper group tree are ephemeral, so they
disappear once the session is lost. I think the underlying client
manages to re-establ
github.com/jimrobinson/activemq/commit/d272a116ff5c0916a6044d657f99df48f264bd2a
On Tue, Mar 10, 2015 at 4:57 PM, James A. Robinson wrote:
> Working my way through the code and the debug log from
> the test, I see that the ZooKeeper group is getting emptied
> out after session expiration occ
uot;:null,"position":-1,"weight":1,"elected":null}),
(02,{"id":"foo","container":null,"address":null,"position":-1,"weight":1,"elected":null})))
after the timeout:
2015-03-10 12:10:53,490 | DEBUG
On Fri, Mar 6, 2015 at 6:07 AM, underflow wrote:
> - Another idea was to create a network of brokers with the original instance
> (w/ kahadb persistence) and the new instance (w/ leveldb persistence) +
> resending all content, if required...
I'll be curious to see what advice you get. I'm new to
On Wed, Mar 4, 2015 at 12:29 PM, James A. Robinson wrote:
> Thanks. I'm pretty sure AMQ-5082 is what I'm seeing on 5.11.1.
> I'll see if I can get the cycles to set up a unit test to replicate the
> issue.
I think I've got the use a case represented for
https://issu
On Fri, Mar 6, 2015 at 8:37 AM, arun196 wrote:
> I have a consumer that processes orders from messages received from activemq.
> I would like to set a timeout in case the order processing takes over 5
> minutes.
I'm sorry, it's not clear to me from your description what
it is that you are intendi
On Thu, Mar 5, 2015 at 2:28 PM, arun196 wrote:
>
> brokerURL="failover:(tcp://localhost:61616?keepAlive=true,tcp://localhost:61617?keepAlive=true,tcp://localhost:61618?keepAlive=true)?connectionTimeout=1000&jms.redeliveryPolicy.maximumRedeliveries=4"
I don't see connectionTimeout listed in
http:
k (you can search the archives on Nabble), and I don't recall any
> discussion of anyone finding a solution. But JIRA is the authoritative
> place to find out whether anyone has reported and/or fixed this issue (or
> any other).
> On Mar 3, 2015 8:23 AM, "James A. Robinson"
I suspect one issue with figuring out the time a message was
received by a subscriber is that an efficient client will typically
buffer up messages.
The page
http://activemq.apache.org/activemq-message-properties.html
says that JMSTimestamp is the time the message was sent,
and I believe that me
Hi folks,
While testing out ActiveMQ I've been building clusters
VirtualBox. I've been spinning up two 3-node Replicated
LevelDB stores on my laptop.
I've noticed that the clusters can sometimes get into a
state where none of the nodes is the master. It appears
to me as though it's an issue wit
80 matches
Mail list logo
