Re: Apache Artemis - Stress-test time

[alessandro.zann...@bticino.it]

Hello, 
I'm a colleague of Francesco. 
We just made another test , just connecting, without subscribing, in order
to check if the problem is the authorization procedure... and we discovered
that this step doesn't occupy a lot of cpu (never reached 5% of usage).
So we can say that the "heavy" step is the subscription procedure made after
connection. For each client , we made a subscription to 50 topics (different
for each clients). 
In our test we subscribed 2000 clients (so, 50x2000 = 100K topics).
During this period (that has requested 1h), the CPU of broker during
subscriptions was ever near 100% of usage! 
Can you help us Justin, to understand why ??
Thanks.
  



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724785.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Apache Artemis - Stress-test time

[alessandro.zann...@bticino.it]

Here attached the output of "jstack -F" command.
jstack.zip   



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724787.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Apache Artemis - Stress-test time

[alessandro.zann...@bticino.it]

Hello,

we've made another test , just making some subscriptions (so, no retained 
messages involved) .

Nothing changed: the cpu has reached very quickly 100% of usage!

Attached the new jstack output.

Looking at jstack output it seems that artemis tries to check on retained 
messages also if there aren't retained messages ...

Can you confirm this "strange" behaviour of Artemis ?





Alessandro ZANNINI
Project Leader Portale My Home e Sviluppo Applicazioni WEB
Tel: +(39)0332272966 Ext 9966
Indirizzo: V.le Borri 231 Varese 21100 IT
email: alessandro.zann...@bticino.it
WebSite: www.bticino.it<http://www.bticino.it>
<http://www.bticino.it>[1491549876074_OutlookEmoji-1485765781489_logo_bticino.resized.png]
[X]

Please consider your environmental responsibility before printing this Email




Da: Martyn Taylor [via ActiveMQ] 
Inviato: martedì 11 aprile 2017 12.40
A: Alessandro ZANNINI
Oggetto: Re: Apache Artemis - Stress-test time

Clebert: Yes we could optimize on this area of the code to avoid the
bindings query.  To determine whether or not this is the root cause of the
problem experience by Francesco and Alessandro, we just need to follow the
same test without using retained messages.

Francesco, Alessandro is this something you could test for us?

Could you also let us know if staggering the subscribe helps.  Like try
testing connecting 100 clients at a time, wait a while then attach another
100.  Once you have all your clients connected does the system perform as
normal?  (this will help us determine whether or not the bottleneck is the
subscribe or just normal function with that many consumers).

Another quick q, are your subscriptions matching most messages.  e.g. are
you using "#" or similar.  If using "#" or subscriptions that match most
messages I'd expect the performance to drop, since the broker needs to
iterate through every subscription queue.  If you have 20,000 subscription
queues matching every message you might see perf decrease.

Any more information on your use case would be helpful in diagnosing the
bottleneck

Thanks




On Mon, Apr 10, 2017 at 5:33 PM, Clebert Suconic <[hidden 
email]>
wrote:

> only thing I see on the thread dump is reained message is doing a
> query on the queue every time:
>
>  - org.apache.activemq.artemis.api.core.SimpleString.split(char)
> @bci=100, line=314 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.
> AddressImpl.(org.apache.activemq.artemis.api.core.SimpleString,
> org.apache.activemq.artemis.core.config.WildcardConfiguration)
> @bci=31, line=48 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.
> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=77, line=121 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.
> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=5, line=656 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.
> bindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=110, line=722 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.
> executeBindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=9, line=730 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.protocol.mqtt.
> MQTTRetainMessageManager.addRetainedMessagesToQueue(
> org.apache.activemq.artemis.core.server.Queue,
> java.lang.String) @bci=27, line=74 (Compiled frame)
>
>  -
>
>
> @Mtaylor: any way to avoid it?
>
> On Mon, Apr 10, 2017 at 11:42 AM, [hidden 
> email]
> <[hidden email]> wrote:
> > Here attached the output of "jstack -F" command.
> > jstack.zip <http://activemq.2283324.n4.nabble.com/file/n4724787/
> jstack.zip>
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724787.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic
>



If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724831.html
To unsubscribe from Apache Artemis - Stress-test time, click 
here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4723999&code=YWxlc3NhbmRyby56YW5uaW5pQGJ0aWNpbm8uaXR8NDcyMzk5OXw5Mzc4MzU5Nzg=>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail

Re: Apache Artemis - Stress-test time

[alessandro.zann...@bticino.it]

@Mtaylor 
@clebert

Any news for us ?
Thanks.




--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724924.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Apache Artemis - Stress-test time

[alessandro.zann...@bticino.it]

clebertsuconic wrote
> I am very confused here...
> We have a small issue on quering the queues... but from what I saw the
> CPU issue was on your usage of whatever logger framework you were
> using. I'm not sure what we can do there.

Hi,
we use a custom security plugin that loads security roles (for each topic)
on startup; during subscription phase (when cpu usage grows) our custom
plugin is not used (in jstack we can see only RedisPlugin$UpdateThread.run()
wich is a listener on change in our redis server);
We use standard logging (like that in LegacyLDAPSecuritySettingPlugin): what
make you think our classes cause the cpu usage growing?

If you need other information, just let me know! 

Thanks for your support!



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724960.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.