I loaded JavaLite into my IDE and reproduced some test failures. Here are
my observations...
One thing that I noticed is that it's possible for some of your tests to
leak brokers. Any test that has an assertion before the broker is stopped
can leak if that assertion fails because the test will terminate without
stopping the broker. This will negatively impact any tests that follow. You
should stop your brokers in a finally block or perhaps in the @After
method. In any case, you need to make absolutely sure that no matter what
happens in the test the broker is stopped.
Your tests also leak journals. You create the embedded broker's journal
directory using this:
Files.createTempDirectory("async").toFile().getCanonicalPath();
However, you never clean that directory up. I ran thousands of test
iterations in a loop and it filled up over 200GB of disk space.
I think the most important thing is that when you send messages in your
tests you send them as non-persistent which means they will be sent
*asynchronously*. However, your tests don't take this into account which
can lead to race conditions and ultimately assertion/test failures. I see a
couple of options to resolve this. You could send messages as persistent
which will be done synchronously. You could set
"blockOnNonDurableSend=true" on your embedded client's URL (discussed in
the documentation [1]) which will also make sending the messages
synchronous. Or you could add some other kind of Wait.waitFor() to ensure a
meaningful condition is met before proceeding with the test (although you
won't be able to inspect the message-count in most tests since you're using
asynchronous message listeners).
I also think your tests would be more robust if you used Wait.waitFor()
when inspecting HelloCommand.counter() and if you did this *before* you
shut down the broker. Given the asynchronous nature of the message
listeners there may be a discrepancy between the counter's value and the
queue's message count for a short time. Furthermore, leaving the broker up
during the Wait.waitFor() will mean that the message listeners don't get
prematurely stopped.
When I first started running the tests I could consistently reproduce a
failure in less than 25 runs or so. With the changes I listed above I was
able to run over 12,000 times without a failure.
Ultimately I don't see any issues with the broker at this point, just
issues with the tests.
Justin
[1]
http://activemq.apache.org/components/artemis/documentation/latest/send-guarantees.html#non-transactional-message-sends
On Mon, Nov 30, 2020 at 4:05 PM Igor Polevoy <i...@javalite.io> wrote:
> HI, all.
>
> The question is posted to SO here:
>
>
> https://stackoverflow.com/questions/64991366/how-to-cleanly-close-embedded-activemq-artemis
>
> I'd like to add more details. This error is only detected on some computers
> only during JavaLite tests, and only recently. Nonetheless, it might expose
> an issue.
>
> It happens during execution of this test (others too):
>
> https://github.com/javalite/javalite/blob/java8/javalite-async/src/test/java/org/javalite/async/AsyncSpec.java
>
> As you can see, the broker starts and stops in the context of every test
> method.
>
> Within each test, we start a new broker, ensure that messages sent,
> accounted for, and then we stop the broker.
>
> Here is an example:
>
> This test sends 100 messages:
>
> https://github.com/javalite/javalite/blob/master/javalite-async/src/test/java/org/javalite/async/AsyncSpec.java#L101
>
> This test sends 2 messages:
>
> https://github.com/javalite/javalite/blob/master/javalite-async/src/test/java/org/javalite/async/AsyncSpec.java#L148
>
> They both fail due to the incorrect number of messages tested at the end.
> The 2 messages test fails because it receives 0 messages, and the 100
> message test fails because it receives 102 messages. This is despite the
> fact that every test has a new broker located which uses a new temporary
> directory that is created anew at the start and deleted at the end of
> every test.
>
> This is not a problem with a real production since we do not expect
> multiple embedded brokers in the same JVM instance. Nonetheless, it might
> be an issue in Artemis that manifests itself like this. We are moving
> JavaLite to a new CI server, and this is a show stopper, as the build fails
> there with some random conditions (different tests fail randomly).
>
> Any tests executed individually by Maven on command line suceeds without
> issues, which makes me believe Artemis might leak some data in JVM across
> multiple instances.
>
>
> *Note*: if you want to follow JavaLite code, please ensure to look at the
> java8 branch,
>
>
> Any help is much appreciated.
>
> Igor
>
