Found the second problem, it is in the DataFileAppender & AsyncDatamanager,
when the disk is full, the message is saved in the inflightWrites map but
never taken out since the enqueue will fail due to IOException.
synchronized (this) {
// Find the position where this item will land at.
DataFile dataFile = dataManager.allocateLocation(location);
if( !sync ) {
inflightWrites.put(new WriteKey(location), write);
}
batch = enqueue(dataFile, write);
}
and
synchronized (this) {
// Find the position where this item will land at.
DataFile dataFile = dataManager.allocateLocation(location);
inflightWrites.put(new WriteKey(location), write);
batch = enqueue(dataFile, write);
}
>From the above 5.1 source code, it seems like allocateLocation should
actually make sure that there is enough disk space for the particular
message. However, it actually only create the File handle without actually
allocating any disk space So to fix this problem, we added
nextWriteFile.closeRandomAccessFile(nextWriteFile.openRandomAccessFile(true));
in the AsyncDataManager.allocateLocation method to try and pre allocate the
disk space. We believe that this will not have much impact on the
performance as data file is 32MB default, but please feel free to advice
otherwise as this is our first time looking at ActiveMQ source code.
Regards,
BlueFox wrote:
>
>
>
> BlueFox wrote:
>>
>> Hi, I'm encountering a strange behaviour on ActiveMQ 5.1 broker using
>> AMQStore for persistence. In my setup, an ActiveMQCPP (2.1.3) producer
>> with AUTO_ACKNOWLEDGE is sending persistent message to a queue, which is
>> being processed slowly on a consumer using VMTransport on same VM as the
>> broker. I don't have any special setting for prefetch or memory/disk
>> limit etc.
>>
>> When the broker disk space ran out on the broker, the client would
>> receive out of disk space error, which is fine. Then the client would
>> treat this as a network failure and still keep try to send the message to
>> the broker, this repeats for a while as the broker memory usage goes up
>> (!!!) until it finally ran out of memory. My question is does it mean
>> that the producer need to treat each openwire error differently in order
>> to avoid killing the broker? In this case (the broker ran out of disk
>> space), since the broker cannot accept any more messages, shouldn't it
>> simply drop any new messages coming in until more disk space is available
>> instead of keeping all the new messages in the memory?
>>
>
> I found the problem is a part of issue listed in
> https://issues.apache.org/activemq/browse/AMQ-724. However, I now
> encounter another problem where the persistent message is left in the
> memory, even though the disk error should reject the message....
>
--
View this message in context:
http://www.nabble.com/Strange-behaviour-when-broker-ran-out-of-disk-space-tp17843784p17909860.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
