Right now we are just using the built-in Jetty HTTP server that comes with ActiveMQ. However, this fails after files get any larger than ~1GB, due to some internal problem with Jetty's handling of large file transfers over HTTP. Our project is still in initial stages of development so we haven't had to really face this problem head-on quite yet. Right now what we are doing is sending a URL pointing to the >1GB file on the remote server as a BlobMessage, and ActiveMQ pretends like it was actually responsible for transferring the file. In the future we will be using SFTP or SCP as our out-of-band transfer method. We haven't really figured out the logistics for this yet though.
The good thing about BlobMessages is that you are not restricted by the heap space of the JVM when transferring large files as you might be if you used a stream (since you'd have to read the stream into memory before writing it out unless you wrote your own chunk-and-flush method, which is what the BlobMessage does for you anyway). They were a breeze to set up and use and have been perfect for our needs thus far. Right now we haven't had to do any performance tuning of AMQ besides what the default configuration came with. But again, we aren't production-ready yet so we haven't hit any configuration-related bottlenecks yet. We also aren't sending a large quantity of these large messages. The levels of traffic anticipated for this project are high-read (small AMQ messages for queries), low-write (potentially very large AMQ messages writing big pieces of data), so we haven't run into any producer/consumer flow problems yet. Hope that helped a little. - Marc On Wed, Mar 3, 2010 at 5:26 PM, Fred Moore <fred.moor...@gmail.com> wrote: > Hi Marc, > > > We are sending files >1GB around using the ActiveMQ's BlobMessage. > > ...mmh you have an interesting real-life use case: can you give us a few > more details? > > Things like the "side protocol" you use (FTP? WebDAV?), daily volumes, any > special other ActiveMQ tuning you performed etc > > Cheers, > F. > > > > > On Wed, Mar 3, 2010 at 7:06 PM, Marc Weil <marc.w...@gmail.com> wrote: > > > 100 MB isn't really that large. We are sending files >1GB around using > the > > ActiveMQ's BlobMessage. Why don't you try using that instead of doing > > chunking and serialization/deserialization manually? > > > > http://activemq.apache.org/blob-messages.html > > > > Marc > > > > On Wed, Mar 3, 2010 at 6:17 AM, Cristian Botiza > > <cristian.bot...@endava.com>wrote: > > > > > I would look for some streaming options in ActiveMQ if you want to send > > the > > > full file. But 100 MB is quite huge...anyway see class BytesMessage. > You > > may > > > try using a BufferedStream around the byte array, or read it in chuncks > > > using readBytes(byte[]). > > > > > > ________________________________________ > > > From: dara kok [mrpc.cambo...@gmail.com] > > > Sent: 03 March 2010 13:05 > > > To: users@activemq.apache.org > > > Subject: RE: best approach to transfer large File, modify content on > > > receive > > > > > > So you think sending the file in full size is not performance > efficient? > > I > > > wonder what happens when the receiving app get the file, will the full > > > content of the file load into memory or will just part of it will get > > > loaded > > > and the rest will load on demand? > > > > > > Thanks, > > > > > > > > > Can you chunk the file? In this case you might split it between > multiple > > > producers and consumers > > > and play with the number of producers/consumers and chunck size. You > may > > > send and process chuncks > > > concurrently and also limit the message size. > > > > > > Just one option... > > > -- > > > View this message in context: > > > > > > http://old.nabble.com/best-approach-to-transfer-large-File%2C-modify-content-on-receive-tp27766358p27766785.html > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > > > > > The information in this email is confidential and may be legally > > > privileged. It is intended solely for the addressee. Any opinions > > expressed > > > are mine and do not necessarily represent the opinions of the Company. > > > Emails are susceptible to interference. If you are not the intended > > > recipient, any disclosure, copying, distribution or any action taken or > > > omitted to be taken in reliance on it, is strictly prohibited and may > be > > > unlawful. If you have received this message in error, do not open any > > > attachments but please notify the EndavaIT Service Desk on (+44 (0)870 > > 423 > > > 0187), and delete this message from your system. The sender accepts no > > > responsibility for information, errors or omissions in this email, or > for > > > its use or misuse, or for any act committed or omitted in connection > with > > > this communication. If in doubt, please verify the authenticity of the > > > contents with the sender. Please rely on your own virus checkers as no > > > responsibility is taken by the sender for any damage rising out of any > > bug > > > or virus infection. > > > > > > Endava Limited is a company registered in England under company number > > > 5722669 whose registered office is at 125 Old Broad Street, London, > EC2N > > > 1AR, United Kingdom. Endava Limited is the Endava group holding company > > and > > > does not provide any services to clients. Each of Endava Limited and > its > > > subsidiaries is a separate legal entity and has no liability for > another > > > such entity's acts or omissions. Please refer to the “Legal” section on > > our > > > website for a list of legal entities. > > > > > >
