On 2/14/07, Aleksi Kallio <[EMAIL PROTECTED]> wrote:
There has been discussion regarding FileMessage or something similar: a message for transferring large amounts of data. For background, see: http://issues.apache.org/activemq/browse/AMQ-1075 We need something that offers better service than streaming, of course by building on top of it. That is why we would be interested in this and ready to contribute our effort into making it happen. We have to be capable of moving BIG files over JMS, in the order of couple of gigabytes. Normally they are a lot smaller, but also the big ones must be handled. Features/fixes we are looking for are: 1) streaming requires dedicated destination, which makes things complicated 2) resuming transfers 3) easy monitoring of progress From core developers we would appreciate feedback on how to proceed with developing such a feature. From rest of the communinity we would gladly get feedback on how you would like this to be done and what kind of API would be the best. In the spirit of making ActiveMQ even better,
Sounds great! Welcome! The comments on the JIRA issue describe the kind of API I was thinking of... http://issues.apache.org/activemq/browse/AMQ-1075 There are a few different possible implementations... (i) FileMessage ends up being a wrapper on top of the existing JMS streams http://activemq.apache.org/jms-streams.html (ii) FileMessage uses some out of band transfer mechanism. e.g. the FileMessage essentially just carries around a URL; then the file is transferred using some FTP/HTTP site. e.g. the producer sends to the remote file/web server, then sends the message with a reference to it. i.e. using an out-of-band transfer mechanism. The benefit of this is that existing file/web servers can be used for the actual file transfer piece, then the message broker can be used for reliable message load balancing and failover. i.e. making sure exactly one consumer processes the message properly, in a transactional way etc. (iii) direct connection. This option is similar to (ii) but rather than putting the file on some remote file server, the file stays on the producer until the consumer has received it; also the producer listens on a socket for the consumer. Then the FileMessage includes a URL which when opened on the consumer will essentially connect directly to the producer of the file. So the FileMessage is a normal JMS message sent around the JMS network (over clusters & network store & forward etc). Then when its finally received by a consumer, the consumer opens the URL which in effect causes the consumer to connect directly to the producer, to then stream the file. All of these options have different strengths and weaknesses. I think for many use cases (iii) is good but I could see folks liking (i) and (ii) as well. The nice thing is I think all of these approaches can be handled nicely by the single FileMessage API; then it can be a configuration/policy issue as to exactly which implementation is used. In terms of getting started, the simplest route is probably to add the API in first (to start with assuming just a URL to the file which is a no brainer) then we should be able to start adding different providers to suit. BTW we might also want to support senders providing a File rather than an InputStream if you want to be able to monitor progress (as with a stream you've no idea how long its gonna be) -- James ------- http://radio.weblogs.com/0112098/
