Well, I don't know what I'm doing wrong here but I'm mainly using ruby.
Excon (which Ripple uses) doesn't seem to have a way to pass an IO to a put
request (am I missing something?).
I've also tried NetHTTP in ruby which is actually supposed to be able to do
that and I've tried Javas HttpURLConnection in JRuby (I'm really not a Java
expert). None of them works unfortunately:
# I tried this
require 'net/http'
## this doesn't work and doesn't store all the data
path= "/path/to/file/here" ## 88 MB file
key = "large_file"
uri = URI("http://127.0.0.1:8098";)
req = Net::HTTP::Put.new("/luwak/#{key}")
req.body_stream = File.open(path)
req.content_type = "video/mp4"
req.add_field 'Transfer-Encoding', 'chunked'
res = Net::HTTP.start(uri.host, uri.port) do |http|
http.request(req)
end
# And this - in this case riak started thrashing like crazy until
# I just killed it (it never stored more than 1 MB of data though)
path= "/path/to/file/here" ## 88 MB file
key = "large_file"
uri = URI("http://127.0.0.1:8098";)
req = Net::HTTP::Put.new("/luwak/#{key}")
req.body_stream = File.open(path)
req.content_type = "video/mp4"
req.content_length = File.size(path)
req.add_field 'Transfer-Encoding', 'chunked'
res = Net::HTTP.start(uri.host, uri.port) do |http|
http.request(req)
end
# And in JRuby I tried Javas HttpURLConnection
require 'java'
include_class "java.net.URL"
include_class "java.io.RandomAccessFile"
include_class "java.io.FileInputStream"
include_class "java.net.HttpURLConnection"
CHUNK_LENGTH=1048576
u = URL.new('http://192.168.1.142:8098/luwak/jruby_large_file_upload')
c = u.open_connection
c.request_method = 'PUT'
c.fixed_length_streaming_mode=File.size(FILE)
# c.chunked_streaming_mode = CHUNK_LENGTH ## also tried this
c.do_input = true
c.do_output = true
output_stream = c.output_stream
error_stream = c.error_stream
random_access_file = RandomAccessFile.new(FILE, 'r')
bytes_read = 0
buffer = Java::byte[CHUNK_LENGTH].new
begin
while(bytes_read >= 0) do
bytes_read = random_access_file.read(buffer)
puts "bytes_read: #{bytes_read}"
if bytes_read>=0
## the second write here gets an Exception - I think it's a 400 from
riak
puts "writing..."
output_stream.write(buffer,0,bytes_read)
sleep 1
end
end
rescue Exception => e
puts "Exception: #{e.message}"
end
puts c.response_code rescue nil
puts c.response_message rescue nil
random_access_file.close
output_stream.close
c.disconnect
On Tue, Nov 29, 2011 at 10:53 PM, Greg Stein <gst...@gmail.com> wrote:
>
> On Nov 29, 2011 5:08 AM, "John Axel Eriksson" <j...@insane.se> wrote:
> >
> > Is it possible to incrementally add to a file in Luwak using PUT and the
> Content-Range header. I just assumed that it was but I can't seem to
> > get the expected results, it just overwrites whatever the key contents
> were before. The reason I want to do this is because we have some pretty
> > large files I don't want to load fully into memory before PUTing them.
>
> Hmm? You can read from a file and write to the http socket. There is no
> reason or need to load the entire contents into memory.
>
> I don't know what client you're using, but I do know the Python client is
> broken in this regard. It erroneously loads the full content into memory.
> But there is nothing from the Riak server that demands such an approach.
>
> Cheers,
> -g
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
