Not sure how your binding does it but MyBucket.keys() should be calling the list keys feature of riak via the streaming mode which chunks the key reply instead of buffering and sending one entire list of keys.
@siculars on twitter http://siculars.posterous.com Sent from my iRotaryPhone On Jan 9, 2012, at 17:39, Karthik K <oss....@gmail.com> wrote: > (Changing subject to reflect the problem better and reposting ). > > Any idea , based on the configuration inline as to what explains the > inconsistency number of keys read after a bulk write ( say 1M of payload 1000 > bytes ). > > Any appropriate write flush setting , that is missed on the client / server > side ? > > > -- > Karthik. > > > On Fri, Jan 6, 2012 at 5:54 PM, Karthik K <oss....@gmail.com> wrote: > Further, > ulimit -n is 10K on the box. > > > # tail -100 /var//log/riak/erlang.log.1 > > .... > 17:26:26.960 [info] alarm_handler: {set,{system_memory_high_watermark,[]}} > /usr/lib/riak/lib/os_mon-2.2.6/priv/bin/memsup: Erlang has closed. > Erlang has closed > 17:26:34.443 [info] alarm_handler: {clear,system_memory_high_watermark} > > > So - is there a commit / flush setting that is missed when it comes to high > volume writes ? > > > On Fri, Jan 6, 2012 at 5:37 PM, Karthik K <oss....@gmail.com> wrote: > I am using Riak with LevelDB as the storage engine. > > app.config: > > {storage_backend, riak_kv_eleveldb_backend}, > > > {eleveldb, [ > {data_root, "/var/lib/riak/leveldb"}, > {write_buffer_size, 4194304}, %% 4MB in bytes > {max_open_files, 50}, %% Maximum number of files open at once per > partition > {block_size, 65536}, %% 4K blocks > {cache_size, 33554432}, %% 32 MB default cache size per-partition > {verify_checksums, true} %% make sure data is what we expected it > to be > ]}, > > > > > I want to insert a million keys into the store ( into a given bucket ) . > > pseudo-code: > riakClient = RiakFactory.pbcClient(); > myBucket = riakClient.createBucket("myBucket").nVal(1).execute(); > for (int i = 1; i <= 1000000; ++i) { > final String key = String.valueOf(i); > myBucket.store(key, new String(payload)).returnBody(false); > } > > > after this operation, when I do: > > int count = 0; > for (String key : myBucket.keys() ) { > ++count; > } > return count; > > This returns a total of 14K keys, while I was expecting close to 1 million or > so. > > I am using riak-java-client (pbc). > > Which setting / missing client code can explain the discrepancy ? Thanks. > > > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
