Re: Memory Efficient HashSet

2016-03-10 Thread Steven Schveighoffer via Digitalmars-d-learn
On 3/8/16 3:12 AM, Nordlöw wrote: Has anybody put together a memory-efficient D-implementation of a HashSet Something like sparse_hash_set<> contained in https://github.com/sparsehash/sparsehash but in D. D needs a way to use allocators for hash nodes. In Dcollections, both the performance

Re: Memory Efficient HashSet

2016-03-10 Thread thedeemon via Digitalmars-d-learn
On Wednesday, 9 March 2016 at 22:31:50 UTC, Nordlöw wrote: consumes 842.m MiB on my Ubuntu. Here's mine: https://bitbucket.org/infognition/robinhood/src (you just need one file rbhash.d to use in your apps) The following test takes ~130 MB and can take less with some tweaks in the settings:

Re: Memory Efficient HashSet

2016-03-10 Thread Nordlöw via Digitalmars-d-learn
On Thursday, 10 March 2016 at 10:15:10 UTC, Martin Tschierschke wrote: With how many data sets at the end, you will have to deal? I need something close to the memory overhead of a sorted array of values. That is why I'm considering converting SparseHash's sparse_hash_set to D.

Re: Memory Efficient HashSet

2016-03-10 Thread Ola Fosheim Grøstad via Digitalmars-d-learn
On Wednesday, 9 March 2016 at 22:31:50 UTC, Nordlöw wrote: My knowledge database engine I'm building cannot afford the memory overhead of D's builtin associative arrays. Sounds like a cool project! You could also look into using a trie.

Re: Memory Efficient HashSet

2016-03-10 Thread Nordlöw via Digitalmars-d-learn
On Wednesday, 9 March 2016 at 22:31:50 UTC, Nordlöw wrote: The consumption with Google's sparsehash unordered_set is about 50 MiB. See also: http://smerity.com/articles/2015/google_sparsehash.html

Re: Memory Efficient HashSet

2016-03-10 Thread Martin Tschierschke via Digitalmars-d-learn
On Wednesday, 9 March 2016 at 22:31:50 UTC, Nordlöw wrote: [...] foreach (const i; iota(0, n)) { x[i] = true; // indicate that it's stored } import std.stdio : writeln, readln; writeln("Press return: "); readln; } consumes 842.m MiB on my Ubuntu. The consumption

Re: Memory Efficient HashSet

2016-03-09 Thread Nordlöw via Digitalmars-d-learn
On Tuesday, 8 March 2016 at 12:25:36 UTC, Ola Fosheim Grøstad wrote: On Tuesday, 8 March 2016 at 08:12:04 UTC, Nordlöw wrote: sparse_hash_set<> contained in https://github.com/sparsehash/sparsehash It appears to be very slow? What do you need it for? My knowledge database engine I'm buildin

Re: Memory Efficient HashSet

2016-03-08 Thread Edwin van Leeuwen via Digitalmars-d-learn
On Tuesday, 8 March 2016 at 08:12:04 UTC, Nordlöw wrote: Has anybody put together a memory-efficient D-implementation of a HashSet Something like sparse_hash_set<> contained in https://github.com/sparsehash/sparsehash but in D. There is an implementation in: code.dlang.org/packages/emsi_co

Re: Memory Efficient HashSet

2016-03-08 Thread Ola Fosheim Grøstad via Digitalmars-d-learn
On Tuesday, 8 March 2016 at 08:12:04 UTC, Nordlöw wrote: sparse_hash_set<> contained in https://github.com/sparsehash/sparsehash It appears to be very slow? What do you need it for?

Memory Efficient HashSet

2016-03-08 Thread Nordlöw via Digitalmars-d-learn
Has anybody put together a memory-efficient D-implementation of a HashSet Something like sparse_hash_set<> contained in https://github.com/sparsehash/sparsehash but in D.