Re: hashing binary data

2013-09-04 Thread Karsten Schmidt
On 4 September 2013 17:52, Brian Craft wrote: > This gives me a number of reflection warnings, on field ba, on equals, and > on hashCode. I can eliminate the one on hashCode by type hinting the ba > parameter of BAHashable. > > The one on (.ba g) and equals remains. Is there any way to hint these

Re: hashing binary data

2013-09-04 Thread Brian Craft
On Wednesday, September 4, 2013 1:53:17 PM UTC-7, Karsten Schmidt wrote: > > On 4 September 2013 17:52, Brian Craft > > wrote: > > > This gives me a number of reflection warnings, on field ba, on equals, > and > > on hashCode. I can eliminate the one on hashCode by type hinting the ba > > pa

Re: hashing binary data

2013-09-04 Thread Brian Craft
On Wednesday, September 4, 2013 2:13:12 AM UTC-7, Brian Craft wrote: > > > > On Tuesday, September 3, 2013 10:03:00 PM UTC-7, Mikera wrote: >> >> >> >> On Wednesday, 4 September 2013 12:37:33 UTC+8, Brian Craft wrote: >>> >>> >>> >>> On Tuesday, September 3, 2013 9:14:30 PM UTC-7, Mikera wrote: >

Re: hashing binary data

2013-09-04 Thread Brian Craft
On Tuesday, September 3, 2013 10:03:00 PM UTC-7, Mikera wrote: > > > > On Wednesday, 4 September 2013 12:37:33 UTC+8, Brian Craft wrote: >> >> >> >> On Tuesday, September 3, 2013 9:14:30 PM UTC-7, Mikera wrote: >>> >>> On Wednesday, 4 September 2013 10:00:42 UTC+8, Brian Craft wrote: >>> I'm

Re: hashing binary data

2013-09-03 Thread Mikera
On Wednesday, 4 September 2013 12:37:33 UTC+8, Brian Craft wrote: > > > > On Tuesday, September 3, 2013 9:14:30 PM UTC-7, Mikera wrote: >> >> On Wednesday, 4 September 2013 10:00:42 UTC+8, Brian Craft wrote: >> >>> I'm loading data files of about 1-2G, which are composed of a bunch of >>> numeri

Re: hashing binary data

2013-09-03 Thread Brian Craft
On Tuesday, September 3, 2013 9:14:30 PM UTC-7, Mikera wrote: > > On Wednesday, 4 September 2013 10:00:42 UTC+8, Brian Craft wrote: > >> I'm loading data files of about 1-2G, which are composed of a bunch of >> numeric data blocks. I need to store the data blocks w/o storing >> duplicates. They

Re: hashing binary data

2013-09-03 Thread Mikera
On Wednesday, 4 September 2013 10:00:42 UTC+8, Brian Craft wrote: > I'm loading data files of about 1-2G, which are composed of a bunch of > numeric data blocks. I need to store the data blocks w/o storing > duplicates. They arrive as vectors of floats, and are stored as primitive > byte arrays

Re: hashing binary data

2013-09-03 Thread Softaddicts
Why not keep track of the hash of blocks already stored and discard duplicates blocks from memory asap ? I am talking here of keeping blocks as Clojure structures so the hash will be consistent between two identical blocks. Do you store them away as soon as you can ? Or are you trying to keep th

hashing binary data

2013-09-03 Thread Brian Craft
I'm loading data files of about 1-2G, which are composed of a bunch of numeric data blocks. I need to store the data blocks w/o storing duplicates. They arrive as vectors of floats, and are stored as primitive byte arrays. I first tried memoizing the function that saves a block (returning an id