Re: Assign/update : NA bitmap vs sentinel

2018-11-09 Thread Matt Dowle
On Fri, Nov 9, 2018 at 2:14 PM Wes McKinney wrote: > On Fri, Nov 9, 2018 at 4:51 PM Matt Dowle wrote: > > > > > There is one database that I'm aware of that uses sentinels _and_ > > supports complex types with missing values: Kx's KDB+. > > I read this a

Re: Assign/update : NA bitmap vs sentinel

2018-11-09 Thread Matt Dowle
different > meaning was "okay". I can tell you from working in a large advertising > company, that it is critical to have a reliable way to distinguish null > from not null, and a space had better be not null. It's possible that my > knowledge about kdb+ is out of date, in w

Re: Assign/update : NA bitmap vs sentinel

2018-11-09 Thread Matt Dowle
ple" but the intent is to be a > > suitable lingua franca for language agnostic data interchange and as a > > runtime representation for analytical query engines (where most > > operators are "pure"). If the Arrow community's goal were to create a > > "

Re: Assign/update : NA bitmap vs sentinel

2018-11-05 Thread Matt Dowle
all-null. Being able to > do that without having to read each value individually can be the largest > of all wins, especially where many people declare all data as nullable but > rarely use that nullability. > > > > > > > On Mon, Nov 5, 2018 at 2:31 PM Matt Dowle wrot

Assign/update : NA bitmap vs sentinel

2018-11-05 Thread Matt Dowle
Hi, (First post to this mailing list.) I tweeted here and Wes invited me to follow up on this list : https://twitter.com/wesmckinn/status/1059440916987961346 Wes - it was great to meet you at Stanford in September. There I mentioned the assign/update aspect which is a downside of bitmap for NA, im