What about this Friday? One hour slot from 9:00 to 10:00 am or 10:00 to 11:00 am PST? Some folks are based in London, so meeting later than this is hard. If Friday doesn’t work, we can consider Tuesday or Wednesday next week.
> On 22 May 2019, at 00:54, Jacques Nadeau <jacq...@dremio.com> wrote: > > I agree with Anton that we should probably spend some time on hangouts > further discussing things. Definitely differing expectations here and we seem > to be talking a bit past each other. > -- > Jacques Nadeau > CTO and Co-Founder, Dremio > > > On Tue, May 21, 2019 at 3:44 PM Cristian Opris <cop...@apple.com.invalid> > wrote: > I love a good flame war :P > >> On 21 May 2019, at 22:57, Jacques Nadeau <jacq...@dremio.com >> <mailto:jacq...@dremio.com>> wrote: >> >> >> That's my point, truly independent writers (two Spark jobs, or a Spark job >> and Dremio job) means a distributed transaction. It would need yet another >> external transaction coordinator on top of both Spark and Dremio, Iceberg by >> itself >> cannot solve this. >> >> I'm not ready to accept this. Iceberg already supports a set of semantics >> around multiple writers committing simultaneously and how conflict >> resolution is done. The same can be done here. > > > > MVCC (which is what Iceberg tries to implement) requires a total ordering of > snapshots. Also the snapshots need to be non-conflicting. I really don't see > how any metadata data structures can solve this without an outside > coordinator. > > Consider this: > > Snapshot 0: (K,A) = 1 > Job X: UPDATE K SET A=A+1 > Job Y: UPDATE K SET A=10 > > What should the final value of A be and who decides ? > >> >> By single writer, I don't mean single process, I mean multiple coordinated >> processes like Spark executors coordinated by Spark driver. The coordinator >> ensures that the data is pre-partitioned on >> each executor, and the coordinator commits the snapshot. >> >> Note however that single writer job/multiple concurrent reader jobs is >> perfectly feasible, i.e. it shouldn't be a problem to write from a Spark job >> and read from multiple Dremio queries concurrently (for example) >> >> :D This is still "single process" from my perspective. That process may be >> coordinating other processes to do distributed work but ultimately it is a >> single process. > > Fair enough > >> >> I'm not sure what you mean exactly. If we can't enforce uniqueness we >> shouldn't assume it. >> >> I disagree. We can specify that as a requirement and state that you'll get >> unintended consequences if you provide your own keys and don't maintain this. > > There's no need for unintended consequences, we can specify consistent > behaviour (and I believe the document says what that is) > > >> >> We do expect that most of the time the natural key is unique, but the eager >> and lazy with natural key designs can handle duplicates >> consistently. Basically it's not a problem to have duplicate natural keys, >> everything works fine. >> >> That heavily depends on how things are implemented. For example, we may >> write a bunch of code that generates internal data structures based on this >> expectation. If we have to support duplicate matches, all of sudden we can >> no longer size various data structures to improve performance and may be >> unable to preallocate memory associated with a guaranteed completion. > > Again we need to operate on the assumption that this is a large scale > distributed compute/remote storage scenario. Key matching is done with > shuffles with data movement across the network, such optimizations would > really have little impact on overall performance. Not to mention that most > query engines would already optimize the shuffle already as much as it can be > optimized. > > It is true that if actual duplicate keys would make the key matching join > (anti-join) somewhat more expensive, however it can be done in such a way > that if the keys are in practice unique the join is as efficient as it can be. > > >> >> Let me try and clarify each point: >> >> - lookup for query or update on a non-(partition/bucket/sort) key predicate >> implies scanning large amounts of data - because these are the only data >> structures that can narrow down the lookup, right ? One could argue that the >> min/max index (file skipping) can be applied to any column, but in reality >> if that column is not sorted the min/max intervals can have huge overlaps so >> it may be next to useless. >> - remote storage - this is a critical architecture decision - >> implementations on local storage imply a vastly different design for the >> entire system, storage and compute. >> - deleting single records per snapshot is unfeasible in eager but also >> particularly in the lazy design: each deletion creates a very small >> snapshot. Deleting 1 million records one at a time would create 1 million >> small files, and 1 million RPC calls. >> >> Why is this unfeasible? If I have a dataset of 100mm files including 1mm >> small files, is that a major problem? It seems like your usecase isn't one >> where you want to support single record deletes but it is definitely >> something important to many people. > > 100 mm total files or 1 mm files per dataset is definitely a problem on HDFS, > and I believe on S3 too. Single key delete would work just fine, but it's > simply not optimal to do that on remote storage. This is a very well known > problem with HDFS, and one of the very reasons to have something like Iceberg > in the first place. > > Basically the users would be able to do single key mutation, but it's not the > use case we should be optimizing for, but it's really not advisable. > > >> >> Eager is conceptually just lazy + compaction done, well, eagerly. The logic >> for both is exactly the same, the trade-off is just that with eager you >> implicitly compact every time so that you don't do any work on read, while >> with lazy >> you want to amortize the cost of compaction over multiple snapshots. >> >> Basically there should be no difference between the two conceptually, or >> with regard to keys, etc. The only difference is some mechanics in >> implementation. >> >> I think you have deconstruct the problem too much to say these are the same >> (or at least that is what I'm starting to think given this thread). It seems >> like real world implementation decisions (per our discussion here) are in >> conflict. For example, you just argued against having a 1mm arbitrary >> mutations but I think that is because you aren't thinking about things over >> time with a delta implementation. Having 10,000 mutations a day where we do >> delta compaction once a week >> and local file mappings (key to offset sparse bitmaps) seems like it could >> result in very good performance in a case where we're mutating small amounts >> of data. In this scenario, you may not do major compaction ever unless you >> get to a high enough percentage of records that have been deleted in the >> original dataset. That drives a very different set of implementation >> decisions from a situation where you're trying to restate an entire >> partition at once. > > > We operate on 1 billion mutations per day at least. This is the problem > Iceberg wants to solve, I believe it's stated upfront. 10000/day is not a big > data problem. It can be done fairly trivially and it would be supported, but > there's not much point in extra optimizing for this use case I believe. >
