Re: Merging sorted tables/ record batches

2021-09-01 Thread Antoine Pitrou
Le 01/09/2021 à 03:58, Micah Kornfield a écrit : According to Wikipedia there is a min-heap approach that is O(N log k) not sure if this matches with Niranda's proposal [1]. On the surface the analysis make sense to me but I could be missing something. [1] https://en.m.wikipedia.org/wiki/K-wa

Re: Merging sorted tables/ record batches

2021-08-31 Thread Micah Kornfield
According to Wikipedia there is a min-heap approach that is O(N log k) not sure if this matches with Niranda's proposal [1]. On the surface the analysis make sense to me but I could be missing something. [1] https://en.m.wikipedia.org/wiki/K-way_merge_algorithm On Tuesday, August 31, 2021, Antoi

Re: Merging sorted tables/ record batches

2021-08-31 Thread Antoine Pitrou
On Tue, 31 Aug 2021 17:47:02 -0400 Niranda Perera wrote: > @antoine thanks for the heads up. > This was the example. I want to merge this table into a final table > +---+--+---+ > | Table 0 | | Table 1 | > +---+---+---+ +---+---+---+ > | | A | B | | | A | B | > +---+---+

Re: Merging sorted tables/ record batches

2021-08-31 Thread Niranda Perera
@antoine thanks for the heads up. This was the example. I want to merge this table into a final table +---+--+---+ | Table 0 | | Table 1 | +---+---+---+ +---+---+---+ | | A | B | | | A | B | +---+---+---+ +---+---+---+ | 0 | 3 | x | | 0 | 1 | x | +---+---+---+ +---+---

Re: Merging sorted tables/ record batches

2021-08-31 Thread Antoine Pitrou
Hi Niranda, Le 31/08/2021 à 18:09, Niranda Perera a écrit : Hi all, Is there an efficient way to merge a set of sorted tables/ record batches (like in the merge step of a merge sort)? Ex: Simplest case would be to merge these 2 tables, based on A The formatting of your example looks broken,

Merging sorted tables/ record batches

2021-08-31 Thread Niranda Perera
Hi all, Is there an efficient way to merge a set of sorted tables/ record batches (like in the merge step of a merge sort)? Ex: Simplest case would be to merge these 2 tables, based on A Table 0 Table 1 A B A B 0 3 x 0 1 x 1 3 x 1 2 x 2 4 x 2 2 x 3 5 x 3 5 x I could think of a couple of way