> is this overflow considered a bug? Or is large exec batch something that should be avoided?
This is not a bug and it is something that should be avoided. Some of the hash-join internals expect small batches. I actually thought the limit was 32Ki and not 64Ki because I think there may be some places we are using int16_t as an index. The reasoning is that the hash-join is going to make multiple passes through the data (e.g. first to calculate the hashes from the key columns and then again the encode the key columns, etc.) and you're going to get better performance when your batches are small enough that they fit into the CPU cache. [1] is often given as a reference for this idea. Since this is the case there is not much need for operating on larger batches. > And does acero have any logic preventing that from happening Yes, in the source node we take (potentially large) batches from the I/O side of things and slice them into medium sized batches (about 1Mi rows) to distribute across threads and then each thread iterates over that medium sized batch in even smaller batches (32Ki rows) for actual processing. This all happens here[2]. [1] https://db.in.tum.de/~leis/papers/morsels.pdf [2] https://github.com/apache/arrow/blob/6af660f48472b8b45a5e01b7136b9b040b185eb1/cpp/src/arrow/acero/source_node.cc#L120 On Mon, Jul 3, 2023 at 6:50 AM Ruoxi Sun <zanmato1...@gmail.com> wrote: > Hi folks, > > I've encountered a bug when doing swiss join using a big exec batch, say, > larger than 65535 rows, on the probe side. It turns out to be that in the > algorithm, it is using `uint16_t` to represent the index within the probe > exec batch (the materialize_batch_ids_buf > < > https://github.com/apache/arrow/blob/f951f0c42040ba6f584831621864f5c23e0f023e/cpp/src/arrow/acero/swiss_join.cc#L1897C8-L1897C33 > >), > and row id larger than 65535 will be silently overflow and cause the result > nonsense. > > One thing to note is that I'm not exactly using the acero "the acero way". > Instead I carve out some pieces of code from acero and run them > individually. So I'm just wondering that, is this overflow considered a > bug? Or is large exec batch something that should be avoided? (And does > acero have any logic preventing that from happening, e.g., some wild man > like me just throws it an arbitrary large exec batch?) > > Thanks. > > *Rossi* >
