UnboundedSource.split too early

2023-10-22 Thread gabrywu
Hi, folks, I noticed that this code snippet of UnboundedSourceWrapper tells us we `need` splitting early. // get the splits early. we assume that the generated splits are stable, // this is necessary so that the mapping of state to source is correct // when restoring splitSources = source.split(pa

deploy jars into our maven repo

2023-08-23 Thread gabrywu
Hi, folks, I'm new to gradle, and trying to deploy beam jars to our maven repo, can you help to do it?Is there a wiki about it?

condition missed after a unnest

2023-07-28 Thread gabrywu
Hi, folks, I find a bug that a filter to a unnest result is removed, can you help to look into it? https://github.com/apache/beam/issues/27733

BeamKafkaTable doesn't set commitOffsetsInFinalize to KafkaIO reader

2023-07-24 Thread gabrywu
Hi, folks, I create a issue here, and found that BeamKafkaTable doesn't support all configs of KafkaIO, such as commitOffsetsInFinalize. Who can give me any details of that? https://github.com/apache/beam/issues/27625