Iceberg engine version maintenance lifecycle

2021-12-15 Thread Jack Ye
Hi everyone, As a part of the ongoing 0.13.0 release, we are starting to formally support multiple engine versions for Spark, Flink and Hive. I think it is worth defining a formal process for us to add a new supported version, maintain existing versions and deprecate old versions. We briefly touch

Re: High memory usage with highly concurrent committers

2021-12-15 Thread Ryan Blue
Igor, you might want to take a look at the recently added GCSFileIO: https://github.com/apache/iceberg/pull/3711/files It would be great to get some additional feedback on that! On Tue, Dec 14, 2021 at 8:28 PM Igor Dvorzhak wrote: > Yes, you still can upload files of any size, this property jus