Yes, that always make sense
On Wed, Aug 30, 2023 at 8:30 AM Ryan Blue wrote:
> This is one of the many problems with not using a real catalog. If there
> are any metadata files under the table location, then the table exists so
> we have to delete. The solution is to not use the HadoopCatalog, w
Hey everyone,
I have been one of the supporters for releasing Java 1.4 end of Aug or early
Sep. I think we are getting close and most of the targeted items will be
wrapped within one or two weeks.
I just realized that Spark 3.5 is almost out. A few release candidates failed
but it is convergin
Should I try "fanout-enabled" option within foreachBatch method where I do
dataframe.write ?
On Wed, Aug 30, 2023 at 10:29 AM Nirav Patel wrote:
> Hi,
>
> I am using spark structured streaming and using foreachBatch sink to
> append to iceberg dual hidden partitioned table.
> I got this infamous
Hi,
I am using spark structured streaming and using foreachBatch sink to append
to iceberg dual hidden partitioned table.
I got this infamous error about input dataframe or partition needing to be
clustered:
*Incoming records violate the writer assumption that records are clustered
by spec and by
This is one of the many problems with not using a real catalog. If there
are any metadata files under the table location, then the table exists so
we have to delete. The solution is to not use the HadoopCatalog, which has
been the community's recommendation for several years.
On Wed, Aug 30, 2023
There is no way to drop a Hadoop catalog table without removing the directory so I’m not sure what the alternative would beSent from my iPhoneOn Aug 29, 2023, at 10:10 PM, Manu Zhang wrote:Hi all,The current behavior of dropping a table with HadoopCatalog looks inconsistent to me. When stored at
