One other note: When creating the table, you need `using("iceberg")`. The example should read
data.writeTo("prod.db.table") .using("iceberg") .tableProperty("write.format.default", "orc") .partitionedBy($"level", days($"ts")) .createOrReplace() - Wing Yew On Fri, May 27, 2022 at 11:29 AM Wing Yew Poon <wyp...@cloudera.com> wrote: > That is a typo in the sample code. The doc itself ( > https://iceberg.apache.org/docs/latest/spark-writes/#creating-tables) > says: > "Create and replace operations support table configuration methods, like > partitionedBy and tableProperty" > You could also have looked up the API in Spark documentation: > > https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/DataFrameWriterV2.html > There you would have found that the method is partitionedBy, not > partitionBy. > > - Wing Yew > > > On Fri, May 27, 2022 at 4:32 AM Saulius Pakalka > <saulius.paka...@oxylabs.io.invalid> wrote: > >> Hi, >> >> I am trying to create partitioned iceberg table using scala code below >> based on example in docs. >> >> df_c.writeTo(output_table) >> .partitionBy(days(col("last_updated"))) >> .createOrReplace() >> >> However, this code does not compile and throws two errors: >> >> value partitionBy is not a member of >> org.apache.spark.sql.DataFrameWriterV2[org.apache.spark.sql.Row] >> [error] possible cause: maybe a semicolon is missing before `value >> partitionBy'? >> [error] .partitionBy(days(col("last_updated"))) >> [error] ^ >> [error] not found: value days >> [error] .partitionBy(days(col("last_updated"))) >> [error] ^ >> [error] two errors found >> >> Not sure where to look for a problem. Any advice appreciated. >> >> Best regards, >> >> Saulius Pakalka >> >>