Thanks. The latest example clarifies a few things.
Saulius Pakalka
> On 2022-05-27, at 23:27, Wing Yew Poon <wyp...@cloudera.com.invalid> wrote:
>
> The partitionedBy typo in the doc is already fixed in the master branch of
> the Iceberg repo.
> I filed a PR to add `using("iceberg")` to the `writeTo` examples for creating
> a table (if you want to create an *Iceberg* table).
>
> On Fri, May 27, 2022 at 12:58 PM Wing Yew Poon <wyp...@cloudera.com
> <mailto:wyp...@cloudera.com>> wrote:
> One other note:
> When creating the table, you need `using("iceberg")`. The example should read
>
> data.writeTo("prod.db.table")
> .using("iceberg")
> .tableProperty("write.format.default", "orc")
> .partitionedBy($"level", days($"ts"))
> .createOrReplace()
>
> - Wing Yew
>
>
> On Fri, May 27, 2022 at 11:29 AM Wing Yew Poon <wyp...@cloudera.com
> <mailto:wyp...@cloudera.com>> wrote:
> That is a typo in the sample code. The doc itself
> (https://iceberg.apache.org/docs/latest/spark-writes/#creating-tables
> <https://www.google.com/url?q=https://iceberg.apache.org/docs/latest/spark-writes/%23creating-tables&source=gmail-imap&ust=1654288075000000&usg=AOvVaw3Z4D71Qdmxh8BKvW1XzJrg>)
> says:
> "Create and replace operations support table configuration methods, like
> partitionedBy and tableProperty"
> You could also have looked up the API in Spark documentation:
> https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/DataFrameWriterV2.html
>
> <https://www.google.com/url?q=https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/DataFrameWriterV2.html&source=gmail-imap&ust=1654288075000000&usg=AOvVaw1CUcrwqHqIYkfYSkjkRZ33>
> There you would have found that the method is partitionedBy, not partitionBy.
>
> - Wing Yew
>
>
> On Fri, May 27, 2022 at 4:32 AM Saulius Pakalka
> <saulius.paka...@oxylabs.io.invalid> wrote:
> Hi,
>
> I am trying to create partitioned iceberg table using scala code below based
> on example in docs.
> df_c.writeTo(output_table)
> .partitionBy(days(col("last_updated")))
> .createOrReplace()
> However, this code does not compile and throws two errors:
>
> value partitionBy is not a member of
> org.apache.spark.sql.DataFrameWriterV2[org.apache.spark.sql.Row]
> [error] possible cause: maybe a semicolon is missing before `value
> partitionBy'?
> [error] .partitionBy(days(col("last_updated")))
> [error] ^
> [error] not found: value days
> [error] .partitionBy(days(col("last_updated")))
> [error] ^
> [error] two errors found
>
> Not sure where to look for a problem. Any advice appreciated.
>
> Best regards,
>
> Saulius Pakalka
>
