Thanks. The latest example clarifies a few things. 

Saulius Pakalka

> On 2022-05-27, at 23:27, Wing Yew Poon <wyp...@cloudera.com.invalid> wrote:
> 
> The partitionedBy typo in the doc is already fixed in the master branch of 
> the Iceberg repo.
> I filed a PR to add `using("iceberg")` to the `writeTo` examples for creating 
> a table (if you want to create an *Iceberg* table).
> 
> On Fri, May 27, 2022 at 12:58 PM Wing Yew Poon <wyp...@cloudera.com 
> <mailto:wyp...@cloudera.com>> wrote:
> One other note:
> When creating the table, you need `using("iceberg")`. The example should read
> 
> data.writeTo("prod.db.table")
>     .using("iceberg")
>     .tableProperty("write.format.default", "orc")
>     .partitionedBy($"level", days($"ts"))
>     .createOrReplace()
> 
> - Wing Yew
> 
> 
> On Fri, May 27, 2022 at 11:29 AM Wing Yew Poon <wyp...@cloudera.com 
> <mailto:wyp...@cloudera.com>> wrote:
> That is a typo in the sample code. The doc itself 
> (https://iceberg.apache.org/docs/latest/spark-writes/#creating-tables 
> <https://www.google.com/url?q=https://iceberg.apache.org/docs/latest/spark-writes/%23creating-tables&source=gmail-imap&ust=1654288075000000&usg=AOvVaw3Z4D71Qdmxh8BKvW1XzJrg>)
>  says:
> "Create and replace operations support table configuration methods, like 
> partitionedBy and tableProperty"
> You could also have looked up the API in Spark documentation:
> https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/DataFrameWriterV2.html
>  
> <https://www.google.com/url?q=https://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/DataFrameWriterV2.html&source=gmail-imap&ust=1654288075000000&usg=AOvVaw1CUcrwqHqIYkfYSkjkRZ33>
> There you would have found that the method is partitionedBy, not partitionBy.
> 
> - Wing Yew
> 
> 
> On Fri, May 27, 2022 at 4:32 AM Saulius Pakalka 
> <saulius.paka...@oxylabs.io.invalid> wrote:
> Hi,
> 
> I am trying to create partitioned iceberg table using scala code below based 
> on example in docs.
> df_c.writeTo(output_table)
>   .partitionBy(days(col("last_updated")))
>   .createOrReplace()
> However, this code does not compile and throws two errors:
> 
> value partitionBy is not a member of 
> org.apache.spark.sql.DataFrameWriterV2[org.apache.spark.sql.Row]
> [error] possible cause: maybe a semicolon is missing before `value 
> partitionBy'?
> [error]       .partitionBy(days(col("last_updated")))
> [error]        ^
> [error]  not found: value days
> [error]       .partitionBy(days(col("last_updated")))
> [error]                    ^
> [error] two errors found
> 
> Not sure where to look for a problem. Any advice appreciated.
> 
> Best regards,
> 
> Saulius Pakalka
> 

Reply via email to