Thanks Ryan
I used to put jars into spark.yarn.jars which is hdfs location. By
specifying the --jars argument, I can execute import org.apache.iceberg.xxx
in spark-shell.
Maybe my question is how to create a iceberg table from a existing table or
how to create a iceberg table with parquet format backed. I tried to
execute create table test using iceberg in spark-sql, while it shows
iceberg is not a valid data source. That should not the right usage, looks
like it needs to use the API to do programming. Do we have example code in
the repo or unit tests?
On Thu, May 16, 2019 at 1:58 AM Ryan Blue <rb...@netflix.com.invalid> wrote:
> Hi Junjie,
>
> If you can’t load the classes, I’m not sure what’s going on. You should be
> able to use --jars and --driver-class-path to add the Jar on the command
> line. Try that, and also make sure you’re using a recent Jar built from
> master because we renamed from com.netflix a little while ago.
>
> Right now, master only supports reading from HDFS tables in Spark. That
> looks like this:
>
> val df = spark.read.format("iceberg").load("hdfs://...")
> df.createOrReplaceTempView("table_name")
>
> We’re working on getting better support into Spark with the DataSourceV2
> effort. CTAS was just committed, and we will have a PR for reading v2
> tables from SQL posted to Spark shortly.
>
> I’m also planning on updating the Spark code so that it can use a Hive
> metastore for tables now that we have committed the connection pool in
> iceberg-hive.
>
> rb
>
> On Tue, May 14, 2019 at 8:58 PM 俊杰陈 <cjjnj...@gmail.com> wrote:
>
>> Try to send this again since I can't see this on dev mail list.
>>
>> On Tue, May 14, 2019 at 1:24 PM 俊杰陈 <cjjnj...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> Do we have wiki or doc for spark usage? I tried to put jar in spark
>>> jars dir, but when importing org.apache.iceberg.xxxx in spark-shell, it
>>> shows iceberg is not a member of package org.apache.
>>>
>>> --
>>> Thanks & Best Regards
>>>
>>
>>
>> --
>> Thanks & Best Regards
>>
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>
--
Thanks & Best Regards
