Yes. You are right, but I tried old hadoopFile for OrcInputFormat. In hive12,
OrcStruct is not exposing its api, so spark cannot access it. With Hive13, RDD
can read from OrcFile. Btw, I didn’t see ORCNewOutputFormat in hive-0.13.
Direct RDD manipulation (Hive13)
val inputRead =
sc.hadoopFile("/user/zzhang/orc_demo",classOf[org.apache.hadoop.hive.ql.io.orc.OrcInputFormat],classOf[org.apache.hadoop.io.NullWritable],classOf[org.apache.hadoop.hive.ql.io.orc.OrcStruct])
val v = inputRead.map(pair => pair._2.toString)
val c = v.collect
Thanks.
Zhan Zhabg
On Aug 14, 2014, at 11:12 AM, Yin Huai <huaiyin....@gmail.com> wrote:
> Hi Zhan,
>
> Thank you for trying it. For "directly manipulate ORCFile through RDD", do
> you mean using hadoopFile and saveAsHadoopFile? For "some ORC API", do you
> mean ORCNewInputFormat, ORCNewOutputFormat and ORCStruct?
>
> Thanks,
>
> Yin
>
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader of
> this message is not the intended recipient, you are hereby notified that any
> printing, copying, dissemination, distribution, disclosure or forwarding of
> this communication is strictly prohibited. If you have received this
> communication in error, please contact the sender immediately and delete it
> from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
