Hi,
I wrote a custom serde that stores takes data and stores it in objects called
DocumentWritable that inherits from Writable. When I try to run code to insert
into the table that uses my serde, I get an error saying that DocumentWritable
cannot be cast to BytesWritable. Does my DocumentWritable need to inherit from
org.apache.hadoop.io.BytesWritable?
I'm creating a table using a custom serde that I wrote:
CREATE TABLE avro_test (
NOTEBOOK_ID STRING,
…
DLC_XID STRING
)
ROW FORMAT SERDE 'com.merck.huron.hive.LuceneSerde';
When I try to insert data using a select statement I am storing it into a class
I have called DocumentWritable:
INSERT OVERWRITE TABLE avro_test SELECT * FROM avro
DocumentWritable inherits from Writable. When I do the select and my serde code
gets executed I get the following error:
Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
{"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO
DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL
SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null}
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
{"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO
DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL
SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:565)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
... 8 more
Caused by: java.lang.ClassCastException:
com.comp.proj.indexing.DocumentWritable cannot be cast to
org.apache.hadoop.io.BytesWritable
at
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat$1.write(HiveIgnoreKeyTextOutputFormat.java:90)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:606)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
... 9 more
Notice: This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from
your system.
