As the exceptions says your outputformat isn't serializable; is the
carboncontext marked as transient?
On 17.04.2017 09:12, Sangeeta Gulia wrote:
Hi Team,
Please ignore the previous mail.
I am trying to create a connector for carbondata. Currently i am working on
creating OutputFormat for carbondata. For that i need to create a
carboncontext which i initialise in configure method. Internally
CarbonContext uses sparkContext.
I am able to create carbonContext through my code but when it run statement
"executionEnvironment.execute()
it throws the below error:
Exception in thread "main" org.apache.flink.optimizer.CompilerException:
Error translating node 'Data Sink
"org.apache.flink.CarbonDataOutputFormat@76225435"
: NONE [[ GlobalProperties [partitioning=RANDOM_PARTITIONED] ]] [[
LocalProperties [ordering=null, grouped=null, unique=null] ]]': Could not
write the user code wrapper class org.apache.flink.api.common.
operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
org.apache.spark.sql.hive.client.ClientWrapper
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
JobGraphGenerator.java:381)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
JobGraphGenerator.java:106)
at org.apache.flink.optimizer.plan.SingleInputPlanNode.
accept(SingleInputPlanNode.java:198)
at org.apache.flink.optimizer.plan.OptimizedPlan.accept(
OptimizedPlan.java:128)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
compileJobGraph(JobGraphGenerator.java:192)
at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:188)
at org.apache.flink.api.java.LocalEnvironment.execute(
LocalEnvironment.java:91)
at org.apache.flink.api.java.ExecutionEnvironment.execute(
ExecutionEnvironment.java:926)
at CarbonJDBCExample.main(CarbonJDBCExample.java:86)
Caused by:
org.apache.flink.runtime.operators.util.CorruptConfigurationException:
Could not write the user code wrapper class org.apache.flink.api.common.
operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
org.apache.spark.sql.hive.client.ClientWrapper
at org.apache.flink.runtime.operators.util.TaskConfig.
setStubWrapper(TaskConfig.java:281)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
createDataSinkVertex(JobGraphGenerator.java:903)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
JobGraphGenerator.java:278)
... 8 more
Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive.
client.ClientWrapper
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(
ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(
ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(
ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(
ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(
ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(
ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.flink.util.InstantiationUtil.serializeObject(
InstantiationUtil.java:317)
at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(
InstantiationUtil.java:254)
at org.apache.flink.runtime.operators.util.TaskConfig.
setStubWrapper(TaskConfig.java:279)
... 10 more
Help me with the same.
On Mon, Apr 17, 2017 at 12:39 PM, Sangeeta Gulia <sangeeta.gu...@knoldus.in>
wrote:
Hi Team,
I am trying to create a connector for carbondata. Currently i am working
on creating OutputFormat for carbondata. For that i need to create a
carboncontext which i initialise in configure method. Internally
CarbonContext uses sparkContext.
I am able to create carbonContext through my code but when it run statement
"executionEnvironment.execute()
Exception in thread "main" org.apache.flink.optimizer.CompilerException:
Error translating node 'Data Sink "org.apache.flink.
CarbonDataOutputFormat@76225435" : NONE [[ GlobalProperties
[partitioning=RANDOM_PARTITIONED] ]] [[ LocalProperties [ordering=null,
grouped=null, unique=null] ]]': Could not write the user code wrapper class
org.apache.flink.api.common.operators.util.UserCodeObjectWrapper : java.io
.NotSerializableException: org.apache.spark.sql.hive.client.ClientWrapper
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
JobGraphGenerator.java:381)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
JobGraphGenerator.java:106)
at org.apache.flink.optimizer.plan.SingleInputPlanNode.
accept(SingleInputPlanNode.java:198)
at org.apache.flink.optimizer.plan.OptimizedPlan.accept(
OptimizedPlan.java:128)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
compileJobGraph(JobGraphGenerator.java:192)
at org.apache.flink.client.LocalExecutor.executePlan(
LocalExecutor.java:188)
at org.apache.flink.api.java.LocalEnvironment.execute(
LocalEnvironment.java:91)
at org.apache.flink.api.java.ExecutionEnvironment.execute(
ExecutionEnvironment.java:926)
at CarbonJDBCExample.main(CarbonJDBCExample.java:86)
Caused by:
org.apache.flink.runtime.operators.util.CorruptConfigurationException:
Could not write the user code wrapper class org.apache.flink.api.common.
operators.util.UserCodeObjectWrapper : java.io.NotSerializableException:
org.apache.spark.sql.hive.client.ClientWrapper
at org.apache.flink.runtime.operators.util.TaskConfig.
setStubWrapper(TaskConfig.java:281)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.
createDataSinkVertex(JobGraphGenerator.java:903)
at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(
JobGraphGenerator.java:278)
... 8 more
Caused by: java.io.NotSerializableException: org.apache.spark.sql.hive.
client.ClientWrapper
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.defaultWriteFields(
ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(
ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(
ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(
ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(
ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(
ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(
ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(
ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(
ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.flink.util.InstantiationUtil.serializeObject(
InstantiationUtil.java:317)
at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(
InstantiationUtil.java:254)
at org.apache.flink.runtime.operators.util.TaskConfig.
setStubWrapper(TaskConfig.java:279)
... 10 more
--
Warm Regards,
Sangeeta Gulia
Software Consultant
Knoldus Software LLP
