EvenGui opened a new issue, #4472: URL: https://github.com/apache/incubator-seatunnel/issues/4472
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened 运行PG 同步至 Kudu 遇到Kerberos 问题. 集群环境是CDH6.3.3，集群开启了Keberos Spark版本是2.4 Scala版本是2.11 Spark运行脚本带上--keytab /opt/test/xxx.keytab --principal [email protected] 应该是能够解决Kerberos问题的 使用Seatunnel同步PG到Kudu时报错，但是PG到HDFS,HIVE 是没有问题的 在命令提交前，又尝试通过kinit 认证，再提交，依旧抱错 ### SeaTunnel Version 2.3.0 ### SeaTunnel Config ```conf env { job.mode = "BATCH" job.name = "pg_to_kudu_s9mltCeI" } source { Jdbc { connection_check_timeout_sec = 30 driver = "org.postgresql.Driver" fetch_size = 0 partition_num = 1 password = "123456" query = "SELECT * FROM "public".student" url = "jdbc:postgresql://192.168.0.90:5432/xxx_test" user = "postgres" } } transform = { } sink { Kudu { kudu_master = "cdh3.xxx.com:7051" kudu_table = "student" save_mode = "append" } } ``` ### Running Command ```shell ./bin/start-seatunnel-spark-connector-v2.sh --master yarn --deploy-mode client --keytab /opt/test/xxx.keytab --principal [email protected] --config ./config/kudu.conf ``` ### Error Exception ```log Previous exception in task: ErrorCode:[KUDU-06], ErrorDescription:[Initialize the Kudu client failed] - org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (cdh3.macro.com:7051). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available] at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:424) at org.apache.kudu.client.KuduClient.openTable(KuduClient.java:267) at org.apache.seatunnel.connectors.seatunnel.kudu.kuduclient.KuduOutputFormat.init(KuduOutputFormat.java:172) at org.apache.seatunnel.connectors.seatunnel.kudu.kuduclient.KuduOutputFormat.<init>(KuduOutputFormat.java:64) at org.apache.seatunnel.connectors.seatunnel.kudu.sink.KuduSinkWriter.<init>(KuduSinkWriter.java:47) at org.apache.seatunnel.connectors.seatunnel.kudu.sink.KuduSink.createWriter(KuduSink.java:67) at org.apache.seatunnel.connectors.seatunnel.kudu.sink.KuduSink.createWriter(KuduSink.java:39) at org.apache.seatunnel.translation.spark.sink.SparkDataWriterFactory.createDataWriter(SparkDataWriterFactory.java:45) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:113) at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:67) at org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:66) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:244) at org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49) at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:363) at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:352) at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275) at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:329) at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:247) at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145) at org.apache.kudu.client.Connection.cleanup(Connection.java:725) at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:470) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at org.apache.kudu.client.Connection.handleUpstream(Connection.java:239) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) at org.apache.kudu.shaded.org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.apache.kudu.shaded.org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.apache.kudu.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.apache.kudu.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.apache.kudu.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.apache.kudu.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.apache.kudu.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.apache.kudu.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ... 3 more org.apache.seatunnel.connectors.seatunnel.kudu.kuduclient.KuduOutputFormat.init(KuduOutputFormat.java:174) org.apache.seatunnel.connectors.seatunnel.kudu.kuduclient.KuduOutputFormat.<init>(KuduOutputFormat.java:64) org.apache.seatunnel.connectors.seatunnel.kudu.sink.KuduSinkWriter.<init>(KuduSinkWriter.java:47) org.apache.seatunnel.connectors.seatunnel.kudu.sink.KuduSink.createWriter(KuduSink.java:67) org.apache.seatunnel.connectors.seatunnel.kudu.sink.KuduSink.createWriter(KuduSink.java:39) org.apache.seatunnel.translation.spark.sink.SparkDataWriterFactory.createDataWriter(SparkDataWriterFactory.java:45) org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:113) org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:67) org.apache.spark.sql.execution.datasources.v2.WriteToDataSourceV2Exec$$anonfun$doExecute$2.apply(WriteToDataSourceV2Exec.scala:66) org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) org.apache.spark.scheduler.Task.run(Task.scala:121) org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407) org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:138) at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 23/04/01 15:29:41 ERROR scheduler.TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job 23/04/01 15:29:41 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 23/04/01 15:29:41 INFO cluster.YarnScheduler: Cancelling stage 0 23/04/01 15:29:41 INFO cluster.YarnScheduler: Killing all running tasks in stage 0: Stage cancelled 23/04/01 15:29:41 INFO scheduler.DAGScheduler: ResultStage 0 (save at SinkExecuteProcessor.java:85) failed in 48.289 s due to Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, cdh3.macro.com, executor 4): org.apache.spark.util.TaskCompletionListenerException: java.lang.NullPointerException ``` ### Flink or Spark Version 2.4 ### Java or Scala Version scala:2.11.4 java:1.8 ### Screenshots _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
