Hi Dan, Can you verify from the pod that jobmanager and *10.98.253.58:8081 <http://10.98.253.58:8081> *is actually accessible (e.g., with curl)? I'd probably also try out localhost:8081 as you are connecting to the respective pod directly.
On Fri, Sep 11, 2020 at 9:59 PM Dan Hill <quietgol...@gmail.com> wrote: > Hi Robert! > > I have Flink running locally on minikube. I'm running SQL client using > exec on the jobmanager. > > kubectl exec pod/flink-jobmanager-0 -i -t -- /opt/flink/bin/sql-client.sh > embedded -e /opt/flink/sql-client-defaults.yaml > > Here's the sql-client-defaults.yaml. I didn't specify a session. > execution: > type: batch > result-mode: table > max-table-result-rows: 1000000 > > I'm prototyping the Table SQL interface. I got blocked using the Table > SQL interface and figured I'd try the SQL Client to see if I could get > unblocked. > > > On Fri, Sep 11, 2020 at 11:18 AM Robert Metzger <rmetz...@apache.org> > wrote: > >> Hi Dan, >> >> the notation of "flink-jobmanager/10.98.253.58:8081" is not a problem. >> It is how java.net.InetAddress stringifies a resolved address (with both >> hostname and IP). >> >> How did you configure the SQL client to work with a Kubernetes Session? >> Afaik this is not a documented, tested and officially supported feature >> (this doesn't mean we should not support it -- apparently it is something >> we should do rather soon ;) ). >> >> Best, >> Robert >> >> On Fri, Sep 11, 2020 at 5:25 AM Dan Hill <quietgol...@gmail.com> wrote: >> >>> I just tried using the Flink SQL Client. A simple job is not running >>> because it cannot hit jobmanager. I'm not sure why Flink SQL Client is >>> hitting "flink-jobmanager/10.98.253.58:8081". I'd expect either >>> "flink-jobmanager:8081" or "10.98.253.58:8081" (which should work with >>> my kubernetes setup). >>> >>> I'm using riskfocus's Flink helm chart >>> <https://github.com/riskfocus/helm-charts-public/tree/master/flink>. >>> >>> The last SELECT errors out. I have an environment file that indicates >>> this "execution.type: batch". My setup works when using DataSet and >>> DataStream. The jobmanager and taskmanager logs look fine. This seems >>> like a weird configuration with SQL Client that is either broken with that >>> Flink helm chart or with SQL Client. >>> >>> >>> Flink SQL> DROP TABLE `default_catalog.mydb.user`; >>> >>> [INFO] Table has been removed. >>> >>> >>> Flink SQL> CREATE TABLE `default_catalog.mydb.user` (`platformId` >>> BIGINT, `userId` STRING) WITH ('connector' = 'filesystem', 'path' = >>> 's3://mys3bucket/users.csv','format' = 'csv'); >>> >>> [INFO] Table has been created. >>> >>> >>> Flink SQL> SELECT * FROM `default_catalog.mydb.user` LIMIT 10; >>> >>> *[ERROR] Could not execute SQL statement. Reason:* >>> >>> *org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException: >>> connection timed out: flink-jobmanager/10.98.253.58:8081 >>> <http://10.98.253.58:8081>* >>> >>> My config logs from jobmanager. >>> >>> 2020-09-11 02:33:07,962 INFO >>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend >>> [] - Using predefined options: DEFAULT. >>> >>> 2020-09-11 02:33:07,962 INFO >>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend >>> [] - Using default options factory: >>> DefaultConfigurableOptionsFactory{configuredOptions={}}. >>> >>> 2020-09-11 02:33:08,100 INFO >>> org.apache.flink.table.client.gateway.local.ProgramDeployer [] - >>> Submitting job org.apache.flink.streaming.api.graph.StreamGraph@3bd08435 >>> for query default: SELECT * FROM `default_catalog.mydb.user` LIMIT 10` >>> >>> 2020-09-11 02:33:08,112 INFO org.apache.flink.configuration.Configuration >>> [] - Config uses fallback configuration key >>> 'jobmanager.rpc.address' instead of key 'rest.address' >>> >>> tail: log/flink--sql-client-flink-jobmanager-0.log: file truncated >>> >>> 2020-09-11 02:34:35,848 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> taskmanager.numberOfTaskSlots, 2 >>> >>> 2020-09-11 02:34:35,884 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: blob.server.port, 6124 >>> >>> 2020-09-11 02:34:35,884 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: taskmanager.rpc.port, >>> 6122 >>> >>> 2020-09-11 02:34:35,885 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: jobmanager.heap.size, 1g >>> >>> 2020-09-11 02:34:35,886 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> taskmanager.memory.process.size, 1g >>> >>> 2020-09-11 02:34:35,886 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: state.backend, rocksdb >>> >>> 2020-09-11 02:34:35,886 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: state.checkpoints.dir, >>> file:///flink_state/checkpoints >>> >>> 2020-09-11 02:34:35,887 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: state.savepoints.dir, >>> file:///flink_state/savepoints >>> >>> 2020-09-11 02:34:35,887 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: state.backend.async, true >>> >>> 2020-09-11 02:34:35,887 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.fs.memory-threshold, 1024 >>> >>> 2020-09-11 02:34:35,888 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.fs.write-buffer-size, 4096 >>> >>> 2020-09-11 02:34:35,888 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.incremental, true >>> >>> 2020-09-11 02:34:35,888 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.local-recovery, true >>> >>> 2020-09-11 02:34:35,889 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.checkpoints.num-retained, 1 >>> >>> 2020-09-11 02:34:35,889 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> taskmanager.state.local.root-dirs, file:///flink_state/local-recovery >>> >>> 2020-09-11 02:34:35,889 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.checkpoint.transfer.thread.num, 1 >>> >>> 2020-09-11 02:34:35,890 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.localdir, /flink_state/rocksdb >>> >>> 2020-09-11 02:34:35,890 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.options-factory, >>> org.apache.flink.contrib.streaming.state.DefaultConfigurableOptionsFactory >>> >>> 2020-09-11 02:34:35,890 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.predefined-options, DEFAULT >>> >>> 2020-09-11 02:34:35,891 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.timer-service.factory, HEAP >>> >>> 2020-09-11 02:34:35,891 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.ttl.compaction.filter.enabled, false >>> >>> 2020-09-11 02:34:35,892 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: jobmanager.rpc.address, >>> flink-jobmanager >>> >>> 2020-09-11 02:34:35,892 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: jobmanager.rpc.port, 6123 >>> >>> 2020-09-11 02:34:35,892 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> taskmanager.memory.jvm-metaspace.size, 256mb >>> >>> 2020-09-11 02:34:35,893 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: s3a.endpoint, >>> http://minio:9000 >>> >>> 2020-09-11 02:34:35,893 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: s3a.path.style.access, >>> true >>> >>> 2020-09-11 02:34:35,894 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: s3a.access-key, >>> YOURACCESSKEY >>> >>> 2020-09-11 02:34:35,894 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: s3a.secret-key, ****** >>> >>> 2020-09-11 02:34:35,895 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> s3a.aws.credentials.provider, >>> org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider >>> >>> 2020-09-11 02:34:36,492 INFO org.apache.flink.client.cli.CliFrontend >>> [] - Loading FallbackYarnSessionCli >>> >>> 2020-09-11 02:34:36,498 INFO >>> org.apache.flink.table.client.gateway.local.LocalExecutor [] - Using >>> default environment file: file:/opt/flink/conf/sql-client-defaults.yaml >>> >>> 2020-09-11 02:34:37,786 INFO org.apache.flink.table.client.SqlClient >>> [] - Using session environment file: >>> file:/opt/flink/sql-client-defaults.yaml >>> >>> 2020-09-11 02:34:37,931 INFO >>> org.apache.flink.table.client.config.entries.ExecutionEntry [] - >>> Property 'execution.restart-strategy.type' not specified. Using default >>> value: fallback >>> >>> 2020-09-11 02:34:43,269 INFO >>> org.apache.flink.table.client.gateway.local.ExecutionContext >>> [] - Executor config: {blob.server.port=6124, s3a.secret-key=YOURSECRETKEY, >>> state.checkpoints.num-retained=1, s3a.access-key=YOURACCESSKEY, >>> state.backend.rocksdb.options-factory=org.apache.flink.contrib.streaming.state.DefaultConfigurableOptionsFactory, >>> jobmanager.rpc.address=flink-jobmanager, >>> taskmanager.memory.jvm-metaspace.size=256mb, >>> state.backend.rocksdb.predefined-options=DEFAULT, >>> state.savepoints.dir=file:///flink_state/savepoints, >>> state.backend.fs.memory-threshold=1024, >>> state.backend.rocksdb.timer-service.factory=HEAP, >>> execution.savepoint.ignore-unclaimed-state=false, >>> taskmanager.numberOfTaskSlots=2, state.backend.async=true, >>> pipeline.classpaths=[], >>> state.backend.rocksdb.ttl.compaction.filter.enabled=false, >>> jobmanager.heap.size=1g, taskmanager.memory.process.size=1g, s3a.endpoint= >>> http://minio:9000, >>> state.backend.rocksdb.checkpoint.transfer.thread.num=1, >>> state.backend.local-recovery=true, >>> state.backend.rocksdb.localdir=/flink_state/rocksdb, >>> state.backend.incremental=true, execution.target=remote, >>> taskmanager.rpc.port=6122, jobmanager.rpc.port=6123, >>> s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider, >>> state.backend.fs.write-buffer-size=4096, execution.attached=true, >>> execution.shutdown-on-attached-exit=false, >>> pipeline.jars=[file:/opt/flink/opt/flink-sql-client_2.12-1.11.1.jar], >>> state.backend=rocksdb, s3a.path.style.access=true, >>> taskmanager.state.local.root-dirs=file:///flink_state/local-recovery, >>> state.checkpoints.dir=file:///flink_state/checkpoints} >>> >>> 2020-09-11 02:34:43,286 INFO >>> org.apache.flink.client.deployment.DefaultClusterClientServiceLoader >>> [] - Could not load factory due to missing dependencies. >>> >>> 2020-09-11 02:34:44,079 INFO >>> org.apache.flink.table.client.cli.CliClient [] - >>> Command history file path: /root/.flink-sql-history >>> >>> 2020-09-11 02:35:15,396 INFO >>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend >>> [] - Using predefined options: DEFAULT. >>> >>> 2020-09-11 02:35:15,397 INFO >>> org.apache.flink.contrib.streaming.state.RocksDBStateBackend >>> [] - Using default options factory: >>> DefaultConfigurableOptionsFactory{configuredOptions={}}. >>> >>> 2020-09-11 02:35:17,529 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> taskmanager.numberOfTaskSlots, 2 >>> >>> 2020-09-11 02:35:17,530 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: blob.server.port, 6124 >>> >>> 2020-09-11 02:35:17,530 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: taskmanager.rpc.port, >>> 6122 >>> >>> 2020-09-11 02:35:17,530 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: jobmanager.heap.size, 1g >>> >>> 2020-09-11 02:35:17,530 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> taskmanager.memory.process.size, 1g >>> >>> 2020-09-11 02:35:17,530 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: state.backend, rocksdb >>> >>> 2020-09-11 02:35:17,530 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: state.checkpoints.dir, >>> file:///flink_state/checkpoints >>> >>> 2020-09-11 02:35:17,531 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: state.savepoints.dir, >>> file:///flink_state/savepoints >>> >>> 2020-09-11 02:35:17,531 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: state.backend.async, true >>> >>> 2020-09-11 02:35:17,531 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.fs.memory-threshold, 1024 >>> >>> 2020-09-11 02:35:17,531 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.fs.write-buffer-size, 4096 >>> >>> 2020-09-11 02:35:17,531 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.incremental, true >>> >>> 2020-09-11 02:35:17,531 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.local-recovery, true >>> >>> 2020-09-11 02:35:17,532 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.checkpoints.num-retained, 1 >>> >>> 2020-09-11 02:35:17,532 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> taskmanager.state.local.root-dirs, file:///flink_state/local-recovery >>> >>> 2020-09-11 02:35:17,532 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.checkpoint.transfer.thread.num, 1 >>> >>> 2020-09-11 02:35:17,532 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.localdir, /flink_state/rocksdb >>> >>> 2020-09-11 02:35:17,532 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.options-factory, >>> org.apache.flink.contrib.streaming.state.DefaultConfigurableOptionsFactory >>> >>> 2020-09-11 02:35:17,532 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.predefined-options, DEFAULT >>> >>> 2020-09-11 02:35:17,532 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.timer-service.factory, HEAP >>> >>> 2020-09-11 02:35:17,533 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> state.backend.rocksdb.ttl.compaction.filter.enabled, false >>> >>> 2020-09-11 02:35:17,533 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: jobmanager.rpc.address, >>> flink-jobmanager >>> >>> 2020-09-11 02:35:17,533 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: jobmanager.rpc.port, 6123 >>> >>> 2020-09-11 02:35:17,533 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> taskmanager.memory.jvm-metaspace.size, 256mb >>> >>> 2020-09-11 02:35:17,533 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: s3a.endpoint, >>> http://minio:9000 >>> >>> 2020-09-11 02:35:17,534 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: s3a.path.style.access, >>> true >>> >>> 2020-09-11 02:35:17,534 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: s3a.access-key, >>> YOURACCESSKEY >>> >>> 2020-09-11 02:35:17,534 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: s3a.secret-key, ****** >>> >>> 2020-09-11 02:35:17,535 INFO >>> org.apache.flink.configuration.GlobalConfiguration >>> [] - Loading configuration property: >>> s3a.aws.credentials.provider, >>> org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider >>> >>> 2020-09-11 02:35:18,047 INFO >>> org.apache.flink.table.client.gateway.local.ProgramDeployer [] - >>> Submitting job org.apache.flink.streaming.api.graph.StreamGraph@2f95653f >>> for query default: SELECT * FROM `default_catalog.mydb.user` LIMIT 10` >>> >>> 2020-09-11 02:35:18,382 INFO org.apache.flink.configuration.Configuration >>> [] - Config uses fallback configuration key >>> 'jobmanager.rpc.address' instead of key 'rest.address' >>> >>> 2020-09-11 02:41:39,539 WARN org.apache.flink.runtime.rest.RestClient >>> [] - Rest endpoint shutdown failed. >>> >>> java.util.concurrent.TimeoutException: null >>> >> -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
