Java 21 support and apache/flink docker images

2024-11-06 Thread Bjarke Tornager
Hi all, What is the status of Apache Flink Java 21 support and apache/flink images? I am currently using Java 17 in production but would like to update to Java 21 if possible. For the 1.19 release I can see that it was announced that Java 21 has beta support. However, it looks like the associated

Re: Reminder: Help required to fix security vulnerabilities in Flink Docker image

2024-07-02 Thread elakiya udhayanan
4 at 10:12 PM Gabor Somogyi > wrote: > >> Hi Elakiya, >> >> I've just double checked the story and seems like the latest 1.17 gosu >> release is not vulnerable. >> Can you please try it out on your side? Alexis has written down how you >> can bump the docke

Re: Reminder: Help required to fix security vulnerabilities in Flink Docker image

2024-06-23 Thread elakiya udhayanan
and seems like the latest 1.17 gosu > release is not vulnerable. > Can you please try it out on your side? Alexis has written down how you > can bump the docker version locally: > > ---CUT-HERE--- > ENV GOSU_VERSION 1.17 > ---CUT-HERE--- > > Please report back and

Re: Reminder: Help required to fix security vulnerabilities in Flink Docker image

2024-06-23 Thread Gabor Somogyi
Hi Elakiya, I've just double checked the story and seems like the latest 1.17 gosu release is not vulnerable. Can you please try it out on your side? Alexis has written down how you can bump the docker version locally: ---CUT-HERE--- ENV GOSU_VERSION 1.17 ---CUT-HERE--- Please report bac

Re: Reminder: Help required to fix security vulnerabilities in Flink Docker image

2024-06-21 Thread Alexis Sarda-Espinosa
Hi Elakiya, just to be clear, I'm not a Flink maintainer, but here my 2 cents. I imagine the issues related to Go come from 'gosu', which is installed in the official Flink Docker images. You can see [1] for some thoughts from the gosu maintainer regarding CVEs (and the m

Reminder: Help required to fix security vulnerabilities in Flink Docker image

2024-06-21 Thread elakiya udhayanan
Hi Team, I would like to remind about the request for the help required to fix the vulnerabilities seen in the Flink Docker image. Any help is appreciated. Thanks in advance. Thanks, Elakiya U On Tue, Jun 18, 2024 at 12:51 PM elakiya udhayanan wrote: > Hi Community, > > In o

Help required to fix security vulnerabilities in Flink Docker image

2024-06-18 Thread elakiya udhayanan
Hi Community, In one of our applications we are using a Fink Docker image and running Flink as a Kubernetes pod. As per policy, we tried scanning the Docker image for security vulnerabilities using JFrog XRay and we find that there are multiple critical vulnerabilities being reported as seen in

Re: ARM flink docker image

2024-03-14 Thread Sharil Shafie
The docker image is build for both linux/amd64 or linux/arm64/v8 platform. The image you'll pull is based on the platform or the flag you set. Regards. On Fri, 15 Mar 2024, 2:01 am Yang LI, wrote: > Dear Flink Community, > > Do you know if we have somewhere a tested ARM base

ARM flink docker image

2024-03-14 Thread Yang LI
Dear Flink Community, Do you know if we have somewhere a tested ARM based flink docker image? I think we can already run locally on an ARM macbook. But we don't have a ARM specified docker image yet. Regards, Yang LI

Re: Flink HA with Zookeeper and Docker Compose: unable to startup a working setup.

2024-01-15 Thread Yang Wang
wrote: > Hello, > i'm trying to setup a testing environment using: > > - Flink HA with Zookeeper > - Docker Compose > > While starting the TaskManager generates an exception and then after some > restarts if fails. > > The exception is: > "Caused by: org.apache.f

Flink HA with Zookeeper and Docker Compose: unable to startup a working setup.

2023-12-29 Thread Alessio Bernesco Làvore
Hello, i'm trying to setup a testing environment using: - Flink HA with Zookeeper - Docker Compose While starting the TaskManager generates an exception and then after some restarts if fails. The exception is: "Caused by: org.apache.flink.runtime.rpc.exceptions.FencingTokenExceptio

Re: Local process and Docker have different behavior

2023-08-11 Thread liu ron
ty, > > When I run Flink locally in my test cases on my Mac, I observe different > behavior compared to running it in my Docker-backed build instance or using > the official Docker-compose image. The processes complete as expected when > i run it in-process but not always whe

Local process and Docker have different behavior

2023-08-10 Thread Daniel Henneberger
Dear Apache Flink community, When I run Flink locally in my test cases on my Mac, I observe different behavior compared to running it in my Docker-backed build instance or using the official Docker-compose image. The processes complete as expected when i run it in-process but not always when i

Re: Failed docker-entrypoint.sh in Flink-Operator Logs

2023-04-05 Thread Weihua Hu
Hi, That's because the ConfigMap volume is always read-only. Currently /docker-entrypoint.sh will try to update some configs in docker Environment. But these are not needed in kubernetes. So I think we can ignore those errors safely when using the operator/native kubernetes integration. The

Failed docker-entrypoint.sh in Flink-Operator Logs

2023-04-04 Thread Ivan Webber via user
flink/conf/sedcDS30D: Read-only file system /docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file system /docker-entrypoint.sh: line 89: /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system Starting kubernetes-application as a console application on host test-replay-run

Docker Flink Official Image with Docker Swarm cannot register TaskManager to JobManager's Resource Manager

2023-03-29 Thread chyp long
Hello, I am having trouble when deploying Apache Flink 1.16.1 on 2 Google Cloud instances with Docker Swarm. The JobManager is deployed on the manager node and the TaskManager is deployed in the worker node. The TaskManager seems to have trouble to communicate with ResourceManager on JobManager

apache/flink docker images arm64

2023-02-10 Thread Roberts, Ben (Senior Developer) via user
Hi, Would it be possible for the arm64/v8 architecture images to be published to dockerhub apache/flink:1.16 and 1.16.1 please? I’m aware that the official docker flink image is now published in the arm64 arch, but that image doesn’t include a JDK, so it’d be super helpful to have the apache

Flink 1.16 arm64/v8 docker images

2023-01-31 Thread Roberts, Ben (Senior Developer) via user
Hi, Is it possible for the arm64/v8 architecture images to be published for >1.16 flink-docker (apache/flink)? I’m aware that the official docker flink image is now published in the arm64 arch, but that image doesn’t include a JDK, so it’d be super helpful to have the apache/flink ima

Re: Docker image Flink 1.15.4

2023-01-26 Thread Chesnay Schepler
1.15.4 is not released yet. On 26/01/2023 16:06, Peng Zhang wrote: Hi, We would like to use Flink 1.15.4 docker image. The latest seems 1.15.3. Could you make a docker release Flink 1.1.5.4? Thanks! There is a blocking bug https://issues.apache.org/jira/browse/FLINK-28695 <ht

Docker image Flink 1.15.4

2023-01-26 Thread Peng Zhang
Hi, We would like to use Flink 1.15.4 docker image. The latest seems 1.15.3. Could you make a docker release Flink 1.1.5.4? Thanks! There is a blocking bug https://issues.apache.org/jira/browse/FLINK-28695 in 1.5.3 and fixed in 1.15.4 BR, Peng

Re: [EXTERNAL] Re: Flink 1.15.3 Docker image

2022-11-30 Thread Roberts, Ben (Senior Developer) via user
That’s great, thanks Márton! -- Ben Roberts From: Márton Balassi Date: Tuesday, 29 November 2022 at 18:55 To: Roberts, Ben (Senior Developer) Cc: user@flink.apache.org Subject: [EXTERNAL] Re: Flink 1.15.3 Docker image Done, please let me know if you see anything unexpected. On Tue, Nov 29

Re: Flink 1.15.3 Docker image

2022-11-29 Thread Márton Balassi
Done, please let me know if you see anything unexpected. On Tue, Nov 29, 2022 at 7:07 PM Márton Balassi wrote: > Hi Ben, > > Thanks for reaching out. Since the image repo has been updated [1] I can > pick this up. Will let you know when done. > > [1] > https://github.co

Re: Flink 1.15.3 Docker image

2022-11-29 Thread Márton Balassi
Hi Ben, Thanks for reaching out. Since the image repo has been updated [1] I can pick this up. Will let you know when done. [1] https://github.com/apache/flink-docker/commit/a22c0f04972a1d8539d9213b52fc0728eac8c1fa On Tue, Nov 29, 2022 at 4:28 PM Roberts, Ben (Senior Developer) via user < u

Flink 1.15.3 Docker image

2022-11-29 Thread Roberts, Ben (Senior Developer) via user
Hi, Is there anyone able to publish the apache/flink docker image for release 1.15.3? I see that the release was announced on 2022-11-10, and the PR for the 1.15.3 dockerfiles (linked: https://github.com/apache/flink-docker/pull/140) has been approved and merged on 2022-11-25, but it looks

Re: Flink SQL on Docker

2022-11-01 Thread Tristan Culp via user
Hi, sorry about the late response. Currently the sql client queries connect to a Kafka topic to read events, so I don't think it'll be reproducible. I'm using Flink 1.15, and running it in a Docker image on Mac OS with M1 chip. I also noticed that multiple queries in a sql

Re: Flink SQL on Docker

2022-10-24 Thread Shengkai Fang
, Shengkai Tristan Culp via user 于2022年10月25日周二 05:30写道: > Hello all, > > I've been working with the Flink SQL Client lately, and have been trying > to use it on a Docker image. I've noticed that when it runs on an image, > however, it performs much slower and after 1 or 2 sql

Flink SQL on Docker

2022-10-24 Thread Tristan Culp via user
Hello all, I've been working with the Flink SQL Client lately, and have been trying to use it on a Docker image. I've noticed that when it runs on an image, however, it performs much slower and after 1 or 2 sql queries, stops working altogether, leading me to have to restart the SQL cl

Re: [NOTICE] Switch docker image base to Eclipse Temurin

2022-09-08 Thread Chesnay Schepler
At first glance this might happen when an older docker version is used: https://github.com/adoptium/temurin-build/issues/2974 You may need to upgrade to Docker 20.10.5+. On 08/09/2022 12:33, Sigalit Eliazov wrote: Hi all, We pulled the new image and we are facing an issue to start the job

Re: [NOTICE] Switch docker image base to Eclipse Temurin

2022-09-08 Thread Sigalit Eliazov
d not help any guidance will be appreciated Thanks Sigalit On Mon, Sep 5, 2022 at 1:21 PM Chesnay Schepler wrote: > * September 7th > > On 05/09/2022 11:27, Chesnay Schepler wrote: > > On Wednesday, September 9th, the Flink 1.14.5/1.15.2 Docker images > > will switch base

Re: [NOTICE] Switch docker image base to Eclipse Temurin

2022-09-05 Thread Chesnay Schepler
* September 7th On 05/09/2022 11:27, Chesnay Schepler wrote: On Wednesday, September 9th, the Flink 1.14.5/1.15.2 Docker images will switch bases FROM openjdk:8/11-jar (Debian-based) TO eclipse-temurin:8/11-jre-jammy (Ubuntu-based) due to the deprecation of the OpenJDK images. Users that

[NOTICE] Switch docker image base to Eclipse Temurin

2022-09-05 Thread Chesnay Schepler
On Wednesday, September 9th, the Flink 1.14.5/1.15.2 Docker images will switch bases FROM openjdk:8/11-jar (Debian-based) TO eclipse-temurin:8/11-jre-jammy (Ubuntu-based) due to the deprecation of the OpenJDK images. Users that customized the images are advised to check for breaking changes

Re: Needed help with skipping savepoint state (unsure how to set --allowNonRestoredState in Docker)

2022-06-07 Thread Chesnay Schepler
You are on the right path with using the --allowNonRestoredState flag; we'll just have to find the right place to put it w.r.t. your setup. Which docker images are you using (flink/statefun/something custom), and how do you submit the job? On 03/06/2022 01:17, Bhavani Balasubramanyam

Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)

2022-03-30 Thread 胡伟华
do some wiping >>> and metadata writing from the job submitting flink program before it >>> actually executes/submits the job to the job manager. when i don't do this >>> redis preparation, the redis sink works completely fine. that is, the >>

Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)

2022-03-30 Thread Jin Yi
is >>>> redis preparation, the redis sink works completely fine. that is, the >>>> redis commands work fine from the taskmanager tasks. however, if i enable >>>> the option redis preparation from within the flink job program, it fails to >>>> contact redis and hits a timeout exception. >>>> >>>> previously, we were using 1.12.3, and this behavior worked fine. is >>>> the 1.14.4 flink docker image pretty restrictive when it comes to network >>>> access at the job submission client layer? >>>> >>>> thanks. >>>> >>> >> >

Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)

2022-03-29 Thread 胡伟华
s the job to the job manager. when i don't do this >> redis preparation, the redis sink works completely fine. that is, the redis >> commands work fine from the taskmanager tasks. however, if i enable the >> option redis preparation from within the flink job program, it

Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)

2022-03-29 Thread Jin Yi
tasks. however, if i enable >>> the option redis preparation from within the flink job program, it fails to >>> contact redis and hits a timeout exception. >>> >>> previously, we were using 1.12.3, and this behavior worked fine. is the >>> 1.14.4 flink docker image pretty restrictive when it comes to network >>> access at the job submission client layer? >>> >>> thanks. >>> >> >

Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)

2022-03-29 Thread 胡伟华
nk job program, it fails to > contact redis and hits a timeout exception. > > previously, we were using 1.12.3, and this behavior worked fine. is the > 1.14.4 flink docker image pretty restrictive when it comes to network access > at the job submission client layer? > > thanks.

Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)

2022-03-29 Thread Jin Yi
is and hits a timeout exception. >> >> previously, we were using 1.12.3, and this behavior worked fine. is the >> 1.14.4 flink docker image pretty restrictive when it comes to network >> access at the job submission client layer? >> >> thanks. >> >

Re: flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)

2022-03-29 Thread huweihua
ob program, it fails to > contact redis and hits a timeout exception. > > previously, we were using 1.12.3, and this behavior worked fine. is the > 1.14.4 flink docker image pretty restrictive when it comes to network > access at the job submission client layer? > > thanks. >

flink docker image (1.14.4) unable to access other pods from flink program (job and task manager access is fine)

2022-03-28 Thread Jin Yi
redis commands work fine from the taskmanager tasks. however, if i enable the option redis preparation from within the flink job program, it fails to contact redis and hits a timeout exception. previously, we were using 1.12.3, and this behavior worked fine. is the 1.14.4 flink docker image p

Re: flink-playground docker/mvn clean install Unknown host repo.maven.apache.org: Name or service not known

2021-12-21 Thread David Morávek
act, that even with >> this system property, some custom java HTTP clients, may simply decide to >> ignore this... :) >> >> does it explain your concerns? >> >> [1] >> https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html >> >> D

Re: flink-playground docker/mvn clean install Unknown host repo.maven.apache.org: Name or service not known

2021-12-21 Thread HG
fact, that even with > this system property, some custom java HTTP clients, may simply decide to > ignore this... :) > > does it explain your concerns? > > [1] > https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html > > D. > > On Tue, Dec 21, 202

Re: flink-playground docker/mvn clean install Unknown host repo.maven.apache.org: Name or service not known

2021-12-21 Thread David Morávek
oracle.com/javase/8/docs/technotes/guides/net/proxies.html D. On Tue, Dec 21, 2021 at 11:17 AM HG wrote: > Hi David > When I start a docker container based on the image that is created shortly > before the mvn command fails, it is resolvable. > I can do : apt update > This basical

Re: flink-playground docker/mvn clean install Unknown host repo.maven.apache.org: Name or service not known

2021-12-21 Thread HG
Hi David When I start a docker container based on the image that is created shortly before the mvn command fails, it is resolvable. I can do : apt update This basically means that the outside world is reachable When I install wget the following succeeds without any issue: wget https

Re: flink-playground docker/mvn clean install Unknown host repo.maven.apache.org: Name or service not known

2021-12-21 Thread David Morávek
Hello Hans, it's DNS ;) You need to make sure, that "repo.maven.apache.org" can be resolved from your docker container (you can use tools such as host, dig, nslookup to verify that). This is may be tricky to debug, unless you're familiar with networking. A good place to sta

flink-playground docker/mvn clean install Unknown host repo.maven.apache.org: Name or service not known

2021-12-20 Thread HG
Hello I am trying to the flink-playground examples. The docker-compose build fails on the mvn clean install command . I am behind a proxy. To diagnose this I started a container based on the already created image docker run -it --name my-maven-project -v "$(pwd)":/usr/src/mymaven -

Re: Flink docker on k8s job submission timeout

2021-11-11 Thread Chesnay Schepler
You should be able to increase the timeout by setting client.timeout. On 10/11/2021 15:32, dhanesh arole wrote: Hello all, We are trying to run a Flink job in standalone mode using the official docker image on k8s. As per this documentation <https://ci.apache.org/projects/flink/flink-d

Flink docker on k8s job submission timeout

2021-11-10 Thread dhanesh arole
Hello all, We are trying to run a Flink job in standalone mode using the official docker image on k8s. As per this documentation <https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#advanced-customization> we have created our custom

Re: Application mode - Custom Flink docker image with Python user code

2021-10-26 Thread Dian Fu
any location and invoked? > > Thanks, > Sumeet > > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/docker/#enabling-python > [2] > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#application-mode >

Re: Application mode - Custom Flink docker image with Python user code

2021-10-26 Thread Shuiqiang Chen
Hi Sumeet, Actually, running pyflink jobs in application mode on kubernetes has been supported since release 1.13. To build a docker image with PyFlink installed, please refer to Enabling Python[1]. In order to run the python code in application mode, you also need to COPY the code files into

Application mode - Custom Flink docker image with Python user code

2021-10-26 Thread Sumeet Malhotra
at any location and invoked? Thanks, Sumeet [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/docker/#enabling-python [2] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#application-mode

Re: How can I build the flink docker image from source code?

2021-08-19 Thread Caizhi Weng
tle bit code in flink, and want to build a docker image > to test it. Could you tell me how can I build the image from source code? > > > > Thx! >

How can I build the flink docker image from source code?

2021-08-19 Thread Chenyu Zheng
Hi contributors, I’ve changed a little bit code in flink, and want to build a docker image to test it. Could you tell me how can I build the image from source code? Thx!

Re: [table-walkthrough] docker-compose build issues

2021-06-11 Thread Lingfeng Pu
Hi, I apologize that I forgot the attachments in my last post. I'll repost my question with attachments this time: *I have successfully run the project "table-walkthrough" on IDEA (w/t errors but warnings)*, *I'm now trying to build this project by using the "docker

[table-walkthrough] docker-compose build issues

2021-06-10 Thread Lingfeng Pu
Hi, *I have successfully run the project "table-walkthrough" on IDEA (w/t errors but warnings)*, *I'm now trying to build this project by using the "docker-compose" command* as the tutorial does. However, when I run the "docker-compose build" command in the dire

Re: Flink 1.12.4 docker image

2021-05-26 Thread Arvid Heise
2021 at 3:08 PM Arvid Heise wrote: >> >>> Hi Nikola, >>> >>> I'm looking into it. I might have missed a step during release. >>> >>> Best, >>> >>> Arvid >>> >>> On Mon, May 24, 2021 at 3:47 PM Nikola H

Re: Flink 1.12.4 docker image

2021-05-26 Thread Nikola Hrusov
n Mon, May 24, 2021 at 3:47 PM Nikola Hrusov wrote: >> >>> Hello, >>> >>> I saw that flink 1.12.4 just got released. However I am struggling to >>> find the docker image. >>> >>> I checked both: >>> - https://hub.docker.com/_/

Re: Flink 1.12.4 docker image

2021-05-25 Thread Arvid Heise
. I might have missed a step during release. > > Best, > > Arvid > > On Mon, May 24, 2021 at 3:47 PM Nikola Hrusov wrote: > >> Hello, >> >> I saw that flink 1.12.4 just got released. However I am struggling to >> find the docker image. >>

Re: Flink 1.12.4 docker image

2021-05-25 Thread Arvid Heise
Hi Nikola, I'm looking into it. I might have missed a step during release. Best, Arvid On Mon, May 24, 2021 at 3:47 PM Nikola Hrusov wrote: > Hello, > > I saw that flink 1.12.4 just got released. However I am struggling to find > the docker image. > > I

Flink 1.12.4 docker image

2021-05-24 Thread Nikola Hrusov
Hello, I saw that flink 1.12.4 just got released. However I am struggling to find the docker image. I checked both: - https://hub.docker.com/_/flink - https://hub.docker.com/r/apache/flink but on both 1.12.4 is not available. Are there plans to publish it as a docker image? Regards , Nikola

Re: docker based taskmanager can't connect to job/resource manager

2021-05-12 Thread Guowei Ma
Hi, Here is the link[1]. [image: image.png] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#starting-a-session-cluster-on-docker Best, Guowei On Thu, May 13, 2021 at 1:53 PM guenterh.lists wrote: > Hi Guowei, > > thanks

Re: docker based taskmanager can't connect to job/resource manager

2021-05-12 Thread guenterh.lists
seems that you might need add the "jobmanager.rpc.address: jobmanager" to the FLINK_PROPERTIES before creating a network. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/ <https://ci.apache.org/projects/flink/flin

Re: docker based taskmanager can't connect to job/resource manager

2021-05-12 Thread Guowei Ma
Hi, I do not try it. But from the documentation[1] it seems that you might need add the "jobmanager.rpc.address: jobmanager" to the FLINK_PROPERTIES before creating a network. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/doc

docker based taskmanager can't connect to job/resource manager

2021-05-12 Thread guenterh.lists
Hi, I'm trying to start a mini cluster following the explanations given in a flink forward presentation [1] Starting a jobmanager task is possible FLINK_PROPERTIES="jobmanager.memory.process.size: 2048m parallelism.default: 4 " docker network create flink-network doc

Re: Flink Hadoop config on docker-compose

2021-04-22 Thread Matthias Pohl
nk-dist/src/main/flink-bin/bin/config.sh#L256 >> [2]. >> https://github.com/apache/flink/blob/release-1.11/flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapred/utils/HadoopUtils.java#L64 >> >> >> Best, >> Yang >>

Re: Flink Hadoop config on docker-compose

2021-04-22 Thread Flavio Pompermaier
t;> [2]. >>> https://github.com/apache/flink/blob/release-1.11/flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapred/utils/HadoopUtils.java#L64 >>> >>> >>> Best, >>> Yang >>> >>> Best,

Re: Flink Hadoop config on docker-compose

2021-04-16 Thread Flavio Pompermaier
> https://github.com/apache/flink/blob/release-1.11/flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapred/utils/HadoopUtils.java#L64 > > > Best, > Yang > > Best, > Yang > > Flavio Pompermaier 于2021年4月16日周五 上午3:55写道: >

Re: Flink Hadoop config on docker-compose

2021-04-15 Thread Yang Wang
pred/utils/HadoopUtils.java#L64 Best, Yang Best, Yang Flavio Pompermaier 于2021年4月16日周五 上午3:55写道: > Hi Robert, > indeed my docker-compose does work only if I add also Hadoop and yarn home > while I was expecting that those two variables were generated automatically > just setting env.xxx variabl

Re: Flink Hadoop config on docker-compose

2021-04-15 Thread Flavio Pompermaier
Hi Robert, indeed my docker-compose does work only if I add also Hadoop and yarn home while I was expecting that those two variables were generated automatically just setting env.xxx variables in FLINK_PROPERTIES variable.. I just want to understand what to expect, if I really need to specify

Re: Flink Hadoop config on docker-compose

2021-04-15 Thread Robert Metzger
Hi, I'm not aware of any known issues with Hadoop and Flink on Docker. I also tried what you are doing locally, and it seems to work: flink-jobmanager| 2021-04-15 18:37:48,300 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Starting StandaloneSessionClusterEntry

Flink Hadoop config on docker-compose

2021-04-14 Thread Flavio Pompermaier
Hi everybody, I'm trying to set up reading from HDFS using docker-compose and Flink 1.11.3. If I pass 'env.hadoop.conf.dir' and 'env.yarn.conf.dir' using FLINK_PROPERTIES (under environment section of the docker-compose service) I see in the logs the following line

Re: Flink docker 1.11.3 actually runs 1.11.2

2021-04-14 Thread Chesnay Schepler
Works properly for me. I think your suspicion about the .env is correct; it is probably not considered when checking whether something has changed, so docker juse re-uses the previous image. On 4/13/2021 9:51 PM, Flavio Pompermaier wrote: Hi Chesnay, my tests were done using docker-compose

Re: Flink docker 1.11.3 actually runs 1.11.2

2021-04-13 Thread Flavio Pompermaier
Hi Chesnay, my tests were done using docker-compose (with the command 'docker-compose up --build -d flink-jobmanager flink-taskmanager'). These are the necessary files (./flink/db-libs/* contains the jdbc libraries I use while /opt/flink/data is used as a volume to share files with oth

Re: Flink docker 1.11.3 actually runs 1.11.2

2021-04-13 Thread Chesnay Schepler
.11.3 like they are supposed to do. On 4/13/2021 6:31 PM, Flavio Pompermaier wrote: Hi to all, I've just build a docker that use the image flink:1.11.3-scala_2.12-java11 but the web UI (and logs too) display Flink 1.11.2 (Commit: fe36135). Was there an error with the release? Best, Flavio

Flink docker 1.11.3 actually runs 1.11.2

2021-04-13 Thread Flavio Pompermaier
Hi to all, I've just build a docker that use the image flink:1.11.3-scala_2.12-java11 but the web UI (and logs too) display Flink 1.11.2 (Commit: fe36135). Was there an error with the release? Best, Flavio

Re: [EXTERNAL] Re: 1.12.2 docker image

2021-03-04 Thread Yang Wang
e. > > > > I’m seeing the following: > > ``` > > ➜ docker pull flink:1.12.2 > > 1.12.2: Pulling from library/flink > > no matching manifest for linux/amd64 in the manifest list entries > > ``` > > > > Best > > kevin > > > > *From:

Re: [EXTERNAL] Re: 1.12.2 docker image

2021-03-04 Thread Bohinski, Kevin
Hi, Actually that seemed to use a cached 1.12.1 image. I’m seeing the following: ``` ➜ docker pull flink:1.12.2 1.12.2: Pulling from library/flink no matching manifest for linux/amd64 in the manifest list entries ``` Best kevin From: "Bohinski, Kevin" Date: Thursday, March 4, 2021

Re: [EXTERNAL] Re: 1.12.2 docker image

2021-03-04 Thread Bohinski, Kevin
Hi, I see the images on docker hub. I tried to launch a k8s session with the following options, and I am getting the following logs: ``` -Dkubernetes.container.image=flink:1.12-scala_2.12-java8 \ -Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-1.12.2.jar

Re: 1.12.2 docker image

2021-03-03 Thread Chesnay Schepler
they should be released in a day or two. On 3/3/2021 11:18 PM, Bohinski, Kevin wrote: Hi, Are there plans to provide a docker image for 1.12.2? Best kevin

1.12.2 docker image

2021-03-03 Thread Bohinski, Kevin
Hi, Are there plans to provide a docker image for 1.12.2? Best kevin

Re: Flink docker in session cluster mode - is a local distribution needed?

2021-02-17 Thread Yang Wang
I am not aware of some simple solution we could use the Flink runtime jars within the docker images, except for "docker run/exec". So if we want to provide some easy commands to submit Flink jobs, I think they are also a wrapper of "docker run/exec". Best, Yang Till Rohrman

Re: Flink docker in session cluster mode - is a local distribution needed?

2021-02-17 Thread Till Rohrmann
Yes, agreed. This could be better streamlined. If you wanna help with this, then feel free to open a JIRA issue for it. Cheers, Till On Wed, Feb 17, 2021 at 11:37 AM Manas Kale wrote: > Hi Till, > Oh I see... I managed to do what you said using a bunch of docker exec > commands. H

Re: Flink docker in session cluster mode - is a local distribution needed?

2021-02-17 Thread Manas Kale
Hi Till, Oh I see... I managed to do what you said using a bunch of docker exec commands. However, I think this solution is quite hacky and could be improved by providing some simple command to submit jobs using the Flink runtime within the docker images. I believe this will achieve full

Re: Flink docker in session cluster mode - is a local distribution needed?

2021-02-16 Thread Till Rohrmann
Hi Manas, I think the documentation assumes that you first start a session cluster and then submit jobs from outside the Docker images. If your jobs are included in the Docker image, then you could log into the master process and start the jobs from within the Docker image. Cheers, Till On Tue

Flink docker in session cluster mode - is a local distribution needed?

2021-02-16 Thread Manas Kale
creating a custom Docker image that has Python. For the last step, the guide asks us to submit the job using a local distribution of Flink: $ ./bin/flink run ./examples/streaming/TopSpeedWindowing.jar I am probably missing something here because I have the following questions: Why do I need to use

[ANNOUNCE] Stateful Functions Docker images are now hosted on Dockerhub at apache/flink-statefun

2021-01-18 Thread Tzu-Li (Gordon) Tai
Hi, We have created an "apache/flink-statefun" Dockerhub repository managed by the Flink PMC, at: https://hub.docker.com/r/apache/flink-statefun The images for the latest stable StateFun release, 2.2.2, have already been pushed there. Going forward, it will be part of the release process to make

Re: Flink Docker job fails to launch

2021-01-15 Thread Chesnay Schepler
immediately? On 1/15/2021 2:28 PM, Manas Kale wrote: You mean taskmanager? I tried using this command: docker run --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" flink_pipeline taskmanager after running above script but got: 2021-01-15 13:03:0

Re: Flink Docker job fails to launch

2021-01-15 Thread Manas Kale
You mean taskmanager? I tried using this command: docker run --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" flink_pipeline taskmanager after running above script but got: 2021-01-15 13:03:05,069 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils [] - Trying to select t

Re: Flink Docker job fails to launch

2021-01-15 Thread Chesnay Schepler
Where are you starting the task executor? On 1/15/2021 11:57 AM, Manas Kale wrote: Hi all, I've got a job that I am trying to run using docker as per [1]. Here's the dockerfile: # Start from base Flink image. FROM flink:1.11.0 # Add fat JAR and logger properties file to image. ADD

Flink Docker job fails to launch

2021-01-15 Thread Manas Kale
Hi all, I've got a job that I am trying to run using docker as per [1]. Here's the dockerfile: # Start from base Flink image. FROM flink:1.11.0 # Add fat JAR and logger properties file to image. ADD ./target/flink_POC-0.1.jar /opt/flink/usrlib/flink_POC-0.1.jar ADD ./targ

Re: Flink 1.12.0 docker image MISSING

2021-01-05 Thread Chesnay Schepler
This is a known issue: FLINK-20632 <https://issues.apache.org/jira/browse/FLINK-20632> On 1/5/2021 11:27 AM, Alexandru Vasiu wrote: Hi, Docker image for flink 1.12 is missing from Docker Hub. Thank you, Alex Vasiu ComplyAdvantage is a trading name of IVXS TECHNOLOGY ROMANIA. This m

Flink 1.12.0 docker image MISSING

2021-01-05 Thread Alexandru Vasiu
Hi, Docker image for flink 1.12 is missing from Docker Hub. Thank you, Alex Vasiu -- ComplyAdvantage is a trading name of IVXS TECHNOLOGY ROMANIA. This message, including any attachments, is intended only for the use of the individual(s) to whom it is addressed and may contain information

Re: cannot pull statefun docker image

2020-11-06 Thread Tzu-Li (Gordon) Tai
:*` to `FROM ververica/flink-statefun:*` and the examples should work. Alternatively, you can build the images yourself, using the Dockerfiles for released StateFun versions at: https://github.com/apache/flink-statefun-docker Cheers, Gordon On Sat, Nov 7, 2020, 12:17 AM Lian Jiang wrote: > Hi, &g

cannot pull statefun docker image

2020-11-06 Thread Lian Jiang
Hi, I tried to build statefun-greeter-example docker image with "docker build ." but cannot pull the base statefun docker image due to access denied. Any idea? Thanks. $ docker login Authenticating with existing credentials... Login Succeeded lianj:~/repo/flink-statefun/statefu

Re: Error on deploying Flink docker image with Kubernetes (minikube) and automatically launch a stream WordCount job.

2020-09-24 Thread Yang Wang
standalone job/application Flink > cluster on K8s via yamls. > > You should have your own image with user jar baked located at > /opt/flink/usrlib. It could not be > > specified via config option. Usually, you could add new layer on the > official docker image to > > bu

Re: Error on deploying Flink docker image with Kubernetes (minikube) and automatically launch a stream WordCount job.

2020-09-23 Thread Felipe Gutierrez
t; Currently, if you want to deploy a standalone job/application Flink cluster > on K8s via yamls. > You should have your own image with user jar baked located at > /opt/flink/usrlib. It could not be > specified via config option. Usually, you could add new layer on the official >

Re: Error on deploying Flink docker image with Kubernetes (minikube) and automatically launch a stream WordCount job.

2020-09-23 Thread Yang Wang
Hi Felipe, Currently, if you want to deploy a standalone job/application Flink cluster on K8s via yamls. You should have your own image with user jar baked located at /opt/flink/usrlib. It could not be specified via config option. Usually, you could add new layer on the official docker image to

Error on deploying Flink docker image with Kubernetes (minikube) and automatically launch a stream WordCount job.

2020-09-18 Thread Felipe Gutierrez
Hi community, I am trying to deploy the default WordCount stream application on minikube using the official documentation at [1]. I am using minikube v1.13.0 on Ubuntu 18.04 and Kubernetes v1.19.0 on Docker 19.03.8. I could sucessfully start 1 job manager and 3 task managers using the yaml files

Re: Hostname for taskmanagers when running in docker

2020-08-13 Thread Xintong Song
Hi Nikola, I'm not entirely sure about how this happened. Would need some more information to investigate, such as the complete configurations for taskmanagers in your docker compose file, and the taskmanager logs. One quick thing you may try is to explicitly set the configuration o

Hostname for taskmanagers when running in docker

2020-08-12 Thread Nikola Hrusov
Hello, After upgrading the docker image for flink to 1.11.1 from 1.9 the hostname of the taskmanagers reported to our metrics show as IPs (e.g. 10.0.23.101) instead of hostnames. In the docker compose file we specify the hostname as such: *hostname: "taskmanager-{{ '{{' }

  1   2   3   >