knaufk commented on a change in pull request #18725: URL: https://github.com/apache/flink/pull/18725#discussion_r807701590
########## File path: docs/content/docs/deployment/resource-providers/standalone/docker.md ########## @@ -247,68 +246,175 @@ You can see that certain tags include the version of Hadoop, e.g. (e.g. `-hadoop Beginning with Flink 1.5, image tags that omit the Hadoop version correspond to Hadoop-free releases of Flink that do not include a bundled Hadoop distribution. +## Flink with Docker Compose -### Passing configuration via environment variables +[Docker Compose](https://docs.docker.com/compose/) is a way to run a group of Docker containers locally. +The next sections show examples of configuration files to run Flink. -When you run Flink image, you can also change its configuration options by setting the environment variable `FLINK_PROPERTIES`: +### General -```sh -$ FLINK_PROPERTIES="jobmanager.rpc.address: host -taskmanager.numberOfTaskSlots: 3 -blob.server.port: 6124 -" -$ docker run --env FLINK_PROPERTIES=${FLINK_PROPERTIES} flink:{{< stable >}}{{< version >}}-scala{{< scala_version >}}{{< /stable >}}{{< unstable >}}latest{{< /unstable >}} <jobmanager|standalone-job|taskmanager> -``` +* Create the docker-compose.yaml file. Please check the examples in the sections below: + * [Application Mode](#app-cluster-yml) + * [Session Mode](#session-cluster-yml) + * [Session Mode with SQL Client](#session-cluster-sql-yaml) -The [`jobmanager.rpc.address`]({{< ref "docs/deployment/config" >}}#jobmanager-rpc-address) option must be configured, others are optional to set. +* Launch a cluster in the foreground (use `-d` for background) -The environment variable `FLINK_PROPERTIES` should contain a list of Flink cluster configuration options separated by new line, -the same way as in the `flink-conf.yaml`. `FLINK_PROPERTIES` takes precedence over configurations in `flink-conf.yaml`. + ```sh + $ docker-compose up + ``` -### Provide custom configuration +* Scale the cluster up or down to `N` TaskManagers -The configuration files (`flink-conf.yaml`, logging, hosts etc) are located in the `/opt/flink/conf` directory in the Flink image. -To provide a custom location for the Flink configuration files, you can + ```sh + $ docker-compose scale taskmanager=<N> + ``` -* **either mount a volume** with the custom configuration files to this path `/opt/flink/conf` when you run the Flink image: +* Access the JobManager container ```sh - $ docker run \ - --mount type=bind,src=/host/path/to/custom/conf,target=/opt/flink/conf \ - flink:{{< stable >}}{{< version >}}-scala{{< scala_version >}}{{< /stable >}}{{< unstable >}}latest{{< /unstable >}} <jobmanager|standalone-job|taskmanager> + $ docker exec -it $(docker ps --filter name=jobmanager --format={{.ID}}) /bin/sh ``` -* or add them to your **custom Flink image**, build and run it: - +* Kill the cluster - ```dockerfile - FROM flink - ADD /host/path/to/flink-conf.yaml /opt/flink/conf/flink-conf.yaml - ADD /host/path/to/log4j.properties /opt/flink/conf/log4j.properties + ```sh + $ docker-compose kill ``` -{{< hint info >}} -The mounted volume must contain all necessary configuration files. -The `flink-conf.yaml` file must have write permission so that the Docker entry point script can modify it in certain cases. -{{< /hint >}} +* Access Web UI -### Using filesystem plugins + When the cluster is running, you can visit the web UI at [http://localhost:8081](http://localhost:8081). -As described in the [plugins]({{< ref "docs/deployment/filesystems/plugins" >}}) documentation page: In order to use plugins they must be -copied to the correct location in the Flink installation in the Docker container for them to work. +### Application Mode -If you want to enable plugins provided with Flink (in the `opt/` directory of the Flink distribution), you can pass the environment variable `ENABLE_BUILT_IN_PLUGINS` when you run the Flink image. -The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names separated by `;`. A valid plugin name is for example `flink-s3-fs-hadoop-{{< version >}}.jar` +In application mode you start a Flink cluster that is dedicated to run only the Flink Jobs which have been bundled with the images. +Hence, you need to build a dedicated Flink Image per application. +Please check [here](#application-mode-on-docker) for the details. Review comment: Fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
