Thanks a lot ! I used TASK_MANAGER_NUMBER_OF_TASK_SLOTS in my docker-compose.yml, it works perfectly :) In which format I could provide parameters via FLINK_PROPERTIES ? I'm thinking of abandoning the idea to copy flink-conf in Dockerfile. Is it limited to a specific set of parameters or generic ?
ср, 12 мая 2021 г. в 15:20, Chesnay Schepler <ches...@apache.org>: > You could also configure the number of slots via the > TASK_MANAGER_NUMBER_OF_TASK_SLOTS > environment variable. > > On 5/12/2021 2:19 PM, Chesnay Schepler wrote: > > I believe this is due to FLINK-21037; we did not consider the possibility > of users mounting the configuration directly, and instead assumed that > modifications to the config always go through the FLINK_PROPERTIES > environment variable. > > That would also be the workaround for your issue. > > On 5/12/2021 2:06 PM, Alex Drobinsky wrote: > > Dear flink community, > > First I need provide some minimum information about my deployment scenario: > I'm running application inside of Flink docker, below original Dockerfile: > > ----------------------------------------------------------------------------------------------------------- > > FROM flink:1.13.0-scala_2.11-java11 > > # Copy log and monitoring related JARs to flink lib dir > COPY kafka-clients-2.4.1.jar /opt/flink/lib/ > > RUN chmod 777 /tmp > RUN apt-get update && apt-get install -y htop > > # configuration files > COPY Log4cpp.properties /opt/flink/ > COPY Log4j.properties /opt/flink/conf/log4j.properties > COPY SessionOrganizer.json /opt/flink/*COPY flink-conf.yaml /opt/flink/conf/ > COPY slaves /opt/flink/conf/ > * > # job file > COPY KafkaToSessions-shade.jar /opt/flink/lib/ > > # libraries > ADD libs /usr/local/lib/ > > # Add /usr/local/lib to ldconfig > RUN echo "/usr/local/lib/" > /etc/ld.so.conf.d/ips.conf && \ > ldconfig && \ > ulimit -c 0 > > RUN mkdir /opt/flink/ip-collection/ && \ > mkdir /opt/flink/checkpoints/ && \ > mkdir /opt/flink/ip-collection/incorrectIcs && \ > mkdir /opt/flink/ip-collection/storage && \ > mkdir /opt/flink/ip-collection/logs > > CMD /opt/flink/bin/start-cluster.sh && /opt/flink/bin/flink run > /opt/flink/lib/KafkaToSessions-shade.jar > > > ------------------------------------------------------------------------------------------------------------------------------- > If we will ignore irrelevant parts of Dockerfile, the only 2 things > remains ( beside FROM statement) > 1. Overwritten flink-conf.yml + slaves > 2. CMD which executes start-cluster and job. > > My flink-conf.yml: > > --------------------------------------------------------------------------------------------------------- > > rest.address: localhostrest.port: 8088state.backend: > filesystemstate.checkpoints.dir: > file:///opt/flink/checkpointsjobmanager.memory.process.size: > 2224mjobmanager.rpc.port: 6123jobmanager.rpc.address: > localhosttaskmanager.memory.flink.size: > 2224mtaskmanager.memory.task.heap.size: 1000mtaskmanager.numberOfTaskSlots: > 12taskmanager.rpc.port: 50100taskmanager.data.port: 50000parallelism.default: > 6heartbeat.timeout: 120000heartbeat.interval: 20000env.java.opts: > "-XX:+UseG1GC -XX:MaxGCPauseMillis=300" > > --------------------------------------------------------------------------------------------------------- > > Slaves file contain single line with localhost. > After start of docker, I noticed that application doesn't work due lack of > slots. When I checked flink-conf.yml I noticed that > taskmanager.numberOfTaskSlots is set to 1. > P.S. during first time, daemon.sh complained that it doesn't have write > permissions to change flink-conf.yml, when I added chown flink.flink > /opt/flink/flink-conf.yml - > it stopped to complain & taskmanager.numberOfTaskSlots change occured. > > Any suggestions ? > > Best regards, > Alexander > > > > >
