Re: How many task managers to launch for a job?

2019-05-26 Thread Xintong Song
Well, it depends on how many resource are needed for one pipeline of you job and how many resource are are configured for each TaskExecutor. In addition, the resource of each TaskManager also depends on the job's resource needs and your environment. So having one slot for each TaskManager would be

Minicluster missing in scala runtime in version 1.8 source code

2019-05-26 Thread Ravi Pullareddy
Hi There I notice mini cluster folder missing under scala runtime in version 1.8. I am able to compile version 1.7 for zeppelin interpreter but not flink version 1.8. Is there a way to compile scala interpreter in zeppelin for version 1.8 Thanks Ravi

Replacing AWS credentials provider for S3 filesystem

2019-05-26 Thread Rafi Aroch
Hi, I'm seeing an issue when trying to set a different credentials provider for AWS S3. I'm setting in flink-conf.json the *fs.s3a.aws.credentials.provider *to a different value. I'm using the *flink-s3-fs-hadoop* dependency and I get an exception when job starts: *RuntimeException: Failed to st