>Fink will not run natively in windows - that is why I use Github CLI Do you mean Git Bash?
> I made test with Flink version 1.14.4 - Taskmanager is running. But no chance with 1.15.0 or 1.15.1 - taskmanager does not start. Something's wrong with Flink? I do not use Windows, so can't verify. start-cluster.sh in Flink 1.15.x works fine on *nix systems . Best, Alexander Fedulov On Fri, Jul 8, 2022 at 7:17 PM <pod...@gmx.com> wrote: > > Fink will not run natively in windows - that is why I use Github CLI > > I made test with Flink version 1.14.4 - Taskmanager is running. But no > chance with 1.15.0 or 1.15.1 - taskmanager does not start. > Something's wrong with Flink? > > > *Sent:* Friday, July 08, 2022 at 5:15 PM > *From:* "Alexander Fedulov" <alexan...@ververica.com> > *To:* pod...@gmx.com > *Cc:* "user" <user@flink.apache.org> > *Subject:* Re: Is Flink able to read a CSV file or just like in Blink > this function does not work? > You do not need to run anything beyond start-cluster for a simple test. It > starts one jobmanager (this is what also hosts the UI you see) and one > taskmanager: > > [image: image.png] > > > What does Github CLI has to do with the Flink startup? > > > > > On Fri, Jul 8, 2022 at 2:25 PM <pod...@gmx.com> wrote: > >> Yep, sorry, my fault. >> >> I see now that this is not enough to just start cluster; taskmanager have >> to be started as well. >> How I start it (Github CLI): >> >> export FLINK_HOME=/C/Flink/flink-1.15.0 >> $FLINK_HOME/bin/start-cluster.sh >> >> Flink is running (I can reach it going to http://localhost:8081) >> >> I know that now I should start jobmanager, so i execute" >> $FLINK_HOME/bin/jobmanager.sh start >> >> I get message: >> >> [INFO] 1 instance(s) of standalonesession are already running on >> MNKH-WjKRv8Leb8. >> Starting standalonesession daemon on host MNKH-WjKRv8Leb8. >> >> So it's OK I think >> >> Now I should start taskmanager: >> >> $FLINK_HOME/bin/taskmanager.sh start >> Starting taskexecutor daemon on host MNKH-WjKRv8Leb8 >> >> But I see in logs this process fail: >> >> ... >> 2022-07-08 14:07:32,403 INFO >> org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor >> system started at akka.tcp://flink@localhost:63030 >> 2022-07-08 14:07:32,427 ERROR >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - >> Terminating TaskManagerRunner with exit code 1. >> org.apache.flink.util.FlinkException: Failed to start the >> TaskManagerRunner. >> at >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:483) >> ~[flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerProcessSecurely$5(TaskManagerRunner.java:525) >> ~[flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) >> ~[flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:525) >> [flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:505) >> [flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:463) >> [flink-dist-1.15.0.jar:1.15.0] >> Caused by: java.io.IOException: Could not create the working directory >> C:\Users\MIKE~1\AppData\Local\Temp\tm_localhost:63030-ddbffb. >> at >> org.apache.flink.runtime.entrypoint.WorkingDirectory.createDirectory(WorkingDirectory.java:58) >> ~[flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.entrypoint.WorkingDirectory.<init>(WorkingDirectory.java:39) >> ~[flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.entrypoint.WorkingDirectory.create(WorkingDirectory.java:88) >> ~[flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypointUtils.lambda$createTaskManagerWorkingDirectory$0(ClusterEntrypointUtils.java:152) >> ~[flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.entrypoint.DeterminismEnvelope.map(DeterminismEnvelope.java:49) >> ~[flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.entrypoint.ClusterEntrypointUtils.createTaskManagerWorkingDirectory(ClusterEntrypointUtils.java:150) >> ~[flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.startTaskManagerRunnerServices(TaskManagerRunner.java:210) >> ~[flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.start(TaskManagerRunner.java:288) >> ~[flink-dist-1.15.0.jar:1.15.0] >> at >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:481) >> ~[flink-dist-1.15.0.jar:1.15.0] >> ... 5 more >> >> I'm trying to find the reason. >> >> >> *Sent:* Friday, July 08, 2022 at 12:21 PM >> *From:* "Alexander Fedulov" <alexan...@ververica.com> >> *To:* pod...@gmx.com >> *Cc:* "user" <user@flink.apache.org> >> *Subject:* Re: Is Flink able to read a CSV file or just like in Blink >> this function does not work? >> Hi Mike, >> >> please do not forget to include the mailing list address/reply to all, >> otherwise, this becomes a private conversation. >> How do you start your cluster? >> >> Best, >> Alex >> >> >> On Fri, Jul 8, 2022 at 10:45 AM <pod...@gmx.com> wrote: >> >>> Thanks for support. >>> As you suggested, I did run it in IDE: >>> >>> WARNING: An illegal reflective access operation has occurred >>> WARNING: Illegal reflective access by >>> org.apache.flink.api.java.ClosureCleaner >>> (file:/C:/Flink/flink-1.15.0/lib/flink-dist-1.15.0.jar) to field >>> java.lang.String.value >>> WARNING: Please consider reporting this to the maintainers of >>> org.apache.flink.api.java.ClosureCleaner >>> WARNING: Use --illegal-access=warn to enable warnings of further illegal >>> reflective access operations >>> WARNING: All illegal access operations will be denied in a future release >>> +----------------------+ >>> | some_value | >>> +----------------------+ >>> | 3 | >>> +----------------------+ >>> 1 row in set >>> >>> So it works (path is 'file:///c:/Temp/test4.txt' - when path was wrong I >>> got error... unfortunately Flink do not display any error in such case) >>> >>> You say 'Check in the UI that after you start your cluster you have >>> TaskManagers registered successfully'. >>> If I go to 'Task Managers' managers menu ( >>> http://localhost:8081/#/task-manager) I do not see any - list is empty. >>> >>> No idea what it should be there or how to make one. >>> >>> *Sent:* Thursday, July 07, 2022 at 4:27 PM >>> *From:* "Alexander Fedulov" <alexan...@ververica.com> >>> *To:* pod...@gmx.com >>> *Subject:* Re: Is Flink able to read a CSV file or just like in Blink >>> this function does not work? >>> Did you try running in an IDE? >>> Check in the UI that after you start your cluster you have TaskManagers >>> registered successfully and they have enough slots to fulfill your job >>> parallelism requirements. >>> >>> Best, >>> Alexander Fedulov >>> >>> On Thu, Jul 7, 2022 at 2:53 PM <pod...@gmx.com> wrote: >>> >>>> >>>> Thank you very much Alex for your help. >>>> My environment is Windows and Github CLI. >>>> >>>> I tried different directory syntax: >>>> 'file:///c:/temp/test4.txt' >>>> 'file:///c://temp//test4.txt' >>>> '//c:/temp/test4.txt' >>>> 'c:/temp/test4.txt' >>>> 'c:\\temp\\test4.txt' >>>> ... >>>> dozens of others >>>> Nothing helps >>>> >>>> I run this job with command: >>>> $ $FLINK_HOME/bin/flink run c:/Temp/flinkTest2.jar >>>> >>>> $ $FLINK_HOME/bin/flink run c:/Temp/flinkTest2.jar >>>> WARNING: An illegal reflective access operation has occurred >>>> WARNING: Illegal reflective access by >>>> org.apache.flink.api.java.ClosureCleaner >>>> (file:/C:/Flink/flink-1.15.0/lib/flink-dist-1.15.0.jar) to field >>>> java.lang.String.value >>>> WARNING: Please consider reporting this to the maintainers of >>>> org.apache.flink.api.java.ClosureCleaner >>>> WARNING: Use --illegal-access=warn to enable warnings of further >>>> illegal reflective access operations >>>> WARNING: All illegal access operations will be denied in a future >>>> release >>>> Job has been submitted with JobID a9eaf86337ca14d0d80e6d6ef70dab6e >>>> >>>> >>>> >>>> But maybe problem is in another place - even if i put non existing path >>>> - there is no error; job is subited to Flink, has status RUNNING and it >>>> fails after 5 minuts or so. >>>> >>>> Source: Custom File source: FAILED >>>> CsvTableSource(read fields: ) -> SourceConversion[1] -> >>>> HashAggregate[2]: CANCELED >>>> HashAggregate[4] -> ConstraintEnforcer[5] -> Sink: Collect table sink: >>>> CANCELED >>>> >>>> In console I see: >>>> NoResourceAvailableException: Slot request bulk is not fulfillable! >>>> Could not allocate the required slot within slot request timeout >>>> >>>> >>>> >>>> *Sent:* Thursday, July 07, 2022 at 1:10 AM >>>> *From:* "Alexander Fedulov" <alexan...@ververica.com> >>>> *To:* pod...@gmx.com >>>> *Cc:* "user" <user@flink.apache.org> >>>> *Subject:* Re: Is Flink able to read a CSV file or just like in Blink >>>> this function does not work? >>>> Hi Mike, >>>> >>>> I do not see any issues with your code. With a sample csv file like this >>>> a,1.0 >>>> b,2.0 >>>> c,3.0 >>>> >>>> it produces the expected result >>>> >>>> +----------------------+ >>>> | some_value | >>>> +----------------------+ >>>> | 3 | >>>> +----------------------+ >>>> 1 row in set >>>> >>>> Process finished with exit code 0 >>>> >>>> -- >>>> >>>> I do not use Windows, so I am not 100% sure what is the correct way to >>>> specify the file path there, however, some observations: >>>> - You are using a forward slash notation, I assume you are running in >>>> WSL or using cygwin or something like that? Notice that in that notation >>>> an additional forward slash is missing 'file:///foo/bar' >>>> - Omitting the schema works with the local filesystem by default. On >>>> *nix systems this works fine 'connector.path' = >>>> '/home/alex/tmp/test.csv' . Maybe you could just try a standard windows >>>> backward slash notation instead. >>>> - That said, when a non-existent file is specified, the job fails >>>> immediately, so I would actually expect that behavior if the issue was >>>> indeed with the file path. >>>> Which version of Flink are you running? >>>> >>>> Best, >>>> Alexander Fedulov >>>> >>>> >>>> On Wed, Jul 6, 2022 at 10:39 PM <pod...@gmx.com> wrote: >>>> >>>>> If I'm reading Flink manul correctly (and this is not simple - no >>>>> examples), this code should read CSV file: >>>>> >>>>> >>>>> *package flinkTest2;* >>>>> >>>>> >>>>> *import org.apache.flink.table.api.EnvironmentSettings; import >>>>> org.apache.flink.table.api.TableEnvironment;* >>>>> >>>>> *public class flinkTest2 {* >>>>> >>>>> * public static void main(String[] args) throws Exception {* >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> * EnvironmentSettings settings = EnvironmentSettings >>>>> .newInstance() //.inStreamingMode() .inBatchMode() >>>>> .build();* >>>>> * final TableEnvironment tEnv = TableEnvironment.create(settings);* >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> * tEnv.executeSql("CREATE TABLE test_table (column_name1 >>>>> STRING, column_name2 DOUBLE) WITH ('connector.type' = 'filesystem', >>>>> 'connector.path' = 'file://C/temp/test4.txt', 'format.type' = 'csv')"); >>>>> tEnv.sqlQuery("SELECT COUNT(*) AS some_value FROM test_table") >>>>> .execute() .print();* >>>>> >>>>> * } }* >>>>> >>>>> Job is running and running and running... >>>>> I think Flink is not able to open file. Does not matter what is here: >>>>> 'connector.path' = 'file://C/temp/test4.txt' Flink does not display any >>>>> error or something. 'connector.path' = 'file:blah/blah/blah' could be fine >>>>> for flink as well. >>>>> >>>>> Anyone could help me with that? >>>>> Thanks >>>>> Mike >>>>> >>>>> >>>>
