Hi Guys, I have another issue when I submit the python beam pipeline ( wordcount example provided by apache beam team) directly on flink cluster running local.
PipelineOptions are : "--runner=FlinkRunner", "--flink_version=1.9", "--environment_type=LOOPBACK", "--flink_master=192.168.99.100:8081", versions: Apache Beam SDK=2.21.0 python=3.7.6 Flink=1.9.3 Please suggest , if I am missing something. Ashish Raghav
DEBUG:apache_beam.io.filesystem:translate_pattern: 'C:\\Users\\ashish.raghav\\Desktop\\projects\\test-flink\\input.txt' -> 'C:\\\\Users\\\\ashish\\.raghav\\\\Desktop\\\\projects\\\\test\\-flink\\\\input\\.txt' lines_read: PCollection[ReadFromText/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn).None] DEBUG:root:Unhandled type_constraint: Union[] count complete output prepared writing output DEBUG:root:Unhandled type_constraint: Union[] writing complete WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter. INFO:root:Using Python SDK docker image: apache/beam_python3.7_sdk:2.21.0. If the image is not available at local, we will try to pull from hub.docker.com INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x000002A93F75C558> ==================== DEBUG:apache_beam.runners.portability.fn_api_runner.translations:20 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] DEBUG:apache_beam.runners.portability.fn_api_runner.translations:Stages: ['ref_AppliedPTransform_ReadFromText/Read/_SDFBoundedSourceWrapper/Impulse_5\n ReadFromText/Read/_SDFBoundedSourceWrapper/Impulse:beam:transform:impulse:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_ReadFromText/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn)_6\n ReadFromText/Read/_SDFBoundedSourceWrapper/ParDo(SDFBoundedSourceDoFn):beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Split_7\n Split:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_PairWithOne_8\n PairWithOne:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'GroupAndSum/Precombine\n GroupAndSum/Precombine:beam:transform:combine_per_key_precombine:v1\n must follow: \n downstream_side_inputs: <unknown>', 'GroupAndSum/Group\n GroupAndSum/Group:beam:transform:group_by_key:v1\n must follow: \n downstream_side_inputs: <unknown>', 'GroupAndSum/Merge\n GroupAndSum/Merge:beam:transform:combine_per_key_merge_accumulators:v1\n must follow: \n downstream_side_inputs: <unknown>', 'GroupAndSum/ExtractOutputs\n GroupAndSum/ExtractOutputs:beam:transform:combine_per_key_extract_outputs:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_Format_16\n Format:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_WriteToText/Write/WriteImpl/DoOnce/Impulse_21\n WriteToText/Write/WriteImpl/DoOnce/Impulse:beam:transform:impulse:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_WriteToText/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:2706>)_22\n WriteToText/Write/WriteImpl/DoOnce/FlatMap(<lambda at core.py:2706>):beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_WriteToText/Write/WriteImpl/DoOnce/Map(decode)_24\n WriteToText/Write/WriteImpl/DoOnce/Map(decode):beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_WriteToText/Write/WriteImpl/InitializeWrite_25\n WriteToText/Write/WriteImpl/InitializeWrite:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_WriteToText/Write/WriteImpl/WriteBundles_26\n WriteToText/Write/WriteImpl/WriteBundles:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_WriteToText/Write/WriteImpl/Pair_27\n WriteToText/Write/WriteImpl/Pair:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_WriteToText/Write/WriteImpl/WindowInto(WindowIntoFn)_28\n WriteToText/Write/WriteImpl/WindowInto(WindowIntoFn):beam:transform:window_into:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_WriteToText/Write/WriteImpl/GroupByKey_29\n WriteToText/Write/WriteImpl/GroupByKey:beam:transform:group_by_key:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_WriteToText/Write/WriteImpl/Extract_33\n WriteToText/Write/WriteImpl/Extract:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_WriteToText/Write/WriteImpl/PreFinalize_34\n WriteToText/Write/WriteImpl/PreFinalize:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>', 'ref_AppliedPTransform_WriteToText/Write/WriteImpl/FinalizeWrite_35\n WriteToText/Write/WriteImpl/FinalizeWrite:beam:transform:pardo:v1\n must follow: \n downstream_side_inputs: <unknown>'] INFO:apache_beam.runners.portability.flink_runner:Adding HTTP protocol scheme to flink_master parameter: http://192.168.99.100:8081 INFO:apache_beam.utils.subprocess_server:Using cached job server jar from https://repo.maven.apache.org/maven2/org/apache/beam/beam-runners-flink-1.9-job-server/2.21.0/beam-runners-flink-1.9-job-server-2.21.0.jar INFO:apache_beam.utils.subprocess_server:Starting service with ['java' '-jar' 'C:\\Users\\ashish.raghav/.apache_beam/cache/jars\\beam-runners-flink-1.9-job-server-2.21.0.jar' '--flink-master' 'http://192.168.99.100:8081' '--artifacts-dir' 'C:\\Users\\ASHISH~1.RAG\\AppData\\Local\\Temp\\beam-tempzvygf8j0\\artifactsqaa76epq' '--job-port' '65158' '--artifact-port' '0' '--expansion-port' '0'] ERROR:apache_beam.utils.subprocess_server:Error bringing up service Traceback (most recent call last): File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\utils\subprocess_server.py", line 88, in start cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) File "C:\Users\ashish.raghav\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 800, in __init__ restore_signals, start_new_session) File "C:\Users\ashish.raghav\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1207, in _execute_child startupinfo) FileNotFoundError: [WinError 2] The system cannot find the file specified Traceback (most recent call last): File "C:\Users\ashish.raghav\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "C:\Users\ashish.raghav\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\ashish.raghav\Desktop\projects\test-flink\pipeline_flink_instream.py", line 104, in <module> run() File "C:\Users\ashish.raghav\Desktop\projects\test-flink\pipeline_flink_instream.py", line 98, in run print("writing complete") File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\pipeline.py", line 524, in __exit__ self.run().wait_until_finish() File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\pipeline.py", line 497, in run self._options).run(False) File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\pipeline.py", line 510, in run return self.runner.run_pipeline(self, self._options) File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\runners\portability\flink_runner.py", line 49, in run_pipeline return super(FlinkRunner, self).run_pipeline(pipeline, options) File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\runners\portability\portable_runner.py", line 404, in run_pipeline job_service_handle = self.create_job_service(options) File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\runners\portability\portable_runner.py", line 311, in create_job_service return self.create_job_service_handle(server.start(), options) File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\runners\portability\job_server.py", line 85, in start self._options).run(False) File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\pipeline.py", line 510, in run return self.runner.run_pipeline(self, self._options) File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\runners\portability\flink_runner.py", line 49, in run_pipeline return super(FlinkRunner, self).run_pipeline(pipeline, options) File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\runners\portability\portable_runner.py", line 404, in run_pipeline job_service_handle = self.create_job_service(options) File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\runners\portability\portable_runner.py", line 311, in create_job_service return self.create_job_service_handle(server.start(), options) File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\runners\portability\job_server.py", line 85, in start self._endpoint = self._job_server.start() File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\runners\portability\job_server.py", line 114, in start return self._server.start() File "C:\Users\ashish.raghav\Desktop\projects\test-flink\.venv\lib\site-packages\apache_beam\utils\subprocess_server.py", line 88, in start cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) File "C:\Users\ashish.raghav\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 800, in __init__ restore_signals, start_new_session) File "C:\Users\ashish.raghav\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 1207, in _execute_child startupinfo) FileNotFoundError: [WinError 2] The system cannot find the file specified startupinfo) FileNotFoundError: [WinError 2] The system cannot find the file specified
