I try again maybe someone can help me with this?
 
How to run Beam on Flink?
 
I have code:
 
def run():
 import apache_beam as beam
 from apache_beam.options.pipeline_options import PipelineOptions
 
 options = PipelineOptions([
        "--runner=FlinkRunner",
        "--flink_version=1.14",
        "--flink_master=localhost:8081",
        "--environment_config=localhost:50000"
 ])
 output_file = 'E:\\directory\\output.txt'
 with beam.Pipeline(options=options) as p:
    (p
        | 'Create file lines' >> beam.Create([
          'Each element must be a string.',
          'It writes one element per line.',
          'There are no guarantees on the line order.',
          'The data might be written into multiple files.',
        ])
        | 'Write to files' >> beam.io.WriteToText(output_file)
    )
if __name__ == "__main__":
    run()
 
Should work. But for some reason Flink is not able to save to file:
 
CHAIN MapPartition (MapPartition at [2]Write to files/Write/WriteImpl/DoOnce/{FlatMap(<lambda at core.py:3320>), Map(decode)}) -> FlatMap (FlatMap at ExtractOutput[0]) FAILED
 
Same problem if I want to open some file.
What is wrong here? I tried several example scripts - none is working.
If you could help me to take first step in Beam and Flink.
Regards
Mike

Reply via email to