Hi Lydian,
I do not know how you setup your FlinkRunner. Recently, I tested several
ways (you can check https://github.com/liferoad/beam-ml-flink) to run a
simple Beam ML job on FlinkRunner and they all seem to be working fine.
XQ
On Mon, Jan 27, 2025 at 12:28 PM Ahmet Altay <al...@google.com> wrote:
> I do not know the answer. Adding @Kenneth Knowles <k...@google.com> and @XQ
> Hu <x...@google.com> in case they might be able to answer it.
>
> On Sun, Jan 26, 2025 at 3:54 PM Lydian Lee <tingyenlee...@gmail.com>
> wrote:
>
>> Hi,
>>
>> We are trying to bump an old pipeline using flink runner and beam python
>> SDK.
>> The version changes are:
>> - Flink: 1.15.4 -> 1.18.0
>> - Beam: 2.41.0 -> 2.60.0
>> - python: 3.9. (no change)
>>
>> However, after bumping the version, we noticed that the
>> python-worker-harness is unable to process function properly. Here's the
>> error log we kept received:
>>
>> ```
>> Starting worker with command ['/opt/apache/beam/boot', '--id=6-1',
>> '--logging_endpoint=localhost:46759',
>> '--artifact_endpoint=localhost:44123',
>> '--provision_endpoint=localhost:42447', '--control_endpoint=localho
>> 2025/01/26 23:41:26 Provision info:
>> pipeline_options:{fields:{key:"beam:option:allow_non_deterministic_key_coders:v1"
>> value:{bool_value:false}}
>> fields:{key:"beam:option:allow_non_restored_state:v1"
>> value:{bool_value:false}} fields:{key:"beam:o
>> 2025/01/26 23:41:44 boot.go: error logging message over FnAPI. endpoint
>> localhost:46759 error: EOF message follows
>> 2025/01/26 23:41:44 DEBUG Received signal: terminated 2025/01/26 23:41:44
>> boot.go: error logging message over FnAPI. endpoint localhost:46759 error:
>> EOF message follows
>> 2025/01/26 23:41:44 WARN Python (worker 6-1) exited 1 times: signal:
>> terminated restarting SDK process
>> ```
>> It probably unrelated to the job itself as the job was working fine in
>> the earlier version:
>> [image: Untitled.jpeg]
>> Wondering if anyone have any idea how I can fix this issue? Thanks
>>
>
