Hi John, Because I can't see your code, I can only provide some possible reasons for this error: 1. This error generally occurs in jobs where there are transfers between Table and datastream. But given that you said you just used the sql + python udf, this shouldn't be the case. 2. The default value of `taskmanager.memory.managed.consumer-weights` is `OPERATOR:70,STATE_BACKEND:70,PYTHON:30`, so in your case, there is actually no need to set it to `PYTHON:30` 3. In fact, for pure sql+python udf jobs, if you don't set error value `PYTHON:0` in `taskmanager.memory.managed.consumer-weights`, I really can't think of any situation where this problem will occur.
Best, Xingbo John Tipper <john_tip...@hotmail.com> 于2022年6月16日周四 19:41写道： > Hi Xingbo, > > Yes, there are a number of temporary views being created, where each is > being created using SQL (CREATE TEMPORARY VIEW ...) rather than explicit > calls to the Table and DataStream APIs. > > Is this a good pattern or are there caveats I should be aware of please? > > Many thanks, > > John > > > ------------------------------ > *From:* Xingbo Huang <hxbks...@gmail.com> > *Sent:* 16 June 2022 12:34 > *To:* John Tipper <john_tip...@hotmail.com> > *Cc:* user@flink.apache.org <user@flink.apache.org> > *Subject:* Re: The configured managed memory fraction for Python worker > process must be within (0, 1], was: %s > > Hi John, > > Does your job logic include conversion between Table and DataStream? For > example, methods such as `create_temporary_view(path: str, data_stream: > DataStream): -> Table` are used. > > Best, > Xingbo > > John Tipper <john_tip...@hotmail.com> 于2022年6月16日周四 18:31写道： > > Hi Xingbo, > > I’m afraid I can’t share my code but Flink is 1.13. The main Flink code is > running inside Kinesis on AWS so I cannot change the version. > > Many thanks, > > John > > Sent from my iPhone > > On 16 Jun 2022, at 10:37, Xingbo Huang <hxbks...@gmail.com> wrote: > > > Hi John, > > Could you provide the code snippet and the version of pyflink you used? > > Best, > Xingbo > > > John Tipper <john_tip...@hotmail.com> 于2022年6月16日周四 17:05写道： > > Hi all, > > I'm trying to run a PyFlink unit test to test some PyFlink SQL and where > my code uses a Python UDF. I can't share my code but the test case is > similar to the code here: > https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py > When > I have some simple SQL everything is fine. When I add a more complex query > I get an error, which looks like it's memory related. > > java.lang.IllegalArgumentException: The configured managed memory fraction > for Python worker process must be within (0, 1], was: %s. It may be because > the consumer type "Python" was missing or set to 0 for the config option > "taskmanager.memory.managed.consumer-weights".0.0 > > > > In my test case setUp(), I try to set that value like this, but it seems > to have no effect: > > self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights", > "PYTHON:30") > > > Am I not setting it correctly, or is there something else I need to do to > fix this error? > > Many thanks, > > John > >
