[
https://issues.apache.org/jira/browse/FLINK-33531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787824#comment-17787824
]
Xingbo Huang commented on FLINK-33531:
--------------------------------------
After doing some experiments, I came to the following conclusions:
1. In Python 3.9 + Cython 0.29.36 environment, the `test_denpendency.py` test
will fail stably in my private Azure pipeline. Although I don't think Python
and Cython versions have anything to do with this test failure.
2. Change the Python or Cython version of this test and the failure case will
no longer appear.
3. This problem cannot be reproduced locally using the same versions of all
packages such as Python and Cython.
4. After reverting the commit that may cause the problem, this case will still
fail in Azure.（I didn't revert all the commits because I don't think these are
the root causes.）
My preferred solution right now is to upgrade Cython to address testing issues
caused by the Azure environment.
> Nightly Python fails with NPE at metadataHandlerProvider on AZP
> (StreamDependencyTests.test_add_python_archive)
> ---------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-33531
> URL: https://issues.apache.org/jira/browse/FLINK-33531
> Project: Flink
> Issue Type: Bug
> Components: API / Python
> Affects Versions: 1.19.0
> Reporter: Sergey Nuyanzin
> Assignee: Xingbo Huang
> Priority: Blocker
> Labels: test-stability
>
> It seems starting 02.11.2023 every master nightly fails with this (that's why
> it is a blocker)
> for instance
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54512&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=c67e71ed-6451-5d26-8920-5a8cf9651901]
> {noformat}
> 2023-11-12T02:10:24.5082784Z Nov 12 02:10:24 if is_error(answer)[0]:
> 2023-11-12T02:10:24.5083620Z Nov 12 02:10:24 if len(answer) > 1:
> 2023-11-12T02:10:24.5084326Z Nov 12 02:10:24 type = answer[1]
> 2023-11-12T02:10:24.5085164Z Nov 12 02:10:24 value =
> OUTPUT_CONVERTER[type](answer[2:], gateway_client)
> 2023-11-12T02:10:24.5086061Z Nov 12 02:10:24 if answer[1] ==
> REFERENCE_TYPE:
> 2023-11-12T02:10:24.5086850Z Nov 12 02:10:24 > raise
> Py4JJavaError(
> 2023-11-12T02:10:24.5087677Z Nov 12 02:10:24 "An
> error occurred while calling {0}{1}{2}.\n".
> 2023-11-12T02:10:24.5088538Z Nov 12 02:10:24
> format(target_id, ".", name), value)
> 2023-11-12T02:10:24.5089551Z Nov 12 02:10:24 E
> py4j.protocol.Py4JJavaError: An error occurred while calling
> o3371.executeInsert.
> 2023-11-12T02:10:24.5090832Z Nov 12 02:10:24 E :
> java.lang.NullPointerException: metadataHandlerProvider
> 2023-11-12T02:10:24.5091832Z Nov 12 02:10:24 E at
> java.util.Objects.requireNonNull(Objects.java:228)
> 2023-11-12T02:10:24.5093399Z Nov 12 02:10:24 E at
> org.apache.calcite.rel.metadata.RelMetadataQueryBase.getMetadataHandlerProvider(RelMetadataQueryBase.java:122)
> 2023-11-12T02:10:24.5094480Z Nov 12 02:10:24 E at
> org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:118)
> 2023-11-12T02:10:24.5095365Z Nov 12 02:10:24 E at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:844)
> 2023-11-12T02:10:24.5096306Z Nov 12 02:10:24 E at
> org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:307)
> 2023-11-12T02:10:24.5097238Z Nov 12 02:10:24 E at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
> 2023-11-12T02:10:24.5098014Z Nov 12 02:10:24 E at
> org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:556)
> 2023-11-12T02:10:24.5098753Z Nov 12 02:10:24 E at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:420)
> 2023-11-12T02:10:24.5099517Z Nov 12 02:10:24 E at
> org.apache.calcite.plan.hep.HepPlanner.executeRuleInstance(HepPlanner.java:243)
> 2023-11-12T02:10:24.5100373Z Nov 12 02:10:24 E at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance$State.execute(HepInstruction.java:178)
> 2023-11-12T02:10:24.5101313Z Nov 12 02:10:24 E at
> org.apache.calcite.plan.hep.HepPlanner.lambda$executeProgram$0(HepPlanner.java:211)
> 2023-11-12T02:10:24.5102410Z Nov 12 02:10:24 E at
> org.apache.flink.calcite.shaded.com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
> 2023-11-12T02:10:24.5103343Z Nov 12 02:10:24 E at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:210)
> 2023-11-12T02:10:24.5104105Z Nov 12 02:10:24 E at
> org.apache.calcite.plan.hep.HepProgram$State.execute(HepProgram.java:118)
> 2023-11-12T02:10:24.5104868Z Nov 12 02:10:24 E at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:205)
> 2023-11-12T02:10:24.5105616Z Nov 12 02:10:24 E at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:191)
> 2023-11-12T02:10:24.5106421Z Nov 12 02:10:24 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepProgram.optimize(FlinkHepProgram.scala:64)
> 2023-11-12T02:10:24.5107359Z Nov 12 02:10:24 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepRuleSetProgram.optimize(FlinkHepRuleSetProgram.scala:78)
> 2023-11-12T02:10:24.5108346Z Nov 12 02:10:24 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:59)
> 2023-11-12T02:10:24.5109407Z Nov 12 02:10:24 E at
> scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:156)
> 2023-11-12T02:10:24.5110241Z Nov 12 02:10:24 E at
> scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:156)
> 2023-11-12T02:10:24.5111078Z Nov 12 02:10:24 E at
> scala.collection.Iterator.foreach(Iterator.scala:937)
> 2023-11-12T02:10:24.5111734Z Nov 12 02:10:24 E at
> scala.collection.Iterator.foreach$(Iterator.scala:937)
> 2023-11-12T02:10:24.5112410Z Nov 12 02:10:24 E at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
> 2023-11-12T02:10:24.5113145Z Nov 12 02:10:24 E at
> scala.collection.IterableLike.foreach(IterableLike.scala:70)
> 2023-11-12T02:10:24.5113840Z Nov 12 02:10:24 E at
> scala.collection.IterableLike.foreach$(IterableLike.scala:69)
> 2023-11-12T02:10:24.5114528Z Nov 12 02:10:24 E at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> 2023-11-12T02:10:24.5115232Z Nov 12 02:10:24 E at
> scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:156)
> 2023-11-12T02:10:24.5116054Z Nov 12 02:10:24 E at
> scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:154)
> 2023-11-12T02:10:24.5116775Z Nov 12 02:10:24 E at
> scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> 2023-11-12T02:10:24.5117583Z Nov 12 02:10:24 E at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:55)
> 2023-11-12T02:10:24.5118581Z Nov 12 02:10:24 E at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.optimizeTree(StreamCommonSubGraphBasedOptimizer.scala:176)
> 2023-11-12T02:10:24.5119603Z Nov 12 02:10:24 E at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.doOptimize(StreamCommonSubGraphBasedOptimizer.scala:83)
> 2023-11-12T02:10:24.5120689Z Nov 12 02:10:24 E at
> org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:87)
> 2023-11-12T02:10:24.5121602Z Nov 12 02:10:24 E at
> org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:324)
> 2023-11-12T02:10:24.5122403Z Nov 12 02:10:24 E at
> org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:182)
> 2023-11-12T02:10:24.5123301Z Nov 12 02:10:24 E at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:1277)
> 2023-11-12T02:10:24.5124195Z Nov 12 02:10:24 E at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:862)
> 2023-11-12T02:10:24.5125097Z Nov 12 02:10:24 E at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1097)
> 2023-11-12T02:10:24.5125945Z Nov 12 02:10:24 E at
> org.apache.flink.table.api.internal.TablePipelineImpl.execute(TablePipelineImpl.java:59)
> 2023-11-12T02:10:24.5126720Z Nov 12 02:10:24 E at
> org.apache.flink.table.api.Table.executeInsert(Table.java:1074){noformat}
> UPD:
> It looks like it is a very similar to what has already been reported at
> FLINK-32424 and FLINK-24241
> However something made it happening every nightly
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
