Hi John, I have written an example on how to write unit tests of Flink functionalities with PyFlink in [1]. Hope it is helpful for you. Feel free to let me know if there are any problems.
Regards, Dian [1] https://github.com/dianfu/pyflink-faq/tree/main/testing On Sun, Apr 24, 2022 at 9:25 AM Dian Fu <[email protected]> wrote: > Hi John, > > >> I don't know how to fix this. I've tried adding `flink-table-planner` > and `flink-table-planner-blink` dependencies with `<type>test-jar</type>` > to my dummy pom.xml, but it still fails. > What's the failure after doing this? The flink-table-planner*-tests.jar > should be available in maven repository[1]. > > >> This is starting to feel like a real pain to do something that should > be trivial: basic TDD of a PyFlink project. Is there a real-world example > of a Python project that shows how to set up a testing environment for unit > testing SQL with PyFlink? > I'm not aware of such a project, however I agree that this may be a very > important aspect which should be improved. I will look into this. > > Regards, > Dian > > [1] > https://repo1.maven.org/maven2/org/apache/flink/flink-table-planner_2.11/1.13.6/ > > > On Sun, Apr 24, 2022 at 4:44 AM John Tipper <[email protected]> > wrote: > >> Hi all, >> >> Is there an example of a self-contained repository showing how to perform >> SQL unit testing of PyFlink (specifically 1.13.x if possible)? I have >> cross-posted the question to Stack Overflow here: >> https://stackoverflow.com/questions/71983434/is-there-an-example-of-pyflink-sql-unit-testing-in-a-self-contained-repo >> >> >> There is a related SO question ( >> https://stackoverflow.com/questions/69937520/pyflink-sql-local-test), >> where it is suggested to use some of the tests from PyFlink itself. The >> issue I'm running into is that the PyFlink repo assumes that a bunch of >> things are on the Java classpath and that some Python utility classes are >> available (they're not distributed via PyPi apache-flink). >> >> I have done the following: >> >> >> 1. Copied `test_case_utils.py` and `source_sink_utils.py` from >> PyFlink ( >> >> https://github.com/apache/flink/tree/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing) >> into my project. >> 2. Copy an example unit test ( >> >> https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/table/tests/test_sql.py#L39) >> as suggested by the related SO question. >> 3. >> >> When I try to run the test, I get an error because the test case cannot >> determine what version of Avro jars to download (`download_apache_avro()` >> fails, because pyflink_gateway_server.py tries to evaluate the value of >> `avro.version` by running `mvn help:evaluate -Dexpression=avro.version`) >> >> I then added a dummy `pom.xml` defining a Maven property of >> `avro.version` (with a value of `1.10.0`) and my unit test case is loaded. >> >> I now get a new error and my test is skipped: >> >> 'flink-table-planner*-tests.jar' is not available. Will skip the >> related tests. >> >> I don't know how to fix this. I've tried adding `flink-table-planner` and >> `flink-table-planner-blink` dependencies with `<type>test-jar</type>` to my >> dummy pom.xml, but it still fails. >> >> This is starting to feel like a real pain to do something that should be >> trivial: basic TDD of a PyFlink project. Is there a real-world example of >> a Python project that shows how to set up a testing environment for unit >> testing SQL with PyFlink? >> >> Many thanks, >> >> John >> >
