No, thanks! I used JobClient to getJobStatus and sleep if it was not terminal. I'll switch to this.
On Sat, Oct 10, 2020 at 12:50 AM Aljoscha Krettek <aljos...@apache.org> wrote: > Hi Dan, > > did you try using the JobClient you can get from the TableResult to wait > for job completion? You can get a CompletableFuture for the JobResult > which should help you. > > Best, > Aljoscha > > On 08.10.20 23:55, Dan Hill wrote: > > I figured out the issue. The join caused part of the job's execution to > be > > delayed. I added my own hacky wait condition into the test to make sure > > the join job finishes first and it's fine. > > > > What common test utilities exist for Flink? I found > > flink/flink-test-utils-parent. I implemented a simple sleep loop to wait > > for jobs to finish. I'm guessing this can be done with one of the other > > utilities. > > > > Are there any open source test examples? > > > > How are watermarks usually sent with Table API in tests? > > > > After I collect some answers, I'm fine updating the Flink testing page. > > > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#testing-flink-jobs > > > > On Thu, Oct 8, 2020 at 8:52 AM Austin Cawley-Edwards < > > austin.caw...@gmail.com> wrote: > > > >> Can't comment on the SQL issues, but here's our exact setup for Bazel > and > >> Junit5 w/ the resource files approach: > >> > https://github.com/fintechstudios/vp-operator-demo-ff-virtual-2020/tree/master/tools/junit > >> > >> Best, > >> Austin > >> > >> On Thu, Oct 8, 2020 at 2:41 AM Dan Hill <quietgol...@gmail.com> wrote: > >> > >>> I was able to get finer grained logs showing. I switched from > >>> -Dlog4j.configuration to -Dlog4j.configurationFile and it worked. > With my > >>> larger test case, I was hitting a silent log4j error. When I created a > >>> small test case to just test logging, I received a log4j error. > >>> > >>> Here is a tar > >>> < > https://drive.google.com/file/d/1b6vJR_hfaRZwA28jKNlUBxDso7YiTIbk/view?usp=sharing > > > >>> with the info logs for: > >>> - (test-nojoin.log) this one works as expected > >>> - (test-join.log) this does not work as expected > >>> > >>> I don't see an obvious issue just by scanning the logs. I'll take a > >>> deeper in 9 hours. > >>> > >>> > >>> > >>> > >>> On Wed, Oct 7, 2020 at 8:28 PM Dan Hill <quietgol...@gmail.com> wrote: > >>> > >>>> Switching to junit4 did not help. > >>>> > >>>> If I make a request to the url returned from > >>>> > MiniClusterWithClientResource.flinkCluster.getClusterClient().getWebInterfaceURL(), > >>>> I get > >>>> {"errors":["Not found."]}. I'm not sure if this is intentional. > >>>> > >>>> > >>>> > >>>> > >>>> On Tue, Oct 6, 2020 at 4:16 PM Dan Hill <quietgol...@gmail.com> > wrote: > >>>> > >>>>> @Aljoscha - Thanks! That setup lets fixing the hacky absolute path > >>>>> reference. However, the actual log calls are not printing to the > console. > >>>>> Only errors appear in my terminal window and the test logs. Maybe > console > >>>>> logger does not work for this junit setup. I'll see if the file > version > >>>>> works. > >>>>> > >>>>> On Tue, Oct 6, 2020 at 4:08 PM Austin Cawley-Edwards < > >>>>> austin.caw...@gmail.com> wrote: > >>>>> > >>>>>> What Aljoscha suggested is what works for us! > >>>>>> > >>>>>> On Tue, Oct 6, 2020 at 6:58 PM Aljoscha Krettek < > aljos...@apache.org> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi Dan, > >>>>>>> > >>>>>>> to make the log properties file work this should do it: assuming > the > >>>>>>> log4j.properties is in //src/main/resources. You will need a > >>>>>>> BUILD.bazel > >>>>>>> in that directory that has only the line > >>>>>>> "exports_files(["log4j.properties"]). Then you can reference it in > >>>>>>> your > >>>>>>> test via "resources = ["//src/main/resources:log4j.properties"],". > Of > >>>>>>> course you also need to have the right log4j deps (or slf4j if > you're > >>>>>>> using that) > >>>>>>> > >>>>>>> Hope that helps! > >>>>>>> > >>>>>>> Aljoscha > >>>>>>> > >>>>>>> On 07.10.20 00:41, Dan Hill wrote: > >>>>>>>> I'm trying to use Table API for my job. I'll soon try to get a > test > >>>>>>>> working for my stream job. > >>>>>>>> - I'll parameterize so I can have different sources and sink for > >>>>>>> tests. > >>>>>>>> How should I mock out a Kafka source? For my test, I was planning > >>>>>>> on > >>>>>>>> changing the input to be from a temp file (instead of Kafka). > >>>>>>>> - What's a good way of forcing a watermark using the Table API? > >>>>>>>> > >>>>>>>> > >>>>>>>> On Tue, Oct 6, 2020 at 3:35 PM Dan Hill <quietgol...@gmail.com> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Thanks! > >>>>>>>>> > >>>>>>>>> Great to know. I copied this junit5-jupiter-starter-bazel > >>>>>>>>> < > >>>>>>> > https://github.com/junit-team/junit5-samples/tree/main/junit5-jupiter-starter-bazel > > > >>>>>>> rule > >>>>>>>>> into my repository (I don't think junit5 is supported directly > with > >>>>>>>>> java_test yet). I tried a few ways of bundling > `log4j.properties` > >>>>>>> into the > >>>>>>>>> jar and didn't get them to work. My current iteration hacks the > >>>>>>>>> log4j.properties file as an absolute path. My failed attempts > >>>>>>> would spit > >>>>>>>>> an error saying log4j.properties file was not found. This route > >>>>>>> finds it > >>>>>>>>> but the log properties are not used for the java logger. > >>>>>>>>> > >>>>>>>>> Are there a better set of rules to use for junit5? > >>>>>>>>> > >>>>>>>>> # build rule > >>>>>>>>> java_junit5_test( > >>>>>>>>> name = "tests", > >>>>>>>>> srcs = glob(["*.java"]), > >>>>>>>>> test_package = "ai.promoted.logprocessor.batch", > >>>>>>>>> deps = [...], > >>>>>>>>> jvm_flags = > >>>>>>>>> > >>>>>>> > ["-Dlog4j.configuration=file:///Users/danhill/code/src/ai/promoted/logprocessor/batch/log4j.properties"], > >>>>>>>>> ) > >>>>>>>>> > >>>>>>>>> # log4j.properties > >>>>>>>>> status = error > >>>>>>>>> name = Log4j2PropertiesConfig > >>>>>>>>> appenders = console > >>>>>>>>> appender.console.type = Console > >>>>>>>>> appender.console.name = LogToConsole > >>>>>>>>> appender.console.layout.type = PatternLayout > >>>>>>>>> appender.console.layout.pattern = %d [%t] %-5p %c - %m%n > >>>>>>>>> rootLogger.level = info > >>>>>>>>> rootLogger.appenderRefs = stdout > >>>>>>>>> rootLogger.appenderRef.stdout.ref = LogToConsole > >>>>>>>>> > >>>>>>>>> On Tue, Oct 6, 2020 at 3:34 PM Austin Cawley-Edwards < > >>>>>>>>> austin.caw...@gmail.com> wrote: > >>>>>>>>> > >>>>>>>>>> Oops, this is actually the JOIN issue thread [1]. Guess I should > >>>>>>> revise > >>>>>>>>>> my previous "haven't had issues" statement hah. Sorry for the > >>>>>>> spam! > >>>>>>>>>> > >>>>>>>>>> [1]: > >>>>>>>>>> > >>>>>>> > apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Streaming-SQL-Job-Switches-to-FINISHED-before-all-records-processed-td38382.html > >>>>>>>>>> > >>>>>>>>>> On Tue, Oct 6, 2020 at 6:32 PM Austin Cawley-Edwards < > >>>>>>>>>> austin.caw...@gmail.com> wrote: > >>>>>>>>>> > >>>>>>>>>>> Unless it's related to this issue[1], which was w/ my JOIN and > >>>>>>> time > >>>>>>>>>>> characteristics, though not sure that applies for batch. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Austin > >>>>>>>>>>> > >>>>>>>>>>> [1]: > >>>>>>>>>>> > >>>>>>> > apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SQL-Streaming-Join-Creates-Duplicates-td37764.html > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Oct 6, 2020 at 6:20 PM Austin Cawley-Edwards < > >>>>>>>>>>> austin.caw...@gmail.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hey Dan, > >>>>>>>>>>>> > >>>>>>>>>>>> We use Junit5 and Bazel to run Flink SQL tests on a mini > >>>>>>> cluster and > >>>>>>>>>>>> haven’t had issues, though we’re only testing on streaming > jobs. > >>>>>>>>>>>> > >>>>>>>>>>>> Happy to help setting up logging with that if you’d like. > >>>>>>>>>>>> > >>>>>>>>>>>> Best, > >>>>>>>>>>>> Austin > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Oct 6, 2020 at 6:02 PM Dan Hill < > quietgol...@gmail.com> > >>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> I don't think any of the gotchas apply to me (at the bottom > of > >>>>>>> this > >>>>>>>>>>>>> link). > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>> > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html#junit-rule-miniclusterwithclientresource > >>>>>>>>>>>>> > >>>>>>>>>>>>> I'm assuming for a batch job that I don't have to do anything > >>>>>>> for: > >>>>>>>>>>>>> "You can implement a custom parallel source function for > >>>>>>> emitting > >>>>>>>>>>>>> watermarks if your job uses event time timers." > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Tue, Oct 6, 2020 at 2:42 PM Dan Hill < > quietgol...@gmail.com> > >>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> I've tried to enable additional logging for a few hours > >>>>>>> today. I > >>>>>>>>>>>>>> think something with junit5 is swallowing the logs. I'm > >>>>>>> using Bazel and > >>>>>>>>>>>>>> junit5. I setup MiniClusterResourceConfiguration using a > >>>>>>> custom > >>>>>>>>>>>>>> extension. Are there any known issues with Flink and > >>>>>>> junit5? I can try > >>>>>>>>>>>>>> switching to junit4. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> When I've binary searched this issue, this failure happens > if > >>>>>>> my > >>>>>>>>>>>>>> query in step 3 has a join it. If I remove the join, I can > >>>>>>> remove step 4 > >>>>>>>>>>>>>> and the code still works. I've renamed a bunch of my tables > >>>>>>> too and the > >>>>>>>>>>>>>> problem still exists. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Tue, Oct 6, 2020, 00:42 Aljoscha Krettek < > >>>>>>> aljos...@apache.org> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi Dan, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> there were some bugs and quirks in the MiniCluster that we > >>>>>>> recently > >>>>>>>>>>>>>>> fixed: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> - https://issues.apache.org/jira/browse/FLINK-19123 > >>>>>>>>>>>>>>> - https://issues.apache.org/jira/browse/FLINK-19264 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> But I think they are probably unrelated to your case. Could > >>>>>>> you > >>>>>>>>>>>>>>> enable > >>>>>>>>>>>>>>> logging and see from the logs whether the 2) and 3) jobs > >>>>>>> execute > >>>>>>>>>>>>>>> correctly on the MiniCluster? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>> Aljoscha > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 06.10.20 08:08, Dan Hill wrote: > >>>>>>>>>>>>>>>> I'm writing a test for a batch job using > >>>>>>>>>>>>>>> MiniClusterResourceConfiguration. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Here's a simple description of my working test case: > >>>>>>>>>>>>>>>> 1) I use TableEnvironment.executeSql(...) to create a > >>>>>>> source and > >>>>>>>>>>>>>>> sink table > >>>>>>>>>>>>>>>> using tmp filesystem directory. > >>>>>>>>>>>>>>>> 2) I use executeSql to insert some test data into the > >>>>>>> source tabel. > >>>>>>>>>>>>>>>> 3) I use executeSql to select from source and insert into > >>>>>>> sink. > >>>>>>>>>>>>>>>> 4) I use executeSql from the same source to a different > >>>>>>> sink. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> When I do these steps, it works. If I remove step 4, no > >>>>>>> data gets > >>>>>>>>>>>>>>> written > >>>>>>>>>>>>>>>> to the sink. My actual code is more complex than this > (has > >>>>>>> create > >>>>>>>>>>>>>>> view, > >>>>>>>>>>>>>>>> join and more tables). This is a simplified description > but > >>>>>>>>>>>>>>> highlights the > >>>>>>>>>>>>>>>> weird error. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Has anyone hit issues like this? I'm assuming I have a > >>>>>>> small code > >>>>>>>>>>>>>>> bug in > >>>>>>>>>>>>>>>> my queries that's causing issues. These queries appear to > >>>>>>> work in > >>>>>>>>>>>>>>>> production so I'm confused. Are there ways of viewing > >>>>>>> failed jobs > >>>>>>>>>>>>>>> or > >>>>>>>>>>>>>>>> queries with MiniClusterResourceConfiguration? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks! > >>>>>>>>>>>>>>>> - Dan > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > > > >
