Thanks! I have a few follow up questions. I have searched but was unable to find where to set the log level just for "org.apache.flink.table.runtime.generated.CompileUtils", so I set "rootLogger.level = DEBUG" in "log4j-cli.properties" and got the logs, but was unable to find generated code in "flink-root-sql-client.xxx.internal.log". So my questions are:
- My understanding is that Flink SQL will generate Java code equivalent and submit the compiled jar to the cluster as a job, is that correct? Can I get the source code to see what kind of low level API the SQL query generated? - How do I set the log level for "org.apache.flink.table.runtime.generated.CompileUtils" only? Also, when I start a local Flink cluster, I am able to query the table and immediately see the results. I did the same on a Zeppelin notebook with a remote cluster which also worked. However after 8-10 hours, I tried the same simple SELECT again in Zeppelin, no data were shown. I INSERT'd a new row to the source table and the result are showing again. Is this related to the behavior of checkpointing? Because I already had "'scan.startup.mode' = 'earliest-offset'" so I expected it to work even with no new data coming in for a long time. Maybe a bit much to ask...but thank you again for the help! On Mon, Mar 14, 2022 at 11:14 AM Caizhi Weng <tsreape...@gmail.com> wrote: > Sorry for misleading. I mean if you enable checkpointing then selected > results are only visible after the checkpoint completes. If there is no > checkpointing the results will be instantly visible, just as the document > records. > > Caizhi Weng <tsreape...@gmail.com> 于2022年3月14日周一 11:12写道： > >> Hi! >> >> I see. So you're running a streaming job. "select" in a streaming job >> will only produce visible data when you enable checkpointing (this is due >> to the exactly-once guarantee of Flink), see [1] for more detail. See [2] >> on how to enable checkpointing for Flink SQL. >> >> Generated code are also in the logs if you set the appropriate logging >> level. >> >> [1] >> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/overview/#execute-a-query >> [2] https://stackoverflow.com/a/65681975 >> >> >> dz902 <dz9...@gmail.com> 于2022年3月14日周一 11:08写道： >> >>> Hi Caizhi, >>> >>> Thanks for the quick reply. I was just running a simple "SELECT * FROM >>> tbl_name" against a table with Kafka connector, but no data showed up and >>> no errors. >>> >>> Where can I find the generated code if I'm using SQL client? >>> >>> Thanks! >>> >>> >>> On Mon, Mar 14, 2022 at 10:58 AM Caizhi Weng <tsreape...@gmail.com> >>> wrote: >>> >>>> Hi! >>>> >>>> For stages and logs you can refer to the web UI. For generated code set >>>> logging level of org.apache.flink.table.runtime.generated.CompileUtils to >>>> debug. >>>> >>>> What query are you running? If possible can you share your SQL in the >>>> mailing list? >>>> >>>> dz902 <dz9...@gmail.com> 于2022年3月14日周一 10:42写道： >>>> >>>>> Hi, >>>>> >>>>> I'm trying to debug SQL queries, from SQL client or Zeppelin notebook >>>>> (job submitted to remote cluster). >>>>> >>>>> I have a query not getting any data. How do I debug? Can I see the >>>>> actual code generated from the SQL query? Or is it possible to show all >>>>> the >>>>> stages, actions or logs generated by the query? >>>>> >>>>> Thanks, >>>>> Dai >>>>> >>>>
