Because in my use case the parallelism was not based on a range on keys/numbers but on a range of dates, so I needed a custom Parameter Provider. For what regards pushdown I don't know how Flink/Blink currently works..for example, let's say I have a Postgres catalog containing 2 tables (public.A and public.B). If I do the following query : SELECT public.A.x, public.B.y FROM public.A JOIN public.B on public.A.pk = public.B.fk. Will this be pushdown as a single query or will Flink fetch both tables and the perform the join? Talking with Bowen I understood that to avoid this I could define a VIEW in the db (but this is not alway possible) or in Flink (but from what I know this is very costly). In this case a parameter "scan.query.statement" without a "scan.parameter.values.provider.class" is super helpful and could improve performance a lot!
On Wed, Apr 22, 2020 at 11:06 AM Jingsong Li <jingsongl...@gmail.com> wrote: > Hi, > > You are right about the lower and upper, it is a must to parallelize the > fetch of the data. > And filter pushdown is used to filter more data at JDBC server. > > Yes, we can provide "scan.query.statement" and > "scan.parameter.values.provider.class" for jdbc connector. But maybe we > need be careful about this too flexible API. > > Can you provide more about your case? Why can not been solved by lower and > upper with filter pushdown? > > Best, > Jingsong Lee > > On Wed, Apr 22, 2020 at 4:45 PM Flavio Pompermaier <pomperma...@okkam.it> > wrote: > >> Maybe I am wrong but support pushdown for JDBC is one thing (that is >> probably useful) while parameters providers are required if you want to >> parallelize the fetch of the data. >> You are not mandated to use NumericBetweenParametersProvider, you can use >> the ParametersProvider you prefer, depending on the statement you have. >> Or do you have in mind something else? >> >> On Wed, Apr 22, 2020 at 10:33 AM Jingsong Li <jingsongl...@gmail.com> >> wrote: >> >>> Hi, >>> >>> Now in JDBCTableSource.getInputFormat, It's written explicitly: WHERE >>> XXX BETWEEN ? AND ?. So we must use `NumericBetweenParametersProvider`. >>> I don't think this is a good and long-term solution. >>> I think we should support filter push-down for JDBCTableSource, so in >>> this way, we can write the filters that we want, what do you think? >>> >>> Best, >>> Jingsong Lee >>> >>> >>> On Tue, Apr 21, 2020 at 10:00 PM Flavio Pompermaier < >>> pomperma...@okkam.it> wrote: >>> >>>> Hi all, >>>> we have a use case where we have a prepared statement that we >>>> parameterize using a custom parameters provider (similar to what happens in >>>> testJDBCInputFormatWithParallelismAndNumericColumnSplitting[1]). >>>> How can we handle this using the JDBC table API? >>>> What should we do to handle such a use case? Is there anyone willing to >>>> mentor us in its implementation? >>>> >>>> Another question: why flink-jdbc has not been renamed to >>>> flink-connector-jdbc? >>>> >>>> Thanks in advance, >>>> Flavio >>>> >>>> [1] >>>> https://github.com/apache/flink/blob/master/flink-connectors/flink-jdbc/src/test/java/org/apache/flink/api/java/io/jdbc/JDBCInputFormatTest.java >>>> >>> >>> >>> -- >>> Best, Jingsong Lee >>> >> >> > > -- > Best, Jingsong Lee >
