er
> >>>> 1.source handoff
> >>>>
> >>>> We both support Fixed-Start-Position And Switched-start-Position. The
> >>>> default is Fixed-Start-Position. Use option
> >> switched-start-position-enabled
> >>>> to control
Hello Ran Tao,
Thanks for this FLIP.
I have a comment about the handover of switching context between sources.
You have proposed to define interfaces named around timestamps,
SupportsGetEndTimestamp
and SupportsSwitchedStartTimestamp.
These work well with KafkaSource as the downstream child sour
; enumerator
>>>> endTimestamp
>>>> and pass it to the next source. and Next source uses
>>>> SupportsSwitchedStartTimestamp to apply it.
>>>>
>>>> 2.more child sources
>>>>
>>>> Yes, this is consistent with the hybr
1 varchar,
> > > f2 bigint
> > > ) with(
> > > 'connector'='hybrid',
> > > 'source-identifiers'='historical01,historical02,realtime',
> > > 'historical01.connector'='filesystem'
> > > &
; 'historical01.format' = 'csv',
> > 'historical02.connector'='filesystem'
> > 'historical02.path' = '/tmp/a.csv',
> > 'historical02.format' = 'csv',
> > 'realtime.connector'='
',
> 'realtime.format' = 'csv'
> );
> ```
>
> more details can be found at [1] & [2].
> Looking forward to your more concerns and opinions.
>
> 1.
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235836225
> 2.https://github.c
found at [1] & [2].
Looking forward to your more concerns and opinions.
1.https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235836225
2.https://github.com/apache/flink/pull/21841
Best Regards,
Ran Tao
Ran Tao 于2022年12月15日周四 16:02写道：
> Hi guys. HybridSource is a good feature,
Hi, all. i have updated flip-278[1]. I think all problems or comments has
been addressed.
1.about option prefix, we use identifiers.
2.table api implementation and demo
3.about switched dynamic position (hybrid source use it auto switch from
previous to next source)
More details can be found at d
Hi, Martijn. i have updated the flip about table api & switched start
timestamp.
thanks.
Martijn Visser 于2022年12月16日周五 16:59写道：
> Hi Ran,
>
> For completeness, this is a new thread that was already previously started
> at https://lists.apache.org/thread/xptn2ddzj34q9f5vtbfb62lsybmvcwjq. I'm
> li
a mistake,
childSources.get(sourceIndex).setStartTimetamp(switchedTimestamp);
Ran Tao 于2022年12月19日周一 16:10写道：
> Hi, John. thanks for your comments.
> About question-2 the "handoff" is using for switching next source
> seamlessly. but it's an option. Not every hybrid source job need to using
> th
Hi, John. thanks for your comments.
About question-2 the "handoff" is using for switching next source
seamlessly. but it's an option. Not every hybrid source job need to using
this mode.
The hybrid source sql or table need to implement two ways like DataStream
api below. One for fixed position, u
Hello all,
Thanks for the FLIP, Ran!
The HybridSource is a really cool feature, and I was glad to see a proposal to
expose it in the Table and SQL APIs.
My main question is also about the switching control (question 2). It seems
like the existing Kafka connector has all the options you'd want
Hi, Martijn, thanks for your comments.
Using identifier as child source prefix may be a good way instead of index.
i will update the flip to illustrate how we can read from hybrid schema to
generate child schemas for the question1.
question2 is start position for the next kafka source. But curre
Hi Ran,
For completeness, this is a new thread that was already previously started
at https://lists.apache.org/thread/xptn2ddzj34q9f5vtbfb62lsybmvcwjq. I'm
linking them because I think Timo's comments are relevant to be kept with
this discussion thread.
I agree with Timo's comments from there tha
Fyi.
This flip using index as child source option prefix because we may use the
same connector as hybrid child sources.
e.g.
create table hybrid_source(
f0 varchar,
f1 varchar,
f2 bigint
) with(
'connector'='hybrid',
'sources'='filesystem,filesystem',
'0.path' = '/tmp/a.csv',
'0.format' =
the
respond from the Timo and sent this email. look forward to your comments.
```
Hi Ran,
Thanks for proposing a FLIP. Btw according to the process, the subject
of this email should be `[DISCUSS] FLIP-278: Hybrid Source Connector` so
that people can identify this discussion as a FLIP discus
16 matches
Mail list logo
