As Chesnay said, it not necessary to use a pool as the connection is reused
across split. However, if you had to customize it for some reasons, you can
do it starting from the JDBC Input and Output format.
cheers!
2016-07-05 13:27 GMT+02:00 Harikrishnan S :
> Awesome ! Thanks a lot ! I should pr
Awesome ! Thanks a lot ! I should probably write a blog post somewhere
explaining this.
On Tue, Jul 5, 2016 at 4:28 PM, Chesnay Schepler wrote:
> They serve a similar purpose.
>
> OutputFormats originate from the Batch API, whereas SinkFunctions are a
> Streaming API concept.
>
> You can however
They serve a similar purpose.
OutputFormats originate from the Batch API, whereas SinkFunctions are a
Streaming API concept.
You can however use OutputFormats in the Streaming API using the
DataStrea#writeUsingOutputFormat.
Regards,
Chesnay
On 05.07.2016 12:51, Harikrishnan S wrote:
Ah tha
Ah that makes send. Also what's the difference between a RichOutputFormat
and a RichSinkFunction ? Can I use JDBCOutputFormat as a sink in a stream ?
On Tue, Jul 5, 2016 at 3:53 PM, Chesnay Schepler wrote:
> Hello,
>
> an instance of the JDBCOutputFormat will use a single connection to send
> al
Hello,
an instance of the JDBCOutputFormat will use a single connection to send
all values.
Essentially
- open(...) is called at the very start to create the connection
- then all invoke/writeRecord calls are executed (using the same connection)
- then close() is called to clean up.
The total
The basic idea was that I would create a pool of connections in the open()
method in a custom sink and each invoke() method gets one connection from
the pool and does the upserts needed. I might have misunderstood how sinks
work in flink though.
On Tue, Jul 5, 2016 at 2:22 PM, Flavio Pompermaier
Oh. So you mean if I write a custom sink for a db, I just need to create
one connection in the open() method and then the invoke() method will reuse
it ? Basically I need to do 35k-50k+ upserts in postgres. Can I reuse
JDBCOutputFormat
for this purpose ? I couldn't find a proper document describing
The connection will be managed by the splitManager, no need of using a
pool. However, if you had to, probably you should look into
establishConnection() method of the JDBCInputFormat.
2016-07-05 10:52 GMT+02:00 Flavio Pompermaier :
> why do you need a connection pool?
> On 5 Jul 2016 11:41, "Ha
why do you need a connection pool?
On 5 Jul 2016 11:41, "Harikrishnan S" wrote:
> Hi,
>
> Are there any examples of implementing a jdbc sink in flink using a
> connection pool ?
>
> Thanks
>
> On Tue, Jul 5, 2016 at 2:00 PM, Harikrishnan S
> wrote:
>
>> Hi,
>>
>> Are there any examples of implem
Hi,
Are there any examples of implementing a jdbc sink in flink using a
connection pool ?
Thanks
On Tue, Jul 5, 2016 at 2:00 PM, Harikrishnan S wrote:
> Hi,
>
> Are there any examples of implementing a jdbc sink in flink using a
> connection pool ?
>
> Thanks
>
> On Tue, Jul 5, 2016 at 1:57 PM
10 matches
Mail list logo
