Maybe you can share a bit more about why you need only one connection to
Cassandra across all TaskManagers, so we can better help?
On Wed, May 2, 2018 at 4:08 AM, Piotr Nowojski <pi...@data-artisans.com>
wrote:
> Hi,
>
> The only way that I can think of is if you keep your flatMap operator with
> parallelism 1, but that might defeat the purpose. Otherwise there is no way
> to open one single connection and share it across multiple TaskManagers
> (which can be running on different physical machines). Please rethink your
> solution/approach with respect to distributed nature of Flink.
>
> However there are some valid use cases where one would like to have some
> part of his job graph distributed and some part(s) non distributed - like
> issuing one single commit after a distributed write, or processing a data
> in parallel but writing them to a relational database like MySQL via one
> single Sink operator..
>
> Piotrek
>
>
> On 26 Apr 2018, at 15:23, Soheil Pourbafrani <soheil.i...@gmail.com>
> wrote:
>
> Here is my code
>
> stream.flatMap(new FlatMapFunction<byte[], Void>() {
>
> @Override
> public void flatMap(byte[] value, Collector<Void> out) throws Exception {
> Parser.setInsert(true);
> CassandraConnection.connect();
> Parser.setInsert(true);
> System.out.println("\n*********** New Message ***********\n");
> System.out.println("Row Number : " + i ++ );
> System.out.println("Message : " + HexUtiles.bytesToHex(value));
> Parser.parse(ByteBuffer.wrap(value), ConfigHashMap);
> }
> });
>
>
>
> On Thu, Apr 26, 2018 at 5:22 PM, Soheil Pourbafrani <soheil.i...@gmail.com
> > wrote:
>
>> I want to use Cassandra native connection (Not Flink Cassandra
>> connection) to insert some data into Cassandra. According to the design of
>> the code, the connection to Cassandra will open once at the start and all
>> taskmanager use it to write data. It's ok running in local mode.
>>
>> The problem is when I submit the code on YARN cluster, as each
>> taskmanager has it's own JVM, the connection to the Cassandra will not
>> share and I should open and close it for each taskmanager. Is there any way
>> to have a connection for all taskmanagers?
>>
>
>
>
--
"So you have to trust that the dots will somehow connect in your future."
