Re: Job Manager HA manual setup

> the task manager configuration is out of sync. Could you check this > please? > > Moreover, it will be helpful to have a look at the jobmanager and task > manager logs to further investigate this. Can you share these? > (Privately works as well of course.) > > – Ufuk > >

Re: Job Manager HA manual setup

typos We have tried this one the job manager can failover, but the task manager CAN'T be relocated to the new task manager. Is there some settings for this one ? Or is the task manager also can be relocate to the new job manager ? Cheers On Sat, Feb 27, 2016 at 7:27 AM, Welly Tambunan

Job Manager HA manual setup

master ? Thanks for your help. Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Optimal Configuration for Cluster

the machines have plenty of > memory, I would configure a bigger heap than the 4 GB you had > previously. Furhermore, you can also consider adding more network > buffers, which should improve job throughput. > > – Ufuk > > On Tue, Feb 23, 2016 at 11:57 AM, Welly Tambunan

Re: Optimal Configuration for Cluster

Hi Ufuk and Fabian, Is that better to start 48 task manager ( one slot each ) in one machine than having single task manager with 48 slot ? Any trade-off that we should know etc ? Cheers On Tue, Feb 23, 2016 at 3:03 PM, Welly Tambunan wrote: > Hi Ufuk, > > Thanks for the explanation

Re: Optimal Configuration for Cluster

: true > > But you are using the DataStream API anyways, right? > > – Ufuk > > > On Tue, Feb 23, 2016 at 6:36 AM, Welly Tambunan wrote: > > Hi Fabian, > > > > Previously when using flink 0.9-0.10 we start the cluster with streaming > > mode or batch mode

Re: Optimal Configuration for Cluster

ory > pools. So the "taskmanager.network.numberOfBuffers" configuration whill > hopefully disappear at some point in the future. > > Best, Fabian > > 2016-02-19 9:34 GMT+01:00 Welly Tambunan : > >> Hi All, >> >> We are trying to running our job in clust

Re: Optimal Configuration for Cluster

cutionMode(ExecutionMode.BATCH)). > > Eventually, we want to merge the network buffer and the managed memory > pools. So the "taskmanager.network.numberOfBuffers" configuration whill > hopefully disappear at some point in the future. > > Best, Fabian > > 2016-02-19 9

Optimal Configuration for Cluster

t automatically resolve by memory/network buffer manager ? Thanks a lot for the help Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Flink v0.10.2

> > > On Wed, Jan 13, 2016 at 8:39 AM, Welly Tambunan wrote: > >> Hi All, >> >> We currently using snapshot version for development as we face Data >> Stream union error. For deployment we may need to built the flink from >> the master. >> >> &g

Flink v0.10.2

Hi All, We currently using snapshot version for development as we face Data Stream union error. For deployment we may need to built the flink from the master. I want to ask when this version will be released ? Any roadmap and plan i can look for this release ? Thanks a lot Cheers -- Welly

Re: Security in Flink

ng for are - >>>> >>>> 1. How do I do user level authentication to ensure that a job is >>>> submitted/deleted/modified by the right user ? Is it possible though the >>>> web client ? >>>> 2. Authentication across multiple slave nodes (where the task managers >>>> are running) and driver program so that they can communicate with each >>>> other >>>> 3. Support for SSL/encryption for data exchanged happening across the >>>> slave nodes >>>> 4. Support for pluggable authentication with existing solution like LDAP >>>> >>>> If not there today is there a roadmap for these security features ? >>>> >>>> Regards, >>>> Sourav >>>> >>> >>> >> > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Security in Flink

;> submitted/deleted/modified by the right user ? Is it possible though the >>> web client ? >>> 2. Authentication across multiple slave nodes (where the task managers >>> are running) and driver program so that they can communicate with each other >>> 3. Support for SSL/encryption for data exchanged happening across the >>> slave nodes >>> 4. Support for pluggable authentication with existing solution like LDAP >>> >>> If not there today is there a roadmap for these security features ? >>> >>> Regards, >>> Sourav >>> >> >> > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Material on Apache flink internals

that followed by hadoop or spark. Is any >>> good documentation/blog/video avalable which talks about this internal. I >>> am ok from a batch or streaming point of view. >>> >>> It will be great if some one can share this info. Thank you for your >>> excellent work. >>> >>> -- >>> Regards, >>> Madhukara Phatak >>> http://datamantra.io/ >>> >> >> -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Documentation for Fold

Hi All, Currently i'm going through the documentation for DataStream here and minor error in the docs. I thought i should inform you. I think fold only works for keyed data stream. [image: Inline image 1] Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com

Re: Data Stream union error after upgrading from 0.9 to 0.10.1

.10 branch which will eventually be Flink 0.10.2. >> >> Cheers, >> Max >> >> On Tue, Dec 1, 2015 at 12:04 PM, Welly Tambunan >> wrote: >> > Thanks a lot Aljoscha. >> > >> > When it will be released ? >> > >> > Cheers >>

Re: Running WebClient from Windows

lient though. > Would you mind and open a JIRA for that? > > Thanks, > Fabian > > 2015-12-02 3:00 GMT+01:00 Welly Tambunan : > >> Hi All, >> >> Is there any way to run WebClient for uploading the job from windows ? >> >> I try to run that from m

Running WebClient from Windows

e 261: `if [[ "$SLAVE" =~ ^.*/([0-9a-zA-Z. -]+)$ ]]; then' Starting Flink webclient [Terminate] Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Data Stream union error after upgrading from 0.9 to 0.10.1

e next 2-3 weeks. > > > On Tue, Dec 1, 2015 at 11:48 PM, Welly Tambunan wrote: > >> Hi Aljoscha, >> >> Is this fix has already been available on 0.10-SNAPSHOT ? >> >> >> Cheers >> >> On Tue, Dec 1, 2015 at 6:04 PM, Welly Tambunan wrote: &

Re: Data Stream union error after upgrading from 0.9 to 0.10.1

Hi Aljoscha, Is this fix has already been available on 0.10-SNAPSHOT ? Cheers On Tue, Dec 1, 2015 at 6:04 PM, Welly Tambunan wrote: > Thanks a lot Aljoscha. > > When it will be released ? > > Cheers > > On Tue, Dec 1, 2015 at 5:48 PM, Aljoscha Krettek > wrote: &

Re: Data Stream union error after upgrading from 0.9 to 0.10.1

Thanks a lot Aljoscha. When it will be released ? Cheers On Tue, Dec 1, 2015 at 5:48 PM, Aljoscha Krettek wrote: > Hi, > I relaxed the restrictions on union. This should make it into an upcoming > 0.10.2 bugfix release. > > Cheers, > Aljoscha > > On 01 Dec 2015,

Data Stream union error after upgrading from 0.9 to 0.10.1

e.org/jira/browse/FLINK-3080 Is there any plan which release this will be ? Another issue i have after upgrading is can't union with different level of parallelism. I think we will need to fall back to 0.9 again for the time being. Cheers -- Welly Tambunan Triplelands http://weltam.wor

Re: Standalone Cluster vs YARN

gt; Regards, > Andreas > > On Wed, Nov 25, 2015 at 10:02 AM, Welly Tambunan > wrote: > >> Hi Ufuk >> >> >In failure cases I find YARN more convenient, because it takes care of >> restarting failed task manager processes/containers for you. >> >> So

Re: Standalone Cluster vs YARN

rsistance. > > With YARN, Flink can automatically start new Taskmanagers (and > JobManagers) to compensate for failures. In cluster mode, you need stand-by > TMs and JMs and manually take care that these are "filled-up" again in case > of a failure. > > 2015-11-25

Re: Standalone Cluster vs YARN

ebi : > >> > On 25 Nov 2015, at 02:35, Welly Tambunan wrote: >> > >> > Hi All, >> > >> > I would like to know if there any feature differences between using >> Standalone Cluster vs YARN ? >> > >> > Until now we are using St

Re: Standalone Cluster vs YARN

Hi Ufuk >In failure cases I find YARN more convenient, because it takes care of restarting failed task manager processes/containers for you. So this mean that we don't need zookeeper ? Cheers On Wed, Nov 25, 2015 at 3:46 PM, Ufuk Celebi wrote: > > On 25 Nov 2015, at 02:35,

Standalone Cluster vs YARN

at if there's some value to that. Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Adding TaskManager on Cluster

On 24 Nov 2015, at 10:39, Welly Tambunan wrote: > > > > Hi Till, > > > > I've just tried that. It's works like a charm. Thanks a lot. > > > > Is there any documentation on taskmanager.sh and other script and the > parameters ? I try to look at the docs

Re: Adding TaskManager on Cluster

te: > Hi Welly, > > you can always start a new TaskManager by simply calling taskmanager.sh > start [streaming|batch], depending whether you are running a streaming > cluster or a batch cluster. You can find the script in /bin. > > Cheers, > Till > ​ > > On Tue, N

Re: Adding TaskManager on Cluster

What i'm looking for here is the ability to add a node to the cluster (scale out) when there's no task slot left for use. On Tue, Nov 24, 2015 at 4:24 PM, Welly Tambunan wrote: > Hi All, > > Currently we are running flink using standalone mode. > > Is there any w

Adding TaskManager on Cluster

Hi All, Currently we are running flink using standalone mode. Is there any way to add one node ( task manager ) to the cluster without bringing the cluster down ? Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Cancel Streaming Job

> > On 24 Nov 2015, at 08:26, Gyula Fóra wrote: > > > > Hi! > > > > This issue has been fixed very recently and the fix will go into the > upcoming bugfix release. (0.10.1) > > > > Should be out in the next few days :) > > > > Cheers > >

Re: Cancel Streaming Job

t; but it's not working. > > > > Cheers > > > -- > Welly Tambunan > Triplelands > > http://weltam.wordpress.com > http://www.triplelands.com <http://www.triplelands.com/blog/> > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Cancel Streaming Job

Hi All, Is there any way to stop/cancel the job that's restarting ? I have already stop the cluster and start it again but seems it's still restarting in dashboard. I also try to cancel the job via CLI by running bin/flink cancel but it's not working. Cheers -- Welly Tambu

Re: Specially introduced Flink to chinese users in CNCC(China National Computer Congress)

6050.n4.nabble.com/Specially-introduced-Flink-to-chinese-users-in-CNCC-China-National-Computer-Congress-tp3254p3574.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Apache Flink 0.10.0 released

0)6 55 53 04 01 | *T*: +31 (0)88 10 44 555 > *E*: leon...@sagent.io | *W*: sagent.io | Disclaimer > <http://sagent.io/email-disclaimer> | Sagent BV > Herengracht 504 | 1017CB Amsterdam | Netherlands > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Apache Flink Operator State as Query Cache

ality to make the Key/Value operator state >>> usable in CoMap/CoFlatMap as well (currently it only works in windows and >>> in Map/FlatMap/Filter functions over the KeyedStream). >>> Until the, you should be able to use a simple Java HashMap and use the >>> "Chec

Re: Apache Flink Operator State as Query Cache

gt; > You can see the API reference and an example here: > https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#working-with-time > > Is this what you are looking for? > > Kostas > > > On Sat, Nov 14, 2015 at 1:54 AM, Welly Tambunan wrote

Re: Apache Flink Operator State as Query Cache

Hi Robert, Is this version has already handle the stream perfection or out of order event ? Any resource on how this work and the API reference ? Cheers On Fri, Nov 13, 2015 at 4:00 PM, Welly Tambunan wrote: > Awesome ! > > This is really the best weekend gift ever. :) > > Ch

Re: Multilang Support on Flink

> reflected in flink-language-binding-generic module. I'm not aware how > Spark or Storm communication protocols work but this protocol is > rather low level. > > Cheers, > Max > > On Fri, Nov 13, 2015 at 9:49 AM, Welly Tambunan wrote: > > Hi All, > > > &g

Re: Flink, Kappa and Lambda

se. This context is > far too young that it provides you with additional value strictly following > a certain pattern, eg to make it more easier to integrate with third-party > software. > > Best > Christian > > > 2015-11-13 9:51 GMT+01:00 Welly Tambunan : > >> Hi r

Re: Apache Flink Forward Videos

; > Cheers, > Max > > On Fri, Nov 13, 2015 at 11:08 AM, Welly Tambunan > wrote: > > Hi All, > > > > I've just notice that the video has already available for this one. > > > > http://flink-forward.org/?post_type=session > > > > > > A

Apache Flink Forward Videos

Hi All, I've just notice that the video has already available for this one. http://flink-forward.org/?post_type=session Another weekend gift for all. Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Apache Flink Operator State as Query Cache

example has the update already: > http://apache.mirror.digionline.de/flink/flink-0.10.0/ > > On Fri, Nov 13, 2015 at 9:50 AM, Welly Tambunan wrote: > >> Hi Aljoscha, >> >> Thanks for this one. Looking forward for 0.10 release version. >> >> Cheers &

Re: Flink, Kappa and Lambda

community to prove to me it doesn't lose data in > these corner cases, so too do I expect Kafka to prove it is resilient. In > the absence of software formally proven correct, I look to empirical > evidence in the form of chaos monkey type tests. > > > On Wednesday, November 11,

Re: Flink, Kappa and Lambda

cture there is a following book - > https://www.manning.com/books/big-data (Big Data. Principles and best > practices of scalable realtime data systems > Nathan Marz and James Warren). > > Regards, > Roman > > 2015-11-12 4:47 GMT+03:00 Welly Tambunan : > >>

Re: Apache Flink Operator State as Query Cache

ostly to hardening and some > infrastructure changes (for example annotations that specify API stability) > for the 1.0 release. > > Let us know if you need more information. > > Cheers, > Aljoscha > > On 12 Nov 2015, at 02:42, Welly Tambunan wrote: > > > > Hi S

Multilang Support on Flink

Hi All, I want to ask if there's multilang support ( like in Storm and pipeTo in Spark ) in flink ? I try to find it in the docs but can't find it. Any link or direction would be really appreciated. Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.tripl

Re: Flink, Kappa and Lambda

record-sane-distributed-development-in-the-modern-era-2/ >> >> Another approach is about lambda architecture for maintaining the >> correctness of the system. >> >> >> https://lostechies.com/ryansvihla/2015/09/17/real-time-analytics-with-spark-streaming-and-cassa

Re: Apache Flink Operator State as Query Cache

ap as well (currently it only works in windows and > in Map/FlatMap/Filter functions over the KeyedStream). > Until the, you should be able to use a simple Java HashMap and use the > "Checkpointed" interface to get it persistent. > > Greetings, > Stephan > >

Flink, Kappa and Lambda

Given that he's using Spark for the streaming processor, do we have to do the same thing with Apache Flink ? Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Apache Flink Operator State as Query Cache

ct(dataStream). coFlatMap( ) Then you can keep > the latest model in a CoFlatMapRichFunction, not necessarily as Operator > State, although maybe OperatorState is a good choice too. > > Does it make sense to you ? > > Anwar > > On Fri, Nov 6, 2015 at 10:21 AM, Welly Tambunan wro

Apache Flink Operator State as Query Cache

subsequent query. We are considering using Flink Operator state for that one. Is that the right approach to use that for memory cache ? Or if that preferable using memory cache like redis etc. Any comments will be appreciated. Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http

Re: Question on flink and hdfs

and > process files in a distributed fashion. So can flink run standalone > without hdfs? > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Efficiency for Filter then Transform ( filter().map() vs flatMap() )

chaining as in the filter is doing very little work. By minimizing the pipeline we can minimize db/external source hit and cached local data efficiently. Cheers On Fri, Sep 4, 2015 at 2:58 PM, Welly Tambunan wrote: > Hi Stephan, > > Cheers > > On Fri, Sep 4, 2015 at 2:31 PM, Step

Re: Efficiency for Filter then Transform ( filter().map() vs flatMap() )

Hi Stephan, Cheers On Fri, Sep 4, 2015 at 2:31 PM, Stephan Ewen wrote: > We will definitely also try to get the chaining overhead down a bit. > > BTW: To reach this kind of throughput, you need sources that can produce > very fast... > > On Fri, Sep 4, 2015 at 12:20 AM, Wel

Re: Efficiency for Filter then Transform ( filter().map() vs flatMap() )

h performance, collapsing the code into one > function helps. > > On Thu, Sep 3, 2015 at 5:59 AM, Welly Tambunan wrote: > >> Hi Gyula, >> >> Thanks for your response. Seems i will use filter and map for now as that >> one is really make the intention clear, and not a big

Re: Efficiency for Filter then Transform ( filter().map() vs flatMap() )

) than you can > disable copying altogether by calling env.getConfig().enableObjectReuse(), > in which case they will have exactly the same performance. > > Cheers, > Gyula > > Welly Tambunan ezt írta (időpont: 2015. szept. 3., > Cs, 4:33): > >> Hi All, >> >

Efficiency for Filter then Transform ( filter().map() vs flatMap() )

operator instance. Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Keep Model in Operator instance up to date

perators like you > described. > > If the cached data is the same for all operators and is small enough you > can centralize the updates in a dedicated operator and push the updated > data to the operators every once in a while. > > Cheers > Gyula > > > > On Thu, Aug 20,

Re: Keep Model in Operator instance up to date

se? > > I might be slow to answer as I am currently on vacation without stable > internet connection. > > Cheers, > Gyula > > On Thu, Aug 20, 2015 at 5:36 AM Welly Tambunan wrote: > >> Hi Gyula, >> >> I have another question. So if i cache something on

Re: Keep Model in Operator instance up to date

Hi Gyula, I have another question. So if i cache something on the operator, to keep it up to date, i will always need to add and connect another stream of changes to the operator ? Is this right for every case ? Cheers On Wed, Aug 19, 2015 at 3:21 PM, Welly Tambunan wrote: > Hi Gy

Re: Keep Model in Operator instance up to date

s/stream_checkpointing.html > > Cheers, > Gyula > > On Wed, Aug 19, 2015 at 9:56 AM Welly Tambunan wrote: > >> Thanks Gyula, >> >> Another question i have.. >> >> > ... while external model updates would be *tricky *to keep consistent. >> Is that

Re: Keep Model in Operator instance up to date

ate more about this one ? Cheers On Wed, Aug 19, 2015 at 2:52 PM, Gyula Fóra wrote: > In that case I would apply a map to wrap in some common type, like a n > Either before the union. > > And then in the coflatmap you can unwrap it. > On Wed, Aug 19, 2015 at 9:50 AM Welly Tambunan wr

Re: Keep Model in Operator instance up to date

ne input. The events >>> should probably be broadcasted in this case so you can check in parallel. >>> >>> This approach can be used effectively with Flink's checkpoint mechanism, >>> while external model updates would be tricky to keep consistent. >>

Re: Keep Model in Operator instance up to date

eers, > Gyula > > > > > On Wed, Aug 19, 2015 at 8:44 AM Welly Tambunan wrote: > >> Hi All, >> >> We have a streaming computation that required to validate the data stream >> against the model provided by the user. >> >> Right now what I have d

Keep Model in Operator instance up to date

from external event ? Is there some kind of best practice for maintaining model up to date on streaming operator ? Thanks a lot Cheers -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Flink Streaming : PartitionBy vs GroupBy differences

Thanks Gyula Cheers On Fri, Jul 3, 2015 at 6:19 PM, Gyula Fóra wrote: > Yes, you can think of it that way. Each Operator has parallel instances > and each parallel instance receives input from multiple channels (FIFO from > each) and produces output. > > Welly Tambunan ez

Re: Flink Streaming : PartitionBy vs GroupBy differences

multiple > inputs than there is no ordering guarantee there, only inside one input. > > Gyula > > Welly Tambunan ezt írta (időpont: 2015. júl. 3., P, > 11:51): > >> Hi Gyula, >> >> Thanks for your response. >> >> So if i use partitionBy the

Re: Flink Streaming : PartitionBy vs GroupBy differences

laborate ? >> >> I think this one is really confusing for me that come from Spark world. >> Any >> help would be really appreciated. >> >> Cheers >> >> >> >> >> >> -- >> View this message in context: >&

Re: Open method is not called with custom implementation RichWindowMapFunction

Thanks Chiwan Great Job ! Cheers On Fri, Jul 3, 2015 at 3:32 PM, Chiwan Park wrote: > I found that the patch had been merged to upstream. [1] :) > > Regards, > Chiwan Park > > [1] https://github.com/apache/flink/pull/855 > > > On Jul 3, 2015, at 5:26 PM, Welly Tamb

Re: Open method is not called with custom implementation RichWindowMapFunction

t; .mapWindow(new MyWindowFunction) > > > > Any thought on this one ? > > > > > > Cheers > > > > > > > > -- > > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Open-method-is-not-called-with-custom-implementation-RichWindowMapFunction-tp1924.html > > Sent from the Apache Flink User Mailing List archive. mailing list > archive at Nabble.com. > > > > > > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>

Re: Batch Processing as Streaming

ming >> >> 2. I want to ask about operator lifecyle ? is that short live or long >> live ? >> Any docs where i can read about this more ? >> >> >> Cheers >> >> >> >> -- >> View this message in context: >> http://apac

Re: Data Source from Cassandra

guide on how to do this robustly ? >> >> >> Cheers >> >> >> >> -- >> View this message in context: >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Data-Source-from-Cassandra-tp1908.html >> Sent from the Apache Flink Use