Re: Data Transfer between TM should be encrypted

2016-08-30 Thread vinay patil
Hi Vijay, That's a good news for me. Eagerly waiting for this change so that I can integrate and test it before going live. Regards, Vinay Patil On Tue, Aug 30, 2016 at 4:06 PM, Vijay Srinivasaraghavan [via Apache Flink User Mailing List archive.] wrote: > Hi Stephan, > > The dev work is almos

Re: Firing windows multiple times

2016-08-30 Thread Shannon Carey
I appreciate your suggestion! However, the main problem with your approach is the amount of time that goes by without an updated value from minuteAggregate and hourlyAggregate (lack of a continuously updated aggregate). For example, if we use a tumbling window of 1 month duration, then we only

Re: Data Transfer between TM should be encrypted

2016-08-30 Thread Vijay Srinivasaraghavan
Hi Stephan, The dev work is almost complete except the Yarn mode deployment stuff that needs to be patched. We are expecting to send a PR in a week or two. RegardsVijay On Tuesday, August 30, 2016 12:39 AM, Stephan Ewen wrote: Let me loop in Vijay, I think he is the one working on this

"select as" in Flink SQL

2016-08-30 Thread Davran Muzafarov
I am try to execute simple sql like this: DataSet dataSet0 = env.fromCollection( infos0 ); tableEnv.registerDataSet( "table0", dataSet0 ); Table table = tableEnv.sql( "select assetClass as \"asset class\" from tabel0" ); I am getting: org.apache.calcite.sql.parser.SqlParseExcept

Setting EventTime window width using stream data

2016-08-30 Thread Paul Joireman
Hi all, Is it possible to dynamically set the size/width of a SlidingEventTimeWindow based on a data coming from the stream? Our use case is as follows. We create a stream sourced from external system and coming in as a JSON string which is deserialized to a stream of POJO.The deseria

Re: Flink long-running YARN configuration

2016-08-30 Thread Maximilian Michels
Yes, it will exist also in the Yarn session and continue to run across jobs. Its address is also printed on the console when the cluster is brought up. On Mon, Aug 29, 2016 at 2:44 PM, Robert Metzger wrote: > The JobManager UI starts when running Flink on YARN. > The address of the UI is register

Re: CountTrigger FIRE or FIRE_AND_PURGE

2016-08-30 Thread Paul Joireman
Fabian, Thanks for the reference, I think I was incorrectly interpreting the results I was getting using the CountTrigger, it looks like it does keep the data. However, I'm running into some unexpected (at least by me) behavior. Given a keyed data stream keyedStream and event timing

ApacheCon Seville CFP closes September 9th

2016-08-30 Thread Rich Bowen
It's traditional. We wait for the last minute to get our talk proposals in for conferences. Well, the last minute has arrived. The CFP for ApacheCon Seville closes on September 9th, which is less than 2 weeks away. It's time to get your talks in, so that we can make this the best ApacheCon yet. I

Re: Accessing state in connected streams

2016-08-30 Thread Aljoscha Krettek
Ah I see, I'm afraid StatefulFunction is more of an internal implementation detail that cannot be used like that. This is a small example that shows how you could do a stateful Co-FlatMap function: object StateExample { trait Base { def id: Int } case class EventA(id: Int, info: String) c

flink dataStream operate dataSet

2016-08-30 Thread rimin515
Hi, i have a problem,a dataStream read from rabbitMQ,and others data from a hbase table,which is a dataSet.Those two data from follow: val words=connectHelper.readFromRabbitMq(...) // words is DataStream[String] val dataSet=HBaseWrite.fullScan() //dataSet is DataSet[(int,Strin

Re: Accessing state in connected streams

2016-08-30 Thread aris kol
Hi Aljoscha, I removed business objects and logic etc.. I am happy to post here [😊] I am sure this is a common issue when you start to seriously mess with state. Assuming a type for the Output And assuming that there is a function (EventA :=> String) in the mapWithState operator of typeAStre

Re: Firing windows multiple times

2016-08-30 Thread Aljoscha Krettek
Hi, I think this can be neatly expressed by using something like a tree of windowed aggregations, i.e. you specify your smallest window computation first and then specify larger window computations based smaller windows. I've written an example that showcases this approach: https://gist.github.com/

Re: Accessing state in connected streams

2016-08-30 Thread Aljoscha Krettek
Hi Aris, I think you're on the right track with using a CoFlatMap for this. Could you maybe post the code of your CoFlatMapFunction (or you could send it to me privately if you have concerns with publicly posting it) then I could have a look. Cheers, Aljoscha On Mon, 29 Aug 2016 at 15:48 aris kol

Re: Data Transfer between TM should be encrypted

2016-08-30 Thread Stephan Ewen
Let me loop in Vijay, I think he is the one working on this and can probably give the best estimate when it can be expected. @vijay: For the SSL/TLS transport encryption - do you have an estimate for the timeline of that feature? On Mon, Aug 29, 2016 at 8:54 PM, vinay patil wrote: > Hi Stephan