this is done at some point in Flink, maybe for savepoints.
Best, Fabian
2018-02-23 14:44 GMT+01:00 Radu Tudoran :
> Hi,
>
>
>
> I was using the RowSerializer (package org.apache.flink.api.java.
> typeutils.runtime;) recently to serialize Rows to file (for reading
> them
...rather than having an error for this
process.
Hence, I wanted to double check if this is an intended behavior for some reason
and if we should consider fixing the rowserializer to guarantee integrity of
the objects that are deserialized.
Best regards,
Dr. Radu Tudoran
Staff Research Engineer - Big
Hi,
Any thoughts of why would I get such an error when trying to run within eclipse
a simple (i.e., any) flink program.
...I did start the local cluster before - where I was able to run the job...but
after that it did not work anymore to run flink from eclipse.
Exception in thread "main" java.
f I am not mistaken you can introduce in
database columns the result of calling now().
Dr. Radu Tudoran
Staff Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
German Research Center
Munich Office
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.c
) or the timestamp from the main stream (...although I am not sure which
one is the main stream in the case of a full join:) )
Dr. Radu Tudoran
Staff Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
German Research Center
Munich Office
Riesstrasse 25, 8
JOIN T1,
T2...)...but they would not have the timestamp logic
..my 2 cents
Dr. Radu Tudoran
Staff Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
German Research Center
Munich Office
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mo
ValueState that inside stored the whole list.
Whenever you state#get() you get the whole list and you can sort it.
Kostas
> On May 18, 2017, at 3:31 AM, Radu Tudoran wrote:
>
> Hi Aljoscha,
>
> Thanks for the clarification. I understand that there might be advantages in
>
rtain implementations very hard to to
efficiently.
Best,
Aljoscha
> On 16. May 2017, at 21:56, Radu Tudoran wrote:
>
> Hi,
>
> I would like to work with ListState, more specifically I would need to access
> the contents and sort them. For this I would need a collection
Hi,
I would like to work with ListState, more specifically I would need to access
the contents and sort them. For this I would need a collection type (e.g., the
List, Array...).
However, I see that if I have a variable of type <> the
only interfaces I have are:
state.get -> which returns an Ite
Hi,
In general I believe it is a good idea to expose the state backend to the
functions. You can always optimize the data processing based on the data
storage. Hence, as the level of the processing (aggregation here) you would be
able to control the access to data, you can implement this in a s
Sorry in case I missed something in the discussion of coming up with the list
of open threads, but there is also
[FLINK-6075] Support Order By for Stream SQL (Support Limit/Top(Sort) for
Stream SQL)
...in case you want to consider it (it only needs a review)
Best regards,
-Original Messa
it comes to which records are dropped.
The watermark documentation might be helpful as well [1].
Cheers,
Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/event_time.html#watermarks-in-parallel-streams
2017-04-27 22:09 GMT+02:00 Radu Tudoran :
> Re-hi,
>
> I debug
over range windows use the TimerServices to group
elements by time.
In case of event-time, the timers are triggered by the event-time of the
operator which is derived from the received watermarks.
In case of processing-time, the timers are triggered based on the wallclock
time of the operator.
processing-time, the timers are triggered based on the wallclock
time of the operator.
So by using event-tim timers, we implicitly rely on the watermarks because the
timers are triggered based on the received watermarks.
Best, Fabian
2017-04-27 10:51 GMT+02:00 Radu Tudoran :
> Hi,
>
Hi,
I am looking at the implementation of RowTimeBoundedRangeOver (in the context
of Stream SQL). I see that the logic is that the progress happens based on the
timestamps of the rowevent - i.e., when an even arrives we register to be
processed based on it's timestamp
(ctx.timerService.regist
Hi,
You need to implement your own timer. You do this when you create your window
by assigning the timer. In your custom timer you would need to implement the
desired logic in the onElement method.
You can keep a counter that you increment for each element up to your desired
number of elements
extractTimestamp(
element: type...,
previousElementTimestamp: Long): Long = {
System.currentTimeMillis()
}
Alternatively you can play around with the stream source and control the time
when the events come
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&a
needs the assignment of rowtime
within a query?
Am 20/03/17 um 13:39 schrieb Radu Tudoran:
> Hi,
>
> As suggested by Timo - I am forwarding this to the mailing list. Sorry for
> not having the conversation directly here - I initially thought it might not
> be of interest...
&
the way to do it?
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
Telephone: +49 891588344173
HUAWEI TECHNOLO
n most
of the user-facing API does not change. It only tries to make time more
explicit.
Timo
Am 20/03/17 um 10:34 schrieb Radu Tudoran:
> Hi Timo,
>
> I have some questions regarding your implementation:
>
> " The timestamp (not an indicator anymore) becomes part of the phy
>>>>>> streams.
>>>>>>>>>
>>>>>>>>> Under this circumstance, I just consider adding a "monotonic
>>>>>>>>> hint"(INC or
>>>>>>>>> DEC) to the field of a (gene
p(Sort) for Stream SQL
6. FLINK-6073<https://issues.apache.org/jira/browse/FLINK-6073> - Support
for SQL inner queries for proctime
7.
Please let me know what you think and of course any feedback about the
design/priority for the roadmap/implementation is more than welcomed.
Dr. Radu Tudoran
Hi,
I propose that we consider also the type of connectivity to be supported in the
Flink API Gateway. I would propose to support a couple of calls option to
ingest also events. I am thinking of:
- callback mechanism
- REST
- RPC
-Original Message-
From: Chen Qin [mailto:qinnc...@gma
Hi,
I am struggling to move a working implementation from Java to Scala :(...this
is for computing window aggregates (sliding window).
As I am not proficient in Scala I got block in (probably a stupid
error)...maybe someone can help me.
I am trying to create a simple window function to be appl
151042>
JDK: oraclejdk8 <https://travis-ci.org/apache/flink/jobs/208151042>
PROFILE="-Dhadoop.version=2.6.3
-Pinclude-yarn-tests,flink-fast-tests-b,include-kinesis
-Dmaven.javadoc.skip=true" <https://travis-ci.org/apache/flink/jobs/208151042>
Dr. Radu Tudoran
ure as you would say we
need to augment the data with two fields whether needed or not...this is nto
necessary very efficient
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 M
t when you translate back a table to a stream /
write it to a dynamic table?
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile: +
Hi,
I made some comments over the Dynamic table document. Not sure how to ask for
feedback for them...therefore my email.
Please let me know what do you think
https://docs.google.com/document/d/1qVVt_16kdaZQ8RTfA_f4konQPW4tnl8THw6rzGUdaqU/edit#heading=h.3eo2vkvydld6
Dr. Radu Tudoran
Senior
Hi,
Should we discuss also about the design of distinct for the stream case?
It might go well in the context of tables as well as in the context of
aggregates over windows...
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf
refinements over the implementation (potentially to start from very simple and
naïve functioning and then extend it to provide more elegant semantics).
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Ce
an update in the operator
> that keeps the single value - if this is possible)
>
> Best regards,
>
> Dr. Radu Tudoran
> Senior Research Engineer - Big Data Expert IT R&D Division
>
>
> HUAWEI TECHNOLOGIES Duesseldorf GmbH
> European Research Center
> Riesstr
apply the inner query has an ID, then option 1
(recomputed and apply updates based on retraction and all others); otherwise
then option 2 (only make an update in the operator that keeps the single value
- if this is possible)
Best regards,
Dr. Radu Tudoran
Senior Research Engineer - Big Data
ironment
to control the clean-up timeout.
Cheers,
Fabian
2017-01-30 18:14 GMT+01:00 Radu Tudoran :
> Hi Fabian,
>
> Thanks for the clarifications. I have a follow up question: you say
> that operations are expected to be bounded in space and time (e.g.,
> the optimizer will d
aling
problem initially. In the long run, the Flink community will work to improve
the handling of large state with features such as incremental checkpoints and
new state backends.
Looking forward to your comments.
Best,
Fabian
2017-01-27 11:01 GMT+01:00 Radu Tudoran :
> Hi,
>
>
less of whether it is an inner or outer query.
This is one of the core principles of SQL, that I would not change.
Best, Fabian
2017-01-30 12:54 GMT+01:00 Radu Tudoran :
> Hi Fabian,
>
> Thanks for the link and for the remarks.
>
> I do not imagine the behavior of the inner query
n to have. Please
let me know what do you think.
I agree with the approach of starting small - even with some very limited cases
when we support inner queries and then extend or define the general cases.
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Division
HUAWEI
ep.
I will give some comments and some thoughts in the document about this.
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile: +49 15
pecific semantic
and then I will push this in a jira issue to be furthered review and validated.
Best regards,
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail
gt;> a retract API for UDAGG. With aggregate accumulator and retract
> >> API, I
> >>>> am
> >>>> proposing a runtime approach to implement the OVER window as
> >>> followings.
> >>>> 4.
> >>>> - We f
Hi,
Can you assign please the jira issue to me. I already can start integrating the
implementation that for this.
-Original Message-
From: Fabian Hueske (JIRA) [mailto:j...@apache.org]
Sent: Thursday, January 26, 2017 2:44 PM
To: dev@flink.apache.org
Subject: [jira] [Created] (FLINK-56
)
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com<mailto:radu.tudo...@huawei.com>
Mobile: +49 1
n Wed, Jan 25, 2017 at 9:49 PM, Radu Tudoran
wrote:
> Hi all,
>
> I would like to open a jira issue (and then provide the
> implementation) for supporting inner queries. The idea is to be able
> to support SQL queries as the ones presented in the scenarios below.
> The key i
Hi all,
I would like to open a jira issue (and then provide the implementation) for
supporting inner queries. The idea is to be able to support SQL queries as the
ones presented in the scenarios below. The key idea is that supporting inner
queries would require to have the implementation for:
nt I see the following six issues to start with:
> > >
> > > - streaming SQL OVER ROW for processing time
> > > - bounded PRECEDING
> > > - unbounded PRECEDING
> > >
> > > - streaming SQL OVER RANGE for processing time
> > >
rich semantics that exists in window. We can define
this SlidingRowWindow or RollingWindow..whatever the name would be - which is a
specialized window for this. We can perhaps enhance the window with a sorting
function. Let me know what do you think about this?
Dr. Radu Tudoran
Senior Research
K-4697, right? "
Why would you need this operator? The window buffer can act to some extent as a
priority queue as long as the trigger and evictor is set to work based on the
rowtime - or maybe I am missing something... Can you please clarify this.
Dr. Radu Tudoran
Senior Research Engineer
existing ones...?
Dr. Radu Tudoran
Senior Research Engineer - Big Data Expert
IT R&D Division
[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com<mailto:radu.tudo...@huawei.com>
M
;> > > > > DataStream input = ...
>> > > > > // this makes the current processing time explicit in the tuples:
>> > > > > DataStream> withTimestamps = input.map(new
>> > > > > ReifyProcessingTIme());
>> > > > >
Hi Aljoscha,
My 2 cents on this would be that it is worth maintaining the access to the
watermarks. I think having the option to customize this is a strong point of
Flink.
Regarding the solution you proposed based on 2 input timers " would fire if the
watermark from both inputs advances suffic
Hi,
IMHO I think we should still maintain user specific triggers and I think there
will always be corner cases where a very specific trigger will be needed to be
constructed. With this being said, I think the idea of supporting also some
state machine to be generated for the trigger is very goo
Hi Aljoscha,
Can you point us to the way it is handled now. Is there anything else for the
removing of elements other than the skip in EvictingWindowOperator. Is there
something as it was before version 1.x where you had an explicit remove from
window buffers?
Dr. Radu Tudoran
Research
added a
fix for that. (You can see a mail regarding that in this mail chain)
Thanks,
Vishnu
On Fri, Jul 22, 2016 at 1:03 PM, Radu Tudoran
wrote:
> Hi,
>
> Overall I believe that the interfaces and the proposal is good. I have the
> following question though: can you delete via
his.. some reference to classes/code
where this happens would be useful
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
ronment(env);
>
> tableEnv.registerDataStream("Words", ds, "frequency, word, pos");
> // run a SQL query on the Table and retrieve the result as a new Table
> Table result = tableEnv.sql("SELECT STREAM word, pos FROM Words WHERE
> frequency > 2");
>
SQL.
On Tue, Jul 19, 2016 at 6:14 PM, Radu Tudoran wrote:
> Hi,
>
> I am not sure that this problem was solved. I am using the last pom to
> compile the table API.
>
> I was trying to run a simple program.
>
>
> ArrayList> input = new ArrayList String, Integer>
ment.sql(StreamTableEnvironment.scala:127)
at TestStreamSQL.main(TestStreamSQL.java:69)
Any thoughts on how this can be solved?
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 Mün
[mailto:aljos...@apache.org]
Sent: Monday, July 18, 2016 5:54 PM
To: dev@flink.apache.org
Subject: Re: [DISCUSS] FLIP-2 Extending Window Function Metadata
What about the collector? This is only used for emitting elements to the
downstream operation.
On Mon, 18 Jul 2016 at 17:52 Radu Tudoran wrote:
>
w, people should also not rely on the fact that elements are "FIFO".
Some state backends might keep the elements in a different order and when you
have merging windows/session windows the order of the elements will also not be
preserved.
Cheers,
Aljoscha
On Wed, 13 Jul 2016 at 18:40 Ra
r stacks are not actually used,
perhaps we can just pass policies to the iterator - or have custom itrators
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@hu
e a binary search for insertion and then you
further reduce the complexity to O(logN).
If M is proportional to N then you can sort M and use merge sort for combining.
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Resea
has finished triggering or
to be evicted in the next iteration.
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile: +49 15209084330
Telephone:
Hi,
Do you have also a linkedin post that I could share - or should I make a
blogpost in which I take this announcement?
Dr. Radu Tudoran
Research Engineer - Big Data Expert
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-
Hi,
Is there some info, description about how this off-heap memory is managed and
its goals?
Thanks
Dr. Radu Tudoran
Research Engineer
IT R&D Division
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München
E-mail: radu.tudo...@huawei.com
Mobile:
63 matches
Mail list logo
