ute to the
documentation? Thank you in advance.
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
h to the end of your mail where
> you offered to fix it yourself.
>
> If you find more issues, ping me, I can give you edit permissions.
>
> Thanks,
> Stephan
>
>
> On Tue, Jan 12, 2016 at 11:49 AM, Stefano Baghino <
> stefano.bagh...@radicalbit.io> wrote:
>
&
training-exercises/blob/master/pom.xml>
> but
> there is no addition of library. So there is error: the import
> com.dataartisans cannot be resolved.
> Below is my pom.xml file. Please could you suggest what am I doing wrong?.
>
>
>
> Regards,
> Subash
>
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
e list:
> Tuple3 newElement = new Tuple3 Boolean>();
> finalElements.add(newElement);
> Now I am stuck on how to convert this 'finalElements' to
> DataSet> fElements,
> so that I could use
> fElements.writeAsCsv(outputPath, "\n"," ");
>
> Best Regards,
> Subash Basnet
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
ed. Is this
> understanding right? If so, is using a *Broadcast variable*, the solution?
>
> Please help me plug the gap in understanding, if any.
>
> -- Nirmalya
>
> --
> Software Technologist
> http://www.linkedin.com/in/nirmalyasengupta
> "If you have built castles
> avgReadings transformation, it should aggregate the streams from all the 4
> earlier windows, and then compute the single average value. It is quite
> apparent that there is a gap in my understanding. Could you please point
> out the mistake that I am making?
>
> Many thanks in a
nager, but before digging deeper I wanted to
ask if my expectation of having the job going despite the job manager
failure is unreasonable.
Thanks in advance.
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
in the masters file? With the
> local file system state backend only something like
>
> host-X
> host-X
> host-X
>
> will be a valid masters configuration.
>
> Can you please share the job manager logs of all started job managers?
>
> – Ufuk
>
>
>
HA
> mode. You can enable it on the StreamExecutionEnvironment.
>
> Otherwise, you have probably already found the documentation:
>
> https://ci.apache.org/projects/flink/flink-docs-master/setup/jobmanager_high_availability.html#configuration
>
> Cheers,
> Max
>
> On Mon, Feb 15, 2016 at 12:
; configured. We will update it.
> >
> > Instead of setting the number of execution retries on the
> > StreamExecutionEnvironment, you may also set
> > "execution-retries.default" in the flink-conf.yaml. Let us know if
> > that fixes your setup.
> >
>
Ok, simply turning up HDFS on the cluster and using it as the state backend
fixed the issue. Thank you both for the help!
On Mon, Feb 15, 2016 at 5:45 PM, Stefano Baghino <
stefano.bagh...@radicalbit.io> wrote:
> You can find the log of the recovering job manager here:
> https://gis
irmalyasengupta
> "If you have built castles in the air, your work need not be lost. That is
> where they should be.
> Now put the foundation under them."
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
Software Technologist
> http://www.linkedin.com/in/nirmalyasengupta
> "If you have built castles in the air, your work need not be lost. That is
> where they should be.
> Now put the foundation under them."
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
ps. Does that make sense? It’s a very ad-hoc description
> and I could make up a drawing or something if that helped. :D
>
>
> > On 18 Feb 2016, at 10:04, Stefano Baghino
> wrote:
> >
> > Thanks, Aljosha, for the explanation. Isn't there a way to apply the
> co
gt; file:///`pwd`/hamlet.txt
> --output file:///`pwd`/wordcount-result.txt
>
> Similarly, you could add
>
> $ bin/flink run ./examples/streaming/WordCount.jar --output file:///
> `pwd`/streaming-wordcount-result.txt
>
> Finally, to complete the demo please add the i
re
> flexible?
> How would be the smartest way to do so?
>
> Best regards,
> Lydia
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
RN?
Thank you so much in advance.
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
In the initial description, I meant "I'm trying to access a private folder
of the latter", so not the service account. Sorry for the mistake.
On Sun, Mar 6, 2016 at 8:54 PM, Stefano Baghino <
stefano.bagh...@radicalbit.io> wrote:
> Hello everybody,
>
> I'm r
I've added a second log with this error to the gist
<https://gist.github.com/stefanobaghino/6605ec33a1c4b632fb78>.
On Sun, Mar 6, 2016 at 9:01 PM, Stefano Baghino <
stefano.bagh...@radicalbit.io> wrote:
> In the initial description, I meant "I'm trying to access a
a per-job Yarn cluster.
> There is currently no plan to delegate the user token per job but we
> could certainly think about implementing this in the future.
>
>
> https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#kerberos
>
> Cheers,
> Max
>
>
dle jobs?
Thank you in advance!
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
ing on integrating proper resource allocation into
> the JobManager. As of now, everything is static, i.e. the JobManager
> won't allocate more than the initial requested resources.
>
> Cheers,
> Max
>
> On Mon, Mar 7, 2016 at 1:38 PM, Stefano Baghino
> wrote:
> >
One last question: running multiple jobs mean that each one has its own
JobManager, right?
On Mon, Mar 7, 2016 at 3:14 PM, Stefano Baghino <
stefano.bagh...@radicalbit.io> wrote:
> Good, thank you for the explanation!
>
> On Mon, Mar 7, 2016 at 2:38 PM, Maximilian Michels wrote:
eshoot this issue? Thank you so much in advance!
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
class QueryOneSource(filenames: Array[String]) extends SourceFunction[Tuple]
>> {
>>
>> val nextTuple: Tuple // case class Tuple(id: Long, value: Int)
>>
>> override def run(ctx: SourceContext[Tuple]) = {
>> while (true) {
>> nextRecord()
>> ctx.collect(this.nextTuple)
>> }
>> }
>>
>> override def cancel() = { }
>>
>> }
>>
>> override def nextRecord() = {
>>
>> }
>>
>> }
>>
>> Best,
>> *Ankur Sharma*
>> *Information Systems Group*
>> *3.15 E1.1 Universität des Saarlandes*
>> *66123, Saarbrücken Germany*
>> *Email: ankur.sha...@mpi-inf.mpg.de *
>> *an...@stud.uni-saarland.de *
>>
>>
>
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
AggregatableTimeslice {
> int getAccountId();
> int getAgentId();
> long getWideMetricId();
> AggregatableTimesliceStats getTimesliceStats();
> }
>
> Ron
> —
> Ron Crocker
> Principal Engineer & Architect
> ( ( •)) New Relic
> rcroc...@newrelic.com
> M: +1 630 363 8835
>
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
where I log, I put error
level logs at the very first line of my application. Still nothing.
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
Ok, my bad, I was simply looking in the wrong place. I though the logs were
sent to YARN but they were actually stored in the Flink logs folder.
Problem solved, sorry for the mix up.
On Sun, Mar 13, 2016 at 8:48 PM, Stefano Baghino <
stefano.bagh...@radicalbit.io> wrote:
> There'
des it sshs to when it starts the TaskManagers?
>
> Best,
> Max
>
>
> On Fri, Mar 11, 2016 at 8:17 AM, Stefano Baghino
> wrote:
> > Hello everybody,
> >
> > me and my colleagues have been running some tests on Flink 1.0.0 in a
> secure
> > environment
s a task of B (which would result in
> the swapped counts you see in the log).
>
> Do you have time to repeat your experiment with different ZooKeeper root
> paths?
>
> – Ufuk
>
>
> On Thu, Mar 17, 2016 at 11:51 AM, Stefano Baghino
> wrote:
> > Hi Ufuk,
> >
try the same thing without -yt, but a yarn-session?
>
> – Ufuk
>
> On Thu, Mar 17, 2016 at 12:29 PM, Stefano Baghino
> wrote:
> > I have another interesting test result on this matter, running again an
> > individual job on a YARN cluster.
> >
> > W
at
> org.apache.flink.streaming.examples.kafka.ReadFromKafka.main(ReadFromKafka.java:54)
>
> How to set property 'zookeeper.connect', or what am I doing wrong?
>
> Best Regards,
> Subash Basnet
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
Yes, but each job runs his own cluster, right? We have to run them on a
secure cluster and on a per-user basis, thus we can't run a YARN session
but have to run each job independently.
On Thu, Mar 17, 2016 at 12:09 PM, Ufuk Celebi wrote:
> On Thu, Mar 17, 2016 at 11:51 AM, Stefano
re cluster in HA mode, would you think setting the
recovery.zookeeper.path.root from the startup script could be regarded as a
good practice?
On Thu, Mar 17, 2016 at 11:51 AM, Stefano Baghino <
stefano.bagh...@radicalbit.io> wrote:
> Hi Ufuk,
>
> does the recovery.zookeeper.pa
m to work fine as well.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/akka-pattern-AskTimeoutException-tp4253p5572.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
>
>
> --
> Frederick Ayala
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
ti
> >>> >> >> wrote:
> >>> >> >> > Hello,
> >>> >> >> >
> >>> >> >> > I'm testing the checkpointing functionality with hdfs as a
> >>> >> >> > backend.
> >>> >> >> >
> >>> >> >> > For what I can see it uses different checkpointing files and
> >>> >> >> > resume
> >>> >> >> > the
> >>> >> >> > computation from different points and not from the latest
> >>> >> >> > available.
> >>> >> >> > This is
> >>> >> >> > to me an unexpected behaviour.
> >>> >> >> >
> >>> >> >> > I log every second, for every worker, a counter that is
> increased
> >>> >> >> > by
> >>> >> >> > 1
> >>> >> >> > at
> >>> >> >> > each step.
> >>> >> >> >
> >>> >> >> > So for example on node-1 the count goes up to 5, then I kill a
> >>> >> >> > job
> >>> >> >> > manager
> >>> >> >> > or task manager and it resumes from 5 or 4 and it's ok. The
> next
> >>> >> >> > time
> >>> >> >> > I
> >>> >> >> > kill
> >>> >> >> > a job manager the count is at 15 and it resumes at 14 or 15.
> >>> >> >> > Sometimes
> >>> >> >> > it
> >>> >> >> > may happen that at a third kill the work resumes at 4 or 5 as
> if
> >>> >> >> > the
> >>> >> >> > checkpoint resumed the second time wasn't there.
> >>> >> >> >
> >>> >> >> > Once I even saw it jump forward: the first kill is at 10 and it
> >>> >> >> > resumes
> >>> >> >> > at
> >>> >> >> > 9, the second kill is at 70 and it resumes at 9, the third kill
> >>> >> >> > is at
> >>> >> >> > 15
> >>> >> >> > but
> >>> >> >> > it resumes at 69 as if it resumed from the second kill
> >>> >> >> > checkpoint.
> >>> >> >> >
> >>> >> >> > This is clearly inconsistent.
> >>> >> >> >
> >>> >> >> > Also, in the logs I can find that sometimes it uses a
> checkpoint
> >>> >> >> > file
> >>> >> >> > different from the previous, consistent resume.
> >>> >> >> >
> >>> >> >> > What am I doing wrong? Is it a known bug?
> >>> >> >
> >>> >> >
> >>> >
> >>> >
> >>
> >>
> >
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
like in Yarn, in standalone mode every job is run under the same
> user which started the cluster.
>
> Cheers,
> Max
>
> On Wed, Mar 16, 2016 at 10:50 AM, Stefano Baghino
> wrote:
> > Hi Max,
> >
> > thanks for the tips. What we did has been running kinit on e
>> 2016-03-11 19:23 GMT+01:00 Ufuk Celebi :
> >>>>>
> >>>>> Everything in the lib folder should be added to the classpath. Can
> you
> >>>>> check the YARN client logs that the files are uploaded? Furthermore,
> >>>>> you can check the classpath of the JVM in the YARN logs of the
> >>>>> JobManager/TaskManager processes.
> >>>>>
> >>>>> – Ufuk
> >>>>>
> >>>>>
> >>>>> On Fri, Mar 11, 2016 at 5:33 PM, Andrea Sella
> >>>>> wrote:
> >>>>> > Hi,
> >>>>> >
> >>>>> > There is a way to add external dependencies to Flink Job, running
> on
> >>>>> > YARN,
> >>>>> > not using HADOOP_CLASSPATH?
> >>>>> > I am looking for a similar idea to standalone mode using lib
> folder.
> >>>>> >
> >>>>> > BR,
> >>>>> > Andrea
> >>>>
> >>>>
> >>>
> >>
> >
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
and after looking at
the code (FlinkYarnClientBase:522-536) it seems to me that the
flink-conf.yaml key, if set, overrides the yarn-site.xml, which in turn
overrides the fallback value of 1. Is this right? Is the documentation
wrong?
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
w can I easily convert
> "org.apache.flink.streaming.scala.DataStream" to
> "org.apache.flink.streaming.api.datastream.DataStream"?
>
> Thanks.
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
11:51 schrieb Lydia Ickler :
> >
> > Hi,
> >
> > If i have 2 DataSets A and B of Type Tuple3 how
> would I get a subset of A (based on the fields (0,1)) that does not occur
> in B?
> > Is there maybe an already implemented method?
> >
> > Best regards,
the snapshotting mechanism in case of a node failure actually
> serialize this map?
>
> And out of curiousity, can I actually see what data exists inside the
> snapshot data?
>
> Thanks.
>
> Bart
>
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
times within X seconds.
> Per default, the failure validity interval is configured to the Akka
> timeout (which is per default 10s). I actually think it would make
> sense to increase this a little and leave the attempts at 1 or 2 (in
> the interval).
>
> Does this help?
>
nd big +1 for making it more
>>>> prominent.
>>>> >>>>>>
>>>> >>>>>> On Mon, Oct 19, 2015 at 2:53 PM, Fabian Hueske <
>>>> mailto:fhue...@gmail.com>>
>>>> >>>>>
>>>> >>>>> fhue...@gmail.com <mailto:fhue...@gmail.com>> wrote:
>>>> >>>>>>>
>>>> >>>>>>> Thanks for starting this Kostas.
>>>> >>>>>>>
>>>> >>>>>>> I think the list is quite hidden in the wiki. Should
>>>> we link from
>>>> >>>>>>> flink.apache.org <http://flink.apache.org/> to that
>>>> page?
>>>> >>>>>>>
>>>> >>>>>>> Cheers, Fabian
>>>> >>>>>>>
>>>> >>>>>>> 2015-10-19 14:50 GMT+02:00 Kostas Tzoumas <
>>>> mailto:ktzou...@apache.org>>
>>>> >>>>>
>>>> >>>>> ktzou...@apache.org <mailto:ktzou...@apache.org>>:
>>>> >>>>>>>>
>>>> >>>>>>>> Hi everyone,
>>>> >>>>>>>>
>>>> >>>>>>>> I started a "Powered by Flink" wiki page, listing
>>>> some of the
>>>> >>>>>>>> organizations that are using Flink:
>>>> >>>>>>>>
>>>> >>>>>>>>
>>>>
>>>> https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink
>>>> >>>>>>>>
>>>> >>>>>>>> If you would like to be added to the list, just send
>>>> me a short email
>>>> >>>>>>>> with your organization's name and a description and I
>>>> will add you to
>>>> >>>>>
>>>> >>>>> the
>>>> >>>>>>>>
>>>> >>>>>>>> wiki page.
>>>> >>>>>>>>
>>>> >>>>>>>> Best,
>>>> >>>>>>>> Kostas
>>>> >>>>>>>>
>>>> >>>>>>>
>>>> >>>>
>>>> >>>>
>>>> >>
>>>>
>>>>
>>>>
>>>
>>>
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
Thanks to you! :)
On Tue, Apr 12, 2016 at 4:16 PM, Fabian Hueske wrote:
> Done, thanks Stefano!
>
> 2016-04-12 15:38 GMT+02:00 Stefano Baghino
> :
>
>> Hi Robert, thanks for bringing up the page.
>> We're in the process of releasing our first version and
er_e02_1461077293721_0016_01_02'} not
registered with resource manager.. Retrying later...
2016-04-19 12:16:09,664 INFO org.apache.flink.yarn.YarnTaskManager
- Trying to register at JobManager akka.tcp://
flink@172.31.20.101:57379/user/jobmanager (attempt 1, timeout: 500
milliseconds)
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
>
> On Tue, Apr 19, 2016 at 6:31 PM, Stefano Baghino
> wrote:
> > Hi everyone,
> >
> > I'm currently experiencing a weird situation, I hope you can help me out
> > with this.
> >
> > I've cloned and built from the master, then I've edited th
;> .keyBy({s => s})
>>>
>>>
>>>
>>> *.mapWithState((in:String, count:Option[Int]) => {val newCount =
>>> count.getOrElse(0) + 1((in, newCount), Some(newCount))})*
>>> .print()
>>
>>
>>
>> Thanks,
>>
>> Jack Huang
>>
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
My bad, thanks for pointing that out.
On Wed, Apr 20, 2016 at 1:49 PM, Aljoscha Krettek
wrote:
> Hi,
> the *withState() family of functions use the Key/Value state interface
> internally, so that should work.
>
> On Wed, 20 Apr 2016 at 12:33 Stefano Baghino <
> stefano.
1 fire because of A is different to "null"
>> Event º4 fire because of B is different to A
>> Event º8 fire because of C is different to B
>>
>> Could it be possible?
>>
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
> each event, like that:
>>>>
>>>> Event.: 1 2 3 4 5 6 7 8 9...
>>>> Data..: A A A B B B B C C...
>>>>
>>>> I would like to do a "trigger" when the character is different than
>>>> before. For example:
>>&
pointed[Integer] {
>> private var count: Integer = 0
>>
>> def map(in: String): (String,Int) = {
>> count += 1
>> return (in, count)
>> }
>> def snapshotState(l: Long, l1: Long): Integer = {
>> count
>> }
>> def restoreS
gt;
> Nirmalya Sengupta
> [image: https://]about.me/sengupta.nirmalya
>
> <https://about.me/sengupta.nirmalya?promo=email_sig&utm_source=email_sig&utm_medium=external_link&utm_campaign=chrome_ext>
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
t;
> On Fri, Apr 29, 2016 at 3:59 PM, Punit Naik
> wrote:
>
>> I have a dataset which has maps. I have performed a groupBy on a key and
>> I want to count all the elements in a particular group. How do I do this?
>>
>> --
>> Thank You
>>
>> Regards
DataStream newCentroids = points.map(new
> SelectNearestCenter()).???
>
>
> Best Regards,
> Subash Basnet
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
May 3, 2016 at 10:26 AM, subash basnet
>> wrote:
>>
>>> Hello all,
>>>
>>> Suppose I have the datastream as:
>>> DataStream> *newCentroids*;
>>>
>>> How to get collection of *newCentroids * to be able to loop as below:
>>> private Collection> *centroids*;
>>> for (Centroid cent : *centroids*) {
>>> }
>>>
>>>
>>>
>>> Best Regards,
>>> Subash Basnet
>>>
>>
>>
>>
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
.0.0
>
>
> org.apache.flink
> *flink-streaming-contrib_${flink.version}*
> *${flink.version}*
>
>
> So I instead used the below for contrib dependency as given in maven
> repository:
>
> org.apache.flink
> flink-streaming-contrib
> 0.10.2
>
>
>
> Be
impler way of doing this? All other variants I've tried yield
> some type information errors.
>
> Thanks in advance!
> Robert
>
> --
> My GPG Key ID: 336E2680
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
andy as well. I'm good to go now, all works well and
> looks a bit more Scala-y now :)
>
> Robert
>
> On Wed, May 4, 2016 at 3:42 PM, Stefano Baghino <
> stefano.bagh...@radicalbit.io> wrote:
>
>> The only real noise I see is the usage of a MapFunction, which can
> http://repository.apache.org/snapshots/org/apache/flink/flink-streaming-scala_2.11/1.1-SNAPSHOT/flink-streaming-scala_2.11-1.1-SNAPSHOT.pom
>
> It seems that sbt cannot resolve the snapshot dependency.
>
> Any ideas?
>
> Cheers,
> Aris
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
;
> We will activate namenode HDFS high availability in our cluster, and I
> want to know if there is additional configuration for flink ?
> We actually use YARN for launching our flink application, and hdfs
> filesystem to store the state backend
>
> Thanks
>
> Thomas
>
u may
experience some undefined behavior.
* I'm not really sure about this last one, perhaps some more experienced ML
user can help me out on this.
On Mon, May 23, 2016 at 12:54 PM, Stefano Baghino <
stefano.bagh...@radicalbit.io> wrote:
> I think the only keys of interest for your needs (hi
> Do you have any idea of how to integrate such property files into the jar
> package?
>
> -Thanks
> Simon
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
> Do you guys have any ideas how to integrate the sl4fj logger into scala
> flink?
>
> --Thanks
> Simon
>
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
extending java.util.Serializable in the end but I feel there may
be a way to not do this (without rewriting/extending the macros to make the
generated classes serializable).
--
BR,
Stefano Baghino
Software Engineer @ Radicalbit
65 matches
Mail list logo
