Re: All but one TMs connect when JM has more than 16G of memory

So for anyone who is interested, here are some code references for getting started with Flink on Slurm. I added basic start and stop scripts for Flink on Slurm in my fork: https://github.com/robert-schmidtke/flink/tree/flink-slurm/flink-dist/src/main/flink-bin/bin And I also created an example of

Re: kryo exception due to race condition

This looks to me like a bug where type registrations are not properly forwarded to all Serializers. Can you open a JIRA ticket for this? On Thu, Oct 1, 2015 at 6:46 PM, Stefano Bortoli wrote: > Hi guys, > > I hit a Kryo exception while running a process 'crossing' POJOs datasets. > I am using t

kryo exception due to race condition

Hi guys, I hit a Kryo exception while running a process 'crossing' POJOs datasets. I am using the 0.10-milestone-1. Checking the serializer: org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:210) I have noticed that the Kryo instance is reused along s

Re: DataSet transformation

Hi, for that you have to collect the dataset to your local machine and then transform the collection into the array. Note that this only advised for small data sets. Robert On Thu, Oct 1, 2015 at 2:13 PM, Lydia Ickler wrote: > Hi all, > > so I have a case class Spectrum(mz: Float, intensity:

DataSet transformation

Hi all, so I have a case class Spectrum(mz: Float, intensity: Float) and a DataSet[Spectrum] to read my data in. Now I want to know if there is a smart way to transform my DataSet into a two dimensional Array ? Thanks in advance, Lydia

Re: All but one TMs connect when JM has more than 16G of memory

Feel free to contribute a documentation to Flink on how to run Flink on SLURM. On Thu, Oct 1, 2015 at 11:45 AM, Robert Schmidtke wrote: > I see, thanks for the info. I only have access to my cluster via SLURM and > we don't have ssh between our nodes which is why I haven't really > considered th

Re: All but one TMs connect when JM has more than 16G of memory

I see, thanks for the info. I only have access to my cluster via SLURM and we don't have ssh between our nodes which is why I haven't really considered the Standalone mode. A colleague has set up YARN on SLURM and it was just the easiest to use. I briefly looked into the Flink Standalone mode but d

Re: All but one TMs connect when JM has more than 16G of memory

Hi, there is currently no option for forcing certain containers onto specific machines. For running the JM (or any other YARN container) on the AM host, you first need to have a NodeManager running on the host with the RM. Maybe YARN is smart enough to schedule the small JM container onto that mach

Re: OutOfMemoryError in netty local transport

Great to hear :) On Thu, Oct 1, 2015 at 11:21 AM, Robert Schmidtke wrote: > I pulled the current master branch and rebuilt Flink completely anyway. > Works like a charm. > > On Thu, Oct 1, 2015 at 11:11 AM, Maximilian Michels wrote: >> >> By the way, you might have to use the "-U" flag to force

Re: All but one TMs connect when JM has more than 16G of memory

Hi Robert, I had a job failure yesterday with what I believe is the setup I have described above. However when trying to reproduce now, the behavior is the same: Flink waiting for resources to become available. So no hard error. Ok, the looping makes sense then. I haven't thought about shared set

Re: OutOfMemoryError in netty local transport

I pulled the current master branch and rebuilt Flink completely anyway. Works like a charm. On Thu, Oct 1, 2015 at 11:11 AM, Maximilian Michels wrote: > By the way, you might have to use the "-U" flag to force Maven to > update its dependencies: mvn -U clean install -DskipTests > > On Thu, Oct

Re: OutOfMemoryError in netty local transport

By the way, you might have to use the "-U" flag to force Maven to update its dependencies: mvn -U clean install -DskipTests On Thu, Oct 1, 2015 at 10:19 AM, Robert Schmidtke wrote: > Sweet! I'll pull it straight away. Thanks! > > On Thu, Oct 1, 2015 at 10:18 AM, Maximilian Michels wrote: >> >>

Re: All but one TMs connect when JM has more than 16G of memory

Hi, It is interesting to note that when I set both yarn.nodemanager.resource.memory-mb > and yarn.scheduler.maximum-allocation-mb to 56G I get a proper error when > requesting 56G and 1M, but when setting yarn.nodemanager.resource.memory-mb > to 56G and yarn.scheduler.maximum-allocation-mb to 54G

Re: error message

Hi! The most common cause for "NoSuchMethodError" in Java is a mixup between code versions. You may have compiled against a different version of the code than you are running against. The fix is mostly simply to make sure you have the same version that you program against and that you run, and re

Re: OutOfMemoryError in netty local transport

Sweet! I'll pull it straight away. Thanks! On Thu, Oct 1, 2015 at 10:18 AM, Maximilian Michels wrote: > Hi Robert, > > Just a quick update: The issue has been resolved in the latest Maven > 0.10-SNAPSHOT dependency. > > Cheers, > Max > > On Wed, Sep 30, 2015 at 3:19 PM, Robert Schmidtke > wrote

Re: OutOfMemoryError in netty local transport

Hi Robert, Just a quick update: The issue has been resolved in the latest Maven 0.10-SNAPSHOT dependency. Cheers, Max On Wed, Sep 30, 2015 at 3:19 PM, Robert Schmidtke wrote: > Hi Max, > > thanks for your quick reply. I found the relevant code and commented it out > for testing, seems to be wor