Flink + Marathon (Mesos) Memory Issues

*Background*: We have a setup of Flink 1.4.0. We run this flink cluster via /flink-jobmanager.sh foreground/ and /flink-taskmanager.sh foreground/ command via Marathon (which launches them as mesos jobs). So, basically, jobmanager and taskmanagers run as mesos tasks. Now, say, we run the flink

Re: Flink 1.4.0 Mapr libs issues

Hi Fabian, Thanks a lot for the reply. Setting the classloader.resolve-order configuration seems to have done the trick. For anybody else, having the same problem as this, this is the config that I set: /*classloader.resolve-order: parent-first classloader.parent-first-patterns: java.;org.apache.f

Flink 1.4.0 Mapr libs issues

*Background:* We have a setup of Flink 1.3.2 along with a secure MAPR (v5.2.1) cluster (Flink is running on mapr client nodes). We run this flink cluster via flink-jobmanager.sh foreground and flink-taskmanager.sh foreground command via Marathon. We want to upgrade to Flink 1.4.0. Since, we requir

Re: MapR libraries shading issue

Cool. For future reference, I created a JIRA ticket: https://issues.apache.org/jira/browse/FLINK-7033 Thanks for all the help, guys. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/MapR-l

Re: MapR libraries shading issue

Again as I mentioned in the MapR thread, So, after some more digging, I found out that you can make flink use the default java truststore by passing -Djavax.net.ssl.trustStore=$JAVA_HOME/jre/lib/security/cacerts as JVM_ARGS for Flink. I tested this approach with AWS, datadog along with MapR Stream

Re: MapR libraries shading issue

Pasting my reply from the MapR community thread: So, I have found a temporary workaround for this. Heres what I did: For the client, I found out the password of /opt/mapr/conf/ssl_trustore via ssl-client.xml For both Datadog and Amazon S3, I found out the certs chain that they were using (i could

Re: MapR libraries shading issue

As Stephan pointed out, this seems more like a MapR libs meddling with some jar. As I had mentioned in the original question, I run across the same problem when i use the aws sdk jar in my program. The error is as follows: /shaded.com.amazonaws.SdkClientException: Unable to execute HTTP request: s

MapR libraries shading issue

I am trying to use Flink (1.3.0) with MapR(5.2.1). Accordingly, I built Flink for Mapr as follows with maven 3.1: /mvn clean install -DskipTests -Dscala.version=2.10.6 -Pvendor-repos,mapr -Dhadoop.version=2.7.0-mapr-1703 -Dzookeeper.version=3.4.5-mapr-1604/ I, then added /opt/mapr/lib/* to Flink

Flink with Mesos: Fetcher error

I am trying to configure Flink to work on top of Mesos. I am using Flink release-1.3. I am using DCOS 1.9's underlying mesos which is version 1.2. I am able to start Flink without any issues when the taskmanager starts on the same host as that of appmaster. But when the taskmanager is launched on a

Re: Flink, Yarn and MapR Kerberos issue

So, I was able to circumvent this issue. This is in no way a permanent solution, but I thought I should let you (and anybody who encounters this problem in future) know some of my observations. What I fount out was that, 1. In Mapr's version of hadoop, they do the authentication inside initialize(

Flink, Yarn and MapR Kerberos issue

I am trying to setup Flink 1.2 using yarn on MapR (v5.2.0). The MapR cluster, on which, I am trying to setup this is a secure cluster. But, this cluster does not use Kerberos. Mapr, by default, uses some variant of ssl

Re: Netty issues while deploying Flink with Yarn on MapR

Thanks Robert. I would love to try to solve this problem so that future MapR and Flink users do not face these issues. Should I create a JIRA for it? Let me know how I can be of help. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Netty-issu

Re: Netty issues while deploying Flink with Yarn on MapR

In case anyone is having similar issues with Flink on Yarn on MapR, I managed to solve this issue with help from the MapR community. -- View this message in context: http://apache-flink-user-

Netty issues while deploying Flink with Yarn on MapR

I am trying to run Flink using Yarn on MapR. My previous issue got resolved and I have updated the original post accordingly so. Accordingly, I modified pom.xml to change the zookeeper

Re: Issues while restarting a job on HA cluster

Hi Robert, Thanks for the answer. My code does actually contain both mapr streams and maprdb jars. here are the steps I followed based on your suggestion: 1. I copied only the mapr-streams-*.jar and maprdb*.jar. 2. Then I tried to run my jar, but i got java.lang.noclassdeffounderror for some maprf

Issues while restarting a job on HA cluster

1. We have a HA cluster of 2 masters and 3 slaves. We run a jar through flink cli. Then we cancel that running job. Then we do some changes in the source code of jar, repackage it and deploy it again and run it again through cli. The following error occurs: / java.lang.LinkageError

Flink with Yarn on MapR

Hi, I am trying to setup flink with Yarn on Mapr cluster. I built flink (flink-1.3-SNAPSHOT) as follows: mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.0-mapr-1607 The build is successful. Then I try to run ./bin/yarn-session.sh -n 4 (without changing any config or whatsoever