Partitioning a libsvm format file

Hi, I am using spark-scala system to train distributed svm. For training svm I am using the files in LIBSVM format. I want to partition a file into fixed number of partititions, with each partition having equal number of datapoints(assume that the number of datapoints in the file is exactly divis

Problem in running mosek in spark cluster - java.lang.UnsatisfiedLinkError: no mosekjava7_0 in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738)

We have a small apache spark cluster of 6 computers. We are trying to solve a distributed problem which requires solving a optimization problem at each machine during a spark map operation. We decided to use mosek as the solver and I collected an academic license to this end. We observed that mos

How to set java.library.path in a spark cluster

Hi, I am working on a 3 machine cloudera cluster. Whenever I submit a spark job as a jar file with native dependency on mosek it shows the following error. java.lang.UnsatisfiedLinkError: no mosekjava7_0 in java.library.path How should I set the java.library.path. I printed the environment varia

Re: Kyro deserialisation error

Hi, I am also facing the same problem. Has any one found out the solution yet? It just returns a vague set of characters. Please help.. Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Exception while deserializing and fetching task: com.esotericsof

java.lang.OutOfMemoryError: Java heap space during reduce operation

Hi, *In a reduce operation I am trying to accumulate a list of SparseVectors. The code is given below;* val WNode = trainingData.reduce{(node1:Node,node2:Node) => val wNode = new Node(num1,num2) wNode.WhatList ++= (node1.WList)