Dear,
I am writing a Flink program(Recommender system) needed a matrix as a state
which is the rating matrix, While the matrix is very sparse, I implemented a
sparse binary matrix to save the memory and save only the ones, not all the
matrix and use it as a data type and save it in a value State but unexpectedly
the performance became terrible and the job became very slow, I wonder any
suggestion to know what is the problem?
My first implementation for the rating matrix state :
MapState<String, Map<String, Float>> ratingMatrix;
The second implementation (the slow one) for rating matrix state:
ValueState<SparseBinaryMatrix> userItemRatingHistory;
and this apart from sparseBinaryMatrix class
public class SparseBinaryMatrix implements Serializable {
private ArrayList<Row> content;
private int rowLength;
private HashMap<String, Integer> columnLabels;
private HashMap<Integer, String> inverseColumnLabels;
private HashMap<String, Integer> rowLabels;
private HashMap<Integer, String> inverseRowLabels;
private enum LabelerType{Row, Column};
public Integer colNumber;
public Integer rowNumber;
// This constructor initializes the matrix with zeros
public SparseBinaryMatrix(int rows, int columns)
{
content = new ArrayList<>(rows);
rowLength = columns;
// for (int i = 0; i < rows; i++)
// content.add(new Row(columns));
}
Is depending on other class (Row) may lead to this terrible performance while
Row is class I have implemented and this is part of it
public class Row implements Serializable {
//This is an alternating sorted array
private ArrayList<Integer> content;
private int length=0;
public Row (int numbColumns)
{
length = numbColumns;
for (int i = 0; i < numbColumns;i++)
setColumnToZero(i);
}
public Row (int[] initialValues )
{
length = initialValues.length;
content = new ArrayList<>(length);
for (int i = 0; i < length;i++)
setColumn(i, initialValues[i]);
}
Regards,
Heidy
