Re: In yarn-cluster model, mapState not support transfer value

2019-09-19 Thread shengjk1
Hi, Dian Thanks for your reminder. I saw HeapMaoState.java and TtlMapState.java, You are right. But with objectList get large, every time call "mapState.put(key,objectList);” is very influencing performance, even lead to checkpoint timeout. Now I am not have a better method to improve perf

Re: In yarn-cluster model, mapState not support transfer value

2019-09-19 Thread Dian Fu
Hi Shengjk1, You should call "mapState.put(key,objectList);" manually after calling "objectList.add(stringObjectTuple2.f1);" to write it to the state backend. This is because objectList is just a common Java list object and it will not be synced to state backend automatic when updated. I gues

In yarn-cluster model, mapState not support transfer value

2019-09-19 Thread shengjk1
Hi,all As we know java map support transfer value,such as : HashMap> stringListHashMap = new HashMap<>(); for (int i = 0; i < 10; i++) { List a = stringListHashMap.get("a"+i%2); if (a==null){ a=new ArrayList<>(); stringListHashMap.put("a"+i%2,a); } a.add("a"+i); } stringListHash