asdfgh19 opened a new issue, #10823:
URL: https://github.com/apache/ignite/issues/10823
I created a cache like this IgniteCache<String, BinaryObject>. In order to
do upsert, I used EntryProcessor as follows.
<details>
<summary>EntryProcessor </summary>
```java
public class UpsertEntryProcessorV1
implements EntryProcessor<String, BinaryObject, Void> {
@IgniteInstanceResource
private Ignite ignite;
@SuppressWarnings("unchecked")
@Override
public Void process(MutableEntry<String, BinaryObject> entry,
Object... arguments) throws EntryProcessorException {
if (entry == null || entry.getKey() == null) {
return null;
}
Map<String, Map<String, Object>> params =
(Map<String, Map<String, Object>>) arguments[0];
Map<String, Object> param = params.get(entry.getKey());
if (param == null || param.isEmpty()) {
return null;
}
BinaryObjectBuilder builder;
BinaryObject entryVal = entry.getValue();
if (entryVal != null) {
builder = entryVal.toBuilder();
} else {
String valueType = (String) arguments[1];
builder = ignite.binary().builder(valueType);
}
for (Map.Entry<String, Object> item : param.entrySet()) {
if (item.getKey() == null) {
continue;
}
builder.setField(item.getKey(), item.getValue());
}
entry.setValue(builder.build());
return null;
}
}
```
</details>
In this cache, I wrote 20,000 records, each record has 1 to 1,000 fields.
I found that the ignite service has a lot of full gc, and even crashes
directly. Through heap dump analysis, I found that BinarySchema occupies a lot
of memory.
<details>
<summary>heap</summary>
</details>
**Eventually I figured out that the problem was when I was randomly writing
to some fields of a record every time**
when I write a record for the first time, a BinarySchema will be created.
The next time I update this record and write one more field, a new
BinarySchema will be created and written to ./work/db/ binary_meta/, the old
schema will not be cleared.
In the end, tens of thousands of BinarySchemas will be created, but there
are only dozens of BinarySchemas that are actually valid in serialized storage.
The following are some places I found to store BianrySchema, is there a way
to clean up these BinarySchema?
1. BinaryContext#descByCls#schemaReg
2. BinaryContext#schemas
3. CacheObjectBinaryProcessorImpl#metadataLocCache#metadata#schemas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
