Using FieldCache is discouraged.
Introduce docValues for id field.
On Tue, Apr 30, 2024 at 10:25 AM prasad bezavada <prasadbezav...@gmail.com>
wrote:
> Hi everyone,
>
> I'm currently working on querying Solr and writing the results to a file
> using our Java application. To prevent memory problems, I've opted to use
> CursorMark for pagination instead of querying all the data in one go.
> However, I'm still encountering out-of-memory issues, which is surprising
> even when dealing with small amounts of data. Our Solr instance contains a
> vast amount of indexed data.
>
> Upon investigation, it seems that because CursorMark requires sorting
> (which is mandatory for pagination using CursorMarkParams), it's loading
> the entire 'id' field values into the cache (CACHE.core.fieldCache), which
> may be causing the problem. Despite the large amount of data in Solr, the
> actual data retrieved by my query is very small (below 100,000 documents).
>
> Below is the code snippet I'm using for reference:
>
> SolrClient solr = searchUtil.getSolrClient(userInfo);
> SolrQuery query = new SolrQuery();
> query.clear();
> query.setParam("cache", "false");
> //query.setStart(start);
> query.setSort(SortClause.asc("id"));
> query.set(CursorMarkParams.CURSOR_MARK_PARAM,cursor);
> query.setRows(numRecordsPerPage);
> query.setQuery(searchQuery);
> queryResponse = solr.query(query, METHOD.POST);
> //log.info("Query Status " + queryResponse.getStatus());
> query.clear();
>
> ======================
>
> Attaching the Exception log from Solr Log file.PFA.
>
> I'm actively looking into ways to address this issue. If anyone has any
> insights or suggestions, I'd greatly appreciate it.
> --
> *Thanks&Regards*
>
> *Prasad Bezavada*
>
>
>
--
Sincerely yours
Mikhail Khludnev
