it was helpful,
Then, the OS needs to fill some pressure from the applications
requesting memory to free some memory cache?
Exactly under which circumstances the OS free that memory to give it to
applications requesting it?
I mean if the total memory is 16GB and 10GB are used for OS cache,
We want to extract data from mysql, and calculate in sparksql.
The sql explain like below.
== Parsed Logical Plan ==
> 'Sort ['revenue DESC NULLS LAST], true
> +- 'Aggregate ['n_name], ['n_name, 'SUM(('l_extendedprice * (1 -
> 'l_discount))) AS revenue#329]
>+- 'Filter ('c_custkey = 'o_cu
Hi,
When I issue a "free -m" command in a host, I see a lot of memory used
for cache in OS, however Spark Streaming is not able to request that
memory for its usage, and it fail the execution due to not been able to
launch executors.
What I understand of the OS memory cache (the one in "free
Applying this fix https://github.com/apache/spark/pull/19663
Or
Using --files or --jars /local/path/to/hive-site.xml may works
Thanks,
Kent
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
-
To unsubs
