is there any mentrics to show the usage of executor on memory or CPU

Hi all: i want to ask a question about the metrics to show the executor is fully used the memory. in the log i always saw following in the log, i guess this means i did not fully use the executor 's memory. but i don't want to open the log to view, is there any metrics to show it? my confi

[VOTE] Release Spark 2.4.6 (RC3)

Please vote on releasing the following candidate as Apache Spark version 2.4 .6. The vote is open until May 22nd at 9AM PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.4.6 [ ] -1 Do not release this package because ..

Applying schema dynamically in dataframe

Hi dev, Currently I have a scenario where I am reading the data from Kafka using spark dataframe. Multiple data sources ingest the data into kafka same topic. My dataframe now has got json string from multiple sources. I have got key column from Kafka which can be used to identify which source d

PyCharm 2020 :: pyspark installation issue

Hi, Please help me to resolve the below issue is coming while installing pyspark in PyCharm 2020. Collecting pyspark Using cached https://files.pythonhosted.org/packages/9a/5a/271c416c1c2185b6cb0151b29a91fff6fcaed80173c8584ff6d20e46b465/pyspark-2.4.5.tar.gz Complete output from command pyth