[
https://issues.apache.org/jira/browse/SPARK-52566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-52566.
----------------------------------
Resolution: Invalid
Resolving as Invalid — this is a usage/how-to question rather than a specific
Spark defect or actionable change. Usage questions are best directed to
[email protected] (https://spark.apache.org/community.html) or Stack
Overflow (tag apache-spark). Findings from triage: Verified against
apache/master Logging.scala that this is a support/configuration question, not
a Spark defect. Spark's initializeLogging
(common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:343) only
applies its own default profile (log4j2-defaults.properties,
rootLogger.level=info) when islog4j2DefaultConfigured() is true (line 472) —
which requires the user to have essentially NO real Log4j2 config (empty
appenders, or a single ERROR appender whose active config is literally Log4j2's
built-in DefaultConfiguration class). Any user-provided config file yields a
different Confi
Please reopen with a concrete reproducer or a specific proposed change if this
is actually a bug or an actionable improvement.
> Logging Level is not getting overrided
> ---------------------------------------
>
> Key: SPARK-52566
> URL: https://issues.apache.org/jira/browse/SPARK-52566
> Project: Spark
> Issue Type: Question
> Components: Java API, Spark Core
> Affects Versions: 3.4.0
> Environment: Grails - 4.0.12
> Reporter: Anuradha Bhan
> Priority: Major
> Labels: Java, Spark, grails
>
> Grails Project is using Log4j2. Logging level is set to ERROR using
> Configurator before spark session is started. Even then INFO level logs are
> getting printed by spark. It is picking up the default log properties inside
> the spark-core jar.
>
> Code snippet :
>
> Configurator.setLevel("org.apache.spark", Level.ERROR)
> Logger logger = (Logger) LogManager.getLogger("org.apache.spark");
> System.out.println("Logger instance level: " + logger.getLevel());
> logger.debug("This DEBUG should not appear");
> logger.info("This INFO should not appear");
> logger.error("This ERROR should appear");
> SparkSession spark = null;
> StructType schema = createParquetSchema()
> spark = SparkSession
> .builder()
> .appName("XXXX")
> .config("spark.master", "local")
> .config("spark.io.compression.codec", "snappy")
> .config("spark.io.compression.snappy.blockSize", blockSize)
> .getOrCreate();
>
>
> The output comes out to be :
> Logger instance level: DEBUG
> This DEBUG should not appear
> This INFO should not appear
> This ERROR should appear
> Using Spark's default log4j profile:
> org/apache/spark/log4j2-defaults.properties
> 25/06/24 18:30:43 WARN Utils: Your hostname, AnuradhaB resolves to a loopback
> address: 127.0.1.1; using <IPAddress> instead (on interface wlp0s20f3)
> 25/06/24 18:30:43 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
> another address
> 25/06/24 18:30:43 INFO SparkContext: Running Spark version 3.4.0
> 25/06/24 18:30:43 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 25/06/24 18:30:43 INFO ResourceUtils:
> ==============================================================
> 25/06/24 18:30:43 INFO ResourceUtils: No custom resources configured for
> spark.driver.
> 25/06/24 18:30:43 INFO ResourceUtils:
> ==============================================================
> 25/06/24 18:30:43 INFO SparkContext: Submitted application: TestABC
> 25/06/24 18:30:43 INFO ResourceProfile: Default ResourceProfile created,
> executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: ,
> memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name:
> offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name:
> cpus, amount: 1.0)
> 25/06/24 18:30:43 INFO ResourceProfile: Limiting resource is cpu
> 25/06/24 18:30:43 INFO ResourceProfileManager: Added ResourceProfile id: 0
> 25/06/24 18:30:43 INFO SecurityManager: Changing view acls to: anuradha
> 25/06/24 18:30:43 INFO SecurityManager: Changing modify acls to: anuradha
> 25/06/24 18:30:43 INFO SecurityManager: Changing view acls groups to:
> 25/06/24 18:30:43 INFO SecurityManager: Changing modify acls groups to:
> 25/06/24 18:30:43 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: anuradha; groups
> with view permissions: EMPTY; users with modify permissions: anuradha; groups
> with modify permissions: EMPTY
> 25/06/24 18:30:43 INFO Utils: Successfully started service 'sparkDriver' on
> port 38691.
> 25/06/24 18:30:43 INFO SparkEnv: Registering MapOutputTracker
> 25/06/24 18:30:43 INFO SparkEnv: Registering BlockManagerMaster
> 25/06/24 18:30:43 INFO BlockManagerMasterEndpoint: Using
> org.apache.spark.storage.DefaultTopologyMapper for getting topology
> information
> 25/06/24 18:30:43 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint
> up
> 25/06/24 18:30:43 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
> 25/06/24 18:30:43 INFO DiskBlockManager: Created local directory at
> /tmp/blockmgr-abf8fb69-c6a4-40e5-bd43-ad0ccb47e759
> 25/06/24 18:30:43 INFO MemoryStore: MemoryStore started with capacity 4.1 GiB
> 25/06/24 18:30:44 INFO SparkEnv: Registering OutputCommitCoordinator
> 25/06/24 18:30:44 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI
> 25/06/24 18:30:44 INFO Utils: Successfully started service 'SparkUI' on port
> 4040.
> 25/06/24 18:30:44 INFO Executor: Starting executor ID driver on host
> <IPAddress>
> 25/06/24 18:30:44 INFO Executor: Starting executor with user classpath
> (userClassPathFirst = false): ''
> 25/06/24 18:30:44 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42551.
> 25/06/24 18:30:44 INFO NettyBlockTransferService: Server created on
> <IPAddress>:42551
> 25/06/24 18:30:44 INFO BlockManager: Using
> org.apache.spark.storage.RandomBlockReplicationPolicy for block replication
> policy
> 25/06/24 18:30:44 INFO BlockManagerMaster: Registering BlockManager
> BlockManagerId(driver, <IPAddress>, 42551, None)
> 25/06/24 18:30:44 INFO BlockManagerMasterEndpoint: Registering block manager
> <IPAddress>:42551 with 4.1 GiB RAM, BlockManagerId(driver, <IPAddress>,
> 42551, None)
> 25/06/24 18:30:44 INFO BlockManagerMaster: Registered BlockManager
> BlockManagerId(driver, <IPAddress>, 42551, None)
> 25/06/24 18:30:44 INFO BlockManager: Initialized BlockManager:
> BlockManagerId(driver, <IPAddress>, 42551, None)
>
>
> What else could be done to set the logging level to ERROR
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
