Re: [PR] [SPARK-51758][SS] Fix test case related to extra batch causing empty df due to watermark [spark]

[via GitHub]

LuciferYang commented on PR #50626:
URL: https://github.com/apache/spark/pull/50626#issuecomment-2820143914

   @HyukjinKwon @HeartSaVioR @zhengruifeng Do you know which log4j 
configuration file is used by the corresponding Java process during the PySpark 
test?
   
   I found that although the log of this failed pipeline is only 910MB after 
compression, it will occupy 22.25GB of disk space after decompression, and all 
the logs are at the DEBUG level.
   
   - https://github.com/apache/spark/actions/runs/14567609661/attempts/2
   
   
![image](https://github.com/user-attachments/assets/b70a3221-b8c5-4047-abe1-b3c0a44b4020)
   
   
   
![image](https://github.com/user-attachments/assets/430172da-0b37-4316-b556-bdbef722313f)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org