[I] Incorrect timing information reported for FilterExec in Spark UI [datafusion-comet]

Wed, 11 Sep 2024 10:08:28 -0700 


andygrove opened a new issue, #939:
URL: https://github.com/apache/datafusion-comet/issues/939

   ### Describe the bug
   
   
   I ran TPC-H q1 with native explain enabled. I then grepped the executor log 
for timings for FilterExec. Here is partial output:
   
   ```
       FilterExec: col_6@6 IS NOT NULL AND col_6@6 <= 1998-09-24, 
metrics=[output_rows=3120492, elapsed_compute=9.921355ms]
       FilterExec: col_6@6 IS NOT NULL AND col_6@6 <= 1998-09-24, 
metrics=[output_rows=3120799, elapsed_compute=10.538966ms]
       FilterExec: col_6@6 IS NOT NULL AND col_6@6 <= 1998-09-24, 
metrics=[output_rows=3120290, elapsed_compute=9.235761ms]
   ```
   
   I extracted the individual timings and summed them, and this shows a total 
of 7.2 seconds.
   
   The Spark UI reports 2.6 seconds for this filter:
   
   ![Screenshot from 2024-09-11 
09-57-10](https://github.com/user-attachments/assets/1350ccb2-05c9-4ade-921d-a159e8637fd6)
   
   
   
   ### Steps to reproduce
   
   _No response_
   
   ### Expected behavior
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to