[GitHub] [hudi] jsbali edited a comment on pull request #3946: [HUDI-2711] Fallback to fulltable scan for IncrementalRelation if underlying files have been cleared or moved by cleaner

Sat, 08 Jan 2022 02:12:43 -0800 


jsbali edited a comment on pull request #3946:
URL: https://github.com/apache/hudi/pull/3946#issuecomment-1007552923


   Adding more context here
   ![Screenshot 2022-01-07 at 8 52 02 
PM](https://user-images.githubusercontent.com/1778470/148575437-1bcacaef-deac-4ec4-b76d-0524798116e5.png)
   Left side is the one which we get when HoodieFileIndex does its magic. Right 
side is the one in the commit file
   Since partition is expected in the middle, this 
[union](https://github.com/jsbali/hudi/blob/ftscan/hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/hudi/IncrementalRelation.scala#L197)
 fails 
   Have made the change when we call HadoopFsRelation so that order of schema 
is maintained. 
   
   There is one other way where we skip the HoodieFileIndex path and directly 
take in dir glob pattern but that is one extra config which needs to be 
maintained. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to