Thanks for your reply. It makes sense as to why the option is not provided.
(Since the user is the one who is imperatively asking spark to read the
files.)
Yes, I provide the list of files. I'll try the ignoreCorruptFiles option.
Also, I'll look into how I can avoid missing files or at least check
Where is this list of files coming from?
If you made the list, then yes, the expectation is generally "supply a list
of files which are present" on the basis that general convention is
"missing files are considered bad"
Though you could try setting spark.sql.files.ignoreCorruptFiles=true to see
w
