Hi all, I'd like to bring this to your attention once again. I've seen avatars displaying activity in the document, but haven't received any comments. Hopefully that just means that you found the proposal to be AMAZING!
I would like to start a Vote on this next week, probably Tuesday since Monday is a holiday in the United States. Looking forward to your feedback. Best, Burak On Mon, Aug 31, 2026 at 9:46 AM Burak Yavuz <[email protected]> wrote: > Hi all, > > We would like to start a discussion on introducing a new SparkSQL > data type called FILE for providing better support for unstructured or > large data in Spark. > > Jira: SPARK-59132 <https://issues.apache.org/jira/browse/SPARK-59132> > > Brief context: We want to make it easy and cheap to work with files, such > as PDFs, images, audio and video, from tables using Spark. Today you either > copy the bytes of a file into a table as a binary column, or you keep the > path to the file as a string and hope everything downstream knows what to > do with it. > We propose a new data type, FileType (spelled FILE in SQL), which holds > the path of a file and a small amount of metadata describing it. The > engine, the file format, and user defined functions all understand this > type, so a query can filter, join and project files without reading their > contents, and load the bytes only where it actually needs them. > > Please find more information in the SPIP > <https://docs.google.com/document/d/1pPof896ZwcZ-2Yn-YhC4TbyYWn1umiQJykGDkAxzCWc/edit?tab=t.0> > . > > We've recently introduced a FILE type in Parquet > <https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file> > to provide a consistent representation across engines and table formats to > access large blobs. This SPIP provides the engine interfaces and > implementation to leverage this representation. > > Looking forward to your feedback! > > Best regards, > Burak Yavuz and Dejan Krakovic > > >
