Re: [PySpark] Reader/Writer for bgzipped data

Hello Oliver, Yes, Spark makes this possible using the Hadoop compression codecs and the Hadoop-compatible FileSystem interface [1]. Here is an example of reading: df = spark.read.text("gs:///data/shakespeare-bz2") df.show(10) This is using a test data set of the complete works of Shakespeare, s

[PySpark] Reader/Writer for bgzipped data

Hello, Is it possible to read/write a DataFrame from/to a set of bgzipped files? Can it read from/write to AWS S3? Thanks! Best, Oliver -- Oliver Ruebenacker, Ph.D. (he) Senior Software Engineer, Knowledge Portal Network , Flannick Lab