Re: Parquet File Output Sink - Spark Structured Streaming

2019-03-27 Thread Matt Kuiper
the Parquet file. Matt From: Gabor Somogyi Sent: Wednesday, March 27, 2019 10:20:18 AM To: Matt Kuiper Cc: user@spark.apache.org Subject: Re: Parquet File Output Sink - Spark Structured Streaming Hi Matt, Maybe you could set maxFilesPerTrigger to 1. BR, G

Parquet File Output Sink - Spark Structured Streaming

2019-03-27 Thread Matt Kuiper
Hello, I am new to Spark and Structured Streaming and have the following File Output Sink question: Wondering what (and how to modify) triggers a Spark Sturctured Streaming Query (with Parquet File output sink configured) to write data to the parquet files. I periodically feed the Stream inpu