Inconsistent schema on Encoders.bean (reported issues from user@)

2020-05-04 Thread Jungtaek Lim
Hi devs, There're couple of issues being reported on the user@ mailing list which results in being affected by inconsistent schema on Encoders.bean. 1. Typed datataset from Avro generated classes? [1] 2. spark structured streaming GroupState returns weird values from sate [2] Below is a part of

Handling user-facing metadata issues on file stream source & sink

2020-05-04 Thread Jungtaek Lim
Hi devs, I'm seeing more and more structured streaming end users encountered the metadata issues on file stream source and sink. They have been known-issues and there're even long-standing JIRA issues reported before, end users report them again in user@ mailing list in April. * Spark Structure S