Re: 回复： [DISCUSS] Apache Spark 3.0.1 Release

Another correctness issue: https://issues.apache.org/jira/browse/SPARK-32659 On Tue, Aug 25, 2020 at 11:25 PM Sean Owen wrote: > That isn't a blocker (see comments - not a regression). > That said I think we have a fix ready to merge now, if there are no > objections. > > On Tue, Aug 25, 2020 at

Re: [SparkSql] Casting of Predicate Literals

Hi, So just realized there were already multiple attempts on this issue in the past. From the discussion it seems the preferred approach is to eliminate the cast before they get pushed to data sources, at least for a few common cases such as numeric types. However, a few PRs following this directi

Re: 回复： [DISCUSS] Apache Spark 3.0.1 Release

That isn't a blocker (see comments - not a regression). That said I think we have a fix ready to merge now, if there are no objections. On Tue, Aug 25, 2020 at 10:24 AM Dongjoon Hyun wrote: > > For the correctness blocker, we have the following, Tom. > > - https://issues.apache.org/jira/browse/SP

Re: 回复： [DISCUSS] Apache Spark 3.0.1 Release

For the correctness blocker, we have the following, Tom. - https://issues.apache.org/jira/browse/SPARK-32614 - https://github.com/apache/spark/pull/29516 Bests, Dongjoon. On Tue, Aug 25, 2020 at 6:32 AM Tom Graves wrote: > Hey, > > I'm just curious what the status of the 3.0.1 release is? Do

Re: Renaming blacklisting feature input

Any other feedback here?  The couple I've heard preferred in various conversations are excludeList and blockList.  If not I'll just make proposal on jira and continue discussion there and anyone interested can watch this jira. Thanks,Tom On Tuesday, August 4, 2020, 09:19:01 AM CDT, Tom Grave

Re: Removing references to Master

Thanks for the replies so far, is there any other feedback here?    Of the replies so far I think Leader has been mentioned the most. Tom On Tuesday, August 4, 2020, 09:33:14 AM CDT, Russell Spitzer wrote: I think we should use Scheduler or Comptroller or Leader; something that evokes

Re: 回复： [DISCUSS] Apache Spark 3.0.1 Release

Hey, I'm just curious what the status of the 3.0.1 release is?  Do we have some blockers we are waiting on? Thanks,Tom On Sunday, August 16, 2020, 09:07:44 PM CDT, ruifengz wrote: Thanks for letting us know this issue. On 8/16/20 11:31 PM, Takeshi Yamamuro wrote: I've chec

Re: Question about Expression Encoders

works perfectly! Thanks Herman. Am Di., 25. Aug. 2020 um 12:03 Uhr schrieb Herman van Hovell < her...@databricks.com>: > Hi Robert, > > Your Spark 3.0 code is missing the encoder that converts the Row to an > InternalRow. Your Spark 3.0 code should look like this: > > def rowToCaseClass[C <: Prod

Re: Question about Expression Encoders

Hi everyone Thanks Takeshi. I run into the same issue as Mark for my row to case class converter: def rowToCaseClass[C <: Product : TypeTag](r: Row)(implicit encs: (ExpressionEncoder[Row], ExpressionEncoder[C])): C = { val ir = encs._1.toRow(r) encs._2.fromRow(ir) } So in Spark3.0 I would

Re: [DISCUSS] "latestFirst" option and metadata growing issue in File stream source

Bump this again. On Tue, Aug 18, 2020 at 12:11 PM Jungtaek Lim wrote: > Bump again. > > Unlike file stream sink which has lots of limitations and many of us have > been suggesting alternatives, file stream source is the only way if end > users want to read the data from files. No alternative unl