Hi, Karen.
Are you saying that Spark 3 has to have all deprecated 2.x APIs?
Could you tell us what is your criteria for `unnecessarily` or
`necessarily`?
> the migration process from Spark 2 to Spark 3 unnecessarily painful.
Bests,
Dongjoon.
On Tue, Feb 18, 2020 at 4:55 PM Karen Feng
wrote:
So my understanding would be that to provide a reasonable migration path
we’d want the replacement of the deprecated API to also exist in 2.4 this
way libraries and programs can dual target during the migration process.
Now that isn’t always going to be doable, but certainly worth looking at
the s
Yes, right. We need to choose the best way case by case.
The following level of `Runtime warning` also sounds reasonable to me.
> Maybe we should only log the warning once per Spark application.
Bests,
Dongjoon.
On Tue, Feb 18, 2020 at 1:13 AM Wenchen Fan wrote:
> I don't know what's the best
Like https://github.com/apache/spark/pull/23131, we added back unionAll.
We might need to double check whether we removed some widely used APIs in
this release before RC. If the maintenance costs are small, keeping some
deprecated APIs look reasonable to me. This can help the adoption of Spark
3.0
Sure. I understand the background of the following requests. So, it's a
good time to decide the criteria in order to start discussion.
1. "to provide a reasonable migration path we’d want the replacement of
the deprecated API to also exist in 2.4"
2. "We need to discuss the APIs case by ca
Apache Spark 2.0 was released in July 2016. Assuming the project has been
trying the best to follow the semantic versioning, it is "more than three
years" to wait for the breaking changes. What the community misses to
address necessary breaking changes would be going to be technical debts for
anoth
I think I was too rushed to read and focused on the first sentence of
Karen's input. Sorry about that.
As I said I'm not sure I can agree with the point of deprecation and
breaking changes of APIs, the thread has another topic which seems to be a
good input - practice on new API proposal. I feel i
