OK I just went through the change, and the change breaks bunch of existing
UTs.
https://github.com/apache/spark/pull/28611
Note that I modified all the cases where Spark extracts the columns for
"read method" only properties to both "read" & "write". It doesn't only
change the code path of Encode
Hi,
Spark Streaming has a `mapWithState` API to run a map on a stream while
maintaining a state as elements are read.
The core RDD API does not seem to have anything similar. Given a RDD of
elements of type T, an initial state of type S and a map function (S,T)
-> (S,T), return an RDD of Ts obtai
The API signature would of course be more general (sorry!):
Given a RDD of elements of type T, an initial state of type S and a map
function (S,T) -> (S,U), return an RDD of Us obtained by applying the
map function in sequence, updating the state as elements are mapped.
With this formulation, zip
On 24/05/2020 11:27, Antonin Delpeuch (lists) wrote:
> With this formulation, zipWithIndex would be a special case of
> mapWithState (so it could be refactored to be expressed as such).
Forget about this part, it would obviously not, since zipWithIndex can
compute the size of each partition indepe
Hello,
I need to unsubscribe apache spark's information right now. Thanks a lot!
Best regard,
Tommy
Java Beans are well-defined; it's valid to have a getter- or
setter-only property. That doesn't mean Spark can meaningfully use
such a property, as it typically has to both read and write them. I
guess it depends on context. For example, I don't see how you can have
a deserializer without setters,
I already went ahead with this one, everything is pretty self explanatory +
previous emails seem pretty helpful about how to test things. I don't need
answers on my previous questions any more.
On Fri, May 22, 2020 at 10:12 AM Edgar Klerks
wrote:
> Hi there,
>
> I am a potentially new contributo
Hi Sean,
The fix was to provide explicitly typed versions of the the three generic
methods at the bottom of the code:
i.e
static List createListAndCombine(T v) {
List c = new ArrayList<>();
c.add(v);
return c;
}
becomes
static List createListAndCombine(Nut v) {
List c = new
