; is
> >> >> empty, you will need to go and lookup the previous previous partition
> >> >> value
> >> >> until you find the one you are looking for. (Note this assumes that
> >> the
> >> >> first record in your dataset is valid, if it isn't your code will
> >> still
> >>
;> >> <http:///user/SendEmail.jtp?type=node&node=20516&i=0>;>
>> >> wrote:
>> >>
>> >>
>> >> Hi Georg,
>> >>
>> >> Can you describe your question more clear?
>> >>
>> >> Actua
working on building a custom ML pipeline-model / estimator to
> >> impute
> >> > missing values, e.g. I want to fill with last good known value.
> >> > Using a window function is slow / will put the data into a single
> >> > partition.
> >>
;> > I built some sample code to use the RDD API however, it some None /
>> null
>> > problems with empty partitions.
>> >
>> > How should this be implemented properly to handle such empty
>> partitions?
>> >
>> http://stackoverflow.com/qu
-handling-empty-partitions
> >
> > Kind regards,
> > Georg
>
>
>
>
>
> -
>
>
> Liang-Chi Hsieh | @viirya
> Spark Technology Center
> http://www.spark.tc/
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001
egards,
> > Georg
>
>
>
>
>
> -
> Liang-Chi Hsieh | @viirya
> Spark Technology Center
> http://www.spark.tc/
> --
> View this message in context: http://apache-spark-
> developers-list.1001551.n3.nabble.com/handling-of-empty-
> partitions-tp20496p2051
such empty partitions?
> http://stackoverflow.com/questions/41474175/spark-mappartitionswithindex-handling-empty-partitions
>
> Kind regards,
> Georg
-
Liang-Chi Hsieh | @viirya
Spark Technology Center
http://www.spark.tc/
--
View this message in context:
http://apache-spark-develope
.nabble.com/handling-of-empty-partitions-tp20496.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
