Yes, this does not read raw pickle files. It reads files written in the standard Spark/Hadoop form for binary objects (SequenceFiles) but uses Python pickling for the serialization. See the docs, which say this reads what saveAsPickleFile() writes.
On Mon, Aug 26, 2019 at 12:23 AM hxngillani <f2017279...@umt.edu.pk> wrote: > > Hello Dear Members > i want to train model using Bigdl, i have data set of Medical images in the > form of pickle object files (,pck).that pickle file is a 3D image(3D array) > > i have tried > pickleRdd = sc.pickleFilehome/student/BigDL- > trainings/elephantscale/data/volumetric_data/329637-8.pck > sqlContext = SQLContext(sc) > df = sqlContext.createDataFrame(pickleRdd) > > this code throwing and error > Caused by: java.io.IOException: > file:/home/student/BigDL-trainings/elephantscale/data/volumetric_data/329637-8.pck > not a SequenceFile > > > the things i came to know is that > The function > sc.pickleFile > loads a pickle file that is created by > rdd.saveAsPickleFile > > I am loading a pickle file that is created by Python's "pickle" module > My Question is that is there any way to load that file in spark data frame > > > > -- > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
