Spark Dataset withColumn issue

2020-11-12 Thread Vikas Garg
In Spark Datase, if we add additional column using withColumn then the column is added in the last. e.g. val ds1 = ds.select("Col1", "Col3").withColumn("Col2", lit("sample")) the the order of columns is >> Col1 | Col3 | Col2 I want the order to be >> Col1 | Col2 | Col3 How can I achiev

Re: Spark Dataset withColumn issue

2020-11-12 Thread Vikas Garg
I am deriving the col2 using with colunn which is why I cant use it like you told me On Thu, Nov 12, 2020, 20:11 German Schiavon wrote: > ds.select("Col1", "Col2", "Col3") > > On Thu, 12 Nov 2020 at 15:28, Vikas Garg wrote: > >> In Spark Datase,

Re: Spark Dataset withColumn issue

2020-11-12 Thread Vikas Garg
select(“Col1”, “Col2”, “Col3") > > > Thanks, > Subash > > On Thu, Nov 12, 2020 at 9:19 PM Vikas Garg wrote: > >> I am deriving the col2 using with colunn which is why I cant use it like >> you told me >> >> On Thu, Nov 12, 2020, 20:11 German Schiav

Re: Getting error message

2020-12-17 Thread Vikas Garg
n any program and on restarting the system, program starts running fine. This error goes away on On Thu, 17 Dec 2020, 23:50 Patrick McCarthy, wrote: > my-domain.com/192.168.166.8:63534 probably isn't a valid address on your > network, is it? > > On Thu, Dec 17, 2020 at 3:0

Re: Getting error message

2020-12-17 Thread Vikas Garg
reempted. > > It seems that a node dies or goes off the network, so perhaps you can > also debug the logs on the failing node to see why it disappears and > prevent the failures in the first place. > > On Thu, Dec 17, 2020 at 1:27 PM Vikas Garg wrote: > >> Mydomain is

Re: Getting error message

2020-12-17 Thread Vikas Garg
nux-s-oom-process-killer > > On Thu, Dec 17, 2020 at 1:45 PM Vikas Garg wrote: > >> I am running code in a local machine that is single node machine. >> >> Getting into logs, it looked like the host is killed. This is happening >> very frequently an I am unable to find

Convert Seq[Any] to Seq[String]

2020-12-18 Thread Vikas Garg
Hi, Can someone please help me how to convert Seq[Any] to Seq[String] For line val df = row.toSeq.toDF(newCol.toSeq: _*) I get that error message. I converted Map "val aMap = Map("admit" -> ("description","comments"))" to Seq var atrb = ListBuffer[(String,String,String)]() for((key,value) <- a

Re: Convert Seq[Any] to Seq[String]

2020-12-18 Thread Vikas Garg
ol > names to be > > On Fri, Dec 18, 2020 at 8:37 AM Vikas Garg wrote: > >> Hi, >> >> Can someone please help me how to convert Seq[Any] to Seq[String] >> >> For line >> val df = row.toSeq.toDF(newCol.toSeq: _*) >> I get that error message. >

Re: Issue with executer

2021-01-20 Thread Vikas Garg
l info while sharing logs > > Thanks > Sachit > > On Wed, 20 Jan 2021, 17:35 Vikas Garg, wrote: > >> Hi, >> >> I am facing issue with spark executor. I am struggling with this issue >> since last many days and unable to resolve the issue. >> &

Re: Issue with executer

2021-01-20 Thread Vikas Garg
wn risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, da

Learning Spark

2019-07-04 Thread Vikas Garg
Hi, I am new Spark learner. Can someone guide me with the strategy towards getting expertise in PySpark. Thanks!!!

Re: Learning Spark

2019-07-04 Thread Vikas Garg
of Python also. On Fri, 5 Jul 2019 at 10:32, Kurt Fehlhauer wrote: > Are you a data scientist or data engineer? > > > On Thu, Jul 4, 2019 at 10:34 PM Vikas Garg wrote: > >> Hi, >> >> I am new Spark learner. Can someone guide me with the strategy toward

Re: Learning Spark

2019-07-05 Thread Vikas Garg
ou will find that the error messages > tend to be much more meaningful in Scala because that is the native > language of Spark. If you don’t want to to install the JVM and Scala, I > highly recommend Databricks community edition as a place to start. > > On Thu, Jul 4, 2019 at 11:22 P

Re: Learning Spark

2019-07-05 Thread Vikas Garg
lable in Python. > > On Fri., 5 Jul. 2019, 7:40 pm Vikas Garg, wrote: > >> Is there any disadvantage of using Python? I have gone through multiple >> articles which says that Python has advantages over Scala. >> >> Scala is super fast in comparison but Python has more