hey guys
One of my input records has an problem that makes the code fail.
var demoRddFilter = demoRdd.filter(line =>
!line.contains("ISR$CASE$I_F_COD$FOLL_SEQ") ||
!line.contains("primaryid$caseid$caseversion"))
var demoRddFilterMap = demoRddFilter.map(line => line.split('$')(0) + "~" +
line.split('$')(5) + "~" + line.split('$')(11) + "~" +
line.split('$')(12))demoRddFilterMap.saveAsTextFile("/data/aers/msfx/demo/" +
outFile)
This is possibly happening because perhaps one input record may not have 13
fields.If this were Hadoop mapper code , I have 2 ways to solve this 1. test
the number of fields of each line before applying the map function2. enclose
the mapping function in a try catch block so that the mapping function only
fails for the erroneous recordHow do I implement 1. or 2. in the Spark code
?Thanks
sanjay <!--#yiv7202296517 _filtered #yiv7202296517
{font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv7202296517
{font-family:"Cambria Math";panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered
#yiv7202296517 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2
4;}#yiv7202296517 #yiv7202296517 p.yiv7202296517MsoNormal, #yiv7202296517
li.yiv7202296517MsoNormal, #yiv7202296517 div.yiv7202296517MsoNormal
{margin:0cm;margin-bottom:.0001pt;font-size:11.0pt;font-family:"Calibri",
sans-serif;}#yiv7202296517 a:link, #yiv7202296517
span.yiv7202296517MsoHyperlink
{color:#0563C1;text-decoration:underline;}#yiv7202296517 a:visited,
#yiv7202296517 span.yiv7202296517MsoHyperlinkFollowed
{color:#954F72;text-decoration:underline;}#yiv7202296517
p.yiv7202296517MsoListParagraph, #yiv7202296517
li.yiv7202296517MsoListParagraph, #yiv7202296517
div.yiv7202296517MsoListParagraph
{margin-top:0cm;margin-right:0cm;margin-bottom:0cm;margin-left:36.0pt;margin-bottom:.0001pt;font-size:11.0pt;font-family:"Calibri",
sans-serif;}#yiv7202296517 span.yiv7202296517EstiloCorreo17
{font-family:"Calibri", sans-serif;color:windowtext;}#yiv7202296517
.yiv7202296517MsoChpDefault {font-family:"Calibri", sans-serif;} _filtered
#yiv7202296517 {margin:70.85pt 3.0cm 70.85pt 3.0cm;}#yiv7202296517
div.yiv7202296517WordSection1 {}#yiv7202296517 _filtered #yiv7202296517 {}
_filtered #yiv7202296517 {} _filtered #yiv7202296517 {} _filtered
#yiv7202296517 {} _filtered #yiv7202296517 {} _filtered #yiv7202296517 {}
_filtered #yiv7202296517 {} _filtered #yiv7202296517 {} _filtered
#yiv7202296517 {} _filtered #yiv7202296517 {}#yiv7202296517 ol
{margin-bottom:0cm;}#yiv7202296517 ul {margin-bottom:0cm;}-->
