Can you do remote debugging in Spark? Didn't know that. Do you have a link?
Also noticed isSplittable in
org.apache.hadoop.mapreduce.lib.input.TextInputFormat which checks for
org.apache.hadoop.io.compress.SplittableCompressionCodec. Maybe there
are some way to tell it not to split?
On Tue, Jun 14, 2016 at 2:42 PM, Sean Owen <so...@cloudera.com> wrote:
> It really sounds like the line is being split across partitions. This
> is what TextInputFormat does but should be perfectly capable of
> putting together lines that break across files (partitions). If you're
> into debugging, that's where I would start if you can. Breakpoints
> around how TextInputFormat is parsing lines. See if you can catch it
> when it returns a line that doesn't contain what you expect.
>
> On Tue, Jun 14, 2016 at 1:38 PM, Kristoffer Sjögren <sto...@gmail.com> wrote:
>> That's funny. The line after is the rest of the whole line that got
>> split in half. Every following lines after that are fine.
>>
>> I managed to reproduce without gzip also so maybe it's no gzip's fault
>> after all..
>>
>> I'm clueless...
>>
>> On Tue, Jun 14, 2016 at 12:53 PM, Kristoffer Sjögren <sto...@gmail.com>
>> wrote:
>>> Seems like it's the gzip. It works if download the file, gunzip and
>>> put it back to another directory and read it the same way.
>>>
>>> Hm.. I wonder what happens with the lines after it..
>>>
>>>
>>>
>>> On Tue, Jun 14, 2016 at 11:52 AM, Sean Owen <so...@cloudera.com> wrote:
>>>> What if you read it uncompressed from HDFS?
>>>> gzip compression is unfriendly to MR in that it can't split the file.
>>>> It still should just work, certainly if the line is in one file. But,
>>>> a data point worth having.
>>>>
>>>> On Tue, Jun 14, 2016 at 10:49 AM, Kristoffer Sjögren <sto...@gmail.com>
>>>> wrote:
>>>>> The line is in one file. I did download the file manually from HDFS,
>>>>> read and decoded it line-by-line successfully without Spark.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jun 14, 2016 at 11:44 AM, Sean Owen <so...@cloudera.com> wrote:
>>>>>> The only thing I can think of is that a line is being broken across two
>>>>>> files?
>>>>>> Hadoop easily puts things back together in this case, or should. There
>>>>>> could be some weird factor preventing that. One first place to look:
>>>>>> are you using a weird line separator? or at least different from the
>>>>>> host OS?
>>>>>>
>>>>>> On Tue, Jun 14, 2016 at 10:41 AM, Kristoffer Sjögren <sto...@gmail.com>
>>>>>> wrote:
>>>>>>> I should mention that we're in the end want to store the input from
>>>>>>> Protobuf binary to Parquet using the following code. But this comes
>>>>>>> after the lines has been decoded from base64 into binary.
>>>>>>>
>>>>>>>
>>>>>>> public static <T extends Message> void save(JavaRDD<T> rdd, Class<T>
>>>>>>> clazz, String path) {
>>>>>>> try {
>>>>>>> Job job = Job.getInstance();
>>>>>>> ParquetOutputFormat.setWriteSupportClass(job,
>>>>>>> ProtoWriteSupport.class);
>>>>>>> ProtoParquetOutputFormat.setProtobufClass(job, clazz);
>>>>>>> rdd.mapToPair(order -> new Tuple2<>(null, order))
>>>>>>> .saveAsNewAPIHadoopFile(path, Void.class, clazz,
>>>>>>> ParquetOutputFormat.class, job.getConfiguration());
>>>>>>> } catch (IOException e) {
>>>>>>> throw new RuntimeException(e);
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> <dependency>
>>>>>>> <groupId>org.apache.parquet</groupId>
>>>>>>> <artifactId>parquet-protobuf</artifactId>
>>>>>>> <version>1.8.1</version>
>>>>>>> </dependency>
>>>>>>>
>>>>>>> On Tue, Jun 14, 2016 at 11:37 AM, Kristoffer Sjögren <sto...@gmail.com>
>>>>>>> wrote:
>>>>>>>> I'm trying to figure out exactly what information could be useful but
>>>>>>>> it's all as straight forward.
>>>>>>>>
>>>>>>>> - It's text files
>>>>>>>> - Lines ends with a new line character.
>>>>>>>> - Files are gzipped before added to HDFS
>>>>>>>> - Files are read as gzipped files from HDFS by Spark
>>>>>>>> - There are some extra configuration
>>>>>>>>
>>>>>>>> conf.set("spark.files.overwrite", "true");
>>>>>>>> conf.set("spark.hadoop.validateOutputSpecs", "false");
>>>>>>>>
>>>>>>>> Here's the code using Java 8 Base64 class.
>>>>>>>>
>>>>>>>> context.textFile("/log.gz")
>>>>>>>> .map(line -> line.split("×tamp="))
>>>>>>>> .map(split -> Base64.getDecoder().decode(split[0]));
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jun 14, 2016 at 11:26 AM, Sean Owen <so...@cloudera.com> wrote:
>>>>>>>>> It's really the MR InputSplit code that splits files into records.
>>>>>>>>> Nothing particularly interesting happens in that process, except for
>>>>>>>>> breaking on newlines.
>>>>>>>>>
>>>>>>>>> Do you have one huge line in the file? are you reading as a text file?
>>>>>>>>> can you give any more detail about exactly how you parse it? it could
>>>>>>>>> be something else in your code.
>>>>>>>>>
>>>>>>>>> On Tue, Jun 14, 2016 at 10:24 AM, Kristoffer Sjögren
>>>>>>>>> <sto...@gmail.com> wrote:
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> We have log files that are written in base64 encoded text files
>>>>>>>>>> (gzipped) where each line is ended with a new line character.
>>>>>>>>>>
>>>>>>>>>> For some reason a particular line [1] is split by Spark [2] making it
>>>>>>>>>> unparsable by the base64 decoder. It does this consequently no matter
>>>>>>>>>> if I gives it the particular file that contain the line or a bunch of
>>>>>>>>>> files.
>>>>>>>>>>
>>>>>>>>>> I know the line is not corrupt because I can manually download the
>>>>>>>>>> file from HDFS, gunzip it and read/decode all the lines without
>>>>>>>>>> problems.
>>>>>>>>>>
>>>>>>>>>> Was thinking that maybe there is a limit to number of characters per
>>>>>>>>>> line but that doesn't sound right? Maybe the combination of
>>>>>>>>>> characters
>>>>>>>>>> makes Spark think it's new line?
>>>>>>>>>>
>>>>>>>>>> I'm clueless.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> -Kristoffer
>>>>>>>>>>
>>>>>>>>>> [1] Original line:
>>>>>>>>>>
>>>>>>>>>> CsAJCtwGCghwYWdlVmlldxC4PhjM1v66BSJFaHR0cDovL25hLnNlL3Nwb3J0ZW4vc3BvcnR0dC8xLjM5MTU5MjEtdXBwZ2lmdGVyLXNtZWRlcm5hLW1vdC1rb25rdXJzKjhVcHBnaWZ0ZXI6IFNtZWRlcm5hIG1vdCBrb25rdXJzIC0gU3BvcnQgKFRUKSAtIHd3dy5uYS5zZTJXaHR0cDovL25hLnNlL255aGV0ZXIvb3JlYnJvLzEuMzk2OTU0My1rcnlwaGFsLW9wcG5hci1mb3Itb2JvLWF0dC1iZWhhbGxhLXRqYW5zdGViaWxhcm5hOqECcWdrZWplNGpmLHBkMzBmdDRuNCxwbHB0b3JqNncscGxwczBvamZvLHBkYjVrZGM4eCxwbHBzN293Y3UscGE0czN1bXp5LHBhNHJla25sNyxwYTRyd3dxam4scGE0c21ra2Z4LHBkM2tpa3BmMixwZDNqcjE5dGMscGQ0ZGQ0M2F3LHAwZ3MwbmlqMSxwYTRvZTNrbXoscGE0cWJ3eDZxLHBkM2s2NW00dyxwYTRyazc3Z3IscGQzMHAzdW8wLHBkNGM1ajV5dixwbHB0c211NmcscGM3bXNibmM5LHBhNHFpaTdsZCxwbHB0dnpqdnUscGE0bmlsdmFnLHBhNHB6cjN2cyxwZDNsZDNnYmkscGl1a2M2NmVlLHB5dHoyOThzNErIAgoTNzI0NTY2NzU0MzQxNTUyOTQ4ORAAGAAioQJxZ2tlamU0amYscGQzMGZ0NG40LHBscHRvcmo2dyxwbHBzMG9qZm8scGRiNWtkYzh4LHBscHM3b3djdSxwYTRzM3VtenkscGE0cmVrbmw3LHBhNHJ3d3FqbixwYTRzbWtrZngscGQza2lrcGYyLHBkM2pyMTl0YyxwZDRkZDQzYXcscDBnczBuaWoxLHBhNG9lM2tteixwYTRxYnd4NnEscGQzazY1bTR3LHBhNHJrNzdncixwZDMwcDN1bzAscGQ0YzVqNXl2LHBscHRzbXU2ZyxwYzdtc2JuYzkscGE0cWlpN2xkLHBscHR2emp2dSxwYTRuaWx2YWcscGE0cHpyM3ZzLHBkM2xkM2diaSxwaXVrYzY2ZWUscHl0ejI5OHM0KgkzOTUxLDM5NjAS3gIIxNjxhJTVsJcVEqUBTW96aWxsYS81LjAgKExpbnV4OyBBbmRyb2lkIDUuMS4xOyBTQU1TVU5HIFNNLUczODhGIEJ1aWxkL0xNWTQ4QikgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgU2Ftc3VuZ0Jyb3dzZXIvMy4zIENocm9tZS8zOC4wLjIxMjUuMTAyIE1vYmlsZSBTYWZhcmkvNTM3LjM2IjUKDDYyLjIwLjE5Ni44MBWgd3NBHRgibUIiAlNFKgfDlnJlYnJvMg5UZWxpYVNvbmVyYSBBQigAMdejcD0K1+s/OABCCAiAAhWamRlAQgcIURUAAOBAQggIlAEVzczMP0IHCFQVmpkJQUIICJYBFTMzE0BCBwhYFZqZ+UBCCAj6ARWamdk/QggImwEVzcysQEoHCAYVO6ysPkoHCAQVRYO4PkoHCAEVIg0APw==×tamp=1465887564
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [2] Line as spark hands it over:
>>>>>>>>>>
>>>>>>>>>> CsAJCtwGCghwYWdlVmlldxC4PhjM1v66BSJFaHR0cDovL25hLnNlL3Nwb3J0ZW4vc3BvcnR0dC8xLjM5MTU5MjEtdXBwZ2lmdGVyLXNtZWRlcm5hLW1vdC1rb25rdXJzKjhVcHBnaWZ0ZXI6IFNtZWRlcm5hIG1vdCBrb25rdXJzIC0gU3BvcnQgKFRUKSAtIHd3dy5uYS5zZTJXaHR0cDovL25hLnNlL255aGV0ZXIvb3JlYnJvLzEuMzk2OTU0My1rcnlwaGFsLW9wcG5hci1mb3Itb2JvLWF0dC1iZWhhbGxhLXRqYW5zdGViaWxhcm5hOqECcWdrZWplNGpmLHBkMzBmdDRuNCxwbHB0b3JqNncscGxwczBvamZvLHBkYjVrZGM4eCxwbHBzN293Y3UscGE0czN1bXp5LHBhNHJla25sNyxwYTRyd3dxam4scGE0c21ra2Z4LHBkM2tpa3BmMixwZDNqcjE5dGMscGQ0ZGQ0M2F3LHAwZ3MwbmlqMSxwYTRvZTNrbXoscGE0cWJ3eDZxLHBkM2s2NW00dyxwYTRyazc3Z3IscGQzMHAzdW8wLHBkNGM1ajV5dixwbHB0c211NmcscGM3bXNibmM5LHBhNHFpaTdsZCxwbHB0dnpqdnUscGE0bmlsdmFnLHBhNHB6cjN2cyxwZDNsZDNnYmkscGl1a2M2NmVlLHB5dHoyOThzNErIAgoTNzI0NTY2NzU0MzQxNTUyOTQ4ORAAGAAioQJxZ2tlamU0amYscGQzMGZ0NG40LHBscHRvcmo2dyxwbHBzMG9qZm8scGRiNWtkYzh4LHBscHM3b3djdSxwYTRzM3VtenkscGE0cmVrbmw3LHBhNHJ3d3FqbixwYTRzbWtrZngscGQza2lrcGYyLHBkM2pyMTl0YyxwZDRkZDQzYXcscDBnczBuaWoxLHBhNG9lM2tteixwYTRxYnd4NnEscGQzazY1bTR3LHBhNHJrNzdncixwZDMwcDN1bzAscGQ0YzVqNXl2LHBscHRzbXU2ZyxwYzdtc2JuYzkscGE0cWlpN2xkLHBscHR2emp2dSxwYTRuaWx2YWcscGE0
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>>>>>>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>>>>>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
