Thanks for you help. Really appreciate it!
Give me some time i'll come back after I've tried your suggestions.
On Tue, Jun 14, 2016 at 3:28 PM, Kristoffer Sjögren <sto...@gmail.com> wrote:
> I cannot reproduce it by running the file through Spark in local mode
> on my machine. So it does indeed seems to be something related to
> split across partitions.
>
> On Tue, Jun 14, 2016 at 3:04 PM, Kristoffer Sjögren <sto...@gmail.com> wrote:
>> Can you do remote debugging in Spark? Didn't know that. Do you have a link?
>>
>> Also noticed isSplittable in
>> org.apache.hadoop.mapreduce.lib.input.TextInputFormat which checks for
>> org.apache.hadoop.io.compress.SplittableCompressionCodec. Maybe there
>> are some way to tell it not to split?
>>
>> On Tue, Jun 14, 2016 at 2:42 PM, Sean Owen <so...@cloudera.com> wrote:
>>> It really sounds like the line is being split across partitions. This
>>> is what TextInputFormat does but should be perfectly capable of
>>> putting together lines that break across files (partitions). If you're
>>> into debugging, that's where I would start if you can. Breakpoints
>>> around how TextInputFormat is parsing lines. See if you can catch it
>>> when it returns a line that doesn't contain what you expect.
>>>
>>> On Tue, Jun 14, 2016 at 1:38 PM, Kristoffer Sjögren <sto...@gmail.com>
>>> wrote:
>>>> That's funny. The line after is the rest of the whole line that got
>>>> split in half. Every following lines after that are fine.
>>>>
>>>> I managed to reproduce without gzip also so maybe it's no gzip's fault
>>>> after all..
>>>>
>>>> I'm clueless...
>>>>
>>>> On Tue, Jun 14, 2016 at 12:53 PM, Kristoffer Sjögren <sto...@gmail.com>
>>>> wrote:
>>>>> Seems like it's the gzip. It works if download the file, gunzip and
>>>>> put it back to another directory and read it the same way.
>>>>>
>>>>> Hm.. I wonder what happens with the lines after it..
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jun 14, 2016 at 11:52 AM, Sean Owen <so...@cloudera.com> wrote:
>>>>>> What if you read it uncompressed from HDFS?
>>>>>> gzip compression is unfriendly to MR in that it can't split the file.
>>>>>> It still should just work, certainly if the line is in one file. But,
>>>>>> a data point worth having.
>>>>>>
>>>>>> On Tue, Jun 14, 2016 at 10:49 AM, Kristoffer Sjögren <sto...@gmail.com>
>>>>>> wrote:
>>>>>>> The line is in one file. I did download the file manually from HDFS,
>>>>>>> read and decoded it line-by-line successfully without Spark.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 14, 2016 at 11:44 AM, Sean Owen <so...@cloudera.com> wrote:
>>>>>>>> The only thing I can think of is that a line is being broken across
>>>>>>>> two files?
>>>>>>>> Hadoop easily puts things back together in this case, or should. There
>>>>>>>> could be some weird factor preventing that. One first place to look:
>>>>>>>> are you using a weird line separator? or at least different from the
>>>>>>>> host OS?
>>>>>>>>
>>>>>>>> On Tue, Jun 14, 2016 at 10:41 AM, Kristoffer Sjögren
>>>>>>>> <sto...@gmail.com> wrote:
>>>>>>>>> I should mention that we're in the end want to store the input from
>>>>>>>>> Protobuf binary to Parquet using the following code. But this comes
>>>>>>>>> after the lines has been decoded from base64 into binary.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> public static <T extends Message> void save(JavaRDD<T> rdd, Class<T>
>>>>>>>>> clazz, String path) {
>>>>>>>>> try {
>>>>>>>>> Job job = Job.getInstance();
>>>>>>>>> ParquetOutputFormat.setWriteSupportClass(job,
>>>>>>>>> ProtoWriteSupport.class);
>>>>>>>>> ProtoParquetOutputFormat.setProtobufClass(job, clazz);
>>>>>>>>> rdd.mapToPair(order -> new Tuple2<>(null, order))
>>>>>>>>> .saveAsNewAPIHadoopFile(path, Void.class, clazz,
>>>>>>>>> ParquetOutputFormat.class, job.getConfiguration());
>>>>>>>>> } catch (IOException e) {
>>>>>>>>> throw new RuntimeException(e);
>>>>>>>>> }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <dependency>
>>>>>>>>> <groupId>org.apache.parquet</groupId>
>>>>>>>>> <artifactId>parquet-protobuf</artifactId>
>>>>>>>>> <version>1.8.1</version>
>>>>>>>>> </dependency>
>>>>>>>>>
>>>>>>>>> On Tue, Jun 14, 2016 at 11:37 AM, Kristoffer Sjögren
>>>>>>>>> <sto...@gmail.com> wrote:
>>>>>>>>>> I'm trying to figure out exactly what information could be useful but
>>>>>>>>>> it's all as straight forward.
>>>>>>>>>>
>>>>>>>>>> - It's text files
>>>>>>>>>> - Lines ends with a new line character.
>>>>>>>>>> - Files are gzipped before added to HDFS
>>>>>>>>>> - Files are read as gzipped files from HDFS by Spark
>>>>>>>>>> - There are some extra configuration
>>>>>>>>>>
>>>>>>>>>> conf.set("spark.files.overwrite", "true");
>>>>>>>>>> conf.set("spark.hadoop.validateOutputSpecs", "false");
>>>>>>>>>>
>>>>>>>>>> Here's the code using Java 8 Base64 class.
>>>>>>>>>>
>>>>>>>>>> context.textFile("/log.gz")
>>>>>>>>>> .map(line -> line.split("×tamp="))
>>>>>>>>>> .map(split -> Base64.getDecoder().decode(split[0]));
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 14, 2016 at 11:26 AM, Sean Owen <so...@cloudera.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> It's really the MR InputSplit code that splits files into records.
>>>>>>>>>>> Nothing particularly interesting happens in that process, except for
>>>>>>>>>>> breaking on newlines.
>>>>>>>>>>>
>>>>>>>>>>> Do you have one huge line in the file? are you reading as a text
>>>>>>>>>>> file?
>>>>>>>>>>> can you give any more detail about exactly how you parse it? it
>>>>>>>>>>> could
>>>>>>>>>>> be something else in your code.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jun 14, 2016 at 10:24 AM, Kristoffer Sjögren
>>>>>>>>>>> <sto...@gmail.com> wrote:
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>> We have log files that are written in base64 encoded text files
>>>>>>>>>>>> (gzipped) where each line is ended with a new line character.
>>>>>>>>>>>>
>>>>>>>>>>>> For some reason a particular line [1] is split by Spark [2] making
>>>>>>>>>>>> it
>>>>>>>>>>>> unparsable by the base64 decoder. It does this consequently no
>>>>>>>>>>>> matter
>>>>>>>>>>>> if I gives it the particular file that contain the line or a bunch
>>>>>>>>>>>> of
>>>>>>>>>>>> files.
>>>>>>>>>>>>
>>>>>>>>>>>> I know the line is not corrupt because I can manually download the
>>>>>>>>>>>> file from HDFS, gunzip it and read/decode all the lines without
>>>>>>>>>>>> problems.
>>>>>>>>>>>>
>>>>>>>>>>>> Was thinking that maybe there is a limit to number of characters
>>>>>>>>>>>> per
>>>>>>>>>>>> line but that doesn't sound right? Maybe the combination of
>>>>>>>>>>>> characters
>>>>>>>>>>>> makes Spark think it's new line?
>>>>>>>>>>>>
>>>>>>>>>>>> I'm clueless.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> -Kristoffer
>>>>>>>>>>>>
>>>>>>>>>>>> [1] Original line:
>>>>>>>>>>>>
>>>>>>>>>>>> CsAJCtwGCghwYWdlVmlldxC4PhjM1v66BSJFaHR0cDovL25hLnNlL3Nwb3J0ZW4vc3BvcnR0dC8xLjM5MTU5MjEtdXBwZ2lmdGVyLXNtZWRlcm5hLW1vdC1rb25rdXJzKjhVcHBnaWZ0ZXI6IFNtZWRlcm5hIG1vdCBrb25rdXJzIC0gU3BvcnQgKFRUKSAtIHd3dy5uYS5zZTJXaHR0cDovL25hLnNlL255aGV0ZXIvb3JlYnJvLzEuMzk2OTU0My1rcnlwaGFsLW9wcG5hci1mb3Itb2JvLWF0dC1iZWhhbGxhLXRqYW5zdGViaWxhcm5hOqECcWdrZWplNGpmLHBkMzBmdDRuNCxwbHB0b3JqNncscGxwczBvamZvLHBkYjVrZGM4eCxwbHBzN293Y3UscGE0czN1bXp5LHBhNHJla25sNyxwYTRyd3dxam4scGE0c21ra2Z4LHBkM2tpa3BmMixwZDNqcjE5dGMscGQ0ZGQ0M2F3LHAwZ3MwbmlqMSxwYTRvZTNrbXoscGE0cWJ3eDZxLHBkM2s2NW00dyxwYTRyazc3Z3IscGQzMHAzdW8wLHBkNGM1ajV5dixwbHB0c211NmcscGM3bXNibmM5LHBhNHFpaTdsZCxwbHB0dnpqdnUscGE0bmlsdmFnLHBhNHB6cjN2cyxwZDNsZDNnYmkscGl1a2M2NmVlLHB5dHoyOThzNErIAgoTNzI0NTY2NzU0MzQxNTUyOTQ4ORAAGAAioQJxZ2tlamU0amYscGQzMGZ0NG40LHBscHRvcmo2dyxwbHBzMG9qZm8scGRiNWtkYzh4LHBscHM3b3djdSxwYTRzM3VtenkscGE0cmVrbmw3LHBhNHJ3d3FqbixwYTRzbWtrZngscGQza2lrcGYyLHBkM2pyMTl0YyxwZDRkZDQzYXcscDBnczBuaWoxLHBhNG9lM2tteixwYTRxYnd4NnEscGQzazY1bTR3LHBhNHJrNzdncixwZDMwcDN1bzAscGQ0YzVqNXl2LHBscHRzbXU2ZyxwYzdtc2JuYzkscGE0cWlpN2xkLHBscHR2emp2dSxwYTRuaWx2YWcscGE0cHpyM3ZzLHBkM2xkM2diaSxwaXVrYzY2ZWUscHl0ejI5OHM0KgkzOTUxLDM5NjAS3gIIxNjxhJTVsJcVEqUBTW96aWxsYS81LjAgKExpbnV4OyBBbmRyb2lkIDUuMS4xOyBTQU1TVU5HIFNNLUczODhGIEJ1aWxkL0xNWTQ4QikgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgU2Ftc3VuZ0Jyb3dzZXIvMy4zIENocm9tZS8zOC4wLjIxMjUuMTAyIE1vYmlsZSBTYWZhcmkvNTM3LjM2IjUKDDYyLjIwLjE5Ni44MBWgd3NBHRgibUIiAlNFKgfDlnJlYnJvMg5UZWxpYVNvbmVyYSBBQigAMdejcD0K1+s/OABCCAiAAhWamRlAQgcIURUAAOBAQggIlAEVzczMP0IHCFQVmpkJQUIICJYBFTMzE0BCBwhYFZqZ+UBCCAj6ARWamdk/QggImwEVzcysQEoHCAYVO6ysPkoHCAQVRYO4PkoHCAEVIg0APw==×tamp=1465887564
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [2] Line as spark hands it over:
>>>>>>>>>>>>
>>>>>>>>>>>> CsAJCtwGCghwYWdlVmlldxC4PhjM1v66BSJFaHR0cDovL25hLnNlL3Nwb3J0ZW4vc3BvcnR0dC8xLjM5MTU5MjEtdXBwZ2lmdGVyLXNtZWRlcm5hLW1vdC1rb25rdXJzKjhVcHBnaWZ0ZXI6IFNtZWRlcm5hIG1vdCBrb25rdXJzIC0gU3BvcnQgKFRUKSAtIHd3dy5uYS5zZTJXaHR0cDovL25hLnNlL255aGV0ZXIvb3JlYnJvLzEuMzk2OTU0My1rcnlwaGFsLW9wcG5hci1mb3Itb2JvLWF0dC1iZWhhbGxhLXRqYW5zdGViaWxhcm5hOqECcWdrZWplNGpmLHBkMzBmdDRuNCxwbHB0b3JqNncscGxwczBvamZvLHBkYjVrZGM4eCxwbHBzN293Y3UscGE0czN1bXp5LHBhNHJla25sNyxwYTRyd3dxam4scGE0c21ra2Z4LHBkM2tpa3BmMixwZDNqcjE5dGMscGQ0ZGQ0M2F3LHAwZ3MwbmlqMSxwYTRvZTNrbXoscGE0cWJ3eDZxLHBkM2s2NW00dyxwYTRyazc3Z3IscGQzMHAzdW8wLHBkNGM1ajV5dixwbHB0c211NmcscGM3bXNibmM5LHBhNHFpaTdsZCxwbHB0dnpqdnUscGE0bmlsdmFnLHBhNHB6cjN2cyxwZDNsZDNnYmkscGl1a2M2NmVlLHB5dHoyOThzNErIAgoTNzI0NTY2NzU0MzQxNTUyOTQ4ORAAGAAioQJxZ2tlamU0amYscGQzMGZ0NG40LHBscHRvcmo2dyxwbHBzMG9qZm8scGRiNWtkYzh4LHBscHM3b3djdSxwYTRzM3VtenkscGE0cmVrbmw3LHBhNHJ3d3FqbixwYTRzbWtrZngscGQza2lrcGYyLHBkM2pyMTl0YyxwZDRkZDQzYXcscDBnczBuaWoxLHBhNG9lM2tteixwYTRxYnd4NnEscGQzazY1bTR3LHBhNHJrNzdncixwZDMwcDN1bzAscGQ0YzVqNXl2LHBscHRzbXU2ZyxwYzdtc2JuYzkscGE0cWlpN2xkLHBscHR2emp2dSxwYTRuaWx2YWcscGE0
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>>>>>>>>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>>>>>>>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
