have a few questions to you guys:
1) is this a known effect for the 0.20 version? If so what is the jira
issue?
2) is this specific to gzip?
3) is this effect still present in the MRv2/yarn version of Hadoop?
Thanks.
--
Met vriendelijke groet,
Niels Basjes
(Verstuurd vanaf mobiel )
Yes, this is what i was looking for.
Thanks
--
Met vriendelijke groet,
Niels Basjes
(Verstuurd vanaf mobiel )
Op 27 dec. 2011 12:08 schreef "Koji Noguchi" het
volgende:
> Assuming you're using TextInputFormat, it sounds like
> https://issues.apache.org/jira/browse/MA
I would not expect this. I would expect behaviour that is independent of
the way the splits are created.
--
Met vriendelijke groet,
Niels Basjes
(Verstuurd vanaf mobiel )
Op 26 dec. 2011 07:57 schreef "Anthony Urso" het
volgende:
> Gzip files (unlike uncompressed files) are n
decompression code at all. It
only does the splitting smartness. (It could even be refactored to make any
codec splittable). It has been tested with both the java and the native
decompressors.
What do you think?
Is this a feature that should go in the official release or not?
--
Best regards
Niels Basjes
tely) currently no way to create a codec that can be
splittable/non-splittable by using a config setting.
Does this make things any less potentially confusing?
>
I don't think this would make it less confusing.
Niels
>- Tim.
>
>
>
What does your input look like?
--
Met vriendelijke groet,
Niels Basjes
(Verstuurd vanaf mobiel )
Op 17 apr. 2012 02:57 schreef "Ranjan Banerjee" het
volgende:
> Hello,
> I am running the default map reduce job (word count) in Hadoop. The job
> runs fine and I am abl
nstead of "part of the landslide" which people tend
to ignore.
Now for submitting changes for Hadoop: Is it desirable that I fix these in
my change set or should I leave these as-is to avoid "obfuscating" the
changes that are relevant to the Jira at hand?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
limited set of subprojects (i.e. only the mapreduce ones) ?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
ect.
Should I correct this in the junit test files I'm touching?
--
Best regards,
Niels Basjes
://svn.apache.org/viewvc/hadoop/common/trunk/src/test/core/org/apache/hadoop/io/compress/TestCodec.java?view=markup
Which contains (among others) this method: testSplitableCodecs
As I'm actually touching these methods I'll include the changes in the
patch I'm currently working on.
--
Met vriendelijke groeten,
Niels Basjes
w/where to get this problem fixed?
--
Met vriendelijke groeten,
Niels Basjes
I found where to report this ... so I did:
https://issues.apache.org/jira/browse/INFRA-3340
2011/1/5 Niels Basjes :
> Hi,
>
> I just submitted a patch for the feature I've been working on.
> https://issues.apache.org/jira/browse/HADOOP-7076
>
> This patch works fine on m
vriendelijke groeten,
Niels Basjes
Thanks guys,
I really appreciate your help.
Niels Basjes
2011/1/6 Doug Cutting :
> The problem was that the submitter could transition from "Patch Available"
> to "In Progress", following the "Resume Progress" transition, but the
> submitter cannot th
/ ...
https://issues.apache.org/jira/browse/HADOOP-7076
--
Met vriendelijke groeten,
Niels Basjes
It's in the patch
> available state, which is the flag for reviewers to know there's work
> for them to do. Since this is a volunteer effort, I'm afraid there's
> no firm timeline for when this will get done.
> -Jakob
>
>
> On Fri, Jan 7, 2011 at 6:50 AM
* PoweredBy, a list of sites and applications powered by Apache Hadoop
> * Support
> * [[Help|Getting help from the hadoop community]].
> - * [[Support|People and companies for hire]].
> + * [[Support|People and companies for hire]].
> * [[Conferences|Hadoop Community Events and Conferences]]
> * HadoopUserGroups (HUGs)
> * HadoopSummit
>
--
Met vriendelijke groeten,
Niels Basjes
ot;if" block is the dead code as the string will always contain
at least the text ":" or "null:null"
It will probably fail a bit later when really trying to login with a
wrong uid/password.
So, is this worth reporting as a bug?
--
Met vriendelijke groeten,
Niels Basjes
Done,
https://issues.apache.org/jira/browse/HADOOP-7127
2011/1/29 Konstantin Boudnik :
> Yes, please do. Thanks for looking into this.
> --
> Take care,
> Konstantin (Cos) Boudnik
>
>
>
> On Sat, Jan 29, 2011 at 12:44, Niels Basjes wrote:
>> I was playing around w
Tag @link: reference not found:
http://stackoverflow.com/questions/3960651
Now these warnings are clearly caused by an error I made (I've just fixed it).
Does this classify as a bug in the Hadoop QA?
If so, where do I report it?
--
Best regards,
Niels Basjes
t match the expected package "testjar"
>> ClassWithNoPackage.java Hadoop-MR/src/test/mapred/testjar
>>
>> Any thoughts?
>>
>> Thanks,
>> Bikash
>>
>
>
>
> --
> Harsh J
> http://harshj.com
>
--
Met vriendelijke groeten,
Niels Basjes
this the correct solution? Is it worth
submitting as a patch?
diff --git build.xml build.xml
index 26ccfa0..168b34f 100644
--- build.xml
+++ build.xml
@@ -1571,6 +1571,7 @@
+
--
Met vriendelijke groeten,
Niels Basjes
7305
(Hadoop QA should start in a few minutes to validate this).
--
Met vriendelijke groeten,
Niels Basjes
ijke groeten,
Niels Basjes
> www.tejasoft.com
>
> --
> View this message in context:
> http://old.nabble.com/Hadoop-Annotations-support-tp31977831p31977831.html
> Sent from the Hadoop core-dev mailing list archive at Nabble.com.
>
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
o I'm wondering why this is happening. Is he on a long vacation? Or is
there something else?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Have you tried it to see what diffrence it makes?
--
Met vriendelijke groet,
Niels Basjes
(Verstuurd vanaf mobiel )
Op 3 okt. 2011 07:06 schreef "Jinsong Hu" het
volgende:
> Hi, There:
> I just thought an idea. When we format the disk , the block size is
> usually 1K to 4K.
"old style" Hadoop and the trunk is the basis for the "yarn" Hadoop.
For which branch of the source tree should I make my changes so you guys
will reevaluate it for inclusion?
Thanks.
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Thanks,
I'll get busy creating a new patch over the next few days.
Niels Basjes
On Wed, Nov 30, 2011 at 18:51, Eli Collins wrote:
> Hey Niels,
>
> Thanks for contributing. The nest place to contribute new features is
> to trunk. It's currently an easy merge from tru
.
>>
>
Do I simply make separate Jira (related) issues for these backports?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
e --.patch so
that the committer can clearly see what it was intended for.
Do I have to do something to ensure Jenkins will accept this all correctly?
Perhaps in naming convention? Or in the timing between uploading the
various patches?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
in the
branch-0.22. And I couldn't get the test-patch.sh that does exist to work.
So I did against a clean checkout:
patch -p0 < HADOOP-7076-branch-0.22.patch
followed by
ant clean test jar
Which succeeded.
--
Best regards / Met vriendelijke groeten,
Niels Basjes
do something wrong in the patch I created for branch-0.22?
Or is HADOOP-7435 not yet operational?
Thanks.
Niels Basjes
On Tue, Dec 6, 2011 at 00:17, Niels Basjes wrote:
> Hi,
>
> On Mon, Dec 5, 2011 at 18:54, Eli Collins wrote:
>
>> >> https://issues.apache.org
27;forced' and
'helped' getting this right.
The reason for me to propose this as an incompatible change is that this
way I hope to eradicate some of the existing bugs in custom implementations
'out there'.
P.S. If you agree to this change then I'm willing to put my back into it
and submit a patch.
--
Best regards,
Niels Basjes
wrote:
> On 28 May 2014 20:50, Niels Basjes wrote:
>
> > Hi,
> >
> > Last week I ran into this problem again
> > https://issues.apache.org/jira/browse/MAPREDUCE-2094
> >
> > What happens here is that the default implementation of the isSplitable
> >
e the new functionality?
> >
> > *Apologies if I'm stepping into a Hadoop holy war here
> >
> >
> >> On Thu, May 29, 2014 at 10:47 AM, Niels Basjes wrote:
> >> My original proposal (from about 3 years ago) was to change the
> isSplitable
> >&g
I forgot to ask a relevant question: What made the original proposed
solution "incompatible"?
To me it still seems to be a clean backward compatible solution that fixes
this issue in a simple way.
Perhaps Todd can explain why?
Niels
On May 29, 2014 2:17 PM, "Niels Basjes"
PM, "Doug Cutting" wrote:
> On Thu, May 29, 2014 at 2:47 AM, Niels Basjes wrote:
> > For arguments I still do not fully understand this was rejected by Todd
> and
> > Doug.
>
> Performance is a part of compatibility.
>
> Doug
>
ile. Their applications might suddenly run much slower
> and it would be difficult for them to determine why. Such folks are
> probably few, but perhaps exist. I'd prefer a change that avoided
> that possibility entirely.
>
> Doug
>
> On Fri, May 30, 2014 at 3:02 PM,
uld not be splittable at all.
Niels
On May 31, 2014 11:12 PM, "Chris Douglas" wrote:
> On Fri, May 30, 2014 at 11:05 PM, Niels Basjes wrote:
> > How would someone create the situation you are referring to?
>
> By adopting a naming convention where the filename suffix do
On Mon, Jun 2, 2014 at 1:21 AM, Chris Douglas wrote:
> On Sat, May 31, 2014 at 10:53 PM, Niels Basjes wrote:
> > The Hadoop framework uses the filename extension to automatically insert
> > the "right" decompression codec in the read pipeline.
>
> This would be
On Tue, Jun 10, 2014 at 8:10 PM, Chris Douglas wrote:
> On Fri, Jun 6, 2014 at 4:03 PM, Niels Basjes wrote:
> > and if you then give the file the .gz extension this breaks all common
> > sense / conventions about file names.
>
> That the suffix for all compression codecs
Hi,
On Wed, Jun 11, 2014 at 8:25 PM, Chris Douglas wrote:
> On Wed, Jun 11, 2014 at 1:35 AM, Niels Basjes wrote:
> > That's not what I meant. What I understood from what was described is
> that
> > sometimes people use an existing file extension (like .gz) for a f
still propose making the isSplittable abstract to fix
the problem before it is created (in 3.x)
Niels Basjes
On Jun 13, 2014 11:47 PM, "Chris Douglas" wrote:
> On Fri, Jun 13, 2014 at 2:54 AM, Niels Basjes wrote:
> > Hmmm, people only look at logs when they have a problem. So
t) in this patch, perhaps I
did something wrong? Or is this problem caused by the error I highlighted?
What can I do to fix this?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
/4768/console
So I say Jenkins problem for now.
On Sun, Jul 27, 2014 at 9:01 PM, Niels Basjes wrote:
> Hi,
>
> I just submitted a patch and Jenkins said it failed to apply the patch.
> But when I look at the console output
>
> https://builds.apache.org/job/PreCommit-MA
t;
> On Sun, Jul 27, 2014 at 11:01 AM, Niels Basjes wrote:
>
> > Hi,
> >
> > I just submitted a patch and Jenkins said it failed to apply the patch.
> > But when I look at the console output
> >
> > https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4771/
e to handle it but
> hadoop is currently using svn.
>
> Cheers
>
>
> On Sun, Jul 27, 2014 at 11:01 AM, Niels Basjes wrote:
>
> > Hi,
> >
> > I just submitted a patch and Jenkins said it failed to apply the patch.
> > But when I look at the console out
I created a patch file with SVN and it works now.
I dare to ask: Are there any git created patch files that work?
On Sun, Jul 27, 2014 at 9:44 PM, Niels Basjes wrote:
> I'll look for a workaround regarding the binary file. Thanks.
>
>
> On Sun, Jul 27, 2014 at 9:07
>
> On Sun, Jul 27, 2014 at 12:27 PM, Niels Basjes wrote:
>
> > I created a patch file with SVN and it works now.
> > I dare to ask: Are there any git created patch files that work?
> >
> >
> > On Sun, Jul 27, 2014 at 9:44 PM, Niels Basjes w
ith "git diff
> --binary". I ended up making the same patch without the binary part and it
> could be applied okay.
>
> This does differ in behavior from the old boxes, which were still able to
> apply the non-binary parts of a binary-diff.
>
>
> On Mon, Jul 28,
Gian Merlino
https://issues.apache.org/jira/browse/MAPREDUCE-2094
Niels Basjes
P.S. I still thing that the FileInputFormat.isSplitable() should implement
a safe default instead of an optimistic default.
On Sat, Jun 14, 2014 at 10:33 AM, Niels Basjes wrote:
> I did some digging through the code
y directing the BIGTOP_DEFAULTS_DIR into a
blank directory.
But in /usr/bin/pig sourcing /etc/default/hadoop hardcoded into the script.
Why is this done this way?
P.S. Where is the git(?) repo located where this (apperently HW specific)
scripting is maintained?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
ols/hadoop-distcp/src/main/resources/distcp-default.xml:
mapreduce.reduce.class
Seems to me fixing these removes a lot of senseless clutter in the
messaging in the console for end users.
Or is there a good reason to keep it like this?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
(this one still
talks about the time when there were 3 projects).
Thanks.
--
Best regards / Met vriendelijke groeten,
Niels Basjes
).
Niels Basjes
On Sat, Dec 20, 2014 at 6:09 PM, Raghavendra Vaidya <
raghavendra.vai...@gmail.com> wrote:
> I have been struggling to setup hadoop code with native libraries on mac
> osx this gave me an idea to write a utility which can help setup
> hadoop development enviro
aving such
a "core concept breaking" piece in the main codebase. New people may start
to think it is a good idea to do this.
So I say you should simply make this plugin available to your customers,
just not as a core part of Hadoop.
Niels Basjes
[1] 50 GB/sec = approx 20*12*200MB/sec
ranch for testing.
> >>>
> >>> For example, naming a patch foo-branch-2.01.patch should get
> tested on branch-2. Naming a patch foo-HDFS-7285.00.patch should get
> tested on the HDFS-7285 branch.
> >>>
> >>> This hopefully means that there should really be no more ‘blind’
> +1’s to patches that go to branches. The “we only test against trunk”
> argument is no longer valid. :)
> >>>
> >>>
> >>
> >
>
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
GfbNjCOpfNo==jktq8jiu6efm_...@mail.gmail.com%3E
>
>
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
made on April 20, 1999
So the 80 chars thing was at best reconsidered 16 years ago.
Things have changed ...
Niels Basjes
On Tue, May 5, 2015 at 8:21 PM, Jonathan Eagles wrote:
> More formally, we follow the sun java coding standards which follow
> the 80 character limit. There is
reason to force everyone to
> write code as if our screen sizes can’t support more than 80 characters.
>
> On May 5, 2015, at 12:21 PM, Niels Basjes ni...@basjes.nl>> wrote:
>
> Perhaps we should tell him these screens can also be turned landscape ? ;)
>
> But seriously
Splittable Gzip
---
Key: HADOOP-7076
URL: https://issues.apache.org/jira/browse/HADOOP-7076
Project: Hadoop Common
Issue Type: New Feature
Components: io
Reporter: Niels Basjes
Files compressed with the
Type: Bug
Components: fs
Reporter: Niels Basjes
I was playing around with PMD, just to see what kind of messages it gives on
hadoop.
I noticed a message about "Dead code" in org.apache.hadoop.fs.ftp.FTPFileSystem
Starting at about line 80:
String userA
Reporter: Niels Basjes
Assignee: Niels Basjes
Priority: Minor
After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
I open eclipse, set ANT_HOME and build the project.
At that point the following error appears:
{quote}
The type com.sun.javadoc.RootDoc
[
https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niels Basjes reopened HADOOP-7305:
--
Apparently there are some issues with the first version in combination with OS
X.
> Ecli
Niels Basjes created HADOOP-11843:
-
Summary: Make setting up the build environment easier
Key: HADOOP-11843
URL: https://issues.apache.org/jira/browse/HADOOP-11843
Project: Hadoop Common
66 matches
Mail list logo
