We also issue explicit readahead via fadvise since 2011 or so, so the
typical io sizes hitting the device are large enough to max out the
throughput, at least for typical spinning disks.
Todd
On Wed, Mar 11, 2020, 9:46 AM Kihwal Lee
wrote:
> When Datanode was initially designed, Linux AIO was s
[
https://issues.apache.org/jira/browse/HDFS-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-14535.
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
> The default 8KB buf
[
https://issues.apache.org/jira/browse/HDFS-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-14482.
Resolution: Fixed
> Crash when using libhdfs with bad classp
Todd Lipcon created HDFS-14482:
--
Summary: Crash when using libhdfs with bad classpath
Key: HDFS-14482
URL: https://issues.apache.org/jira/browse/HDFS-14482
Project: Hadoop HDFS
Issue Type: Bug
Todd Lipcon created HDFS-14111:
--
Summary: hdfsOpenFile on HDFS causes unnecessary IO from file
offset 0
Key: HDFS-14111
URL: https://issues.apache.org/jira/browse/HDFS-14111
Project: Hadoop HDFS
[
https://issues.apache.org/jira/browse/HDFS-10369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-10369.
Resolution: Invalid
You're mallocing a buffer of 5 bytes here, seems your C code is just b
doop
> > > RPC
> > > > using SSL). But it looks like both attempts have not been
> progressing.
> > > >
> > > > During the recent Hadoop contributor meetup, Daryn Sharp mentioned
> he's
> > > > working on another approach that l
Todd Lipcon created HDFS-13826:
--
Summary: Add a hidden configuration for NameNode to generate fake
block locations
Key: HDFS-13826
URL: https://issues.apache.org/jira/browse/HDFS-13826
Project: Hadoop
Todd Lipcon created HDFS-13747:
--
Summary: Statistic for list_located_status is incremented
incorrectly by listStatusIterator
Key: HDFS-13747
URL: https://issues.apache.org/jira/browse/HDFS-13747
Project
o bring up kerberos to run it like that. And for more
> sophisticated
> > multinode deployments, should the scripts refuse to work without
> kerberos
> > unless you pass in some argument like "--Dinsecure-clusters-
> permitted"
> >
> > Any other ideas?
> >
> >
> >
> -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Todd Lipcon created HDFS-13703:
--
Summary: Avoid allocation of CorruptedBlocks hashmap when no
corrupted blocks are hit
Key: HDFS-13703
URL: https://issues.apache.org/jira/browse/HDFS-13703
Project
Todd Lipcon created HDFS-13702:
--
Summary: HTrace hooks taking 10-15% CPU in DFS client when disabled
Key: HDFS-13702
URL: https://issues.apache.org/jira/browse/HDFS-13702
Project: Hadoop HDFS
Todd Lipcon created HDFS-13701:
--
Summary: Removal of logging guards regressed performance
Key: HDFS-13701
URL: https://issues.apache.org/jira/browse/HDFS-13701
Project: Hadoop HDFS
Issue Type
[
https://issues.apache.org/jira/browse/HDFS-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-3653.
---
Resolution: Won't Fix
> 1.x: Add a retention period for purged e
[
https://issues.apache.org/jira/browse/HDFS-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-3069.
---
Resolution: Won't Fix
Target Version/s: (was: )
> If an edits file has more edi
it/notify/notifyAll) are
> non-fair. (
> http://stackoverflow.com/questions/11275699/synchronized-release-order),
> It will causes many dfsclient timeout.
> >
> > I’m thinking we can use ReentrantReadWriteLock for synchronization. What
> do you guys think?
>
--
Todd Lipcon
Software Engineer, Cloudera
[
https://issues.apache.org/jira/browse/HDFS-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-957.
--
Resolution: Won't Fix
> FSImage layout version should be only once file is
[
https://issues.apache.org/jira/browse/HDFS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-3528.
---
Resolution: Fixed
Fix Version/s: 2.6.0
> Use native CRC32 in DFS write p
gt; > >> > > > > On Fri, Aug 22, 2014 at 3:48 PM, Karthik Kambatla <
> > > >> > >> > ka...@cloudera.com
> > > >> >
> > > >> > >> > > > > wrote:
> > > >> > >> > > > >
> > > >> > >> > > > >> Hi folks,
> > > >> > >> > > > >>
> > > >> > >> > > > >> For the SCM migration, feel free to follow
> > > >> > >> > > > >> https://issues.apache.org/jira/browse/INFRA-8195
> > > >> > >> > > > >>
> > > >> > >> > > > >> Most of this is planned to be handled this Sunday. As
> a
> > > >> result,
> > > >> > >> the
> > > >> > >> > > > >> subversion repository would be read-only. If this is a
> > > major
> > > >> > >> issue
> > > >> > >> > for
> > > >> > >> > > > you,
> > > >> > >> > > > >> please shout out.
> > > >> > >> > > > >>
> > > >> > >> > > > >> Daniel Gruno, the one helping us with the migration,
> was
> > > >> asking
> > > >> > >> if
> > > >> > >> > we
> > > >> > >> > > > are
> > > >> > >> > > > >> open to renaming "trunk" to "master" to better conform
> > to
> > > >> git
> > > >> > >> > lingo. I
> > > >> > >> > > > am
> > > >> > >> > > > >> tempted to say yes, but wanted to check.
> > > >> > >> > > > >>
> > > >> > >> > > > >> Would greatly appreciate any help with checking the
> git
> > > repo
> > > >> > has
> > > >> > >> > > > >> everything.
> > > >> > >> > > > >>
> > > >> > >> > > > >> Thanks
> > > >> > >> > > > >> Karthik
> > > >> > >> > > > >>
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > >
> > > >> > >> > >
> > > >> > >> > > --
> > > >> > >> > > CONFIDENTIALITY NOTICE
> > > >> > >> > > NOTICE: This message is intended for the use of the
> > individual
> > > or
> > > >> > >> entity
> > > >> > >> > to
> > > >> > >> > > which it is addressed and may contain information that is
> > > >> > >> confidential,
> > > >> > >> > > privileged and exempt from disclosure under applicable law.
> > If
> > > >> the
> > > >> > >> reader
> > > >> > >> > > of this message is not the intended recipient, you are
> hereby
> > > >> > notified
> > > >> > >> > that
> > > >> > >> > > any printing, copying, dissemination, distribution,
> > disclosure
> > > or
> > > >> > >> > > forwarding of this communication is strictly prohibited. If
> > you
> > > >> have
> > > >> > >> > > received this communication in error, please contact the
> > sender
> > > >> > >> > immediately
> > > >> > >> > > and delete it from your system. Thank You.
> > > >> > >> > >
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> --
> > > >> > >> Zhijie Shen
> > > >> > >> Hortonworks Inc.
> > > >> > >> http://hortonworks.com/
> > > >> > >>
> > > >> > >> --
> > > >> > >> CONFIDENTIALITY NOTICE
> > > >> > >> NOTICE: This message is intended for the use of the individual
> or
> > > >> entity
> > > >> > >> to
> > > >> > >> which it is addressed and may contain information that is
> > > >> confidential,
> > > >> > >> privileged and exempt from disclosure under applicable law. If
> > the
> > > >> > reader
> > > >> > >> of this message is not the intended recipient, you are hereby
> > > >> notified
> > > >> > >> that
> > > >> > >> any printing, copying, dissemination, distribution, disclosure
> or
> > > >> > >> forwarding of this communication is strictly prohibited. If you
> > > have
> > > >> > >> received this communication in error, please contact the sender
> > > >> > >> immediately
> > > >> > >> and delete it from your system. Thank You.
> > > >> > >>
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> http://hortonworks.com/download/
> > > >>
> > > >> --
> > > >> CONFIDENTIALITY NOTICE
> > > >> NOTICE: This message is intended for the use of the individual or
> > entity
> > > >> to
> > > >> which it is addressed and may contain information that is
> > confidential,
> > > >> privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > >> of this message is not the intended recipient, you are hereby
> notified
> > > >> that
> > > >> any printing, copying, dissemination, distribution, disclosure or
> > > >> forwarding of this communication is strictly prohibited. If you have
> > > >> received this communication in error, please contact the sender
> > > >> immediately
> > > >> and delete it from your system. Thank You.
> > > >>
> > > >
> > > >
> > >
> > > --
> > > Mobile
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
[
https://issues.apache.org/jira/browse/HDFS-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-3278.
---
Resolution: Fixed
Fix Version/s: 2.1.0-beta
Assignee: Todd Lipcon (was: Sanjay Radia
any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Todd Lipcon
Software Engineer, Cloudera
no public APIs. The existing tests include both
> unit tests and end-to-end tests. They have already covered this
> feature.
>
> Once the feature is merged into trunk, we will continue to test and
> fix any bugs that may be found on trunk.
>
> The bulk of the design and imp
Todd Lipcon created HDFS-5790:
-
Summary: LeaseManager.findPath is very slow when many leases need
recovery
Key: HDFS-5790
URL: https://issues.apache.org/jira/browse/HDFS-5790
Project: Hadoop HDFS
in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Todd Lipcon
Software Engineer, Cloudera
On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe wrote:
> To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
> there. However, I have only been following the HDFS and common side
> of things so I may not have the full picture. Arun, can you give a
> specific example of something you
> >> >> --
> >> >> http://hortonworks.com/download/
> >> >>
> >> >> --
> >> >> CONFIDENTIALITY NOTICE
> >> >> NOTICE: This message is intended for the use of the individual or
> >> >>entity to
> >> >> which it is addressed and may contain information that is
> >>confidential,
> >> >> privileged and exempt from disclosure under applicable law. If the
> >> >>reader
> >> >> of this message is not the intended recipient, you are hereby
> >>notified
> >> >>that
> >> >> any printing, copying, dissemination, distribution, disclosure or
> >> >> forwarding of this communication is strictly prohibited. If you have
> >> >> received this communication in error, please contact the sender
> >> >>immediately
> >> >> and delete it from your system. Thank You.
> >> >>
> >> >
> >> >
> >> >
> >> >--
> >> >Have a Nice Day!
> >> >Lohit
> >>
> >>
> >
> >
> >--
> >http://hortonworks.com/download/
> >
> >--
> >CONFIDENTIALITY NOTICE
> >NOTICE: This message is intended for the use of the individual or entity
> >to
> >which it is addressed and may contain information that is confidential,
> >privileged and exempt from disclosure under applicable law. If the reader
> >of this message is not the intended recipient, you are hereby notified
> >that
> >any printing, copying, dissemination, distribution, disclosure or
> >forwarding of this communication is strictly prohibited. If you have
> >received this communication in error, please contact the sender
> >immediately
> >and delete it from your system. Thank You.
>
>
--
Todd Lipcon
Software Engineer, Cloudera
just trying to find out if it is indeed
abandoned)
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
; as part of persistent state. I
> want to know whether the case 3 is handled in CDH4.3.1 version. If the
> case is hadnled, where is the code?
>
>
> Thanks,
>
> LiuLe
>
--
Todd Lipcon
Software Engineer, Cloudera
Todd Lipcon created HDFS-5287:
-
Summary: JN need not validate finalized log segments in newEpoch
Key: HDFS-5287
URL: https://issues.apache.org/jira/browse/HDFS-5287
Project: Hadoop HDFS
Issue
ormation seems regular. The problem is that between the 2 lines
> there's no log in 12 minutes. There is no long gc happened. It seems the
> code blocked somewhere. Unfortunately, I forgot to print the jstack info
> T_T.
>
> Hope for your response.
>
> Best regards,
> Mickey
>
--
Todd Lipcon
Software Engineer, Cloudera
House
> > Jacklyns Lane
> > Alresford
> > SO24 9JJ
> > Tel: +44 (0)1962 738232
> > Mob: +44 (0)7710 160458
> > Fax: +44 (0)1962 600112
> > Web: www.b <http://www.fosolutions.co.uk/>espokesoftware.com<
> http://bespokesoftware.com/>
> >
[
https://issues.apache.org/jira/browse/HDFS-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-3656.
---
Resolution: Duplicate
Target Version/s: (was: )
Yep, I think you're right. T
cdh/4/hadoop-2.0.0-cdh4.**1.1.tar.gz<http://archive.cloudera.com/cdh4/cdh/4/hadoop-2.0.0-cdh4.1.1.tar.gz>
>
> We would greatly appreciate any advise on whether or not this approach
> sounds reasonable, and if you would consider accepting these modifications
> into the official Hadoop code base.
>
> Thank you,
> Jeff, Alja & Matevz
> UCSD Physics
>
--
Todd Lipcon
Software Engineer, Cloudera
Todd Lipcon created HDFS-5074:
-
Summary: Allow starting up from an fsimage checkpoint in the
middle of a segment
Key: HDFS-5074
URL: https://issues.apache.org/jira/browse/HDFS-5074
Project: Hadoop HDFS
Todd Lipcon created HDFS-5058:
-
Summary: QJM should validate startLogSegment() more strictly
Key: HDFS-5058
URL: https://issues.apache.org/jira/browse/HDFS-5058
Project: Hadoop HDFS
Issue Type
Todd Lipcon created HDFS-5037:
-
Summary: Active NN should trigger its own edit log rolls
Key: HDFS-5037
URL: https://issues.apache.org/jira/browse/HDFS-5037
Project: Hadoop HDFS
Issue Type
t; is
> > 19th July. So, I'm pretty tensed without any mentors :(
> >
> > [1] https://issues.apache.org/jira/browse/ACCUMULO-1197
> > [2] https://issues.apache.org/jira/browse/HDFS-4680
> > [3] https://github.com/cloudera/htrace
> > [4] http://community.apache.org/mentoringprogramme-icfoss-pilot.html
> > [5] https://issues.apache.org/jira/browse/HBASE-6449
> >
> > Thank you,
> > Sreejith R
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Todd Lipcon created HDFS-4982:
-
Summary: JournalNode should relogin from keytab before fetching
logs from other JNs
Key: HDFS-4982
URL: https://issues.apache.org/jira/browse/HDFS-4982
Project: Hadoop
[
https://issues.apache.org/jira/browse/HDFS-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-3182.
---
Resolution: Won't Fix
This was obsoleted by other work in HDFS-3077
> Add
M, Azuryy Yu wrote:
> > Hi dear All,
> >
> > It's long type for the txid currently,
> >
> > FSImage.java:
> >
> > boolean loadFSImage(FSNamesystem target, MetaRecoveryContext recovery)
> > throws IOException{
> >
> > editLog.setNextTxId(lastAppliedTxId + 1L);
> > }
> >
> > Is it possible that (lastAppliedTxId + 1L) exceed Long.MAX_VALUE ?
>
>
>
> --
> Harsh J
>
--
Todd Lipcon
Software Engineer, Cloudera
w what your experience with this format is.
>
> E14
>
> On Jun 24, 2013, at 2:50 PM, Todd Lipcon wrote:
>
> > Hey all,
> >
> > Unfortunately I will have to leave early on Thursday, so will miss the
> slotted time at the design lounge.
> >
> > But, I'l
re a project lead for one of the Apache projects, look at the
> schedule below and suggest a few slots when you think it would be best for
> your project to meet. I'll try to work out a schedule where no more than 2
> projects are using the lounge at once.
> >
> > Day 1, 26th June: 10:30am - 12:30pm, 1:45pm - 3:30pm, 3:45pm - 5:00pm
> >
> > Day 2, 27th June: 10:30am - 12:30pm, 1:45pm - 3:30pm, 3:45pm - 5:00pm
> >
> > It will be up to you, the hadoop contributors, from there.
> >
> > Look forward to seeing you all at the summit,
> >
> > E14
> >
> > PS Please forward to the other -dev lists. This event is for folks on
> the -dev lists.
> >
>
>
>
>
> --
> http://hortonworks.com/download/
>
--
Todd Lipcon
Software Engineer, Cloudera
of TestCLI against a fully distributed
> cluster. At this
> point I'm actually reasonably concerned and will follow up tomorrow.
>
> Thanks,
> Roman.
>
--
Todd Lipcon
Software Engineer, Cloudera
Todd Lipcon created HDFS-4915:
-
Summary: Add config to ZKFC to disable fencing
Key: HDFS-4915
URL: https://issues.apache.org/jira/browse/HDFS-4915
Project: Hadoop HDFS
Issue Type: Bug
en the keepalive, or shorten the socket cache expiry, or both.
>
> thoughts?
> Colin
>
--
Todd Lipcon
Software Engineer, Cloudera
Todd Lipcon created HDFS-4879:
-
Summary: Add "blocked ArrayList" collection to avoid CMS full GCs
Key: HDFS-4879
URL: https://issues.apache.org/jira/browse/HDFS-4879
Project: Hadoop HDFS
[
https://issues.apache.org/jira/browse/HDFS-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4184.
---
Resolution: Duplicate
> Add the ability for Client to provide more hint information for DataN
Todd Lipcon created HDFS-4833:
-
Summary: Corrupt blocks are not invalidated when first processing
repl queues
Key: HDFS-4833
URL: https://issues.apache.org/jira/browse/HDFS-4833
Project: Hadoop HDFS
Todd Lipcon created HDFS-4828:
-
Summary: Make QJM epoch-related errors more understandable
Key: HDFS-4828
URL: https://issues.apache.org/jira/browse/HDFS-4828
Project: Hadoop HDFS
Issue Type
Todd Lipcon created HDFS-4799:
-
Summary: Corrupt replica can be prematurely removed from
corruptReplicas map
Key: HDFS-4799
URL: https://issues.apache.org/jira/browse/HDFS-4799
Project: Hadoop HDFS
} while (thisListing.hasMore());
>
> return listing.toArray(new FileStatus[listing.size()]);
> }
>
> --------
>
>
>
>
>
> Ref:
>
> https://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.4/src/hdfs/org/apache/hadoop/hdfs/DistributedFileSystem.java
> http://docs.oracle.com/javase/6/docs/api/java/util/Iterator.html
>
>
> thanks!
>
> -bc
>
--
Todd Lipcon
Software Engineer, Cloudera
fic APIs
rather than any standard Linux interface. So, I'm in favor of either
putting the APIs in HdfsAdmin, or alternatively in DistributedFileSystem,
forcing a user to down-cast if they want to use the HDFS-specific operation.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
Suresh has also volunteered to maintain the insecure SCR code until secure
> SCR can be implemented for Windows.
>
> Please cast your vote by EOD Monday 4/8.
>
> best,
> Colin
>
--
Todd Lipcon
Software Engineer, Cloudera
as been addressed-- see HDFS-4538 for details.
> Suresh has also volunteered to maintain the insecure SCR code until secure
> SCR can be implemented for Windows.
>
> Please cast your vote by EOD Monday 4/8.
>
> best,
> Colin
>
--
Todd Lipcon
Software Engineer, Cloudera
; I have only three nodes(A,B,C) for test
> A and B are name nodes.
> B and C are data nodes.
> ZK deployed on A, B and C.
>
> A, B and C are all journal nodes.
>
> Thanks.
>
--
Todd Lipcon
Software Engineer, Cloudera
[
https://issues.apache.org/jira/browse/HDFS-4617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4617.
---
Resolution: Duplicate
ATM points out that I already found this bug 3 months ago... resolving as
[
https://issues.apache.org/jira/browse/HDFS-4538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4538.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to HDFS-347 branch.
> al
Todd Lipcon created HDFS-4643:
-
Summary: Fix flakiness in TestQuorumJournalManager
Key: HDFS-4643
URL: https://issues.apache.org/jira/browse/HDFS-4643
Project: Hadoop HDFS
Issue Type: Bug
Todd Lipcon created HDFS-4638:
-
Summary: TransferFsImage should take Configuration as parameter
Key: HDFS-4638
URL: https://issues.apache.org/jira/browse/HDFS-4638
Project: Hadoop HDFS
Issue
Todd Lipcon created HDFS-4621:
-
Summary: additional logging to help diagnose slow QJM logSync
Key: HDFS-4621
URL: https://issues.apache.org/jira/browse/HDFS-4621
Project: Hadoop HDFS
Issue Type
Todd Lipcon created HDFS-4618:
-
Summary: default for checkpoint txn interval is too low
Key: HDFS-4618
URL: https://issues.apache.org/jira/browse/HDFS-4618
Project: Hadoop HDFS
Issue Type: Bug
Todd Lipcon created HDFS-4617:
-
Summary: warning while purging logs with QJM enabled
Key: HDFS-4617
URL: https://issues.apache.org/jira/browse/HDFS-4617
Project: Hadoop HDFS
Issue Type: Bug
; >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and
> > >>>performance
> > >>> and scalability tuned on Windows Server or Windows Azure.
> > >>> We are happy to announce that a lot of progress has been made in this
> > >>> regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen in
> > >>> CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based on
> > trunk.
> > >>> Merge patch for this is available on
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, process/task
> > >>> management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and
> > >>>ownership,
> > >>> user group mapping, hardlinks, symbolic links, chmod, disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts
> > >>> hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support cloud
> > >>> enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win and
> > >>>getting
> > >>> ready for a merge. Currently the merge patch is passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> > >>>into
> > >>> trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> > >>> completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>
--
Todd Lipcon
Software Engineer, Cloudera
;
> [1]
> https://issues.apache.org/jira/browse/HDFS-347?focusedCommentId=13579704&p
> age=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comme
> nt-13579704
>
> [2]
> https://issues.apache.org/jira/browse/HDFS-347?focusedCommentId=13551755&p
> age=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comme
> nt-13551755
>
--
Todd Lipcon
Software Engineer, Cloudera
ckReaderLocal: simpler implementation that uses raw FileChannel
objects. We don't need to cache anything, or make RPCs to the
DataNode."
from his 10/1/2012 patch upload. So, any patch you might have looked
at since then would have clearly removed the old code path.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
small number of developers who know how to program
against that platform.
If that's the axe hanging over our head with the Windows branch, then
I'm all for saying "good, keep it on a branch and don't merge it to
trunk".
I was hoping we could all work together a bit be
optimization on a platform which is not
yet officially supported on trunk, especially when the old code path
is _insecure_ and thus unusable in most environments.
Todd
>
> ____
> From: Todd Lipcon
> To: hdfs-dev@hadoop.apache.org; Tsz Wo Sze
> Sent: Wednesd
On Wed, Feb 20, 2013 at 3:06 PM, Todd Lipcon wrote:
> https://issues.apache.org/jira/browse/HDFS-4476 where I pointed out
sorry, meant to link to:
https://issues.apache.org/jira/browse/HDFS-2246?focusedCommentId=13102013&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-t
ies are owned by an HDFS
user, and users of the cluster run under other unix credentials.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
active
review" to me. On the other hand, I spent probably 40-50% of my time
over the last month reviewing and testing this branch and have voted
+1.
-Todd
>
>
> On Wed, Feb 20, 2013 at 12:16 PM, Todd Lipcon wrote:
>
>> Hi Nicholas,
>>
>> I looked at your comme
currently patch is still not yet ready. It seems to have unnecessarily
> changed the API and protocol. I believe those are important but not trivial
> things.
>
> Tsz-Wo
>
>
>
> From: Todd Lipcon
> To: hdfs-dev@hadoop.apache.org; Ts
gt; -1
> The patch seems not ready yet. I have posted some comments/suggestions on
> the JIRA. Colin also has agreed that there are some bugs to be fixed. Sorry.
>
> Tsz-Wo
>
>
>
>
> ____
> From: Todd Lipcon
> To: hdfs-dev@hadoop.ap
/browse/HDFS-347?focusedCommentId=13551755&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13551755
>
--
Todd Lipcon
Software Engineer, Cloudera
[
https://issues.apache.org/jira/browse/HDFS-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4496.
---
Resolution: Fixed
Hadoop Flags: Reviewed
> DFSClient: don't create a domain socke
Todd Lipcon created HDFS-4486:
-
Summary: Add log category for long-running DFSClient notices
Key: HDFS-4486
URL: https://issues.apache.org/jira/browse/HDFS-4486
Project: Hadoop HDFS
Issue Type
Todd Lipcon created HDFS-4485:
-
Summary: HDFS-347: DN should chmod socket path a+w
Key: HDFS-4485
URL: https://issues.apache.org/jira/browse/HDFS-4485
Project: Hadoop HDFS
Issue Type: Sub-task
in Hadoop. But, with some of the changes in the proposed 2.0.3-alpha, it
wouldn't be wire-protocol-stable.
How can we best devise a scheme that explains the various factors above in
a more detailed way than one big red warning sticker? What of the above
factors does the community think would be implied by "GA?"
Thanks
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
[
https://issues.apache.org/jira/browse/HDFS-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4433.
---
Resolution: Fixed
Hadoop Flags: Reviewed
> make TestPeerCache not fl
[
https://issues.apache.org/jira/browse/HDFS-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4416.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to branch. Thanks, Colin
[
https://issues.apache.org/jira/browse/HDFS-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4418.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to branch.
> HDFS-
Todd Lipcon created HDFS-4418:
-
Summary: HDFS-347: increase default FileInputStreamCache size
Key: HDFS-4418
URL: https://issues.apache.org/jira/browse/HDFS-4418
Project: Hadoop HDFS
Issue Type
Todd Lipcon created HDFS-4417:
-
Summary: HDFS-347: fix case where local reads get disabled
incorrectly
Key: HDFS-4417
URL: https://issues.apache.org/jira/browse/HDFS-4417
Project: Hadoop HDFS
Todd Lipcon created HDFS-4403:
-
Summary: DFSClient can infer checksum type when not provided by
reading first byte
Key: HDFS-4403
URL: https://issues.apache.org/jira/browse/HDFS-4403
Project: Hadoop HDFS
[
https://issues.apache.org/jira/browse/HDFS-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4402.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to branch, thanks.
> s
[
https://issues.apache.org/jira/browse/HDFS-4401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4401.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to branch. Thanks for the fix
[
https://issues.apache.org/jira/browse/HDFS-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4400.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to branch
> DFSInputStr
[
https://issues.apache.org/jira/browse/HDFS-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4390.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to branch
> Bypass U
[
https://issues.apache.org/jira/browse/HDFS-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4388.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Committed to branch. Thanks
[
https://issues.apache.org/jira/browse/HDFS-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4356.
---
Resolution: Fixed
Hadoop Flags: Reviewed
> BlockReaderLocal should use passed f
Todd Lipcon created HDFS-4380:
-
Summary: Opening a file for read before writer writes a block
causes NPE
Key: HDFS-4380
URL: https://issues.apache.org/jira/browse/HDFS-4380
Project: Hadoop HDFS
[
https://issues.apache.org/jira/browse/HDFS-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon reopened HDFS-4352:
---
By the way, I find it _very_ rude to close someone else's ticket as "Invalid"
or &qu
Todd Lipcon created HDFS-4324:
-
Summary: Track and report out-of-date blocks separately from
corrupt blocks
Key: HDFS-4324
URL: https://issues.apache.org/jira/browse/HDFS-4324
Project: Hadoop HDFS
Todd Lipcon created HDFS-4305:
-
Summary: Add a configurable limit on number of blocks per file,
and min block size
Key: HDFS-4305
URL: https://issues.apache.org/jira/browse/HDFS-4305
Project: Hadoop HDFS
Todd Lipcon created HDFS-4304:
-
Summary: Make FSEditLogOp.MAX_OP_SIZE configurable
Key: HDFS-4304
URL: https://issues.apache.org/jira/browse/HDFS-4304
Project: Hadoop HDFS
Issue Type: Bug
Todd Lipcon created HDFS-4301:
-
Summary: 2NN image transfer timeout problematic
Key: HDFS-4301
URL: https://issues.apache.org/jira/browse/HDFS-4301
Project: Hadoop HDFS
Issue Type: Bug
Todd Lipcon created HDFS-4300:
-
Summary: TransferFsImage.downloadEditsToStorage should use a tmp
file for destination
Key: HDFS-4300
URL: https://issues.apache.org/jira/browse/HDFS-4300
Project: Hadoop
Todd Lipcon created HDFS-4298:
-
Summary: StorageRetentionManager spews warnings when used with QJM
Key: HDFS-4298
URL: https://issues.apache.org/jira/browse/HDFS-4298
Project: Hadoop HDFS
Issue
[
https://issues.apache.org/jira/browse/HDFS-4110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-4110.
---
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
Committed backport to branch-2 (same patch
[
https://issues.apache.org/jira/browse/HDFS-4110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon reopened HDFS-4110:
---
Reopening to backport to branch-2
> Refine JNStorage
[
https://issues.apache.org/jira/browse/HDFS-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-3077.
---
Resolution: Fixed
Fix Version/s: 2.0.3-alpha
Committed backport to branch-2. Thanks for
1 - 100 of 1034 matches
Mail list logo
