Re: How to compile fuse-dfs

Resolved. Only fuse-2.9 is not enough, I did following before maven build, then it works. yum -y install fuse fuse-devel fuse-libs On Fri, Mar 28, 2014 at 9:51 PM, Azuryy Yu wrote: > Hi, > > I want to build fuse-dfs from trunk, my command: > > mvn package -DskipTests -Pna

How to compile fuse-dfs

Hi, I want to build fuse-dfs from trunk, my command: mvn package -DskipTests -Pnative,dist -Dtar I've installed fuse-2.9.3 before the build: ./configure; make; make install; modprobe fuse cp -a /usr/local/lib/libfuse.* /usr/lib64 but I find maven complain: main: [exec] -- checking for

Fwd: problem with HDFS caching in Hadoop 2.3

add dev. -- Forwarded message -- From: "hwpstorage" Date: Mar 7, 2014 11:38 PM Subject: problem with HDFS caching in Hadoop 2.3 To: Cc: Hello, It looks like the HDFS caching does not work well. The cached log file is around 200MB. The hadoop cluster has 3 nodes, each has 4GB mem

Re: Thinking ahead to 2.4

Hi, Sorry ignore my inputs. Just keep sub tasks as blocker, because the whole Umbrella tasks may not block release. On Fri, Mar 7, 2014 at 9:44 AM, Azuryy Yu wrote: > Hi Arun, > > I just advice remove some sub tasks from blockers, then add Umbrella > tasks, such as: > >

Re: Thinking ahead to 2.4

Hi Arun, I just advice remove some sub tasks from blockers, then add Umbrella tasks, such as: RM HA - YARN-149 YARN Generic Application Timeline - YARN-1530 Generic application history service- YARN-321 Rolling upgrade(will be ready, no blockers currently, I just mention it) - HDFS-5535 n

Re: [VOTE] Merge HDFS-5535 Rolling Upgrade Improvement

+1 for merging. On Feb 26, 2014 5:42 AM, "Tsz Wo Sze" wrote: > Hi hdfs-dev, > > We propose merging the HDFS-5535 branch to trunk. > > HDFS Rolling Upgrade is a feature to allow upgrading individual HDFS > daemons. In Hadoop v2, HDFS supports highly-available (HA) namenode > services and wire com

Re: [VOTE] Merge HDFS-4685 HDFS ACLs to trunk

+1 It's very useful, Thanks for improve ACL features. On Tue, Feb 11, 2014 at 8:46 AM, Chris Nauroth wrote: > Hello everyone, > > I would like to call a vote to merge HDFS ACLs from branch HDFS-4685 to > trunk. > > HDFS ACLs provide support for finer-grained permissions on files than what > use

Re: confirm expected HA behavior

Hi Todd, I think Arpit's test method is incorrect. we cannot block port 8020 to simulate active NN down. because ZK session is live and NN process is running at the same time. so when unblock 8020, NN1 think himself still is active. On Sat, Feb 8, 2014 at 3:47 AM, Todd Lipcon wrote: > Hi Ar

Re: [VOTE] Merge HDFS-5698 Use protobuf to serialize / deserialize FSImage to trunk

+1 This was very important improvement. hope it release at 2.4 On Jan 31, 2014 6:37 AM, "Haohui Mai" wrote: > Hello all, > > I would like to call a vote to merge of the new protobuf-based FSImage into > trunk. > > The changes introduce a protobuf-based FSImage format into HDFS. It > separates

What will be included in the hadoop-2.4

Hi, Merry Christmas firstly. Do we have release mile stone for Hadoop-2.4? and importantly, doea that the following features will be included? a. Rolling upgrade HDFS b. Heterogeneous storage API support c: The whole central cache features d: Serialize fsImage using protobuf Thanks a lot.

Re: HDFS upgrade problem of fsImage

gt; > Some helpful guidelines for upgrade here - > > http://wiki.apache.org/hadoop/Hadoop_Upgrade > > https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade > > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_rele

Re: HDFS upgrade problem of fsImage

stions, i'd like to extend to you the > opportunity to help the HDFS dev team to implement rolling updates > > https://issues.apache.org/jira/browse/HDFS-5535 > > design reviews, code and tests welcome > > -steve > > > On 21 November 2013 06:27, Azuryy Yu

Re: HDFS upgrade problem of fsImage

; > -Original Message----- > From: Azuryy Yu [mailto:azury...@gmail.com] > Sent: 21 November 2013 09:49 > To: hdfs-dev@hadoop.apache.org; u...@hadoop.apache.org > Subject: HDFS upgrade problem of fsImage > > Hi Dear, > > I have a small test cluster with hadoop-2.0x,

HDFS upgrade problem of fsImage

Hi Dear, I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2. I dont't want to stop cluster during upgrade, so my steps are: 1) on standby NN: hadoop-dameon.sh stop namenode 2) remove HA configuration in the conf 3) hadoop-daemon.sh start na

Re: [Proposal] Pluggable Namespace

Hi Milind, HDFS federation can solve the NN bottle neck and memory limit problem. AbstractNameSystem design sounds good. but distributed meta storage using HBase should bring performance degration. On Oct 4, 2013 3:18 AM, "Milind Bhandarkar" wrote: > Hi All, > > Exec Summary: For the last coupl

Re: Exception while syncing from Flume to HDFS

hi, this is not HDFS issue. you can put your question in the flume ml. On Jul 19, 2013 3:20 PM, "Divya R" wrote: > I'm running hadoop 1.2.0 and flume 1.3. Every thing works fine if its > independently run. When I start my tomcat I get the below exception after > some time. > > 2013-07-17 12:4

Re: Can I move block data directly?

tes all of the ins > > and outs in order successfully complete a block transfer. > > Thus, the balancer is where I'll begin my efforts to learn how to > > manually move blocks. > > > > Any other pointers would be helpful. > > > > Thank you, > >

Re: Can I move block data directly?

t; manually move blocks. > > Any other pointers would be helpful. > > Thank you, > Eitan > > On Mon, Jul 8, 2013 at 2:15 PM, Allan wrote: > > If the imbalance is across data nodes then you need to run the balancer. > > > > Sent from my iPad > > > >

Fwd: Can I move block data directly?

Hi Dear all, There are some unbalanced data nodes in my cluster, some nodes reached more than 95% disk usage. so Can I move some block data from one node to another node directly? such as: from n1 to n2: 1) scp /data//blk_* n2:/data/subdir11/ 2) rm -rf data//blk_* 3) hadoop-dameon.sh

Re: Question for edit log retain

That's detailed. Thanks ATM. On Tue, Jul 2, 2013 at 7:02 AM, Aaron T. Myers wrote: > Hi Azuryy, > > On Wed, Jun 26, 2013 at 6:12 PM, Azuryy Yu wrote: > > > Hi Dear all, > > > > I have some confusion for edit log retain, > > >

Re: [jira] [Created] (HDFS-4945) A Distributed and Cooperative NameNode Cluster for a Highly-Available HDFS

hi, fot your first question, if you deploy QJM ha, it doesnt need share highly-reliable sophisticated storage. --Send from my Sony mobile. On Jun 30, 2013 11:38 AM, "Yonghwan Kim (JIRA)" wrote: > Yonghwan Kim created HDFS-4945: > -- > > Summary: A Di

Question for edit log retain

Hi Dear all, I have some confusion for edit log retain, NNStorageRetentionManager.java: 1) purgeCheckpointsOlderThan() What's mean of check point here? 2)purgeOldStorage() I cannot understand the calculation of minimum txid, I think I can understand it if I know these keys indicates. DFS_NA

Re: A question for txid

d any edit > > logs containing txid -9223372036854775808 > > > > So it looks like it cannot currently handle an overflow. > > > > I've filed https://issues.apache.org/jira/browse/HDFS-4936 to discuss > > this. I don't think this is of immediate concern

A question for txid

Hi dear All, It's long type for the txid currently, FSImage.java: boolean loadFSImage(FSNamesystem target, MetaRecoveryContext recovery) throws IOException{ editLog.setNextTxId(lastAppliedTxId + 1L); } Is it possible that (lastAppliedTxId + 1L) exceed Long.MAX_VALUE ?

Re: [jira] [Resolved] (HDFS-4869) FsShell get command does not support writing to stdout

FsShell was rewrited in hadoop-2, The new FsShell has so many issuses, such as copytolocal cannot keep permission, getmerge doesnt support derectory target etc. --Send from my Sony mobile. On Jun 12, 2013 12:21 AM, "Cristina L. Abad (JIRA)" wrote: > > [ > https://issues.apache.org/jira/brow

Re: Heads up: moving from 2.0.4.1-alpha to 2.0.5-alpha

2.1.0-alpha is 2.0.5-beta actually. --Send from my Sony mobile. On Jun 1, 2013 8:14 AM, "Ted Yu" wrote: > I am currently testing HBase 0.95 using 2.0.5-SNAPSHOT artifacts. > > Would 2.1.0-SNAPSHOT maven artifacts be available after tomorrow's change ? > > Thanks > > On Fri, May 31, 2013 at 12:45

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

good suggestion. On Wed, Apr 17, 2013 at 4:10 PM, Harsh J wrote: > Pardon my late inquisition here but since HBase already shipped out > with a name .snapshots/, why do we force them to change it, and not > rename HDFS' snapshots to use .hdfs-snapshots, given that HDFS > Snapshots has not been

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

I think ".hbase-sanpshot" is good, but we should also disallow user to create ".hbase-sanpshot" under hbase.root sub directories. On Tue, Apr 16, 2013 at 11:18 AM, Ted Yu wrote: > I plan to rename ".sanpshot" in HBase to ".hbase-sanpshot" > > Please suggest better name in future correspondence.

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

This was caused by HDFS-4666, committed yesterday. IMO, it will be in a good place fianlly if HDFS and HBASE all changed naming convention. .hdfs_snapshot and .hbase_snapshot would be better. On Tue, Apr 16, 2013 at 10:13 AM, Ted Yu wrote: > Hi

Re: VOTE: HDFS-347 merge

te/apt/ShortCircuitLocalReads.apt.vm > > best, > Colin > > > On Thu, Apr 11, 2013 at 6:37 PM, Azuryy Yu wrote: > > > It's good to know HDFS-347 win the votes finally. > > > > Does there need some additional configuration to enable these features? > > &g

Re: VOTE: HDFS-347 merge

It's good to know HDFS-347 win the votes finally. Does there need some additional configuration to enable these features? On Fri, Apr 12, 2013 at 2:05 AM, Colin McCabe wrote: > The merge vote is now closed. With three +1s, it passes. > > thanks, > Colin > > > On Wed, Apr 10, 2013 at 10:00 PM,

Re: Which Hadoop version is preferable to make changes in HDFS??

Hadoopv2 support ""dfs.block.replicator.classname", you can implement your replicate policy. --Send from my Sony mobile. On Apr 11, 2013 2:07 PM, "Mohammad Mustaqeem" <3m.mustaq...@gmail.com> wrote: > I want to make changes in replica placement. > Which Hadoop version is preferable to make change

Re: Which Hadoop version is preferable to make changes in HDFS??

Hadoop2.0.x supports "dfs.block.replicator.classname", you can implement your own replicator policy. On Thu, Apr 11, 2013 at 2:07 PM, Mohammad Mustaqeem <3m.mustaq...@gmail.com>wrote: > I want to make changes in replica placement. > Which Hadoop version is preferable to make changes in HDFS?? >

Re: Which java file is responsible for replica placement??

org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault in HadoopV2. On Thu, Apr 11, 2013 at 2:00 PM, Mohammad Mustaqeem <3m.mustaq...@gmail.com>wrote: > Which class is responsible for replica placement?? > Which class chooses the random rack for placement of 3rd replica? > an

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

Sorry, don't include HDFS-4339, because HDFS-4334<https://issues.apache.org/jira/browse/HDFS-4334> is fixed for 3.0.0. On Thu, Apr 11, 2013 at 9:20 AM, Azuryy Yu wrote: > Hi, > > HDFS-4339 should be included, otherwise, HDFS cannot failover > automatically. > > &g

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

Hi, HDFS-4339 should be included, otherwise, HDFS cannot failover automatically. On Thu, Apr 11, 2013 at 12:11 AM, Sangjin Lee wrote: > Hi Arun, > > Would it be possible to include HADOOP-9407 in the release? It's been > resolved for a while, and it'd be good to have this dependency problem >

Re: What do ns_quota and ds_quota mean in an namednode entry

name space, disk space. ns means block numbers limits. ds is total file size limitation. On Apr 4, 2013 3:12 PM, "Bert Yuan" wrote: > Bellow is json format of an namednode entry: > { > inode:{ > inodepath:'/anotherDir/biggerfile', > replication:3, > modificationtime:'2