On Wed, Jul 6, 2011 at 2:48 PM, William Oberman <ober...@civicscience.com>wrote:
> I have a few cassandra/hadoop/pig questions. I currently have things set > up in a test environment, and for the most part everything works. But, > before I start to roll things out to production, I wanted to check > on/confirm some things. > > When I originally set things up, I used: > http://wiki.apache.org/cassandra/HadoopSupport > http://hadoop.apache.org/common/docs/r0.20.203.0/cluster_setup.html > > One difference I noticed between the two guides, which I ignored at the > time, was how "datanodes" are treated. The wiki said "At least one node in > your cluster will also need to be a datanode. That's because Hadoop uses > HDFS to store information like jar dependencies for your job, static data > (like stop words for a word count), and things like that - it's the > distributed cache. It's a very small amount of data but the Hadoop cluster > needs it to run properly". But, the hadoop guide (if you follow it blindly > like I did), creates a datanode on all TaskTracker nodes. I _think_ that is > controlled by the conf/slaves file, but I haven't proved that yet. Is there > any good reason to run datanodes on only the JobTracker vs. on all nodes? > If I should only run it on the JobTracker, how do I properly stop the > datanodes from starting automatically (when both start-dfs and start-mapred > seem to draw from the same slaves file)? > > I noticed a second issue/oddness with datanodes, in that the HDFS data > isn't always small. The other day I ran out of disk running my pig script. > I checked, and by default, hadoop creates HDFS in /tmp, and I'm using EC2 > (and /tmp is on the boot device) which is only 10G by default. Do other > people put HDFS on a different disk? If yes, I'll really want to only run > one datanode, as I don't want to re-template all of my cassandra nodes to > have HDFS disks vs. one new JobTracker node. > > In terms of hardware, I am running small instances (32bit, 2GB) in the test > cluster, while my production cluster is larges (64bit, 7 or 8GB). I was > going to check the performance impact there, but even on smalls in test I > was able to run hadoop jobs while serving web requests. I am wondering if > smalls are causing the high HDFS usage though (I think data might "spill" > more, if I'm understanding things correctly). > > If these are more hadoop then cassandra questions, let me know and I'll > move my questions around. > > I did want to mention that these are small details compared to the amount > of complicated things that worked like a charm during my configuration and > testing of the combination of cassandra/hadoop/pig. It was impressive :-) > > Thanks! > > will > > The logic that "only one datanode is needed" is not an absolute truth. If your jobs use ColumnFamilyInputFormat to read and write to ColumnFamilyOutputFormat then technically you only need one DataNode to hold the distributed cache. However, if you have a large amount of intermediate results or even a multiphase job that has to persist data between phases (this is very very common) then that single DataNode is a bottleneck. Most hadoop clusters run a DataNode and TaskTracker on each slave. Most situations would use datanodes very heavily, for example suppose you have 4 map/reduce jobs to run on the same Cassandra data. Ingesting the data from Cassandra at the beginning of each job might would be wasteful. It might be better to take the data into HDFS during the first job and then save it. Your subsequent jobs could use that instead of re-acquiring it from Cassandra.
