small file bring tow problem: memory of namenode and the utilization ratio of storage.
I think following choice:1. use bigtable 2. archive in application layer. On Wed, Mar 25, 2009 at 1:37 AM, Raghu Angadi <rang...@yahoo-inc.com> wrote: > > What is scale you are thinking of? (10s, 100s or more nodes)? > > The memory for metadata at NameNode you mentioned is that main issue with > small files. There are multiple alternatives for the dealing with that. This > issue is discussed many times here. > > Also please use core-user@ id alone for asking for help.. you don't need > to send to core-devel@ > > Raghu. > > > snehal nagmote wrote: > >> Hello Sir, >> >> I have some doubts, please help me. >> we have requirement of scalable storage system, we have developed one >> agro-advisory system in which farmers will sent the crop pictures >> particularly in sequential manner some >> 6-7 photos of 3-4 kb each would be stored in storage server and these >> photos >> would be read sequentially by scientist to detect the problem, writing to >> images would not be done. >> >> So for storing these images we are using hadoop file system, is it >> feasible >> to use hadoop >> file system for the same purpose. >> >> As also the images are of only 3-4 kb and hadoop reads the data in blocks >> of >> size 64 mb >> how can we increase the performance, what could be the tricks and tweaks >> that should be done to use hadoop for such kind of purpose. >> >> Next problem is as hadoop stores all the metadata in memory,can we use >> some >> mechanism to store the files in the block of some greater size because as >> the files would be of small size,so it will store the lots metadata and >> will >> overflow the main memory >> please suggest what could be done >> >> >> regards, >> Snehal >> >> >
