Hi Emeka, thanks for the reply, I know the old code had some bugs. But taking the suggestion by Stuart i.e instead of using maxtemperature directly I tried using (Class/forName "maxtemperature") and this worked. So I'm not sure if I was referring to different class-paths in my code.
regards, RD On Tue, Jul 28, 2009 at 3:39 AM, Emeka <emekami...@gmail.com> wrote: > RD,I don't know about the gen-class, to be sincere with you I have not > used. But clojure follows a standard and natural sequence if you look > closely. What you used in your former code were referring to different class > path and the paths were not even defined. > > The below is from clojure.org > A stand-alone gen-class <http://clojure.org/API#gen-class> facility is > provided to create named classes for direct use as Java classes, with > facilities for: > > - Naming the generated class > > http://clojure.org/compilation > > So were do you want clojure to place the generated class. My guess is > within the same folder as your 'ns or need near. Now I hope you have seen > why you need that change. > I invite comments from experts here if I have strayed. > > Regards, > Emeka > > > On Mon, Jul 27, 2009 at 9:47 AM, RD <rdsr...@gmail.com> wrote: > >> Hi Stuart, >> Thanks for the link >> Relating to the hadoop code I posted above, I got it to compile by >> putting the code under a namespace and changing >> the class names appropriately , but still I have no clue as to why the >> code didn't work without the namespace. >> >> I'll defenitely check the link though, thks. >> >> regards, >> rdsr >> >> >> Here's the code with the modification >> >> (ns com.rdsr.maxt >> (:gen-class) >> (:import [org.apache.hadoop.io IntWritable Text] >> [org.apache.hadoop.fs Path] >> [org.apache.hadoop.mapred >> JobConf JobClient >> FileInputFormat FileOutputFormat >> Mapper Reducer MapReduceBase]) >> (:use [clojure.contrib.str-utils :only (re-split)])) >> >> (gen-class >> :name com.rdsr.mapper >> :extends org.apache.hadoop.mapred.MapReduceBase >> :implements [org.apache.hadoop.mapred.Mapper] >> :prefix "m-") >> >> (defn m-map [this key value output-collector reporter] >> (let [line (str value) >> year (subs line 15 19) >> air-temperature (if (= (nth line 87) \+) >> (Integer/parseInt (subs line 88 92)) >> (Integer/parseInt (subs line 87 92))) >> quality (subs line 92 93)] >> (if (and (not (= air-temperature 9999)) >> (re-matches #"[01459]" quality)) >> (.collect output-collector (Text. year) (IntWritable. >> air-temperature))))) >> >> (gen-class >> :name com.rdsr.reducer >> :extends org.apache.hadoop.mapred.MapReduceBase >> :implements [org.apache.hadoop.mapred.Reducer] >> :prefix "r-") >> >> (defn r-reduce [this key values output-collector reporter] >> (let [values-seq (iterator-seq values)] >> (.collect output-collector key >> (apply max (map (fn [n] (.get n)) values-seq))))) >> >> (defn -main [& args] >> (let [conf (JobConf. com.rdsr.maxt)] >> (doto conf >> (.setJobName "Max temperature") >> (.setMapperClass com.rdsr.mapper) >> (.setReducerClass com.rdsr.reducer) >> (.setOutputKeyClass Text) >> (.setOutputValueClass IntWritable)) >> (FileInputFormat/addInputPath conf (Path. (first args))) >> (FileOutputFormat/setOutputPath conf (Path. (second args))) >> (JobClient/runJob conf))) >> >> (compile 'com.rdsr.maxt) >> >> >> On Mon, Jul 27, 2009 at 9:46 PM, Stuart Sierra < >> the.stuart.sie...@gmail.com> wrote: >> >>> >>> Hi rdsr, >>> >>> The problem is that you're trying to use the "maxtemperature" as a >>> class name before it's finished compiling. To work around that, use >>> (Class/forName "maxtemperature") instead. >>> >>> You can also see my (very task-specific) Hadoop/Clojure integration at >>> <http://tinyurl.com/mqv2os> >>> and <http://tinyurl.com/lznlg2>. >>> >>> -SS >>> >>> >>> On Jul 27, 4:55 am, RD <rdsr...@gmail.com> wrote: >>> > Hi all, >>> > I'm having trouble converting the basic hadoop example to clojure. >>> > Here's my code. >>> > >>> > (ns maxtemperature >>> > (:gen-class) >>> > (:import [org.apache.hadoop.io IntWritable Text] >>> > [org.apache.hadoop.mapred >>> > JobConf JobClient >>> > FileInputFormat FileOutputFormat >>> > Mapper Reducer MapReduceBase]) >>> > (:use [clojure.contrib.str-utils :only (re-split)])) >>> > >>> > (gen-class >>> > :name mapper >>> > :extends org.apache.hadoop.mapred.MapReduceBase >>> > :implements [org.apache.hadoop.mapred.Mapper] >>> > :prefix "m-") >>> > >>> > (defn m-map [this key value output-collector reporter] >>> > (let [line (str value) >>> > year (subs line 15 19) >>> > air-temperature (if (= (nth line 87) \+) >>> > (Integer/parseInt (subs line 88 92)) >>> > (Integer/parseInt (subs line 87 92))) >>> > quality (subs line 92 93)] >>> > (if (and (not (= air-temperature 9999)) >>> > (re-matches quality "[01459]")) >>> > (.collect output-collector (Text. year) (IntWritable. >>> > air-temperature))))) >>> > >>> > (gen-class >>> > :name reducer >>> > :extends org.apache.hadoop.mapred.MapReduceBase >>> > :implements [org.apache.hadoop.mapred.Reducer] >>> > :prefix "r-") >>> > >>> > (defn r-reduce [this key values output-collector] >>> > (.collect output-collector key (apply max values))) >>> > >>> > (defn -main [s] >>> > (let [conf (JobConf. maxtemperature) >>> > args (re-split " +" s 2)] >>> > (doto conf >>> > (.setJobName "Max temperature") >>> > (.setMapperClass mapper) >>> > (.setReducerClass reducer) >>> > (.setOutputKeyClass Text) >>> > (.setOutputValueClass IntWritable)) >>> > (FileInputFormat/addInputPath conf (first args)) >>> > (FileOutputFormat/setOutputPath conf (second args)) >>> > (.runJob JobClient conf))) >>> > >>> > (compile 'maxtemperature) >>> > >>> > The problem is that, the compilation step fails when it encounters the >>> > maxtemperature in the main function. >>> > Without the main function the code works fine and even the compilation >>> > succeeds. I can't figure out what's wrong. >>> > Greatly appreciate help on this. >>> > >>> > thanks, >>> > rdsr. >>> >>> >> >> >> > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~----------~----~----~----~------~----~------~--~---
