Why aren't you just writing samza tasks directly in clojure using gen-class and implementing the samza interfaces? Works great for us. Much simpler.
> On Feb 10, 2016, at 7:39 PM, Jacob Maes <jacob.m...@gmail.com> wrote: > > Hey Andy, > > That's good news! Do you mind sharing the revelation for any other Samza > users in Clojure-land? > > -Jake > > On Wed, Feb 10, 2016 at 4:07 PM, Andy Chambers < > andy.chamb...@fundingcircle.com> wrote: > >> My co-worker has figured out how to do this without any changes required in >> samza. >> Andy Chambers | Full-Stack Software Developer >> >> andy.chamb...@fundingcircle.com | 707-205-6502 (m) >> >> >> >> >> 747 Front St, 4th Fl | San Francisco, CA 94111 >> >> Our Mission: T o build a better financial world >> >> >> >> >> >> >> Unless specifically indicated, this e-mail is not an offer to sell or a >> solicitation of any investment products or other financial product or >> service, >> an official confirmation of any transaction, or an official statement of >> Funding >> Circle USA. This e-mail is meant only for the intended recipient of this >> transmission, and >> contains trade secret and strictly confidential information belonging to >> the >> sender. It is unlawful for unauthorized individuals to review, use, copy, >> disclose, or disseminate confidential information. If you have received >> this >> e-mail in error, please notify the sender immediately by telephone at >> 857.285.1263 or by return email and promptly delete this message from >> your system. >> >> >> >> On 9 February 2016 at 23:47, Andy Chambers < >> andy.chamb...@fundingcircle.com > wrote: >> No Problem. I'm new to samza so it is entirely possible there's already a >> way to >> do this that I'm missing. >> What I'm aiming for, is to implement the features described in the README >> of my >> project https://github.com/ cddr/samza-config . For discussion purposes, >> I'll copy the example here... >> (ns example.word-counter >> (:require >> [samza-config.core :refer [stateful-task key-value-store]])) >> >> (defn count-words [store sentence output] >> (doseq [word (split sentence)] >> (update-in store word inc) >> (output :word-count {:word word >> :count (get store word)}))) >> >> (defjob word-counter >> {:inputs [(topic “words”)] >> :outputs [(topic “word-counts”)] >> :storage (key-value-store :word-counts) >> :task (stateful-task >> (fn [store input output] >> (count-words store input output)))}) >> >> I thought the easiest way to implement this would be for defjob to setup >> some >> metadata that points to implementations of StreamTask/InitableTask and >> have a >> custom ConfigFactory that can return the config for a named job. >> Here is the (in development) implementation of ConfigFactory >> (defrecord JobConfigFactory [] ConfigFactory (getConfig [this uri] (let >> [job (find-job uri)] (println “found job:” job) job))) >> Unfortunately when I run the job runner, passing the name of this class as >> the >> --config-factory, I get the following stacktrace. I believe this is because >> Clojure uses a special “DynamicClassLoader” to find these classes. That is >> what >> I'd set the classloader to if there was the option to do so. >> $ lein do run -m samza-config.job example.jobs.hello-world Running samza >> job: notifications.jobs.send-email Config Factory: samza_config.job. >> JobConfigFactory Class Loader: #object[sun.misc.Launcher$ AppClassLoader >> 0x4aa298b7 sun.misc.Launcher$ AppClassLoader@4aa298b7] Exception in >> thread “main” java.lang. ClassNotFoundException: samza_config.job. >> JobConfigFactory, compiling:(/private/var/ folders/yt/ >> ch58t4q565g3vjvyj5wjfkt00000gp /T/form-init23293643677066296. clj:1:123) at >> clojure.lang.Compiler.load( Compiler.java:7391) at clojure.lang.Compiler. >> loadFile(Compiler.java:7317) at clojure.main$load_script. >> invokeStatic(main.clj:275) at clojure.main$init_opt. >> invokeStatic(main.clj:277) at clojure.main$init_opt.invoke( main.clj:277) >> at clojure.main$initialize. invokeStatic(main.clj:308) at >> clojure.main$null_opt. invokeStatic(main.clj:342) at >> clojure.main$null_opt.invoke( main.clj:339) at clojure.main$main. >> invokeStatic(main.clj:421) at clojure.main$main.doInvoke( main.clj:384) at >> clojure.lang.RestFn.invoke( RestFn.java:421) at >> clojure.lang.Var.invoke(Var. java:383) at clojure.lang.AFn. >> applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var. java:700) at >> clojure.main.main(main.java: 37) Caused by: java.lang. >> ClassNotFoundException: samza_config.job. JobConfigFactory at >> java.net.URLClassLoader. findClass(URLClassLoader.java: 381) at >> java.lang.ClassLoader. loadClass(ClassLoader.java: 424) at >> sun.misc.Launcher$ AppClassLoader.loadClass( Launcher.java:331) at >> java.lang.ClassLoader. loadClass(ClassLoader.java: 357) at >> java.lang.Class.forName0( Native Method) at java.lang.Class.forName(Class. >> java:264) at org.apache.samza.util. CommandLine.loadConfig( >> CommandLine.scala:66) at org.apache.samza.job. JobRunner$.main(JobRunner. >> scala:65) at org.apache.samza.job. JobRunner.main(JobRunner. scala) at >> samza_config.job$_main. invokeStatic(job.clj:102) at >> samza_config.job$_main. doInvoke(job.clj:96) at clojure.lang.RestFn.invoke( >> RestFn.java:408) at clojure.lang.Var.invoke(Var. java:379) at >> user$eval63.invokeStatic(form- init23293643677066296.clj:1) at >> user$eval63.invoke(form- init23293643677066296.clj:1) at >> clojure.lang.Compiler.eval( Compiler.java:6927) at >> clojure.lang.Compiler.eval( Compiler.java:6917) at >> clojure.lang.Compiler.load( Compiler.java:7379) ... 14 more >> That repo also has similarly defined implementations of serde factories >> that >> serialize/deserialize avro messages and check them against the confluent >> schema >> registry. >> The biggest pain point IMO is the requirement to build a jar to run a job >> during >> development. I think it would be possible to define these classes in Java >> and >> have them call into a Clojure API but that's basically what I'm trying to >> avoid >> as I'm more likely to get the Java wrong. >> Thanks for taking the time to understand this request. >> Cheers, AndyAndy Chambers | Full-Stack Software Developer >> >> andy.chambers@fundingcircle. com | 707-205-6502 (m) >> >> >> >> >> 747 Front St, 4th Fl | San Francisco, CA 94111 >> >> Our Mission: T o build a better financial world >> >> >> >> >> >> >> Unless specifically indicated, this e-mail is not an offer to sell or a >> solicitation of any investment products or other financial product or >> service, >> an official confirmation of any transaction, or an official statement of >> Funding >> Circle USA. This e-mail is meant only for the intended recipient of this >> transmission, and >> contains trade secret and strictly confidential information belonging to >> the >> sender. It is unlawful for unauthorized individuals to review, use, copy, >> disclose, or disseminate confidential information. If you have received >> this >> e-mail in error, please notify the sender immediately by telephone at >> 857.285.1263 or by return email and promptly delete this message from >> your system. >> >> >> On 9 February 2016 at 22:46, Yi Pan < nickpa...@gmail.com > wrote: >> Hi, Andy, >> >> Forgive me for my ignorance on the topic of Clojure. Could you give some >> simple example that Config and Serde Factories are in "Clojure-land" and >> how would a customized ClassLoader help in this case? >> >> Thanks a lot! >> >> -Yi >> >> On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers < >> andy.chambers@fundingcircle. com > wrote: >> >>> That would help but if I understand it correctly, it would still be >>> necessary to implement things like Config and Serde Factories in Java. It >>> would be nice to stay in "Clojure-land" to implement those too. >>> >>> *Andy Chambers* | Full-Stack Software Developer >>> >>> * andy.chambers@fundingcircle. com < andy.chambers@fundingcircle. com >>> * | >>> 707-205-6502 (m) >>> >>> >>> 747 Front St, 4th Fl | San Francisco, CA 94111 >>> >>> *Our Mission: **T**o build a better financial world* >>> >>> >>> Unless specifically indicated, this e-mail is not an offer to sell or a >>> solicitation of any investment products or other financial product or >>> service, an official confirmation of any transaction, or an official >>> statement of Funding Circle USA. This e-mail is meant only for the >>> intended recipient of this transmission, and contains trade secret and >>> strictly confidential information belonging to the sender. It is unlawful >>> for unauthorized individuals to review, use, copy, disclose, or >> disseminate >>> confidential information. If you have received this e-mail in error, >> please >>> notify the sender immediately by telephone at 857.285.1263 or by return >>> email and promptly delete this message from your system. >>> >>>> On 9 February 2016 at 22:25, Yi Pan < nickpa...@gmail.com > wrote: >>>> >>>> Hi, Andy, >>>> >>>> I think that you are looking for the feature in SAMZA-697. Or are you >>>> looking for something even more specific? >>>> >>>> On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers < >>>> andy.chambers@fundingcircle. com > wrote: >>>> >>>>> Hey Folks, >>>>> I'm trying to build some tooling to make writing jobs in Clojure a >>> little >>>>> bit >>>>> more interactive. One feature that I think would help a lot is to >> allow >>>>> specification of a ClassLoader that is able to find classes defined >>>>> dynamically >>>>> in Clojure. >>>>> Would you consider a feature/patch that adds a config parameter for >>> this >>>>> that >>>>> was respected everywhere Class/forName is called? >>>>> Is there anything else I should consider? >>>>> Thanks, >>>>> Andy >>>>> Andy Chambers | Full-Stack Software Developer >>>>> >>>>> andy.chambers@fundingcircle. com | 707-205-6502 (m) >>>>> >>>>> >>>>> >>>>> >>>>> 747 Front St, 4th Fl | San Francisco, CA 94111 >>>>> >>>>> Our Mission: T o build a better financial world >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Unless specifically indicated, this e-mail is not an offer to sell >> or a >>>>> solicitation of any investment products or other financial product or >>>>> service, >>>>> an official confirmation of any transaction, or an official statement >>> of >>>>> Funding >>>>> Circle USA. This e-mail is meant only for the intended recipient of >>> this >>>>> transmission, and >>>>> contains trade secret and strictly confidential information belonging >>> to >>>>> the >>>>> sender. It is unlawful for unauthorized individuals to review, use, >>> copy, >>>>> disclose, or disseminate confidential information. If you have >> received >>>>> this >>>>> e-mail in error, please notify the sender immediately by telephone at >>>>> 857.285.1263 or by return email and promptly delete this message >> from >>>>> your system. >>
