Hey Andy, That's good news! Do you mind sharing the revelation for any other Samza users in Clojure-land?
-Jake On Wed, Feb 10, 2016 at 4:07 PM, Andy Chambers < andy.chamb...@fundingcircle.com> wrote: > My co-worker has figured out how to do this without any changes required in > samza. > Andy Chambers | Full-Stack Software Developer > > andy.chamb...@fundingcircle.com | 707-205-6502 (m) > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > Our Mission: T o build a better financial world > > > > > > > Unless specifically indicated, this e-mail is not an offer to sell or a > solicitation of any investment products or other financial product or > service, > an official confirmation of any transaction, or an official statement of > Funding > Circle USA. This e-mail is meant only for the intended recipient of this > transmission, and > contains trade secret and strictly confidential information belonging to > the > sender. It is unlawful for unauthorized individuals to review, use, copy, > disclose, or disseminate confidential information. If you have received > this > e-mail in error, please notify the sender immediately by telephone at > 857.285.1263 or by return email and promptly delete this message from > your system. > > > > On 9 February 2016 at 23:47, Andy Chambers < > andy.chamb...@fundingcircle.com > wrote: > No Problem. I'm new to samza so it is entirely possible there's already a > way to > do this that I'm missing. > What I'm aiming for, is to implement the features described in the README > of my > project https://github.com/ cddr/samza-config . For discussion purposes, > I'll copy the example here... > (ns example.word-counter > (:require > [samza-config.core :refer [stateful-task key-value-store]])) > > (defn count-words [store sentence output] > (doseq [word (split sentence)] > (update-in store word inc) > (output :word-count {:word word > :count (get store word)}))) > > (defjob word-counter > {:inputs [(topic “words”)] > :outputs [(topic “word-counts”)] > :storage (key-value-store :word-counts) > :task (stateful-task > (fn [store input output] > (count-words store input output)))}) > > I thought the easiest way to implement this would be for defjob to setup > some > metadata that points to implementations of StreamTask/InitableTask and > have a > custom ConfigFactory that can return the config for a named job. > Here is the (in development) implementation of ConfigFactory > (defrecord JobConfigFactory [] ConfigFactory (getConfig [this uri] (let > [job (find-job uri)] (println “found job:” job) job))) > Unfortunately when I run the job runner, passing the name of this class as > the > --config-factory, I get the following stacktrace. I believe this is because > Clojure uses a special “DynamicClassLoader” to find these classes. That is > what > I'd set the classloader to if there was the option to do so. > $ lein do run -m samza-config.job example.jobs.hello-world Running samza > job: notifications.jobs.send-email Config Factory: samza_config.job. > JobConfigFactory Class Loader: #object[sun.misc.Launcher$ AppClassLoader > 0x4aa298b7 sun.misc.Launcher$ AppClassLoader@4aa298b7] Exception in > thread “main” java.lang. ClassNotFoundException: samza_config.job. > JobConfigFactory, compiling:(/private/var/ folders/yt/ > ch58t4q565g3vjvyj5wjfkt00000gp /T/form-init23293643677066296. clj:1:123) at > clojure.lang.Compiler.load( Compiler.java:7391) at clojure.lang.Compiler. > loadFile(Compiler.java:7317) at clojure.main$load_script. > invokeStatic(main.clj:275) at clojure.main$init_opt. > invokeStatic(main.clj:277) at clojure.main$init_opt.invoke( main.clj:277) > at clojure.main$initialize. invokeStatic(main.clj:308) at > clojure.main$null_opt. invokeStatic(main.clj:342) at > clojure.main$null_opt.invoke( main.clj:339) at clojure.main$main. > invokeStatic(main.clj:421) at clojure.main$main.doInvoke( main.clj:384) at > clojure.lang.RestFn.invoke( RestFn.java:421) at > clojure.lang.Var.invoke(Var. java:383) at clojure.lang.AFn. > applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var. java:700) at > clojure.main.main(main.java: 37) Caused by: java.lang. > ClassNotFoundException: samza_config.job. JobConfigFactory at > java.net.URLClassLoader. findClass(URLClassLoader.java: 381) at > java.lang.ClassLoader. loadClass(ClassLoader.java: 424) at > sun.misc.Launcher$ AppClassLoader.loadClass( Launcher.java:331) at > java.lang.ClassLoader. loadClass(ClassLoader.java: 357) at > java.lang.Class.forName0( Native Method) at java.lang.Class.forName(Class. > java:264) at org.apache.samza.util. CommandLine.loadConfig( > CommandLine.scala:66) at org.apache.samza.job. JobRunner$.main(JobRunner. > scala:65) at org.apache.samza.job. JobRunner.main(JobRunner. scala) at > samza_config.job$_main. invokeStatic(job.clj:102) at > samza_config.job$_main. doInvoke(job.clj:96) at clojure.lang.RestFn.invoke( > RestFn.java:408) at clojure.lang.Var.invoke(Var. java:379) at > user$eval63.invokeStatic(form- init23293643677066296.clj:1) at > user$eval63.invoke(form- init23293643677066296.clj:1) at > clojure.lang.Compiler.eval( Compiler.java:6927) at > clojure.lang.Compiler.eval( Compiler.java:6917) at > clojure.lang.Compiler.load( Compiler.java:7379) ... 14 more > That repo also has similarly defined implementations of serde factories > that > serialize/deserialize avro messages and check them against the confluent > schema > registry. > The biggest pain point IMO is the requirement to build a jar to run a job > during > development. I think it would be possible to define these classes in Java > and > have them call into a Clojure API but that's basically what I'm trying to > avoid > as I'm more likely to get the Java wrong. > Thanks for taking the time to understand this request. > Cheers, AndyAndy Chambers | Full-Stack Software Developer > > andy.chambers@fundingcircle. com | 707-205-6502 (m) > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > Our Mission: T o build a better financial world > > > > > > > Unless specifically indicated, this e-mail is not an offer to sell or a > solicitation of any investment products or other financial product or > service, > an official confirmation of any transaction, or an official statement of > Funding > Circle USA. This e-mail is meant only for the intended recipient of this > transmission, and > contains trade secret and strictly confidential information belonging to > the > sender. It is unlawful for unauthorized individuals to review, use, copy, > disclose, or disseminate confidential information. If you have received > this > e-mail in error, please notify the sender immediately by telephone at > 857.285.1263 or by return email and promptly delete this message from > your system. > > > On 9 February 2016 at 22:46, Yi Pan < nickpa...@gmail.com > wrote: > Hi, Andy, > > Forgive me for my ignorance on the topic of Clojure. Could you give some > simple example that Config and Serde Factories are in "Clojure-land" and > how would a customized ClassLoader help in this case? > > Thanks a lot! > > -Yi > > On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers < > andy.chambers@fundingcircle. com > wrote: > > > That would help but if I understand it correctly, it would still be > > necessary to implement things like Config and Serde Factories in Java. It > > would be nice to stay in "Clojure-land" to implement those too. > > > > *Andy Chambers* | Full-Stack Software Developer > > > > * andy.chambers@fundingcircle. com < andy.chambers@fundingcircle. com > >* | > > 707-205-6502 (m) > > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > > > *Our Mission: **T**o build a better financial world* > > > > > > Unless specifically indicated, this e-mail is not an offer to sell or a > > solicitation of any investment products or other financial product or > > service, an official confirmation of any transaction, or an official > > statement of Funding Circle USA. This e-mail is meant only for the > > intended recipient of this transmission, and contains trade secret and > > strictly confidential information belonging to the sender. It is unlawful > > for unauthorized individuals to review, use, copy, disclose, or > disseminate > > confidential information. If you have received this e-mail in error, > please > > notify the sender immediately by telephone at 857.285.1263 or by return > > email and promptly delete this message from your system. > > > > On 9 February 2016 at 22:25, Yi Pan < nickpa...@gmail.com > wrote: > > > > > Hi, Andy, > > > > > > I think that you are looking for the feature in SAMZA-697. Or are you > > > looking for something even more specific? > > > > > > On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers < > > > andy.chambers@fundingcircle. com > wrote: > > > > > > > Hey Folks, > > > > I'm trying to build some tooling to make writing jobs in Clojure a > > little > > > > bit > > > > more interactive. One feature that I think would help a lot is to > allow > > > > specification of a ClassLoader that is able to find classes defined > > > > dynamically > > > > in Clojure. > > > > Would you consider a feature/patch that adds a config parameter for > > this > > > > that > > > > was respected everywhere Class/forName is called? > > > > Is there anything else I should consider? > > > > Thanks, > > > > Andy > > > > Andy Chambers | Full-Stack Software Developer > > > > > > > > andy.chambers@fundingcircle. com | 707-205-6502 (m) > > > > > > > > > > > > > > > > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > > > > > > > Our Mission: T o build a better financial world > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unless specifically indicated, this e-mail is not an offer to sell > or a > > > > solicitation of any investment products or other financial product or > > > > service, > > > > an official confirmation of any transaction, or an official statement > > of > > > > Funding > > > > Circle USA. This e-mail is meant only for the intended recipient of > > this > > > > transmission, and > > > > contains trade secret and strictly confidential information belonging > > to > > > > the > > > > sender. It is unlawful for unauthorized individuals to review, use, > > copy, > > > > disclose, or disseminate confidential information. If you have > received > > > > this > > > > e-mail in error, please notify the sender immediately by telephone at > > > > 857.285.1263 or by return email and promptly delete this message > from > > > > your system. > > > > > >
