My co-worker has figured out how to do this without any changes required in samza. Andy Chambers | Full-Stack Software Developer
andy.chamb...@fundingcircle.com | 707-205-6502 (m) 747 Front St, 4th Fl | San Francisco, CA 94111 Our Mission: T o build a better financial world Unless specifically indicated, this e-mail is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Funding Circle USA. This e-mail is meant only for the intended recipient of this transmission, and contains trade secret and strictly confidential information belonging to the sender. It is unlawful for unauthorized individuals to review, use, copy, disclose, or disseminate confidential information. If you have received this e-mail in error, please notify the sender immediately by telephone at 857.285.1263 or by return email and promptly delete this message from your system. On 9 February 2016 at 23:47, Andy Chambers < andy.chamb...@fundingcircle.com > wrote: No Problem. I'm new to samza so it is entirely possible there's already a way to do this that I'm missing. What I'm aiming for, is to implement the features described in the README of my project https://github.com/ cddr/samza-config . For discussion purposes, I'll copy the example here... (ns example.word-counter (:require [samza-config.core :refer [stateful-task key-value-store]])) (defn count-words [store sentence output] (doseq [word (split sentence)] (update-in store word inc) (output :word-count {:word word :count (get store word)}))) (defjob word-counter {:inputs [(topic “words”)] :outputs [(topic “word-counts”)] :storage (key-value-store :word-counts) :task (stateful-task (fn [store input output] (count-words store input output)))}) I thought the easiest way to implement this would be for defjob to setup some metadata that points to implementations of StreamTask/InitableTask and have a custom ConfigFactory that can return the config for a named job. Here is the (in development) implementation of ConfigFactory (defrecord JobConfigFactory [] ConfigFactory (getConfig [this uri] (let [job (find-job uri)] (println “found job:” job) job))) Unfortunately when I run the job runner, passing the name of this class as the --config-factory, I get the following stacktrace. I believe this is because Clojure uses a special “DynamicClassLoader” to find these classes. That is what I'd set the classloader to if there was the option to do so. $ lein do run -m samza-config.job example.jobs.hello-world Running samza job: notifications.jobs.send-email Config Factory: samza_config.job. JobConfigFactory Class Loader: #object[sun.misc.Launcher$ AppClassLoader 0x4aa298b7 sun.misc.Launcher$ AppClassLoader@4aa298b7] Exception in thread “main” java.lang. ClassNotFoundException: samza_config.job. JobConfigFactory, compiling:(/private/var/ folders/yt/ ch58t4q565g3vjvyj5wjfkt00000gp /T/form-init23293643677066296. clj:1:123) at clojure.lang.Compiler.load( Compiler.java:7391) at clojure.lang.Compiler. loadFile(Compiler.java:7317) at clojure.main$load_script. invokeStatic(main.clj:275) at clojure.main$init_opt. invokeStatic(main.clj:277) at clojure.main$init_opt.invoke( main.clj:277) at clojure.main$initialize. invokeStatic(main.clj:308) at clojure.main$null_opt. invokeStatic(main.clj:342) at clojure.main$null_opt.invoke( main.clj:339) at clojure.main$main. invokeStatic(main.clj:421) at clojure.main$main.doInvoke( main.clj:384) at clojure.lang.RestFn.invoke( RestFn.java:421) at clojure.lang.Var.invoke(Var. java:383) at clojure.lang.AFn. applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var. java:700) at clojure.main.main(main.java: 37) Caused by: java.lang. ClassNotFoundException: samza_config.job. JobConfigFactory at java.net.URLClassLoader. findClass(URLClassLoader.java: 381) at java.lang.ClassLoader. loadClass(ClassLoader.java: 424) at sun.misc.Launcher$ AppClassLoader.loadClass( Launcher.java:331) at java.lang.ClassLoader. loadClass(ClassLoader.java: 357) at java.lang.Class.forName0( Native Method) at java.lang.Class.forName(Class. java:264) at org.apache.samza.util. CommandLine.loadConfig( CommandLine.scala:66) at org.apache.samza.job. JobRunner$.main(JobRunner. scala:65) at org.apache.samza.job. JobRunner.main(JobRunner. scala) at samza_config.job$_main. invokeStatic(job.clj:102) at samza_config.job$_main. doInvoke(job.clj:96) at clojure.lang.RestFn.invoke( RestFn.java:408) at clojure.lang.Var.invoke(Var. java:379) at user$eval63.invokeStatic(form- init23293643677066296.clj:1) at user$eval63.invoke(form- init23293643677066296.clj:1) at clojure.lang.Compiler.eval( Compiler.java:6927) at clojure.lang.Compiler.eval( Compiler.java:6917) at clojure.lang.Compiler.load( Compiler.java:7379) ... 14 more That repo also has similarly defined implementations of serde factories that serialize/deserialize avro messages and check them against the confluent schema registry. The biggest pain point IMO is the requirement to build a jar to run a job during development. I think it would be possible to define these classes in Java and have them call into a Clojure API but that's basically what I'm trying to avoid as I'm more likely to get the Java wrong. Thanks for taking the time to understand this request. Cheers, AndyAndy Chambers | Full-Stack Software Developer andy.chambers@fundingcircle. com | 707-205-6502 (m) 747 Front St, 4th Fl | San Francisco, CA 94111 Our Mission: T o build a better financial world Unless specifically indicated, this e-mail is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Funding Circle USA. This e-mail is meant only for the intended recipient of this transmission, and contains trade secret and strictly confidential information belonging to the sender. It is unlawful for unauthorized individuals to review, use, copy, disclose, or disseminate confidential information. If you have received this e-mail in error, please notify the sender immediately by telephone at 857.285.1263 or by return email and promptly delete this message from your system. On 9 February 2016 at 22:46, Yi Pan < nickpa...@gmail.com > wrote: Hi, Andy, Forgive me for my ignorance on the topic of Clojure. Could you give some simple example that Config and Serde Factories are in "Clojure-land" and how would a customized ClassLoader help in this case? Thanks a lot! -Yi On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers < andy.chambers@fundingcircle. com > wrote: > That would help but if I understand it correctly, it would still be > necessary to implement things like Config and Serde Factories in Java. It > would be nice to stay in "Clojure-land" to implement those too. > > *Andy Chambers* | Full-Stack Software Developer > > * andy.chambers@fundingcircle. com < andy.chambers@fundingcircle. com >* | > 707-205-6502 (m) > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > *Our Mission: **T**o build a better financial world* > > > Unless specifically indicated, this e-mail is not an offer to sell or a > solicitation of any investment products or other financial product or > service, an official confirmation of any transaction, or an official > statement of Funding Circle USA. This e-mail is meant only for the > intended recipient of this transmission, and contains trade secret and > strictly confidential information belonging to the sender. It is unlawful > for unauthorized individuals to review, use, copy, disclose, or disseminate > confidential information. If you have received this e-mail in error, please > notify the sender immediately by telephone at 857.285.1263 or by return > email and promptly delete this message from your system. > > On 9 February 2016 at 22:25, Yi Pan < nickpa...@gmail.com > wrote: > > > Hi, Andy, > > > > I think that you are looking for the feature in SAMZA-697. Or are you > > looking for something even more specific? > > > > On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers < > > andy.chambers@fundingcircle. com > wrote: > > > > > Hey Folks, > > > I'm trying to build some tooling to make writing jobs in Clojure a > little > > > bit > > > more interactive. One feature that I think would help a lot is to allow > > > specification of a ClassLoader that is able to find classes defined > > > dynamically > > > in Clojure. > > > Would you consider a feature/patch that adds a config parameter for > this > > > that > > > was respected everywhere Class/forName is called? > > > Is there anything else I should consider? > > > Thanks, > > > Andy > > > Andy Chambers | Full-Stack Software Developer > > > > > > andy.chambers@fundingcircle. com | 707-205-6502 (m) > > > > > > > > > > > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > > > > > Our Mission: T o build a better financial world > > > > > > > > > > > > > > > > > > > > > Unless specifically indicated, this e-mail is not an offer to sell or a > > > solicitation of any investment products or other financial product or > > > service, > > > an official confirmation of any transaction, or an official statement > of > > > Funding > > > Circle USA. This e-mail is meant only for the intended recipient of > this > > > transmission, and > > > contains trade secret and strictly confidential information belonging > to > > > the > > > sender. It is unlawful for unauthorized individuals to review, use, > copy, > > > disclose, or disseminate confidential information. If you have received > > > this > > > e-mail in error, please notify the sender immediately by telephone at > > > 857.285.1263 or by return email and promptly delete this message from > > > your system. > > >
