Hi, Andy, Great! Looking forward to your uploads!
-Yi On Wed, Feb 10, 2016 at 6:02 PM, Andy Chambers < andy.chamb...@fundingcircle.com> wrote: > We will release some open source when it's ready but the basic idea is to > define a single task that implements all the interfaces, and have that > delegate to clojure handlers. > On Feb 10, 2016 4:39 PM, "Jacob Maes" <jacob.m...@gmail.com> wrote: > > > Hey Andy, > > > > That's good news! Do you mind sharing the revelation for any other Samza > > users in Clojure-land? > > > > -Jake > > > > On Wed, Feb 10, 2016 at 4:07 PM, Andy Chambers < > > andy.chamb...@fundingcircle.com> wrote: > > > > > My co-worker has figured out how to do this without any changes > required > > in > > > samza. > > > Andy Chambers | Full-Stack Software Developer > > > > > > andy.chamb...@fundingcircle.com | 707-205-6502 (m) > > > > > > > > > > > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > > > > > Our Mission: T o build a better financial world > > > > > > > > > > > > > > > > > > > > > Unless specifically indicated, this e-mail is not an offer to sell or a > > > solicitation of any investment products or other financial product or > > > service, > > > an official confirmation of any transaction, or an official statement > of > > > Funding > > > Circle USA. This e-mail is meant only for the intended recipient of > this > > > transmission, and > > > contains trade secret and strictly confidential information belonging > to > > > the > > > sender. It is unlawful for unauthorized individuals to review, use, > copy, > > > disclose, or disseminate confidential information. If you have received > > > this > > > e-mail in error, please notify the sender immediately by telephone at > > > 857.285.1263 or by return email and promptly delete this message from > > > your system. > > > > > > > > > > > > On 9 February 2016 at 23:47, Andy Chambers < > > > andy.chamb...@fundingcircle.com > wrote: > > > No Problem. I'm new to samza so it is entirely possible there's > already a > > > way to > > > do this that I'm missing. > > > What I'm aiming for, is to implement the features described in the > README > > > of my > > > project https://github.com/ cddr/samza-config . For discussion > purposes, > > > I'll copy the example here... > > > (ns example.word-counter > > > (:require > > > [samza-config.core :refer [stateful-task key-value-store]])) > > > > > > (defn count-words [store sentence output] > > > (doseq [word (split sentence)] > > > (update-in store word inc) > > > (output :word-count {:word word > > > :count (get store word)}))) > > > > > > (defjob word-counter > > > {:inputs [(topic “words”)] > > > :outputs [(topic “word-counts”)] > > > :storage (key-value-store :word-counts) > > > :task (stateful-task > > > (fn [store input output] > > > (count-words store input output)))}) > > > > > > I thought the easiest way to implement this would be for defjob to > setup > > > some > > > metadata that points to implementations of StreamTask/InitableTask and > > > have a > > > custom ConfigFactory that can return the config for a named job. > > > Here is the (in development) implementation of ConfigFactory > > > (defrecord JobConfigFactory [] ConfigFactory (getConfig [this uri] (let > > > [job (find-job uri)] (println “found job:” job) job))) > > > Unfortunately when I run the job runner, passing the name of this class > > as > > > the > > > --config-factory, I get the following stacktrace. I believe this is > > because > > > Clojure uses a special “DynamicClassLoader” to find these classes. That > > is > > > what > > > I'd set the classloader to if there was the option to do so. > > > $ lein do run -m samza-config.job example.jobs.hello-world Running > samza > > > job: notifications.jobs.send-email Config Factory: samza_config.job. > > > JobConfigFactory Class Loader: #object[sun.misc.Launcher$ > AppClassLoader > > > 0x4aa298b7 sun.misc.Launcher$ AppClassLoader@4aa298b7] Exception in > > > thread “main” java.lang. ClassNotFoundException: samza_config.job. > > > JobConfigFactory, compiling:(/private/var/ folders/yt/ > > > ch58t4q565g3vjvyj5wjfkt00000gp /T/form-init23293643677066296. > clj:1:123) > > at > > > clojure.lang.Compiler.load( Compiler.java:7391) at > clojure.lang.Compiler. > > > loadFile(Compiler.java:7317) at clojure.main$load_script. > > > invokeStatic(main.clj:275) at clojure.main$init_opt. > > > invokeStatic(main.clj:277) at clojure.main$init_opt.invoke( > main.clj:277) > > > at clojure.main$initialize. invokeStatic(main.clj:308) at > > > clojure.main$null_opt. invokeStatic(main.clj:342) at > > > clojure.main$null_opt.invoke( main.clj:339) at clojure.main$main. > > > invokeStatic(main.clj:421) at clojure.main$main.doInvoke( main.clj:384) > > at > > > clojure.lang.RestFn.invoke( RestFn.java:421) at > > > clojure.lang.Var.invoke(Var. java:383) at clojure.lang.AFn. > > > applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var. java:700) > at > > > clojure.main.main(main.java: 37) Caused by: java.lang. > > > ClassNotFoundException: samza_config.job. JobConfigFactory at > > > java.net.URLClassLoader. findClass(URLClassLoader.java: 381) at > > > java.lang.ClassLoader. loadClass(ClassLoader.java: 424) at > > > sun.misc.Launcher$ AppClassLoader.loadClass( Launcher.java:331) at > > > java.lang.ClassLoader. loadClass(ClassLoader.java: 357) at > > > java.lang.Class.forName0( Native Method) at > > java.lang.Class.forName(Class. > > > java:264) at org.apache.samza.util. CommandLine.loadConfig( > > > CommandLine.scala:66) at org.apache.samza.job. > JobRunner$.main(JobRunner. > > > scala:65) at org.apache.samza.job. JobRunner.main(JobRunner. scala) at > > > samza_config.job$_main. invokeStatic(job.clj:102) at > > > samza_config.job$_main. doInvoke(job.clj:96) at > > clojure.lang.RestFn.invoke( > > > RestFn.java:408) at clojure.lang.Var.invoke(Var. java:379) at > > > user$eval63.invokeStatic(form- init23293643677066296.clj:1) at > > > user$eval63.invoke(form- init23293643677066296.clj:1) at > > > clojure.lang.Compiler.eval( Compiler.java:6927) at > > > clojure.lang.Compiler.eval( Compiler.java:6917) at > > > clojure.lang.Compiler.load( Compiler.java:7379) ... 14 more > > > That repo also has similarly defined implementations of serde factories > > > that > > > serialize/deserialize avro messages and check them against the > confluent > > > schema > > > registry. > > > The biggest pain point IMO is the requirement to build a jar to run a > job > > > during > > > development. I think it would be possible to define these classes in > Java > > > and > > > have them call into a Clojure API but that's basically what I'm trying > to > > > avoid > > > as I'm more likely to get the Java wrong. > > > Thanks for taking the time to understand this request. > > > Cheers, AndyAndy Chambers | Full-Stack Software Developer > > > > > > andy.chambers@fundingcircle. com | 707-205-6502 (m) > > > > > > > > > > > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > > > > > Our Mission: T o build a better financial world > > > > > > > > > > > > > > > > > > > > > Unless specifically indicated, this e-mail is not an offer to sell or a > > > solicitation of any investment products or other financial product or > > > service, > > > an official confirmation of any transaction, or an official statement > of > > > Funding > > > Circle USA. This e-mail is meant only for the intended recipient of > this > > > transmission, and > > > contains trade secret and strictly confidential information belonging > to > > > the > > > sender. It is unlawful for unauthorized individuals to review, use, > copy, > > > disclose, or disseminate confidential information. If you have received > > > this > > > e-mail in error, please notify the sender immediately by telephone at > > > 857.285.1263 or by return email and promptly delete this message from > > > your system. > > > > > > > > > On 9 February 2016 at 22:46, Yi Pan < nickpa...@gmail.com > wrote: > > > Hi, Andy, > > > > > > Forgive me for my ignorance on the topic of Clojure. Could you give > some > > > simple example that Config and Serde Factories are in "Clojure-land" > and > > > how would a customized ClassLoader help in this case? > > > > > > Thanks a lot! > > > > > > -Yi > > > > > > On Tue, Feb 9, 2016 at 10:36 PM, Andy Chambers < > > > andy.chambers@fundingcircle. com > wrote: > > > > > > > That would help but if I understand it correctly, it would still be > > > > necessary to implement things like Config and Serde Factories in > Java. > > It > > > > would be nice to stay in "Clojure-land" to implement those too. > > > > > > > > *Andy Chambers* | Full-Stack Software Developer > > > > > > > > * andy.chambers@fundingcircle. com < andy.chambers@fundingcircle. > com > > > >* | > > > > 707-205-6502 (m) > > > > > > > > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > > > > > > > *Our Mission: **T**o build a better financial world* > > > > > > > > > > > > Unless specifically indicated, this e-mail is not an offer to sell > or a > > > > solicitation of any investment products or other financial product or > > > > service, an official confirmation of any transaction, or an official > > > > statement of Funding Circle USA. This e-mail is meant only for the > > > > intended recipient of this transmission, and contains trade secret > and > > > > strictly confidential information belonging to the sender. It is > > unlawful > > > > for unauthorized individuals to review, use, copy, disclose, or > > > disseminate > > > > confidential information. If you have received this e-mail in error, > > > please > > > > notify the sender immediately by telephone at 857.285.1263 or by > > return > > > > email and promptly delete this message from your system. > > > > > > > > On 9 February 2016 at 22:25, Yi Pan < nickpa...@gmail.com > wrote: > > > > > > > > > Hi, Andy, > > > > > > > > > > I think that you are looking for the feature in SAMZA-697. Or are > you > > > > > looking for something even more specific? > > > > > > > > > > On Tue, Feb 9, 2016 at 10:22 PM, Andy Chambers < > > > > > andy.chambers@fundingcircle. com > wrote: > > > > > > > > > > > Hey Folks, > > > > > > I'm trying to build some tooling to make writing jobs in Clojure > a > > > > little > > > > > > bit > > > > > > more interactive. One feature that I think would help a lot is to > > > allow > > > > > > specification of a ClassLoader that is able to find classes > defined > > > > > > dynamically > > > > > > in Clojure. > > > > > > Would you consider a feature/patch that adds a config parameter > for > > > > this > > > > > > that > > > > > > was respected everywhere Class/forName is called? > > > > > > Is there anything else I should consider? > > > > > > Thanks, > > > > > > Andy > > > > > > Andy Chambers | Full-Stack Software Developer > > > > > > > > > > > > andy.chambers@fundingcircle. com | 707-205-6502 (m) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 747 Front St, 4th Fl | San Francisco, CA 94111 > > > > > > > > > > > > Our Mission: T o build a better financial world > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Unless specifically indicated, this e-mail is not an offer to > sell > > > or a > > > > > > solicitation of any investment products or other financial > product > > or > > > > > > service, > > > > > > an official confirmation of any transaction, or an official > > statement > > > > of > > > > > > Funding > > > > > > Circle USA. This e-mail is meant only for the intended recipient > of > > > > this > > > > > > transmission, and > > > > > > contains trade secret and strictly confidential information > > belonging > > > > to > > > > > > the > > > > > > sender. It is unlawful for unauthorized individuals to review, > use, > > > > copy, > > > > > > disclose, or disseminate confidential information. If you have > > > received > > > > > > this > > > > > > e-mail in error, please notify the sender immediately by > telephone > > at > > > > > > 857.285.1263 or by return email and promptly delete this message > > > from > > > > > > your system. > > > > > > > > > > > > > > >
