Attempting to call unbound function in Hadoop (cluster mode) runs fine on single node.

[cloje]

I have a project that I'm trying to run on Hadoop in cluster mode. 

I have written and executed the code on my local machine and on a single 
node on Hadoop. But when I try to executed the same jar file in cluster 
mode with 5 executors, it bombs at a function and says, Attempting to call 
unbound fn, '#my-func-name.

Has anyone faced the same kind?
Was there a work around?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Attempting to call unbound function in Hadoop (cluster mode) runs fine on single node.

[Derek Troy-West]

Bit of a long-shot since my example is related to Storm not Hadoop, and my 
experience was five years ago, but this blog post might help:

http://derek.troywest.com/articles/trident-in-clojure

and this related ticket: https://github.com/sorenmacbeth/marceline/issues/9

I could run a Storm topology locally, or on a single node, but not when 
deployed to a Storm cluster. It came down the the way Storm chops up 
topologies and serialises them across the network to run on remote nodes. 
There a few different ways of implementing interfaces in Clojure, and (at 
the time at least) they had different semantics in the 
serialise-reconstruct-and-run world.

This may also be helpful, it has been fixed since I wrote the blog 
post: http://dev.clojure.org/jira/browse/CLJ-1208

Good luck!

On Saturday, 5 January 2019 08:35:22 UTC+11, cloje wrote:
>
> I have a project that I'm trying to run on Hadoop in cluster mode. 
>
> I have written and executed the code on my local machine and on a single 
> node on Hadoop. But when I try to executed the same jar file in cluster 
> mode with 5 executors, it bombs at a function and says, Attempting to call 
> unbound fn, '#my-func-name.
>
> Has anyone faced the same kind?
> Was there a work around?
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] Calfpath (v0.7.1), a fast and flexible Ring request router

[Shantanu Kumar]

Hi,

(Cross-posted on Clojure and Ring mailing lists.)

Happy new year to all!

I am pleased to announce Calfpath, a fast Ring-based routing library that 
supports flexible, à la carte request matching. It supports both 
macro-based and data-driven routing.

https://github.com/kumarshantanu/calfpath

At SAP Concur we have been using this library for over 3 years in 
production on REST(ish) API servers. During early 2015 when we were using 
Compojure, we found it was causing 4% of the total internal latency. That 
is when we switched to Calfpath - roughly an order of magnitude faster. The 
benchmarking code is included in the repo - however, you should probably 
test against your own use-case to determine suitability.

The API has matured quite a bit over time and is now more stable than ever 
before. Among downsides, as of now Calfpath is neither bi-directional, nor 
ClojureScript ready.

I would love to receive your feedback and answer any questions. Please let 
me know what you think.


Shantanu

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How does Executors/newScheduledThreadPool know how or where to parallelize work?

[Alan Thompson]

If you haven't seen it before, you can use the excellent Claypoole library
 for many parallel
scheduling tasks.
Alan

On Wed, Jan 2, 2019 at 1:07 PM  wrote:

> I guess this is more of a JVM question than a Clojure question, unless
> Clojure exerts any special magic here. I'm open to a more Clojure approach
> than what I have now.
>
> Someone suggested I use Executors/newScheduledThreadPool for some
> recurring work, so I set it up like this:
>
> (def scheduler-aggregate
>   (Executors/newScheduledThreadPool 32))
>
> at the start I call:
>
>   (.scheduleAtFixedRate  scheduler-aggregate ^Runnable (cycle-aggregate
> to-database-queue) 1 30 TimeUnit/MINUTES)
>
> Aside from a try/catch block (which I just removed to simplify this
> example) the inner function looks like this:
>
> (defn- cycle-aggregate
>   [to-database-queue]
>   (fn []
>  (let [
>transcripts (query @global-database-connection {:item-type
> :transcript :processed { operators/$exists false }})
>]
>(doseq [x transcripts]
>  (aggregate-words x)
>  (set-transcript-processed  @global-database-connection x)))
>
> The function (aggregate-words) counts up a bunch of words, doing some prep
> work for a later NLP engine, and then there is this line:
>
> (log "The end of aggregate-words.")))
>
> The whole process takes about 5 minutes to run, about 300 seconds. I watch
> the database and I see the number of new records increase. About every 10
> seconds I see these words appear in the logs:
>
> "The end of aggregate-words."
>
> At the end of 5 minutes, these words have appeared 30 times, one for each
> of the transcripts I'm importing.
>
> This seems like I've done something wrong? Since the words "The end of
> aggregate-words."
> appear at roughly equal intervals, and the transcripts are all about the
> same size, it seems that all of the transcripts are being handled on one
> thread. After all, if the 30 transcripts were handled on 30 threads, I'd
> expect the 30 calls to aggregate-words would all end at roughly the same
> time, instead of sequentially.
>
> What else do I need to do to parallelize this work? If I call (future)
> inside of aggregate-words, would the new thread come from the pool? Is
> there a way I can call aggregate-words and make sure it runs on its own
> thread from the pool?
>
>
>
>
>
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Calfpath (v0.7.1), a fast and flexible Ring request router

[Alan Thompson]

Hi have been very happy using Pedestal for routing 
chores.  How does this compare to Pedestal?
Alan


On Fri, Jan 4, 2019 at 9:50 PM Shantanu Kumar 
wrote:

> Hi,
>
> (Cross-posted on Clojure and Ring mailing lists.)
>
> Happy new year to all!
>
> I am pleased to announce Calfpath, a fast Ring-based routing library that
> supports flexible, à la carte request matching. It supports both
> macro-based and data-driven routing.
>
> https://github.com/kumarshantanu/calfpath
>
> At SAP Concur we have been using this library for over 3 years in
> production on REST(ish) API servers. During early 2015 when we were using
> Compojure, we found it was causing 4% of the total internal latency. That
> is when we switched to Calfpath - roughly an order of magnitude faster. The
> benchmarking code is included in the repo - however, you should probably
> test against your own use-case to determine suitability.
>
> The API has matured quite a bit over time and is now more stable than ever
> before. Among downsides, as of now Calfpath is neither bi-directional, nor
> ClojureScript ready.
>
> I would love to receive your feedback and answer any questions. Please let
> me know what you think.
>
>
> Shantanu
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] Calfpath (v0.7.1), a fast and flexible Ring request router

[Shantanu Kumar]

Hi Alan,

On Saturday, 5 January 2019 12:25:36 UTC+5:30, Alan Thompson wrote:
>
> Hi have been very happy using Pedestal for routing  
> chores.  How does this compare to Pedestal?
>

I haven't benchmarked Calfpath against Pedestal, but Reitit-ring is 
supposed to be faster than Pedestal as 
per https://github.com/metosin/reitit/blob/master/doc/performance.md and in 
my benchmarks Calfpath outperforms Reitit-ring in most use cases, so I 
guess Calfpath should be much faster than Pedestal. But again, benchmarks 
are tricky, so you should measure the candidate libraries for your actual 
use cases.


Shantanu
 

> Alan
>
>
> On Fri, Jan 4, 2019 at 9:50 PM Shantanu Kumar  > wrote:
>
>> Hi,
>>
>> (Cross-posted on Clojure and Ring mailing lists.)
>>
>> Happy new year to all!
>>
>> I am pleased to announce Calfpath, a fast Ring-based routing library that 
>> supports flexible, à la carte request matching. It supports both 
>> macro-based and data-driven routing.
>>
>> https://github.com/kumarshantanu/calfpath
>>
>> At SAP Concur we have been using this library for over 3 years in 
>> production on REST(ish) API servers. During early 2015 when we were using 
>> Compojure, we found it was causing 4% of the total internal latency. That 
>> is when we switched to Calfpath - roughly an order of magnitude faster. The 
>> benchmarking code is included in the repo - however, you should probably 
>> test against your own use-case to determine suitability.
>>
>> The API has matured quite a bit over time and is now more stable than 
>> ever before. Among downsides, as of now Calfpath is neither bi-directional, 
>> nor ClojureScript ready.
>>
>> I would love to receive your feedback and answer any questions. Please 
>> let me know what you think.
>>
>>
>> Shantanu
>>
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com 
>> 
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com 
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.