One issue we've been having is that we are using Datomic in a non standard way (I think). We're touching all the data all at once for some resolution steps and Datomic wasn't necessarily made for that (at least that has been my impression). I've been trying to understand the Datomic query system better so I know the limitations of the system and how to work with them. ECHELON takes a few hours to run right now and we're looking to increase the amount of data we collect and link by an order of magnitude or two in the next year. It's fine to let ECHELON run for a few days (the batch resolution step only needs to be run once, the streaming approach should always be fast) but it would be nice if we knew for sure that we had found the upper limit for how fast ECHELON can resolve things. If anybody has resources about how to abuse Datomic in this manner correctly, I'd be really happy if you sent them my way. I'll see if I can't push for a technical right up soon so I can provide more information about how ECHELON actually works. -Zack
On Wednesday, September 17, 2014 2:47:10 AM UTC-5, Joshua Ballanco wrote: > > Very interesting project! (and congrats on joining SunlightLabs) > > As someone who has, on occasion, contributed to various SunlightLabs > foundation projects, my only question would be: what do you need help with? > > I understand completely if this project is not at a point where you’re > looking for outside contribution yet, but if there’s any area you’d like > help with, I’d be interested to throw at least a few weekend hours in your > general direction. > > Cheers, > > Josh > > > On Tuesday, September 16, 2014 at 20:48, Zack Maril wrote: > > > This might be of interest to the Clojure/Datomic community: > > > > > http://sunlightfoundation.com/blog/2014/09/16/wrangling-messy-political-data-into-usable-information/ > > > https://github.com/sunlightlabs/echelon > > > > I'm part of the Influence Explorer team at the Sunlight Foundation. > We're building a system with Datomic and Instaparse to disentangle and > analyze the web of relationships between lobbyists, special interest > groups, and legislators. It's been surprisingly successful so far, as in it > works as well as we hoped it would when we first started building it. > Datomic has been a pleasure to use and Instaparse has been excellent so > far. The above link is a results oriented blog post about what we did and > why we did it. We haven't written up the technical details yet as they are > still evolving (we're attempting to move from the current static batch > processing system to hopefully a streaming approach in the near future). > But, if you have any questions about the project I'd be happy to answer > them. > > -Zack > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Clojure" group. > > To post to this group, send email to clo...@googlegroups.com > <javascript:> (mailto:clo...@googlegroups.com <javascript:>) > > Note that posts from new members are moderated - please be patient with > your first post. > > To unsubscribe from this group, send email to > > clojure+u...@googlegroups.com <javascript:> (mailto: > clojure+u...@googlegroups.com <javascript:>) > > For more options, visit this group at > > http://groups.google.com/group/clojure?hl=en > > --- > > You received this message because you are subscribed to the Google > Groups "Clojure" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to clojure+u...@googlegroups.com <javascript:> (mailto: > clojure+u...@googlegroups.com <javascript:>). > > For more options, visit https://groups.google.com/d/optout. > > > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
