I did find this: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSPerformance.html
"Your performance can also be impacted if your application isn’t sending enough I/O requests. This can be monitored by looking at your volume’s queue length and I/O size. The queue length is the number of pending I/O requests from your application to your volume. For maximum consistency, HDD-backed volumes must maintain a queue length (rounded to the nearest whole number) of 4 or more when performing 1 MiB sequential I/O. " So, at least for now, I'll assume I'm dealing with some weird kind of AWS thing. But if anyone has seen this pattern (slow until you get a Socket timeout, then fast) because of something you did in your Clojure code, I would love to hear about it. On Thursday, October 5, 2017 at 2:31:10 PM UTC-4, lawrence...@gmail.com wrote: > > Konstantinov, yes, perhaps, though I can't think where. This is a small > app, about 950 lines of code. There is a limited number of places where I > can make such a mistake. I do aggregate a lot of data into an atom, and I'm > sure there is a lot of contention around the atom, but the Socket timeout > errors are surely some weird AWS or ElasticSearch thing. > > > > On Thursday, October 5, 2017 at 2:12:42 PM UTC-4, Lubomir Konstantinov > wrote: >> >> Holding to a head somewhere perhaps? >> >> On Thursday, 5 October 2017 19:32:11 UTC+3, lawrence...@gmail.com wrote: >>> >>> One last thing, I should mention, if I go through and add " LIMIT 100 " >>> to all the SQL queries, everything works great. That is, when dealing with >>> a few hundred documents, the app seems to work perfectly, and there are no >>> errors. It's only when I try to work with a few million documents that >>> things fall apart. I suspect some kind of contention arises... somewhere. >>> Possibly in several locations. But I'm not yet sure what the problem is. >>> >>> >>> >>> On Wednesday, October 4, 2017 at 12:49:12 AM UTC-4, >>> lawrence...@gmail.com wrote: >>>> >>>> This is probably a stupid question, but is there an obvious way to get >>>> an error message out of Elastisch? I had an app that was working with >>>> MongoDB and I was then told I had to use ElasticSearch instead (something >>>> about only using AWS for everything) so now I'm trying to get an error >>>> message, because my code doesn't seem to work. I read through here without >>>> seeing anything obvious: >>>> >>>> http://clojureelasticsearch.info/articles/getting_started.htm >>>> >>>> I rewrote my MongoDB function, so it should work with Elastisch: >>>> >>>> (defn push-item-to-persistence >>>> [item db] >>>> (let [ >>>> denormalized-id (get-in item [:denormalized-id] :no-id) >>>> item (assoc item :updated-at >>>> (temporal/current-time-as-datetime)) >>>> item (assoc item :permanent-holding-id-for-item-instances >>>> (java.util.UUID/randomUUID)) >>>> item (assoc item :instance-id-for-this-one-item >>>> (java.util.UUID/randomUUID)) >>>> item (assoc item :item-type :deduplication) >>>> ] >>>> (if (= denormalized-id :no-id) >>>> (slingshot/throw+ { >>>> :type >>>> ::no-denormalized-id-in-push-item-into-database >>>> :item item >>>> }) >>>> (slingshot/try+ >>>> (put conn index mapping-type id document) >>>> (esd/put db "facts-over-time" "deduplicaton" (str >>>> denormalized-id) item) >>>> (println " done with put in push-item-to-persistence ") >>>> (catch Object o >>>> (slingshot/throw+ { >>>> :type ::push-item-to-persistence >>>> :error o >>>> :item item >>>> :db db >>>> } >>>> )))))) >>>> >>>> >>>> It doesn't seem that any documents are getting into ElasticSearch. I >>>> was hoping to the (throw) would reveal to me some useful debugging >>>> information, but that doesn't seem to happen. >>>> >>>> The connection appears to be valid, as I can do this: >>>> >>>> (let [conn (persistence/multi-thread-start config) >>>> res (esd/search conn "facts-over-time" >>>> "deduplication" :query {:match_all {}}) >>>> n (esrsp/total-hits res) >>>> hits (esrsp/hits-from res)] >>>> (clojure.pprint/pprint res)) >>>> >>>> and I get: >>>> >>>> {:took 1, >>>> :timed_out false, >>>> :_shards {:total 5, :successful 5, :failed 0}, >>>> :hits {:total 0, :max_score nil, :hits []}} >>>> >>>> So the connection is there. But no records are. >>>> >>>> >>>> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
