Amendments:
* handle pg error 23505 (duplicate primary key race condition)
* some notes on handled errors
* removed dbpm-error condition
I also noticed that the concurrency tests are run automatically
as part of the backend tests. This is not my intention, but I'm
not sure how to fix it right now.
DEFTEST (rt) forms, like in migration.lisp, need to be called
explicitly AFAICS. Tests declared with FiveAM's TEST seem to be
run without being in a suite...
I'll check this out in a week when I get the chance, but
appreciate your comments.
Leslie
New patches:
[DB-POSTMODERN: reap old connections when a new one is requested.
[EMAIL PROTECTED] {
hunk ./src/db-postmodern/pm-controller.lisp 71
+
+(defun reap-orphaned-connections (sc)
+ (let ((n-reaped 0))
+ #+sbcl(maphash (lambda (thread bookkeeper)
+ (let ((alive-p (sb-thread:thread-alive-p thread)))
+ (unless alive-p
+ (cl-postgres:close-database (car bookkeeper))
+ (incf n-reaped)
+ (remhash thread (controller-db-table sc)))))
+ (controller-db-table sc))
+ n-reaped))
+
hunk ./src/db-postmodern/pm-controller.lisp 90
+ (reap-orphaned-connections sc)
}
[added concurrency test cases.
[EMAIL PROTECTED] {
hunk ./elephant-tests.asd 63
+ (:file "testconcurrency")
addfile ./tests/testconcurrency.lisp
hunk ./tests/testconcurrency.lisp 1
+(in-package :ele-tests)
+
+;; not enabled by default
+;(in-suite* testthreads :in elephant-tests)
+
+;;;
+;;; Leslie P. Polzer <[EMAIL PROTECTED]> 2008
+;;;
+;;; These tests will (as of March 2008) fail horribly on
+;;; * BDB without deadlock detection enabled.
+;;; * CLSQL/SQLite
+;;; * Probably other CLSQL backends
+;;;
+;;; The Postmodern backend should handle them correctly,
+;;; and BDB as well (although I noticed a major slowdown).
+;;;
+
+
+(defpclass zork ()
+ ((slot1 :accessor slot1 :initarg :slot1 :initform nil :index t)
+ (slot2 :accessor slot2 :initarg :slot2 :initform nil :index t)))
+
+
+; A basic simulation of a web application using Elephant
+; This is also a test showing whether database connections get cleaned up
+; correctly.
+
+(test threaded-idx-access
+ (dotimes (i 10)
+ (make-instance 'zork :slot1 i :slot2 i))
+
+ (dotimes (batch 20)
+ (dotimes (i 5)
+ (bt:make-thread (lambda ()
+ (dotimes (i 5)
+ (format t "thread ~A: batch ~A, run ~A~%" (bt:current-thread) batch i)
+ (dolist (obj (elephant::get-instances-by-class 'zork))
+ (format t "now handling obj ~A~%" obj)
+ (setf (slot-value obj 'slot1) i)
+ (setf (slot-value obj 'slot2) (slot-value obj 'slot1)))))))
+ #+sbcl(dolist (thr (bt:all-threads))
+ (format t "waiting for thread ~A to finish...~%" thr)
+ (unless (eq thr (bt:current-thread))
+ (sb-thread:join-thread thr)))
+ (format t "batch finished!~%"))
+
+ (drop-instances (get-instances-by-class 'zork))
+ (format t "test finished!~%"))
+
+(test provoke-deadlock ;; sometimes throws a 23505 (primary key constraint violation)
+ ;; I have not tracked this down, yet.
+ (dotimes (i 10)
+ (make-instance 'zork :slot1 i :slot2 i))
+
+ (dotimes (i 30)
+ (bt:make-thread
+ (lambda ()
+ (format t "thread no ~A starting~%" i)
+ (let ((obj (car (get-instances-by-class 'zork))))
+ (setf (slot-value obj 'slot1) i)) ;; this only provokes deadlocks when
+ ;; the slot in question is indexed.
+ (format t "thread finished.~%"))))
+
+ (drop-instances (get-instances-by-class 'zork)))
+
}
[added BORDEAUX-THREADS dependency and changed PM controller to use it instead of SB-THREAD stuff.
[EMAIL PROTECTED] {
hunk ./elephant.asd 327
- :depends-on (:uffi :cl-base64))
+ :depends-on (:uffi :cl-base64 :bordeaux-threads))
hunk ./src/db-postmodern/package.lisp 8
- #+sbcl :sb-thread)
+ :bordeaux-threads)
hunk ./src/db-postmodern/pm-controller.lisp 74
- #+sbcl(maphash (lambda (thread bookkeeper)
- (let ((alive-p (sb-thread:thread-alive-p thread)))
+ (maphash (lambda (thread bookkeeper)
+ (let ((alive-p (thread-alive-p thread)))
hunk ./tests/elet-package.lisp 22
- (:use :common-lisp :elephant :5am)
+ (:use :common-lisp :elephant :5am :bordeaux-threads)
hunk ./tests/elet-package.lisp 24
- (:use :common-lisp :elephant :regression-test)
+ (:use :common-lisp :elephant :regression-test :bordeaux-threads)
hunk ./tests/elet-package.lisp 60
+
}
[DB-POSTMODERN: support transaction retries; handle deadlock; add savepoint utility functions; add warnings to help debugging problematic conditions.
[EMAIL PROTECTED] {
hunk ./src/db-postmodern/pm-btree.lisp 265
+ ;(format t "pm-btree ~S: ~S := ~S~%" bt key value)
hunk ./src/db-postmodern/pm-sql.lisp 3
+
hunk ./src/db-postmodern/pm-sql.lisp 150
+(defun set-savepoint (con savepoint)
+ (ignore-errors
+ (cl-postgres:exec-query con (concatenate 'string "SAVEPOINT " savepoint))))
+
+(defun rollback-to-savepoint (con savepoint)
+ (ignore-errors
+ (cl-postgres:exec-query con (concatenate 'string "ROLLBACK TO " savepoint))))
+
hunk ./src/db-postmodern/pm-sql.lisp 176
+ (warn "42P05: prepared statement already exists!")
+ 'ignoring-this-error)
hunk ./src/db-postmodern/pm-sql.lisp 185
- 'ignoring-this-error)
hunk ./src/db-postmodern/pm-sql.lisp 196
- (handler-case
- (exec-prepared name-string)
- (cl-postgres:database-error (e)
- ;; Sometimes the prepared statement might hold references to old oids,
- ;; which might be have been dropped after a rollback. For safety, try
- ;; to remove the prepared statement and prepare it again
- (cond
- ((string= (cl-postgres:database-error-code e)
- "42P01")
- ;; It seems that this error automatically drops the transaction! Postgresql bug?
- (cl-postgres:exec-query (active-connection) (concatenate 'string "DEALLOCATE " name-string))
- (cl-postgres:prepare-query (active-connection) name-string sql)
- (exec-prepared name-string))
- (t (error e)))))))))
+ (let ((savepoint (princ-to-string (gensym))))
+ ;(set-savepoint (active-connection) savepoint)
+ (handler-case
+ (progn
+ ;(format t "Executing prepared query ~A~%" name-string)
+ (exec-prepared name-string))
+ (cl-postgres:database-error (e)
+ ;; Sometimes the prepared statement might hold references to old oids,
+ ;; which might be have been dropped after a rollback. For safety, try
+ ;; to remove the prepared statement and prepare it again
+ (warn "Error while executing prepared statement ~S (params: ~A).~%"
+ name-string params)
+ (cond
+ ((string= (cl-postgres:database-error-code e)
+ "42P01")
+ ;; It seems that this error automatically drops the transaction! Postgresql bug?
+ (warn "42P01: Prepared statement already exists; trying to remove it.")
+ ;(rollback-to-savepoint (active-connection) savepoint)
+ (cl-postgres:exec-query (active-connection) (concatenate 'string "DEALLOCATE " name-string))
+ (cl-postgres:prepare-query (active-connection) name-string sql)
+ (exec-prepared name-string))
+
+ ((string= (cl-postgres:database-error-code e)
+ "25P02")
+ (warn "25P02: Transaction aborted; some earlier error wasn't handled correctly!")
+ 'ignoring-this-error)
+
+ (t (error e))))))))))
+
hunk ./src/db-postmodern/pm-sql.lisp 321
+
hunk ./src/db-postmodern/pm-transaction.lisp 30
- &key (always-rollback nil) &allow-other-keys)
- ;; SQL doesn't support nested transaction
+ &key (retries 50) (always-rollback nil) &allow-other-keys)
hunk ./src/db-postmodern/pm-transaction.lisp 32
- (if (> (tran-count-of sc) 0)
- (funcall txn-fn)
- (let (tran
- commited
- (*txn-value-cache* (make-value-cache sc)))
- (incf (tran-count-of sc))
- (unwind-protect
- (prog2
- (setf tran (controller-start-transaction sc))
- (funcall txn-fn) ;;this gets returned
- (unless always-rollback ;;automatically commit unless always rollback
- (controller-commit-transaction sc tran)
- (setf commited t)))
- (unless commited (controller-abort-transaction sc tran))
- (decf (tran-count-of sc)))))))
+ (let (savepoint (try 0))
+ (tagbody
+ retry (incf try)
+ (when (>= try retries)
+ (cerror "Retry transaction again?"
+ 'transaction-retry-count-exceeded
+ :format-control "Transaction exceeded the limit of ~A retries"
+ :format-arguments (list retries)
+ :count retries))
+ ;(format t "txn-mgr (thr ~A): try ~A~%" sb-thread::*current-thread* try)
+ ;; XXX honor max retries
+ (if (> (tran-count-of sc) 0) ;; SQL doesn't support nested transaction
+ (progn
+ ;(setf savepoint (princ-to-string (gensym)))
+ ;(set-savepoint (active-connection) savepoint)
+ ;(setf savepoint nil)
+ ;(format t "detected nested transaction~%")
+ (return-from execute-transaction (funcall txn-fn)))
+ (let (tran)
+ (handler-case
+ (let (commited (*txn-value-cache* (make-value-cache sc)))
+ (incf (tran-count-of sc))
+ (unwind-protect
+ (return-from execute-transaction
+ (prog2
+ (setf tran (controller-start-transaction sc))
+ (funcall txn-fn) ;; this gets returned
+ (unless always-rollback ;; automatically commit unless always rollback
+ (controller-commit-transaction sc tran)
+ (setf commited t))))
+ (unless commited (controller-abort-transaction sc tran))
+ (decf (tran-count-of sc))))
+ (cl-postgres:database-error (e)
+ (warn "dbpm txn manager: caught error ~A~%" (cl-postgres:database-error-code e))
+ (cond
+ ((or (string= (cl-postgres:database-error-code e) "40P01") ; deadlock
+ (string= (cl-postgres:database-error-code e) "23505")) ; duplicate primary key
+ ;(if savepoint
+ ;(rollback-to-savepoint (active-connection) savepoint)
+ (controller-abort-transaction sc tran)
+ (go retry)))))))))))
hunk ./src/db-postmodern/pm-transaction.lisp 74
+#|
+
+Notes on error handling:
+
+ 40P01: the correct way to handle a detected deadlock is restarting aborted
+ transactions until the locks are resolved. (Leslie)
+
+ 23505: this occurs due to a race condition we can't really prevent since
+ it's caused by PL/PGSQL code. Rollback until all races are resolved.
+ A more elegant solution will involve savepoints. (Leslie)
+
+|#
+
}
Context:
[Check for unbound slot; potential fix for a compiler error
[EMAIL PROTECTED]
[Fix test dependence for ff-index-test
[EMAIL PROTECTED]
[Improve berkeley DB version agnostic code
[EMAIL PROTECTED]
Added an error message to configure my-config.sexp and made sure we load
it directly from my-config.sexp so that we get it right at load time.
Prior patch didn't override default until after load time.
]
[Support for multiple BDB versions
[EMAIL PROTECTED]
[db-bdb updated to BerkeleyDB 4.6
[EMAIL PROTECTED]
Changed only BDB constants as upgrade 4.5 -> 4.6 they were
changed.
A kind of hack perhaps. But it works. The testing was not excessive,
but it works well for my project.
]
[add test for STRING types (as opposed to SIMPLE-STRING types)
[EMAIL PROTECTED]
[Refactor UTF{16,32}LE serializers.
[EMAIL PROTECTED]
[Enable multiple process connections to a BDB data-store via DB_REGISTER option
[EMAIL PROTECTED]
[Enable multi-store indexed classes
[EMAIL PROTECTED]
[Change semantics of transaction :retry-count from tries to retries
[EMAIL PROTECTED]
[Minor edits, fixed a comment, fixed a db-lisp out of date error
[EMAIL PROTECTED]
[Alex's patch for 8.3
[EMAIL PROTECTED]
I entered here the patch from Alex of 2088/02/16
which apparently makes this compatible with Postgres 8.3.
It is green for me on all tests on Posgres 8.1, so
I am committing it.
]
[mtype change in dcm
[EMAIL PROTECTED]
[moved cache-instance into initial-persistent-setup
[EMAIL PROTECTED]
because it was bypassed by recreate-instance otherwise
]
[accessor name in tests change
[EMAIL PROTECTED]
[db-postmodern: pm-btree initialization fixed
[EMAIL PROTECTED]
[recreate-instance stuff improved
[EMAIL PROTECTED]
[db-postmodern: removed specialized map-index
[EMAIL PROTECTED]
because pure cursor version works fine and is more robust
]
[cursor-duplicate removed from db-postmodern
Henrik Hjelte<[EMAIL PROTECTED]>*-20071124163701]
[db-postmodern fix map-index optimization bug
Henrik Hjelte <[EMAIL PROTECTED]>**20080104151644]
[db-postmodern: cursors re-implemented
[EMAIL PROTECTED]
[controller-doc-improvement
[EMAIL PROTECTED]
[tutorial
[EMAIL PROTECTED]
[non-keyword-accessors
[EMAIL PROTECTED]
allows lispworks to run tests.
]
[function-call-key-form
[EMAIL PROTECTED]
[documentation type fix
[EMAIL PROTECTED]
[Fix the use of internal symbol of sb-kernel in memutils
Leonardo Varuzza <[EMAIL PROTECTED]>**20071230000120
memutil.lisp use the functions sb-kernel::copy-*-from-system-area, which
aren't exported in the latest version of sbcl.
Fix it adding the :: when appropriate
]
[db-bdb bugfix: when bdb key comparison compared only the first half of utf16 strings
[EMAIL PROTECTED]
[Fix instance deserialization to bypass initialization protocol
[EMAIL PROTECTED]
[db-postmodern: optimized map-index for -by-value case
[EMAIL PROTECTED]
[db-postmodern: optimized form-slot-key for persistent-slot-reader
[EMAIL PROTECTED]
it uses SBCL internal function now, for other implementation it's less optimized.
]
[db-postmodern: small example update
[EMAIL PROTECTED]
[added sh script for flushing logs sample
[EMAIL PROTECTED]
[db-postmodern removed possiblity of using NIL as a key in btrees
Henrik Hjelte<[EMAIL PROTECTED]>**20071124163828]
[cursor-duplicate removed from db-postmodern
Henrik Hjelte<[EMAIL PROTECTED]>**20071124163701]
[removed a little compiler warning (typo)
Henrik Hjelte<[EMAIL PROTECTED]>**20071122151929]
[remove kind-hints parameter from add-index
Henrik Hjelte<[EMAIL PROTECTED]>**20071122151046
Probably a coming feature from Ian, but
right now it breaks the generic function add-index
and thus postmodern, so I removed it for now.
]
[Ensure set-db-synch is defined before pset is loaded
[EMAIL PROTECTED]
[Fix instance deserialization to bypass initialization protocol
[EMAIL PROTECTED]
[Fix to from-end traversal of new map-index
[EMAIL PROTECTED]
[New map-index implementation
[EMAIL PROTECTED]
[Cheaper get-instance-by-value
[EMAIL PROTECTED]
[TAG ELEPHANT-0-9-1
[EMAIL PROTECTED]
Patch bundle hash:
0693285213024e3076aa202c33967de47d128f91_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
