Hi All, I am pleased to announce the initial release of Tupelo Datomic, a library of functions to make living and working with Datomic as effortless as possible.
While the native Datomic API is very powerful, there are many details and options which are often unnecessary and make code more cumbersome than necessary. Tupelo Datomic aims to focus on the simple and most common use-cases, so that saving and retrieving data using Datomic is nearly effortless. The following samples are from the Tupelo Datomic documentation at: https://github.com/cloojure/tupelo/blob/master/src/tupelo/datomic.adoc -------------------------------------------------------------------------------------------------------------------- Suppose we’re trying to keep track of information for the world’s premiere spy agency. Let’s create a few attributes that will apply to our heroes & villains (see the executable code in the unit test <https://github.com/cloojure/tupelo/blob/master/test/tst/tupelo/datomic_bond.clj> ). (:require [tupelo.datomic :as td] [tupelo.schema :as ts]) ; Create some new attributes. Required args are the attribute name (an optionally namespaced ; keyword) and the attribute type (full listing at http://docs.datomic.com/schema.html). We wrap ; the new attribute definitions in a transaction and immediately commit them into the DB. (td/transact *conn* ; required required zero-or-more ; <attr name> <attr value type> <optional specs ...> (td/new-attribute :person/name :db.type/string :db.unique/value) ; each name is unique (td/new-attribute :person/secret-id :db.type/long :db.unique/value) ; each secret-id is unique (td/new-attribute :weapon/type :db.type/ref :db.cardinality/many) ; one may have many weapons (td/new-attribute :location :db.type/string) ; all default values (td/new-attribute :favorite-weapon :db.type/keyword )) ; all default values For the :weapon/type attribute, we want to use an enumerated type since there are only a limited number of choices available to our antagonists: ; Create some "enum" values. These are degenerate entities that serve the same purpose as an ; enumerated value in Java (these entities will never have any attributes). Again, we ; wrap our new enum values in a transaction and commit them into the DB. (td/transact *conn* (td/new-enum :weapon/gun) (td/new-enum :weapon/knife) (td/new-enum :weapon/guile) (td/new-enum :weapon/wit)) Let’s create a few antagonists and load them into the DB. Note that we are just using plain Clojure values and literals here, and we don’t have to worry about any Datomic specific conversions. ; Create some antagonists and load them into the db. We can specify some of the attribute-value ; pairs at the time of creation, and add others later. Note that whenever we are adding multiple ; values for an attribute in a single step (e.g. :weapon/type), we must wrap all of the values ; in a set. Note that the set implies there can never be duplicate weapons for any one person. ; As before, we immediately commit the new entities into the DB. (td/transact *conn* (td/new-entity { :person/name "James Bond" :location "London" :weapon/type #{ :weapon/gun :weapon/wit } } ) (td/new-entity { :person/name "M" :location "London" :weapon/type #{ :weapon/gun :weapon/guile } } ) (td/new-entity { :person/name "Dr No" :location "Caribbean" :weapon/type :weapon/gun } )) And, just like that, we have values persisted in the DB! Let’s check that they are really there: ; Verify the antagonists were added to the DB (let [people (get-people (live-db)) ] (is (= people #{ {:person/name "James Bond" :location "London" :weapon/type #{:weapon/wit :weapon/gun} } {:person/name "M" :location "London" :weapon/type #{:weapon/guile :weapon/gun} } {:person/name "Dr No" :location "Caribbean" :weapon/type #{:weapon/gun } } } ))) <https://github.com/cloojure/tupelo/blob/master/src/tupelo/datomic.adoc#entityspec-entityid-and-lookupref>EntitySpec, EntityID, and LookupRef Entities in Datomic are specified using an EntitySpec, which is either an EntityID (EID) or a LookupRef. An EntityID (EID) is a globally unique Long value that uniquely specifies any entity in the Datomic DB. These are always positive for committed entities in Datomic (negative values indicate temporary EIDs used only in building transactions). A LookupRef is an attribute-value pair (wrapped in a vector), which uniquely specifies an entity. If an entity has an attribute specified as either :db.unique/value or :db.unique/identity, that entity may be specified using a LookupRef. Here we verify that we can find James Bond and retrieve all of his attr-val pairs using either type of EntitySpec: ; Using James' name, lookup his EntityId (EID). (let [james-eid (td/query-scalar :let [$ (live-db)] ; like Clojure let :find [?eid] :where [ [?eid :person/name "James Bond"] ] ) ; Retrieve James' attr-val pairs as a map. An entity can be referenced by either EID or LookupRef james-map (td/entity-map (live-db) james-eid) ; use EID james-map2 (td/entity-map (live-db) [:person/name "James Bond"] ) ; use LookupRef ] (is (= james-map {:person/name "James Bond" :location "London" :weapon/type #{:weapon/wit :weapon/gun} } )) (is (= james-map james-map2 )) We can also use either type of EntitySpec for update ; Update the database with more weapons. If we overwrite some items that are already present ; (e.g. :weapon/gun) it is idempotent (no duplicates are allowed). The first arg to td/update ; is an EntitySpec (either EntityId or LookupRef) and determines the Entity that is updated. (td/transact *conn* (td/update james-eid ; update using EID { :weapon/type #{ :weapon/gun :weapon/knife } :person/secret-id 007 } ) ; Note that James has a secret-id but no one else does (td/update [:person/name "Dr No"] ; update using LookupRef { :weapon/type #{ :weapon/gun :weapon/knife :weapon/guile } } ))) As expected, our database contains the updated values for Dr No and James Bond. Notice that, since :weapon/type is implemented as a set in Datomic, duplicate values are not allowed and both antagonists have only a single gun: ; Verify current status. Notice there are no duplicate weapons. (let [people (get-people (live-db)) ] (is (= people #{ { :person/name "James Bond" :location "London" :weapon/type #{:weapon/wit :weapon/knife :weapon/gun} :person/secret-id 7 } { :person/name "M" :location "London" :weapon/type #{:weapon/guile :weapon/gun} } { :person/name "Dr No" :location "Caribbean" :weapon/type #{:weapon/guile :weapon/knife :weapon/gun} } } ))) <https://github.com/cloojure/tupelo/blob/master/src/tupelo/datomic.adoc#enum-values>Enum Values The benefit of using enumerated values in Datomic is that we can easily restrict the the domain of acceptable values more easily than by using plain keyword values. For example, if we try to give James a non-existent weapon, Datomic will generate an exception: ; Try to add non-existent weapon. This throws since the bogus kw does not match up with an entity. (is (thrown? Exception @(td/transact *conn* (td/update [:person/name "James Bond"] ; update using a LookupRef { :weapon/type #{ :there.is/no-such-kw } } )))) ; bogus value for :weapon/type causes exception <https://github.com/cloojure/tupelo/blob/master/src/tupelo/datomic.adoc#query-functions-in-tupelo-datomic>Query Functions in Tupelo Datomic When querying for values using Tupelo Datomic, the fundamental result type is a TupleSet (a Clojure set containing unique Clojure vectors). This overcomes the problem where native result type datomic.query.EntityMap is lazy-loading and may give unexpected results. Here is an example of Tupelo Datomic query in action: ; For general queries, use td/query. It returns a set of tuples (TupleSet). Any duplicated ; tuples will be discarded (let [tuple-set (td/query :let [$ (live-db)] :find [?name ?loc] ; <- shape of output tuples :where [ [?eid :person/name ?name] ; pattern-matching rules specify how the variables [?eid :location ?loc ] ] ) ; must be related (implicit join) ] (is (s/validate ts/TupleSet tuple-set)) ; verify expected type using Prismatic Schema (is (s/validate #{ [s/Any] } tuple-set)) ; literal definition of TupleSet (is (= tuple-set #{ ["Dr No" "Caribbean"] ; Even though London is repeated, each tuple is ["James Bond" "London"] ; still unique. Otherwise, any duplicate tuples ["M" "London"] } ))) ; will be discarded since output is a clojure set. Tupelo Datomic modifies the original Datomic query syntax compared to (datomic.api/q ...) in two ways. For convenience, the query form does not need to be wrapped in a map literal nor is any quoting required. Most importantly, the :in keyword has been replaced with the :let keyword, and the syntax has been copied from the Clojure letspecial form so that each query variables are more closely aligned with their actual values. Also, the implicit DB $ must be explicitly tied to its data source in all cases (as shown above). Receiving a TupleSet result is the most general case, but in many instances we can save some effort. If we are retrieving the value for a single attribute per entity, we don’t need to wrap that result in a tuple. In this case, we can use the functiontd/query-set, which returns a set of scalars as output rather than a set of tuples of scalars: ; If you want just a single attribute as output, you can get a set of values (rather than a set of ; tuples) using td/query-set. As usual, any duplicate values will be discarded. (let [names (td/query-set :let [$ (live-db)] :find [?name] ; <- a single attr-val output allows use of td/query-set :where [ [?eid :person/name ?name] ] ) cities (td/query-set :let [$ (live-db)] :find [?loc] ; <- a single attr-val output allows use of td/query-set :where [ [?eid :location ?loc] ] ) ] (is (= names #{"Dr No" "James Bond" "M"} )) ; all names are present, since unique (is (= cities #{"Caribbean" "London"} ))) ; duplicate "London" discarded A parallel case is when we want results for just a single entity, but multiple values are needed. In this case, we don’t need to wrap the result tuple in a set and we can use the function td/query-tuple, which returns a single tuple as output rather than a set of tuples: ; If you want just a single tuple as output, you can get it (rather than a set of ; tuples) using td/query-tuple. It is an error if more than one tuple is found. (let [beachy (td/query-tuple :let [$ (live-db) ; assign multiple query variables ?loc "Caribbean"] ; just like clojure 'let' special form :find [?eid ?name] ; <- output tuple shape :where [ [?eid :person/name ?name ] [?eid :location ?loc] ] ) busy (try ; error - both James & M are in London (td/query-tuple :let [$ (live-db) ?loc "London"] :find [?eid ?name] ; <- output tuple shape :where [ [?eid :person/name ?name] [?eid :location ?loc ] ] ) (catch Exception ex (.toString ex))) ] (is (matches? beachy [_ "Dr No"] )) ; found 1 match as expected (is (re-seq #"IllegalStateException" busy))) ; Exception thrown/caught since 2 people in London Of course, in some instances you may want only the value of only a single attribute for a single entity. In this case, we may use the function td/query-scalar, which returns a single scalar value instead of a set of tuples of scalars: ; If you know there is (or should be) only a single scalar answer, you can get the scalar value as ; output using td/query-scalar. It is an error if more than one tuple or value is present. (let [beachy (td/query-scalar :let [$ (live-db) ; assign multiple query variables ?loc "Caribbean"] ; just like clojure 'let' special form :find [?name] :where [ [?eid :person/name ?name] [?eid :location ?loc ] ] ) busy (try ; error - multiple results for London (td/query-scalar :let [$ (live-db) ?loc "London"] :find [?eid] :where [ [?eid :person/name ?name] [?eid :location ?loc ] ] ) (catch Exception ex (.toString ex))) multi (try ; error - tuple [?eid ?name] is not scalar (td/query-scalar :let [$ (live-db) ?loc "Caribbean"] :find [?eid ?name] :where [ [?eid :person/name ?name] [?eid :location ?loc ] ] ) (catch Exception ex (.toString ex))) ] (is (= beachy "Dr No")) ; found 1 match as expected (is (re-seq #"IllegalStateException" busy)) ; Exception thrown/caught since 2 people in London (is (re-seq #"IllegalStateException" multi))) ; Exception thrown/caught since 2 people in London <https://github.com/cloojure/tupelo/blob/master/src/tupelo/datomic.adoc#using-the-datomic-pull-api>Using the Datomic Pull API If one wishes to use queries returning possibly duplicate result items, then the Datomic Pull api is required. A Pull query returns results in a List (a Clojure vector), rather than a Set, so that duplicate result items are not discarded. As an example, let’s find the location of all of our entities: ; If you wish to retain duplicate results on output, you must use td/query-pull and the Datomic ; Pull API to return a list of results (instead of a set). (let [result-pull (td/query-pull :let [$ (live-db)] ; $ is the implicit db name :find [ (pull ?eid [:location]) ] ; output :location for each ?eid found :where [ [?eid :location] ] ) ; find any ?eid with a :location attr result-sort (sort-by #(-> % first :location) result-pull) ] (is (s/validate [ts/TupleMap] result-pull)) ; a list of tuples of maps (is (= result-sort [ [ {:location "Caribbean"} ] [ {:location "London" } ] [ {:location "London" } ] ] ))) <https://github.com/cloojure/tupelo/blob/master/src/tupelo/datomic.adoc#using-datomic-partitions>Using Datomic Partitions Datomic allows the user to create *partitions* within the DB. Datomic partitions serve solely as a structural optimization, and do not control or limit how or by whom datoms may be accessed. The effect of a partition in Datomic is to effectively "pre-sort" all entities in that partition so that they are adjacent in storage, which *may* improve access times for related entities that are often accessed together. In Tupelo Datomic, we may easily create and use partitions: ; Create a partition named :people (we could namespace it like :db.part/people if we wished) (td/transact *conn* (td/new-partition :people )) ; Create Honey Rider and add her to the :people partition (let [tx-result @(td/transact *conn* (td/new-entity :people ; <- partition is first arg (optional) to td/new-entity { :person/name "Honey Rider" :location "Caribbean" :weapon/type #{:weapon/knife} } )) [honey-eid] (td/eids tx-result) ; retrieve Honey Rider's EID from the seq (destructuring) ] (is (s/validate ts/Eid honey-eid)) ; verify the expected type (is (= :people ; verify the partition name for Honey's EID (td/partition-name (live-db) honey-eid)))) <https://github.com/cloojure/tupelo/blob/master/src/tupelo/datomic.adoc#future-work>Future Work Lots more to come! <https://github.com/cloojure/tupelo/blob/master/src/tupelo/datomic.adoc#license> License Copyright © 2015 Alan Thompson. Distributed under the Eclipse Public License, the same as Clojure. <https://github.com/cloojure/tupelo/blob/master/src/tupelo/datomic.adoc#todo-list-todo> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
