Hi Joshua,
2014-10-06 22:42 GMT+02:00 Joshua Dunham <joshua_dun...@vrtx.com>:
> Hi Sebastian,
>
> Thank you for chiming in. Some clarifying questions below.
>
Trying to answer :-)
>
> >>> 2. Which is the preferable backend? I’m not familiar with the
> pros/cons of the options but I think looking around at some docs that
> Hazlecast is a ‘safe’ good bet?
> >
> >> We currently support Guava and Ehcache for local caches, Hazelcast,
> and Infinispan for clusters. AFAIK currently Hazelcast is the most stable
> and tested one, and it's currently used in production.
> >
> > Guava for single-instance setups, otherwise Hazelcast. The other
> backends are more experimental. Infinispan is powerful in large setups,
> because it also supports dedicated cluster servers (HotRod Server), but
> this has not been tested extensively and is significantly more complex and
> has more overhead. EHCache has a bit more intelligent memory management
> (i.e. it expires cached objects based on the memory they occupy, while all
> other backends simply take object counts, so when you have many large
> objects you might run into out-of-memory situations), but otherwise
> introduces more overhead than Guava.
> >
>
> So to get started I would make two instances of tomcat and two instances
> of marmotta (ideally separate hardware). I would configure the cluster
> settings on both as such,
>
> clustering.address = 226.6.7.8
> clusting.backend = hazlecast
> clustering.enabled = on
> clustering.mode = replicated
> clustering.name = marmotta
> clustering.port = 46655
>
There is a typo there ("hazelcast"). Also you should change clustering.mode
to "distributed", because Hazelcast does not support "replicated". Not a
problem if you forget it, but you will get a warning in the log. ;-)
>
> while making sure that .name and .port are not in use. I would then
> connect them to the same database backend from the default H2. (I need to
> use mySQL, we predominantly use Oracle, not pSQL).
>
You would then connect to the same database server (not sure if you
referred to this when saying "backend"). More elegant is to set the
database.url in the Zookeeper tree and let Marmotta retrieve it from there.
I am not sure about MySQL (we are not really working much with it), but you
are welcome to try (and maybe tell us your experiences).
> - Or would I connect mySQL db first and then do the cluster config?
>
The order doesn't really matter as long as no users are really accessing
the system in parallel.
> - If I installed and configure one instance to use clustering first,
> configure it to use zookeeper, configure the backend DB; then installed a
> new instance of marmotta and hooked that to zookeeper, is magic going to
> happen and it will sort out all the settings for each?
>
That's the plan (this is how we use it). ;-)
Unfortunately, all configuration you store via the user interface will only
be stored in the local configuration file, because Marmotta does not know
whether your configuration setting should affect all marmotta instances,
those in a cluster, or only the single instance you are configuring.
For this to work you need to make sure you understand the way configuration
is stored in Zookeeper. Look at
http://marmotta.apache.org/platform/cloud.html; the Zookeeper tree contains
three levels of Marmotta configurations:
- global level contains configuration applied to all Marmotta instances
using this Zookeeper server
- cluster level contains configuration applied to all Marmotta instances in
a named cluster (Servlet context init parameter zookeeper.cluster), e.g.
for database URL
- instance level contains configuration applied to a single Marmotta
instance (e.g. for turning on logging or such things)
If you are interested I can send you a sample dump of a Zookeeper tree we
are using.
>
> If the 'master' is data.example.com and the 'replicant' is
> data-02.example.com, I still make my rdf with an IRI of the master? Will
> the replicant have a different IRI and thus I would need to reference a
> different resource at that endpoint? Or do I ‘cheat’ and put the same IRI
> in the system-config.properties
>
You would put a load balancer (e.g. standard Apache HTTPD) in front of your
two tomcats and configure the Marmottas to use the same IRI for resources
(which has to match the IRI of your load balancer). You can do this by
manually setting the configuration variables kiwi.context (sets the prefix
for constructing URIs) and kiwi.host (sets the prefix for accessing the
admin UI).
>
> Since I already have one instance up and containing data, would it work to
> bring the second instance online, configure the cluster settings of both
> and then the database and have it synchronize the data?
>
Since they are both accessing the same database, there is no need to
synchronize manually. Caches are then automatically synchronized by
Hazelcast (as soon as it has finished discovering its peers).
>
> > Clustering in Marmotta generally won't help you with big datasets. But
> it will help you with high concurrent loads. The clustering functionality
> currently implemented essentially provides two features:
> > - a cluster-wide cache so that database lookups for frequently used
> nodes and triples can be reduced; this won't help you if you are always
> requesting different data or run SPARQL queries; it will help you if you
> are repeatedly accessing the same nodes and triples
> > - a cluster-wide synchronization and locking mechanism to make sure the
> cluster members all share the same data and no inconsistencies are created;
> this will actually SLOW DOWN your single-process operations and is useful
> only in highly concurrent setups
> >
> > If you want to improve performance for single-dataset single-user
> situations, don't use the clustering mechanism. Use and tune the PostgreSQL
> database backend instead. Make sure you read
> http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
> >
>
> I'm not super concerned with speed at the moment. My goal for phase 1 is
> to develop an ontology for the
> applications that will hook into the system. Phase 1.5 is making LDPath
> apps that will return select entries via json to use in a webapp. The
> webapp collects data from these curated lists and submits the changes back
> to marmotta. My issue is that even the uniprot dataset is 85GB compressed
> rdf-xml so it will be difficult to work with the amount of data I would
> need to properly connect LDPath apps to clients. I was hoping I could have
> at least different contexts if not an even spread of data among the
> instances.
>
85 GB should not be terribly much, especially for read access. Did you try
loading it with the marmotta loader? It will take some time, but should
work. Make sure to tune your database first, though (not sure about the
optimal settings for MySQL).
>
> >>> I also found the Apache Giraph project which claims to offer native
> node/edge processing for graph databases. Has anyone used this? I would be
> *very* interested to play around if it could connect to Marmotta.
> >
> >> We have a experimental backend that uses Titan DB. If you'd be great if
> someone could evolve Marmotta in that direction!
> >
> > Giraph serves a different purpose, it is a highly scalable graph
> processing framework, not a database. As such, it allows you to parallelize
> typical graph operations (like shortest path computations) and run them on
> a Hadoop cluster. This is totally different to the kind of operations
> needed by Marmotta (e.g. to support SPARQL querying). If you would like to
> have a clustered database backend, you could try the Titan backend with
> HBase or Cassandra, but I am not completely convinced it will be faster
> than PostgreSQL.
> >
>
> Yes, I have been looking into Giraph to ideally serve as a native
> processing engine to Marmotta. As I start to fill the database with
> connections I make (on top of established datasets like uniprot) I was
> looking into apps that could start to find trends in my data. It would be
> ideal to have an app connect in natively rather than trying to load some
> new permutation of data into hadoop and running it there. My use case is
> that I will have **many** triples with a known predicate (middle column)
> and a plain literal value for value (third column). I would like it to work
> back to all known connections and start to find commonalities. It’s easy in
> principle to think about and not super difficult to program. Think
> principle component analysis for each related node of a source node.
>
You could also work directly on the Marmotta database in SQL. The triple
format is easy enough to understand. ;-)
>
> >>> Lastly, What are people using to manage there ontologies? I found
> Protege a while back and installed WebProtege to manage ontologies. Is it
> possible that it connects to marmotta to keep the ontology synchronized?
> Are there any cool things WebProtege (or any ontology manager) can do with
> Marmotta?
> >>>
> > I am using emacs for managing ontologies ;-)
> >
>
> So, using an ‘offline’ ontology manager means you would make your changes
> to terms, save out a versioned copy, diff it against the last export and
> run it against your marmotta to establish the new ontology in DB?
>
Depends, probably nowadays I would use SPARQL Update queries to apply
changes to the ontology. I am not aware of any nice tools for this, though.
Greetings,
Sebastian
