[hibernate-dev] Using a checksum for version properties
Hi, We have some database tables that we can't add version columns to so I would like to instead create a version number by using checksums on the rows. I'm thinking specifically of Oracle here but I assume other platforms may have equivalent features; the sort of query I am looking at using is... select id, name, salary, dbms_crypto.hash(utl_raw.cast_to_raw(id || name || salary), 1) as version from employees In this example, the version column is populated with the MD4 sum of the three properties. I've tried to create a custom EntityPersister but it looks like I'll need to dig deeper. My plan would be to create the checksum code automatically depending on the 'versionability' of the properties. If this sounds useful/sane I am happy to work on a patch. Regards, Tom ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
[hibernate-dev] HSEARCH-679 status
I see that HSEARCH-679 is scheduled to be released in Hibernate Search 3.4.0. I haven't seen any progress on the issue though. If it will help moving the issue along I can definitely test proposed solutions, we can reproduce the issue consistently. Unfortunately we've reproduced the issue during product demonstrations to potential customers, hence the urgency on our end to see the issue resolved. Cheers, Tom ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] HSEARCH-679 status
Sanne, the unit test doesn't make any assertions, so I don't know how useful it would be as a test. It allows a user to view the Hibernate-generated SQL, illustrating the entities loaded. On Mon, Mar 21, 2011 at 2:46 PM, Sanne Grinovero wrote: > Hi Tom, > thanks for the reminder, you might have noticed we're very busy > lately, so thank you for your help proposal. > > I've just committed a solution draft on my HSEARCH-679 branch: > > https://github.com/Sanne/hibernate-search/tree/HSEARCH-679 > > It's a draft but seems to work; I still have to get someone into a > code review, maybe you could start having a look and try it out? > I've run your tests, very useful thank you. Do you think you could > re-shape that in a patch for Search, so we can include that as > functional test? > I would need you to remove the dependencies from Spring, and to think > on a way to have the test fail if it loads the undesired collection > without looking into the logs. > > I think you might try casting it to a > org.hibernate.mapping.PersistentClass > and then see if it ".wasInitialized()". > > Regards, > Sanne > > > > > > 2011/3/21 Tom Waterhouse : > > I see that HSEARCH-679 is scheduled to be released in Hibernate Search > > 3.4.0. I haven't seen any progress on the issue though. If it will help > > moving the issue along I can definitely test proposed solutions, we can > > reproduce the issue consistently. Unfortunately we've reproduced the > issue > > during product demonstrations to potential customers, hence the urgency > on > > our end to see the issue resolved. > > > > Cheers, > > > > Tom > > ___ > > hibernate-dev mailing list > > hibernate-dev@lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hibernate-dev > > > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] HSEARCH-679 status
Sanne, I don't see the method you've identified in PersistentClass. The closest I could find to identifying if a property has been initialized is Hibernate.isPropertyInitialized(), but the method always returns true (object isn't a HibernateProxy, FieldInterceptionHelper.isInstrumented() returns false). Any other ideas on how I can tell if a lazy property has been initialized? Tom On Mon, Mar 21, 2011 at 4:40 PM, Sanne Grinovero wrote: > 2011/3/21 Tom Waterhouse : > > Sanne, the unit test doesn't make any assertions, so I don't know how > useful > > it would be as a test. It allows a user to view the Hibernate-generated > > SQL, illustrating the entities loaded. > > Yes I saw that :) so I was asking if you could make a unit test out of > it to contribute to the project. > The wasInitialized() method I mentioned in the previous email is a > suggestion for a possible implementation. > > Sanne > > > > > On Mon, Mar 21, 2011 at 2:46 PM, Sanne Grinovero > > wrote: > >> > >> Hi Tom, > >> thanks for the reminder, you might have noticed we're very busy > >> lately, so thank you for your help proposal. > >> > >> I've just committed a solution draft on my HSEARCH-679 branch: > >> > >> https://github.com/Sanne/hibernate-search/tree/HSEARCH-679 > >> > >> It's a draft but seems to work; I still have to get someone into a > >> code review, maybe you could start having a look and try it out? > >> I've run your tests, very useful thank you. Do you think you could > >> re-shape that in a patch for Search, so we can include that as > >> functional test? > >> I would need you to remove the dependencies from Spring, and to think > >> on a way to have the test fail if it loads the undesired collection > >> without looking into the logs. > >> > >> I think you might try casting it to a > >> org.hibernate.mapping.PersistentClass > >> and then see if it ".wasInitialized()". > >> > >> Regards, > >> Sanne > >> > >> > >> > >> > >> > >> 2011/3/21 Tom Waterhouse : > >> > I see that HSEARCH-679 is scheduled to be released in Hibernate Search > >> > 3.4.0. I haven't seen any progress on the issue though. If it will > >> > help > >> > moving the issue along I can definitely test proposed solutions, we > can > >> > reproduce the issue consistently. Unfortunately we've reproduced the > >> > issue > >> > during product demonstrations to potential customers, hence the > urgency > >> > on > >> > our end to see the issue resolved. > >> > > >> > Cheers, > >> > > >> > Tom > >> > ___ > >> > hibernate-dev mailing list > >> > hibernate-dev@lists.jboss.org > >> > https://lists.jboss.org/mailman/listinfo/hibernate-dev > >> > > > > > > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] HSEARCH-679 status
Sanne, that testing strategy doesn't work unfortunately. The collection is loaded by virtue of creating test data, so it is intialized. I don't see how HSearch can be tested at the entity level. Perhaps a smaller unit test of FullIndexTextEventListener is the way to go instead. I'm not sure what the Hibernate development testing methodology is, but my suggestion would be to create a unit test of processCollectionEvent with mocks as necessary. Tom On Tue, Mar 22, 2011 at 11:34 AM, Sanne Grinovero wrote: > Hi Tom, > sorry, copy paste error: I meant > org.hibernate.collection.PersistentCollection, all collections > retrieved from the database implement PersistentCollection, which > exposes the wasInitialized() method. > So just cast the collection to PersistentCollection and you should be > able to check if it was fully initialized. > > BTW I fixed the pending issue on github, as soon as you have a > testcase we can proceed with a pull request. > > Sanne > > 2011/3/22 Tom Waterhouse : > > Sanne, > > > > I don't see the method you've identified in PersistentClass. The closest > I > > could find to identifying if a property has been initialized is > > Hibernate.isPropertyInitialized(), but the method always returns true > > (object isn't a HibernateProxy, FieldInterceptionHelper.isInstrumented() > > returns false). > > > > Any other ideas on how I can tell if a lazy property has been > initialized? > > > > Tom > > > > On Mon, Mar 21, 2011 at 4:40 PM, Sanne Grinovero > > wrote: > >> > >> 2011/3/21 Tom Waterhouse : > >> > Sanne, the unit test doesn't make any assertions, so I don't know how > >> > useful > >> > it would be as a test. It allows a user to view the > Hibernate-generated > >> > SQL, illustrating the entities loaded. > >> > >> Yes I saw that :) so I was asking if you could make a unit test out of > >> it to contribute to the project. > >> The wasInitialized() method I mentioned in the previous email is a > >> suggestion for a possible implementation. > >> > >> Sanne > >> > >> > > >> > On Mon, Mar 21, 2011 at 2:46 PM, Sanne Grinovero > > >> > wrote: > >> >> > >> >> Hi Tom, > >> >> thanks for the reminder, you might have noticed we're very busy > >> >> lately, so thank you for your help proposal. > >> >> > >> >> I've just committed a solution draft on my HSEARCH-679 branch: > >> >> > >> >> https://github.com/Sanne/hibernate-search/tree/HSEARCH-679 > >> >> > >> >> It's a draft but seems to work; I still have to get someone into a > >> >> code review, maybe you could start having a look and try it out? > >> >> I've run your tests, very useful thank you. Do you think you could > >> >> re-shape that in a patch for Search, so we can include that as > >> >> functional test? > >> >> I would need you to remove the dependencies from Spring, and to think > >> >> on a way to have the test fail if it loads the undesired collection > >> >> without looking into the logs. > >> >> > >> >> I think you might try casting it to a > >> >> org.hibernate.mapping.PersistentClass > >> >> and then see if it ".wasInitialized()". > >> >> > >> >> Regards, > >> >> Sanne > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> 2011/3/21 Tom Waterhouse : > >> >> > I see that HSEARCH-679 is scheduled to be released in Hibernate > >> >> > Search > >> >> > 3.4.0. I haven't seen any progress on the issue though. If it > will > >> >> > help > >> >> > moving the issue along I can definitely test proposed solutions, we > >> >> > can > >> >> > reproduce the issue consistently. Unfortunately we've reproduced > the > >> >> > issue > >> >> > during product demonstrations to potential customers, hence the > >> >> > urgency > >> >> > on > >> >> > our end to see the issue resolved. > >> >> > > >> >> > Cheers, > >> >> > > >> >> > Tom > >> >> > ___ > >> >> > hibernate-dev mailing list > >> >> > hibernate-dev@lists.jboss.org > >> >> > https://lists.jboss.org/mailman/listinfo/hibernate-dev > >> >> > > >> > > >> > > > > > > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
[hibernate-dev] Mocking for unit tests
I've completed a unit test for HSEARCH-679. Testing the issue was made easier using mock objects, as I needed to mock SearchFactoryImplementor and Worker to determine if indexing work was created during transaction commit. The mocking library I've used recently is Mockito, and is the library used for the test. I don't know that mocks have been used to this point for Hibernate Search testing. Mockito/mocking is acceptable? You can see the test I created as an attachment to HSEARCH-679. Tom ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] Mocking for unit tests
Emmanuel, The unit test in question is functional, in that it extends SearchTestCase. The test requires verification that searchFactoryImplementor.getWorker() is never called by FullTextIndexEventListener. Mockito is used in the current test 1) to mock both SearchFactoryImplementor and Worker, and 2) to verify that searchFactoryImplementor.getWorker() is never called A bona fide data model with annotations, a Hibernate session, CRUD operations, are performed by the test as well. Only the above listed items are mocked. Mockito makes the above trivial to implement, hence its usage. If that is palatable then I will leave the test as-is and introduce Mockito as a dependency. Tom On Tue, Mar 29, 2011 at 8:48 AM, Emmanuel Bernard wrote: > Hi Tom, > > Most tests in HSearch are more integration / functional tests than true > unit tests. Since we can tests everything with in-memory components it ends > up quite fast and more "real life". > > If you feel that there is no way to reproduce the issue at hand in a > functional test, is that: > - because you can't reproduce > - because it would require too much work? > > All in all, if you need Mockito, we can add it as a test dependency. but if > we can find a way to test the problem with a functional / integration test > that would be preferable. > > Emmanuel > > On 28 mars 2011, at 21:12, Tom Waterhouse wrote: > > > I've completed a unit test for HSEARCH-679. Testing the issue was made > > easier using mock objects, as I needed to mock SearchFactoryImplementor > and > > Worker to determine if indexing work was created during transaction > commit. > > > > The mocking library I've used recently is Mockito, and is the library > used > > for the test. I don't know that mocks have been used to this point for > > Hibernate Search testing. Mockito/mocking is acceptable? > > > > You can see the test I created as an attachment to HSEARCH-679. > > > > Tom > > ___ > > hibernate-dev mailing list > > hibernate-dev@lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hibernate-dev > > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
[hibernate-dev] How do I determine that the underlying database is a MySQL database?
I currently do not specify the hibernate.dialect in my configuration (which means Hibernate must be determining the dialect based upon my url, right?) but even if I did, I don't know how to get the property value at runtime. Whatever the case, what's the best way to determine the underlying database at runtime? Thanks in advance, Tom ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] HSEARCH-679 status
Sanne, I've added a comment to HSEARCH-679 that includes the instructions necessary to add the attached unit test to the hibernate-search code base. Let me know if you need anything more to check in the fix for HSEARCH-679. Tom On Tue, Mar 22, 2011 at 12:17 PM, Tom Waterhouse wrote: > Sanne, that testing strategy doesn't work unfortunately. The collection is > loaded by virtue of creating test data, so it is intialized. I don't see > how HSearch can be tested at the entity level. > > Perhaps a smaller unit test of FullIndexTextEventListener is the way to go > instead. I'm not sure what the Hibernate development testing methodology > is, but my suggestion would be to create a unit test of > processCollectionEvent with mocks as necessary. > > Tom > > > On Tue, Mar 22, 2011 at 11:34 AM, Sanne Grinovero wrote: > >> Hi Tom, >> sorry, copy paste error: I meant >> org.hibernate.collection.PersistentCollection, all collections >> retrieved from the database implement PersistentCollection, which >> exposes the wasInitialized() method. >> So just cast the collection to PersistentCollection and you should be >> able to check if it was fully initialized. >> >> BTW I fixed the pending issue on github, as soon as you have a >> testcase we can proceed with a pull request. >> >> Sanne >> >> 2011/3/22 Tom Waterhouse : >> > Sanne, >> > >> > I don't see the method you've identified in PersistentClass. The >> closest I >> > could find to identifying if a property has been initialized is >> > Hibernate.isPropertyInitialized(), but the method always returns true >> > (object isn't a HibernateProxy, FieldInterceptionHelper.isInstrumented() >> > returns false). >> > >> > Any other ideas on how I can tell if a lazy property has been >> initialized? >> > >> > Tom >> > >> > On Mon, Mar 21, 2011 at 4:40 PM, Sanne Grinovero >> > wrote: >> >> >> >> 2011/3/21 Tom Waterhouse : >> >> > Sanne, the unit test doesn't make any assertions, so I don't know how >> >> > useful >> >> > it would be as a test. It allows a user to view the >> Hibernate-generated >> >> > SQL, illustrating the entities loaded. >> >> >> >> Yes I saw that :) so I was asking if you could make a unit test out of >> >> it to contribute to the project. >> >> The wasInitialized() method I mentioned in the previous email is a >> >> suggestion for a possible implementation. >> >> >> >> Sanne >> >> >> >> > >> >> > On Mon, Mar 21, 2011 at 2:46 PM, Sanne Grinovero < >> sa...@hibernate.org> >> >> > wrote: >> >> >> >> >> >> Hi Tom, >> >> >> thanks for the reminder, you might have noticed we're very busy >> >> >> lately, so thank you for your help proposal. >> >> >> >> >> >> I've just committed a solution draft on my HSEARCH-679 branch: >> >> >> >> >> >> https://github.com/Sanne/hibernate-search/tree/HSEARCH-679 >> >> >> >> >> >> It's a draft but seems to work; I still have to get someone into a >> >> >> code review, maybe you could start having a look and try it out? >> >> >> I've run your tests, very useful thank you. Do you think you could >> >> >> re-shape that in a patch for Search, so we can include that as >> >> >> functional test? >> >> >> I would need you to remove the dependencies from Spring, and to >> think >> >> >> on a way to have the test fail if it loads the undesired collection >> >> >> without looking into the logs. >> >> >> >> >> >> I think you might try casting it to a >> >> >> org.hibernate.mapping.PersistentClass >> >> >> and then see if it ".wasInitialized()". >> >> >> >> >> >> Regards, >> >> >> Sanne >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> 2011/3/21 Tom Waterhouse : >> >> >> > I see that HSEARCH-679 is scheduled to be released in Hibernate >> >> >> > Search >> >> >> > 3.4.0. I haven't seen any progress on the issue though. If it >> will >> >> >> > help >> >> >> > moving the issue along I can definitely test proposed solutions, >> we >> >> >> > can >> >> >> > reproduce the issue consistently. Unfortunately we've reproduced >> the >> >> >> > issue >> >> >> > during product demonstrations to potential customers, hence the >> >> >> > urgency >> >> >> > on >> >> >> > our end to see the issue resolved. >> >> >> > >> >> >> > Cheers, >> >> >> > >> >> >> > Tom >> >> >> > ___ >> >> >> > hibernate-dev mailing list >> >> >> > hibernate-dev@lists.jboss.org >> >> >> > https://lists.jboss.org/mailman/listinfo/hibernate-dev >> >> >> > >> >> > >> >> > >> > >> > >> > > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] HSEARCH-679 status
Sanne, 1) fixed - sequence of checking method interaction was incorrect, that has been fixed, along with a test to verify that indexing DOES occur as expected 2) fixed - the test was moved to org.hibernate.search.test.event. This required the registration of an extension of FullTextIndexEventListener as suggested below 3) fixed - ooops! The tests are attached to HSEARCH-679 as HSEARCH-679-test-updated. Tom On Mon, Apr 4, 2011 at 12:34 PM, Sanne Grinovero wrote: > Hi Tom, > thank you, I'm playing with it right now. > > As I mentioned on JIRA, the main reason for which I didn't fix this > yet is that my proposed solution is incomplete, I'm working on a real > solution and a good test would help a lot. > > I'm having some problems with your code: > > 1) The test is always green: > even if I don't fix anything, the test is fine. > > 2) We're having all tests in the package org.hibernate.search.TEST, I > see you had to use org.hibernate.search.event to be able to assign > protected fields. > I've had a similar issue other times, so I'll propose to change this > convention, but for the short time you have many possible workarounds: > - by registering instead an extensions of FullTextIndexEventListener > - by reassigning the field via reflection (you make it public) > - by using Byteman you can avoid all the mocking and have the > framework "blow up" if it ever tries to perform some collections work > > 3) "Copyright (c) 2011 Attensa, Inc. All rights reserved." > this is the most critical. I don't think I can add a class having such > a clause to the Hibernate codebase, I'll need you to provide a "clean" > copy. > > I'm going to try fixing the implementation tonight, do you think > you'll be able to fix the tests for no later than tomorrow afternoon > (release date!) ? > > Cheers, > Sanne > > > 2011/4/4 Tom Waterhouse : > > Sanne, > > > > I've added a comment to HSEARCH-679 that includes the instructions > necessary > > to add the attached unit test to the hibernate-search code base. Let me > > know if you need anything more to check in the fix for HSEARCH-679. > > > > Tom > > > > On Tue, Mar 22, 2011 at 12:17 PM, Tom Waterhouse < > tomwaterho...@gmail.com> > > wrote: > >> > >> Sanne, that testing strategy doesn't work unfortunately. The collection > >> is loaded by virtue of creating test data, so it is intialized. I don't > see > >> how HSearch can be tested at the entity level. > >> > >> Perhaps a smaller unit test of FullIndexTextEventListener is the way to > go > >> instead. I'm not sure what the Hibernate development testing > methodology > >> is, but my suggestion would be to create a unit test of > >> processCollectionEvent with mocks as necessary. > >> > >> Tom > >> > >> On Tue, Mar 22, 2011 at 11:34 AM, Sanne Grinovero > >> wrote: > >>> > >>> Hi Tom, > >>> sorry, copy paste error: I meant > >>> org.hibernate.collection.PersistentCollection, all collections > >>> retrieved from the database implement PersistentCollection, which > >>> exposes the wasInitialized() method. > >>> So just cast the collection to PersistentCollection and you should be > >>> able to check if it was fully initialized. > >>> > >>> BTW I fixed the pending issue on github, as soon as you have a > >>> testcase we can proceed with a pull request. > >>> > >>> Sanne > >>> > >>> 2011/3/22 Tom Waterhouse : > >>> > Sanne, > >>> > > >>> > I don't see the method you've identified in PersistentClass. The > >>> > closest I > >>> > could find to identifying if a property has been initialized is > >>> > Hibernate.isPropertyInitialized(), but the method always returns true > >>> > (object isn't a HibernateProxy, > >>> > FieldInterceptionHelper.isInstrumented() > >>> > returns false). > >>> > > >>> > Any other ideas on how I can tell if a lazy property has been > >>> > initialized? > >>> > > >>> > Tom > >>> > > >>> > On Mon, Mar 21, 2011 at 4:40 PM, Sanne Grinovero < > sa...@hibernate.org> > >>> > wrote: > >>> >> > >>> >> 2011/3/21 Tom Waterhouse : > >>> >> > Sanne, the unit test doesn'
[hibernate-dev] Where to start with MongoDB support for OMG?
If a person wanted to start down the road of adding MongoDB support for OMG, where would the best place to start be? Tom ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] Where to start with MongoDB support for OMG?
Outstanding, this is a good list. I'll jump in head-first and see what happens (probably lots of bumps and bruises, of course...). Cheers, Tom On Wed, Jun 22, 2011 at 3:05 AM, Emmanuel Bernard wrote: > Hello Tom, > If such person existed, that would be awesome :) > > Here are the steps. > Most if not all operations to the datastore are hidden behind the > GridDialect interface. But there are a few steps towards complete success > > 1. Finish to abstract away GridDialect from Infinispan's Cache API > http://opensource.atlassian.com/projects/hibernate/browse/OGM-79 > > 2. Migrate from a Map to a Tuple object representation > storing diffs (optional but likely useful) > http://opensource.atlassian.com/projects/hibernate/browse/OGM-69 > > 3. Figure out an DatastoreConnectionProvider concept (today it's there but > hard coded somehow and with horrible names) > > 4. Write the GridDialect for MongoDB and especially try and see if the > GridDialect contract needs to evolve and see which MongoDB operations to use > to map the contract > > 5. Have a deep discussion on the MongoDB transactional model and see if > Hibernate OGM needs adjustment and write docs :) > > It looks like a long list but all of these are clearly connected and > working on 1 and 2 for example are great to understand how Hibernate OGM > interacts with the datastore and will help for 4 and 5. > > I'm on and offline in the next couple of days but at least I read my emails > and I try to be on #hibernate-dev > > Emmanuel > > On 21 juin 2011, at 18:25, Tom Waterhouse wrote: > > > If a person wanted to start down the road of adding MongoDB support for > OMG, > > where would the best place to start be? > > > > Tom > > ___ > > hibernate-dev mailing list > > hibernate-dev@lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hibernate-dev > > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
[hibernate-dev] MassIndexer have any known issues when InfinispanDirectory is used?
I'm trying to setup clustering of entities and Lucene indexes for our app with Hibernate 3.6.5, Hibernate Search 3.4.0, Infinispan 5.0. I'm using FileCacheStore for the Infinispan cache loader (InfinispanDirectoryProvider). MassIndexerImpl.startAndWait() never returns with this configuration. A lock is never able to be obtained, see the stack from a thread dump below. The same MassIndexer call works fine when using FSDirectoryProvider. Should MassIndexer work with Infinispan as the directory? Tom java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.lucene.store.Lock.obtain(Lock.java:91) at org.apache.lucene.index.IndexWriter.(IndexWriter.java:1097) at org.hibernate.search.backend.Workspace.createNewIndexWriter(Workspace.java:202) at org.hibernate.search.backend.Workspace.getIndexWriter(Workspace.java:175) - locked <7793180e8> (a org.hibernate.search.backend.Workspace) ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] MassIndexer have any known issues when InfinispanDirectory is used?
Sanne, There aren't any other nodes involved in the cluster. This is the 'just make it work' phase of the project, so the simplest configuration is being used. Note that normal index access is fine. Entity operations populate the Lucene indexes as expected, and search operations work as expected. It is only the MassIndexer that has had trouble to this point. Tom On Fri, Aug 26, 2011 at 7:02 AM, Sanne Grinovero wrote: > Hi Tom, > > the MassIndexer needs to acquire the Directory lock, which is in this > case distributed, i.e. it's a single lock to coordinate writes across > all nodes (searches can happen in parallel, but writes can not). > > Is it possible that another node is writing to the index, or is any > node using exclusive_index_use=true ? > > Regards, > Sanne > > 2011/8/25 Tom Waterhouse : > > I'm trying to setup clustering of entities and Lucene indexes for our app > > with Hibernate 3.6.5, Hibernate Search 3.4.0, Infinispan 5.0. I'm using > > FileCacheStore for the Infinispan cache loader > > (InfinispanDirectoryProvider). > > > > MassIndexerImpl.startAndWait() never returns with this configuration. A > > lock is never able to be obtained, see the stack from a thread dump > below. > > The same MassIndexer call works fine when using FSDirectoryProvider. > > > > Should MassIndexer work with Infinispan as the directory? > > > > Tom > > > > java.lang.Thread.State: TIMED_WAITING (sleeping) > >at java.lang.Thread.sleep(Native Method) > >at org.apache.lucene.store.Lock.obtain(Lock.java:91) > >at org.apache.lucene.index.IndexWriter.(IndexWriter.java:1097) > >at > > > org.hibernate.search.backend.Workspace.createNewIndexWriter(Workspace.java:202) > >at > > org.hibernate.search.backend.Workspace.getIndexWriter(Workspace.java:175) > >- locked <7793180e8> (a org.hibernate.search.backend.Workspace) > > ___ > > hibernate-dev mailing list > > hibernate-dev@lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hibernate-dev > > > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] MassIndexer have any known issues when InfinispanDirectory is used?
I set a breakpoint inside of org.apache.lucene.store.Lock.obtain(long) and
noticed something peculiar - the method is called twice. The first call
succeeds, the second fails, my guess because the first call obtained the
lock.
I added logging/stepped through our code to verify that we only make the
call one time, and in fact that is the case.
We are using JPA and JTA, would that impact the call to obtain the lock in
such a way as to allow two calls?
Here is the call we make, including the logging statements I've added. This
call is done inside of a JTA transaction.
FullTextEntityManager fullTextEntityManager =
org.hibernate.search.jpa.Search.getFullTextEntityManager(em);
logger.info("creating MassIndexer");
MassIndexer massIndexer =
fullTextEntityManager.createIndexer(Article.class);
massIndexer.batchSizeToLoadObjects(30);
massIndexer.threadsForSubsequentFetching(8);
massIndexer.threadsToLoadObjects(4);
logger.info("starting MassIndexer");
massIndexer.startAndWait();
On Sun, Aug 28, 2011 at 9:28 AM, Sanne Grinovero wrote:
> Hi Tom,
> I've created a test checking for both event-driven changes and the
> MassIndexer, double-checking event-driven changes after the
> MassIndexer completion, starting two nodes initially and adding a
> third node dynamically during the test execution.. alls seems to work
> flawlessly, a part of taking so long due the jgroups delays for
> starting a new node (it takes ~14 seconds to run).
> you can find it here:
>
> https://github.com/Sanne/hibernate-search/tree/MassIndexerWithInfinispan5.0-Search3.4
>
> Could you please check this test out and change it as much as you feel
> is needed to reproduce the problem?
>
> Also note the previous commit to change the Infinispan version to 5.0:
> you mentioned you're using 5.0 but Search 3.4 was intended to support
> Infinispan 4.2.x, so I had to apply some minimal changes.
>
> You might want to try out Hibernate Search 4.0.0.Alpha1, intended to
> support Infinispan 5.x, but I've created this test for Search 3.4 as
> the backends interaction in 4.0 is very different: there are not two
> competing backends anymore, but a unified access to the IndexWriter,
> so to try reproducing your issue it was pointless to try it out on
> master.
>
> Sanne
>
> 2011/8/27 Tom Waterhouse :
> > Sanne,
> >
> > There aren't any other nodes involved in the cluster. This is the 'just
> > make it work' phase of the project, so the simplest configuration is
> being
> > used.
> >
> > Note that normal index access is fine. Entity operations populate the
> > Lucene indexes as expected, and search operations work as expected. It
> is
> > only the MassIndexer that has had trouble to this point.
> >
> > Tom
> >
> > On Fri, Aug 26, 2011 at 7:02 AM, Sanne Grinovero
> > wrote:
> >>
> >> Hi Tom,
> >>
> >> the MassIndexer needs to acquire the Directory lock, which is in this
> >> case distributed, i.e. it's a single lock to coordinate writes across
> >> all nodes (searches can happen in parallel, but writes can not).
> >>
> >> Is it possible that another node is writing to the index, or is any
> >> node using exclusive_index_use=true ?
> >>
> >> Regards,
> >> Sanne
> >>
> >> 2011/8/25 Tom Waterhouse :
> >> > I'm trying to setup clustering of entities and Lucene indexes for our
> >> > app
> >> > with Hibernate 3.6.5, Hibernate Search 3.4.0, Infinispan 5.0. I'm
> using
> >> > FileCacheStore for the Infinispan cache loader
> >> > (InfinispanDirectoryProvider).
> >> >
> >> > MassIndexerImpl.startAndWait() never returns with this configuration.
> A
> >> > lock is never able to be obtained, see the stack from a thread dump
> >> > below.
> >> > The same MassIndexer call works fine when using FSDirectoryProvider.
> >> >
> >> > Should MassIndexer work with Infinispan as the directory?
> >> >
> >> > Tom
> >> >
> >> > java.lang.Thread.State: TIMED_WAITING (sleeping)
> >> >at java.lang.Thread.sleep(Native Method)
> >> >at org.apache.lucene.store.Lock.obtain(Lock.java:91)
> >> >at
> org.apache.lucene.index.IndexWriter.(IndexWriter.java:1097)
> >> >at
> >> >
> >> >
> org.hibernate.search.backend.Workspace.createNewIndexWriter(Workspace.java:202)
> >> >at
> >> >
> >> >
> org.hibernate.search.backend.Workspace.getIndexWriter(Workspace.java:175)
> >> >- locked <7793180e8> (a org.hibernate.search.backend.Workspace)
> >> > ___
> >> > hibernate-dev mailing list
> >> > hibernate-dev@lists.jboss.org
> >> > https://lists.jboss.org/mailman/listinfo/hibernate-dev
> >> >
> >
> >
>
___
hibernate-dev mailing list
hibernate-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] MassIndexer have any known issues when InfinispanDirectory is used?
Here is the associated stack at the time of the call, if it would be
helpful. The stack is the same for both the initial call that obtains the
lock and the second and subsequent calls that fail.
BaseLuceneLock(Lock).obtain(long) line: 72
IndexWriter.(Directory, IndexWriterConfig) line: 1115
Workspace.createNewIndexWriter(DirectoryProvider, IndexWriterConfig,
ParameterSet) line: 202
Workspace.getIndexWriter(boolean, ErrorContextBuilder) line: 175
Workspace.getIndexWriter(boolean) line: 210
DirectoryProviderWorkspace.doWorkInSync(LuceneWork) line: 96
LuceneBatchBackend$SyncBatchPerDirectoryWorkProcessor.addWorkToDpProcessor(DirectoryProvider,
LuceneWork) line: 139
DpSelectionVisitor$PurgeAllSelectionDelegate.addAsPayLoadsToQueue(LuceneWork,
IndexShardingStrategy, PerDirectoryWorkProcessor) line: 119
LuceneBatchBackend.sendWorkToShards(LuceneWork,
PerDirectoryWorkProcessor) line: 119
LuceneBatchBackend.doWorkInSync(LuceneWork) line: 80
BatchCoordinator.beforeBatch() line: 153
BatchCoordinator.run() line: 96
MassIndexerImpl.startAndWait() line: 203
ArticleSearchControllerImpl.reindexAllArticles() line: 215
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not
available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25
Method.invoke(Object, Object...) line: 597
AopUtils.invokeJoinpointUsingReflection(Object, Method, Object[]) line:
307
ReflectiveMethodInvocation.invokeJoinpoint() line: 183
ReflectiveMethodInvocation.proceed() line: 150
TransactionInterceptor.invoke(MethodInvocation) line: 110
ReflectiveMethodInvocation.proceed() line: 172
JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 202
$Proxy168.reindexAllArticles() line: not available
ReindexArticlesJob.executeJob(JobExecutionContext) line: 24
ReindexArticlesJob(AbstractJob).execute(JobExecutionContext) line: 79
JobRunShell.run() line: 214
SimpleThreadPool$WorkerThread.run() line: 549
On Mon, Aug 29, 2011 at 2:49 PM, Tom Waterhouse wrote:
> I set a breakpoint inside of org.apache.lucene.store.Lock.obtain(long) and
> noticed something peculiar - the method is called twice. The first call
> succeeds, the second fails, my guess because the first call obtained the
> lock.
>
> I added logging/stepped through our code to verify that we only make the
> call one time, and in fact that is the case.
>
> We are using JPA and JTA, would that impact the call to obtain the lock in
> such a way as to allow two calls?
>
> Here is the call we make, including the logging statements I've added.
> This call is done inside of a JTA transaction.
>
> FullTextEntityManager fullTextEntityManager =
> org.hibernate.search.jpa.Search.getFullTextEntityManager(em);
>
> logger.info("creating MassIndexer");
> MassIndexer massIndexer =
> fullTextEntityManager.createIndexer(Article.class);
> massIndexer.batchSizeToLoadObjects(30);
> massIndexer.threadsForSubsequentFetching(8);
> massIndexer.threadsToLoadObjects(4);
> logger.info("starting MassIndexer");
> massIndexer.startAndWait();
>
>
>
> On Sun, Aug 28, 2011 at 9:28 AM, Sanne Grinovero wrote:
>
>> Hi Tom,
>> I've created a test checking for both event-driven changes and the
>> MassIndexer, double-checking event-driven changes after the
>> MassIndexer completion, starting two nodes initially and adding a
>> third node dynamically during the test execution.. alls seems to work
>> flawlessly, a part of taking so long due the jgroups delays for
>> starting a new node (it takes ~14 seconds to run).
>> you can find it here:
>>
>> https://github.com/Sanne/hibernate-search/tree/MassIndexerWithInfinispan5.0-Search3.4
>>
>> Could you please check this test out and change it as much as you feel
>> is needed to reproduce the problem?
>>
>> Also note the previous commit to change the Infinispan version to 5.0:
>> you mentioned you're using 5.0 but Search 3.4 was intended to support
>> Infinispan 4.2.x, so I had to apply some minimal changes.
>>
>> You might want to try out Hibernate Search 4.0.0.Alpha1, intended to
>> support Infinispan 5.x, but I've created this test for Search 3.4 as
>> the backends interaction in 4.0 is very different: there are not two
>> competing backends anymore, but a unified access to the IndexWriter,
>> so to try reproducing your issue it was pointless to try it out on
>> master.
>>
>> Sanne
>>
>> 2011/8/27 Tom Waterhouse :
>> > Sanne,
>> >
>> > There aren't any other nodes in
Re: [hibernate-dev] MassIndexer have any known issues when InfinispanDirectory is used?
I have a test case available. In the test case a org.apache.lucene.store.LockObtainFailedException instance is thrown. Should I attach the test case to an email sent to this list? On Tue, Aug 30, 2011 at 12:50 AM, Sanne Grinovero wrote: > Is it using exclusive_index_use=true ? Exclusive index use does not > work nicely with the MassIndexer in version 3.4, that's one of the > issues fixed in 4.0 (included in the last alpha release). > Is the same test & configuration working fine if you change it only > from Infinispan to a filesystem Directory ? > > I'm asking these questions as the MassIndexer will grab this exclusive > lock from the main backend, and the "main backend" is supposed to not > have any pending activity when it's started. Are you aware of other > ongoing writes to the index while the MassIndexer starts? > > If there is a limited amount of activity it should be able to acquire > the lock, but in the way Lucene's BaseLuceneLock works it's only > polling, not a fair lock, so if the standard backend is constantly > writing it might timeout waiting for the writes to finish. > > A testcase would help; as pointed out in my previous mail I've written > one and found no issues, so please help me reproducing the issue. You > could also open an issue and attach both your Infinispan and Search > configurations. > > Regards, > Sanne > > 2011/8/29 Tom Waterhouse : > > Here is the associated stack at the time of the call, if it would be > > helpful. The stack is the same for both the initial call that obtains > the > > lock and the second and subsequent calls that fail. > > > > BaseLuceneLock(Lock).obtain(long) line: 72 > > IndexWriter.(Directory, IndexWriterConfig) line: 1115 > > Workspace.createNewIndexWriter(DirectoryProvider, > IndexWriterConfig, > > ParameterSet) line: 202 > > Workspace.getIndexWriter(boolean, ErrorContextBuilder) line: 175 > > Workspace.getIndexWriter(boolean) line: 210 > > DirectoryProviderWorkspace.doWorkInSync(LuceneWork) line: 96 > > > > > LuceneBatchBackend$SyncBatchPerDirectoryWorkProcessor.addWorkToDpProcessor(DirectoryProvider, > > LuceneWork) line: 139 > > > > > DpSelectionVisitor$PurgeAllSelectionDelegate.addAsPayLoadsToQueue(LuceneWork, > > IndexShardingStrategy, PerDirectoryWorkProcessor) line: 119 > > LuceneBatchBackend.sendWorkToShards(LuceneWork, > > PerDirectoryWorkProcessor) line: 119 > > LuceneBatchBackend.doWorkInSync(LuceneWork) line: 80 > > BatchCoordinator.beforeBatch() line: 153 > > BatchCoordinator.run() line: 96 > > MassIndexerImpl.startAndWait() line: 203 > > ArticleSearchControllerImpl.reindexAllArticles() line: 215 > > NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not > > available [native method] > > NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39 > > DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 > > Method.invoke(Object, Object...) line: 597 > > AopUtils.invokeJoinpointUsingReflection(Object, Method, Object[]) > line: > > 307 > > ReflectiveMethodInvocation.invokeJoinpoint() line: 183 > > ReflectiveMethodInvocation.proceed() line: 150 > > TransactionInterceptor.invoke(MethodInvocation) line: 110 > > ReflectiveMethodInvocation.proceed() line: 172 > > JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 202 > > $Proxy168.reindexAllArticles() line: not available > > ReindexArticlesJob.executeJob(JobExecutionContext) line: 24 > > ReindexArticlesJob(AbstractJob).execute(JobExecutionContext) line: 79 > > JobRunShell.run() line: 214 > > SimpleThreadPool$WorkerThread.run() line: 549 > > > > > > On Mon, Aug 29, 2011 at 2:49 PM, Tom Waterhouse > > > wrote: > >> > >> I set a breakpoint inside of org.apache.lucene.store.Lock.obtain(long) > and > >> noticed something peculiar - the method is called twice. The first call > >> succeeds, the second fails, my guess because the first call obtained the > >> lock. > >> > >> I added logging/stepped through our code to verify that we only make the > >> call one time, and in fact that is the case. > >> > >> We are using JPA and JTA, would that impact the call to obtain the lock > in > >> such a way as to allow two calls? > >> > >> Here is the call we make, including the logging statements I've added. > >> This call is done inside of a JTA transaction. > >> > >> FullTextEntityManager fullTextEntit
Re: [hibernate-dev] MassIndexer have any known issues when InfinispanDirectory is used?
https://hibernate.onjira.com/browse/HSEARCH-893 On Wed, Aug 31, 2011 at 12:46 PM, Sanne Grinovero wrote: > 2011/8/31 Tom Waterhouse : > > I have a test case available. In the test case a > > org.apache.lucene.store.LockObtainFailedException instance is thrown. > > > > Should I attach the test case to an email sent to this list? > > It's better if you create an issue and attach it there. > Thank you! > Sanne > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
[hibernate-dev] session-per-application in a rich client an antipattern?
I've searched the boards for a while and seen a few posts alluding to the same issue but never a conclusive answer. The application - swing rich client on a single user local db. In this situation I think a session-per-application is desirable. If I load an entity, say the contact "Tom Hughes", I want only one object instance for that entity at any one time. Therefore if edit 'Tom Hughes" on my search screen, it is updated on my "Favourites Contacts" screen because it is the same object instance. Having a shared instance avoids me having stale data on my application's forms and avoids me having to listen to update events (via the interceptor or event mechanism or other means) and reload my forms as needed - this has a negative impact on performance, robustness and complexity. I do realise the power of sessions and how they are used, please don't point me to here : http://hibernate.org/42.html#A11 I think though in my single useer db, I don't care about multiple sessions. They only serve to complicate. Unfortunately I don't think I can implement a session-per-application because of the behaviour of the first level 'cache', ie. the session. That is, entities loaded into the session stay there until the session is closed. I was thinking it may be possible to write a session that used weak references for all non-dirty entities. Entities could then be garbage collected when my app no longer needed them. I would be willing to contribute this work if feasible. So the questions, * I'm not a hibernate expert so are there any flaws in my reasoning? * Should I be using hibernate this way in a two teir architecture?* Is there an existing method to manage the first level cache? * Is the suggestion of a weakly referenced session implementation feasible? This would be super, super useful and from my research, I'm not the only one having these problems. thanks, Tom Hughes ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
RE: [hibernate-dev] session-per-application in a rich client an antipattern?
Tried that : http://forum.hibernate.org/viewtopic.php?t=965080 no reply. If this is not the correct channel for this follow up apologies and would somebody indicate the correct one please. Thanks, Tom -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Christian Bauer Sent: Monday, 25 September 2006 10:09 AM To: hibernate-dev@lists.jboss.org Subject: Re: [hibernate-dev] session-per-application in a rich client anantipattern? On Sep 25, 2006, at 2:01 AM, Tom Hughes wrote: > This would be super, super useful and from my research, I'm not the > only one having these problems. http://forum.hibernate.org ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
RE: [hibernate-dev] session-per-application in a rich client an antipattern?
Great, thanks for the help... -Original Message- From: Christian Bauer [mailto:[EMAIL PROTECTED] Sent: Monday, 25 September 2006 10:17 AM To: Tom Hughes Cc: hibernate-dev@lists.jboss.org Subject: Re: [hibernate-dev] session-per-application in a rich client an antipattern? On Sep 25, 2006, at 2:14 AM, Tom Hughes wrote: > Tried that : > http://forum.hibernate.org/viewtopic.php?t=965080 > > no reply. > > If this is not the correct channel for this follow up apologies and > would somebody indicate the correct one please. This is the developer list, so you are wrong here. Stay on the forum for usage questions. ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
[hibernate-dev] Shortcomings in org.hibernate.id.enhanced.TableGenerator class
I'm working with the org.hibernate.id.enhanced.TableGenerator class and I've noticed a couple of shortcomings (imho). First and foremost, it creates one row in the hibernate_sequences table for each configured segment_value (as opposed to one row for each "target_table" as implemented within the MultipleHiLoPerTableGenerator). Given the fact that my id column is defined in an abstract BaseEntity class, I can't use the enhanced TableGenerator class as is (since one sequence for all of my entities is not viable for my application). Which leads me to my second point. All of the member variables are declared as private, instead of protected, which makes extending the class for the purpose of overriding the configure method impossible. For what it's worth, here's the change that I had planned to make (in the configure method) ... Change the setting of the segmentValue from ... segmentValue = params.getProperty( SEGMENT_VALUE_PARAM ); ... to segmentValue = params.getProperty(TABLE); Of course, such a change would change the class behavior significantly - using targetTable instead of segmentValue as the variable name would probably make sense. So, with that said, is another class worth-while (assuming that such a change to the existing class would break backwards compatability)? And, on a related note, the enhanced TableGenerator is cluster safe, right? Thoughts? PS. I entered a jira improvement yesterday - here's the link ... http://opensource.atlassian.com/projects/hibernate/browse/HHH-3249 ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
[hibernate-dev] Shortcomings in org.hibernate.id.enhanced.TableGenerator class
I apologize if this is a 2nd posting. From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tom Muldoon Sent: Thursday, April 24, 2008 8:04 AM To: 'hibernate-dev@lists.jboss.org' Subject: [hibernate-dev] Shortcomings in org.hibernate.id.enhanced.TableGenerator class I'm working with the org.hibernate.id.enhanced.TableGenerator class and I've noticed a couple of shortcomings (imho). First and foremost, it creates one row in the hibernate_sequences table for each configured segment_value (as opposed to one row for each "target_table" as implemented within the MultipleHiLoPerTableGenerator). Given the fact that my id column is defined in an abstract BaseEntity class, I can't use the enhanced TableGenerator class as is (since one sequence for all of my entities is not viable for my application). Which leads me to my second point. All of the member variables are declared as private, instead of protected, which makes extending the class for the purpose of overriding the configure method impossible. For what it's worth, here's the change that I had planned to make (in the configure method) ... Change the setting of the segmentValue from ... segmentValue = params.getProperty( SEGMENT_VALUE_PARAM ); ... to segmentValue = params.getProperty(TABLE); Of course, such a change would change the class behavior significantly - using targetTable instead of segmentValue as the variable name would probably make sense. So, with that said, is another class worth-while (assuming that such a change to the existing class would break backwards compatability)? And, on a related note, the enhanced TableGenerator is cluster safe, right? Thoughts? PS. I entered a jira improvement yesterday - here's the link ... http://opensource.atlassian.com/projects/hibernate/browse/HHH-3249 ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
