Re: [hibernate-dev] HHH-1123 - Cannot put more than 1000 elements in a InExpression
> While I do agree wholeheartedly with most of what you say, I do think it's > unfair to say that the requirement is "fundamentally broken". That label > should be reserved for SQL itself ;-) ok, let me rephrase "its a fundamentally broken way to do queries knowing how majority of all relational database servers are implemented to do this kinda of query", better ? :) SQL itself is not the problem here. > If there is some opaque business rule R that operates on a large set of input > data, and that rule is implemented in Java, using Hibernate to get the input > data, it's quite possible that the rule generates a large list of entities to > retrieve. Is there a better way to batch fetch a s***load of specific > entities, given a List or Set of identifiers? Yes, session.load to set of queries that doesn't put 1000 elements into an in list. Or change your query heavy logic into using the database what it is good for (joins) and do the heavy computation in memory…. I also do understand that it would be easier to do some of these things in java - but I would argue that in majority of cases your code will be simpler and much faster. …but that doesn't help users that wants to just use java for computation. > But I DO think it's actually easier on the temp segment than to not use it, > in most cases. yes, but as Steve says - no good generic way of implementing this. /max > > David > > On 12/03/2011 05:23 AM, Max Rydahl Andersen wrote: >>> One technical (and probably way out of scope!) way to handle this would >>> be to use a temp table, do a batch insert of the values, then change the >>> " in (v1, v2, v3...)" to " in (select v from temp)". >> I think I would rather hear people complain about query exceptions happening >> when >> they are doing something fundamentally broken with a database than seeing >> them realize their test >> queries working as expected and then when they go to production with enough >> data to pass the 1000 elements limit >> their read only queries suddenly are trashing the temp table space and their >> user need to be able to have create table permissions. >> >> /max >> >> >> >>> >>> >>> On 12/01/2011 04:20 AM, Emmanuel Bernard wrote: Ah good point, I haven't thought of that problem with query splitting On 30 nov. 2011, at 22:20, Steve Ebersole wrote: > Splitting is not always an option. Consider a predicate like: > > ... where a in (x1, ... x2000) and b in (y1, ... y2000) ... > > If you split this up, you will have misses. Yes, it works if you can > keep it all in one query because you can structure it to maintain the > original semantics. However, please read the comments on that JIRA > issue. For some databases, this restriction is not just on the number of > elements in a in-list, but on the number of parameters overall. > Splitting these 2 in-lists about into 4 does not address that. > > I commented on the issue that I am actually inclined to simply reject > this one. In fact, I thought we already did. Maybe that was another > earlier one? > > > On Wed 30 Nov 2011 04:45:55 AM CST, Emmanuel Bernard wrote: >> Also note that there is a limit for the query size globally in some >> vendors and that people relieved from HHH-1123 cal fall into the second >> limit. >> A solution would be for Hibernate to split one query into several but >> I'm not sure I like the idea. >> >> Emmanuel >> >> On 29 nov. 2011, at 21:29, Łukasz Antoniak wrote: >> >>> Hi all! >>> >>> Recently I had a closer look at HHH-1123 issue. This bug affects both - >>> Criteria API and HQL. I have introduced >>> Dialect#maximumInExpressionElements() method which returns maximum >>> number of allowed elements in a single SQL IN clause, or null treated as >>> infinite. The change of InExpression was very easy. However, fixing this >>> bug for HQL queries requires modification of ParameterMetadata >>> (namedDescriptorMap cannot remain unmodifiable), as well as >>> AbstractQueryImpl (queryString). As I don't see any other solution, I >>> wanted to ask you guys for suggestions. Is it the only possible way of >>> fixing this issue? Finally, shall we really fix this? This is a DB >>> vendor limitation, but 40 user gave their vote for it. >>> >>> Regards, >>> Lukasz Antoniak >>> ___ >>> hibernate-dev mailing list >>> hibernate-dev@lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/hibernate-dev >> ___ >> hibernate-dev mailing list >> hibernate-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hibernate-dev > -- > st...@hibernate.org > http://hibernate.org ___ hibernate-dev
[hibernate-dev] Hibernate Developer Weekly IRC Meeting - 12/6
[08:46] Meeting ended Tue Dec 6 14:46:30 2011 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) [08:46] Minutes: http://transcripts.jboss.org/meeting/irc.freenode.org/hibernate-dev/2011/hibernate-dev.2011-12-06-14.03.html [08:46] Minutes (text): http://transcripts.jboss.org/meeting/irc.freenode.org/hibernate-dev/2011/hibernate-dev.2011-12-06-14.03.txt [08:46] Log: http://transcripts.jboss.org/meeting/irc.freenode.org/hibernate-dev/2011/hibernate-dev.2011-12-06-14.03.log.html -- st...@hibernate.org http://hibernate.org ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] HHH-1123 - Cannot put more than 1000 elements in a InExpression
On 12/05/2011 04:44 PM, Steve Ebersole wrote:
> On Mon 05 Dec 2011 12:59:19 PM CST, David Mansfield wrote:
>> While I do agree wholeheartedly with most of what you say, I do think
>> it's unfair to say that the requirement is "fundamentally broken". That
>> label should be reserved for SQL itself ;-)
>>
>> If there is some opaque business rule R that operates on a large set of
>> input data, and that rule is implemented in Java, using Hibernate to get
>> the input data, it's quite possible that the rule generates a large list
>> of entities to retrieve. Is there a better way to batch fetch a
>> s***load of specific entities, given a List or Set of identifiers?
>
> for ( Long identifier : setOrListOfIdentifiers ) {
>session.get( YourEntity.class, identifier );
> }
>
I guess if you use session.load instead of session.get you won't get
10,000 queries over the network if batch fetching is enabled, which does
exactly what the original issue discusses, it splits the "in" clause
into chunks and executes it in an intelligent manner. It's a shame the
same can't be worked into a solution when a criteria or hql query is
used - to me this is more a breakage in SQL than in hibernate.
If you use session.get your performance will be absolutely horrendous
compared to an "in" or global temp table (I state this mainly for the
record, not because it's some new information to you ;-)
Of course you are completely right that a global temp is unworkable in
reality though.
Thanks,
David
___
hibernate-dev mailing list
hibernate-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] HHH-1123 - Cannot put more than 1000 elements in a InExpression
On Tue 06 Dec 2011 09:00:00 AM CST, David Mansfield wrote: > I guess if you use session.load instead of session.get you won't get > 10,000 queries over the network if batch fetching is enabled, which > does exactly what the original issue discusses, it splits the "in" > clause into chunks and executes it in an intelligent manner. It's a > shame the same can't be worked into a solution when a criteria or hql > query is used - to me this is more a breakage in SQL than in hibernate. > > If you use session.get your performance will be absolutely horrendous > compared to an "in" or global temp table (I state this mainly for the > record, not because it's some new information to you ;-) Great! Then use load(). next... ;) There is a huge difference here wrt the query splitting... We know for certain the structure of the query and the fact that we do not have to worry about AND/OR predicate branches and how splitting might effect the overall semantic of the query. Guys, if there is something more you would like to say, why not say it on the issue? That way we are not having to repeat this same information between the 2 places... -- st...@hibernate.org http://hibernate.org ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] Hibernate OGM split in modules, ready for new integrations!
Hey Sanne, I started creating the layout for the hibernate-ogm-ehcache module. Now I stumbled on something, given I'm no maven expert, I think might not be quite right: from the parent project, it couldn't resolve the dependency to hibernate-ogm-core, so I first had to install that from the ./hibernate-ogm-core module, so I could build from the root. I think this might not be what is wanted though ... Is there a reason for that ? Or is this some weirdness I experience because I used mvn2 still ? Otherwise I started the module on my fork on github and will work to get something passing tests as soon as possible. Thanks a lot for all these changes! Alex PS: Also, I started using my Hibernate-core code style, but look like this might use another, right ? On Sun, Dec 4, 2011 at 7:21 PM, Sanne Grinovero wrote: > Hi Greg, > we've implemented all changes in Hibernate OGM which we discussed at > Devoxx to make it easier for you to create a new module and get > started experimenting with an EHCache integration. > > Actually to proof myself correct I've already created a new module, > and removed Infinispan from the main module relegating it in it's own > extension; so while in the main pom and possibly in some comments and > documentation Infinispan might still be mentioned as the "main" > supported engine, it's not the case anymore as far as the > implementation concerns: the tests of the core module are now using a > "spiced" ConcurrentHashMap instead and we'll improve on the docs too > as soon as we actually have an alternative. I hope this will let you > get started quickly, basically you only have to look into the > Infinispan module and think how to best implement the 6 trivial > classes. > > The structure of the testsuite might need some warnings: since the > tests verify the functionality via the user API, we actually want to > re-run all the tests on each database. That's what Hibernate Core > always did, via a couple of properties it switches the JDBC driver and > connection details, just one class cares for the dialect which > encapsulates the database specifics; but most code is well tested > using a single database, so by default the testsuite runs H2 only, > while Jenkins re-runs the full testsuite on all supported databases. > In the case of Hibernate OGM, the code which we actually want to test > is the integration with each database, so running tests on only one of > them is not really an option. > To re-run the same testsuite multiple times, and to keep all the > database specific code and dependencies in different modules, we're > hacking a bit around Maven by packaging the testsuite from the main > module and referring to the same tests again in each module. > > Basically I'd suggest to have a look at how the > hibernate-ogm-infinispan module reuses the test code from > hibernate-ogm-core; most of the magic is triggered by the > maven-dependency-plugin in the parent pom.xml and the > org.hibernate.ogm.test.utils.InfinispanTestHelper in the test sources. > > Of course you can add additional EHCache tests as you like in the new > hibernate-ogm-ehcache module, but I'd suggest to reuse all tests from > hibernate-ogm-core. > > We can certainly polish this approach if it doesn't proof sound in > practice, but I hope it's a good way to get started with it. It should > provide some convergence of features across "databases"; if some > feature can really not be implemented we can introduce flags to > disable groups of tests in specific modules; for now there exists one > such flag which disables JTA integration [1] : I'm using it with the > Map implementation as I didn't implement transactional support on it. > In theory you could introduce more such flags if you really need to > disable some other tests, but that would be the least desirable option > as we hope to provide a consistent behaviour across different backing > databases as much as possible. > > The basic build information can be found here: > http://docs.jboss.org/hibernate/ogm/3.0/reference/en-US/html/ogm-howtocontribute.html > > For any question, we're looking forward to help you! > > Regards, > Sanne > > 1 - > org.hibernate.ogm.test.utils.TestableGridDialect.backendSupportsTransactions() -- Alex Snaps Senior Software Engineer - Terracotta http://twitter.com/alexsnaps http://www.linkedin.com/in/alexsnaps ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
[hibernate-dev] Where is annotation <-> column name mapped?
Hello, I'm trying to fix some bug on Hibernate (https://hibernate.onjira.com/browse/HHH-6580), and I need to get following information: 1. Which property is an entity ID? 2. Which column name in table corresponds to this property? What class/object has this information? I spent few hours on it, but I can't tie it all together. Maybe some documentation would help? I found that EntityMetamodel.getIdentifierProperty can give me answer to question 1, but I can't find answer for question 2. Best regards -- Paweł Stawicki http://pawelstawicki.blogspot.com http://szczecin.jug.pl ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
