[hibernate-dev] SingleTableEntityPersister memory footprint in 5.3
Hi, On Discourse, a user told us that the memory footprint has increased dramatically between 5.2 and 5.3: https://discourse.hibernate.org/t/batch-fetch-style-recommendations/631/5?u=vlad I think it's worth investigating the cause and see how we could address it in the next release. Vlad ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] Releases and CI setup
On Mon, Apr 30, 2018 at 6:09 PM, Guillaume Smet wrote: > So I would say: > - either we fix the issue we have with all the branches being tested for > each commit that we discussed numerous times > Here it is: https://github.com/gsmet/github-plugin/commit/03328fcf8ecd9117d1c8c2a9b967efa111d6be46 Unit tests are OK and I added new ones. We would need to test it in the wild. It's not intrusive at all and rather clean (there's a small hack related to BranchSpec pattern matching but it's due to a bug in an underlying library). The class GitHubRepositoryBranchSpecContributor that might look a bit obscure is just an adaptation of the original Jenkins class filtering on repository name only. Comments welcome, I would like to test it on our CI at some point. -- Guillaume ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] SingleTableEntityPersister memory footprint in 5.3
On Thu, May 3, 2018 at 9:35 AM, Vlad Mihalcea wrote: > I think it's worth investigating the cause and see how we could address it > in the next release. > Sure, something looks fishy there. Will you do it or do you expect someone else to do it? It's not very clear from your email. -- Guillaume ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] SingleTableEntityPersister memory footprint in 5.3
I asked because someone might have an idea about some change that was doe in 5.3 that might affect it. For the moment, I haven't started doing anything about it. Vlad On Thu, May 3, 2018 at 2:01 PM, Guillaume Smet wrote: > On Thu, May 3, 2018 at 9:35 AM, Vlad Mihalcea > wrote: > >> I think it's worth investigating the cause and see how we could address it >> in the next release. >> > > Sure, something looks fishy there. > > Will you do it or do you expect someone else to do it? It's not very clear > from your email. > > -- > Guillaume > > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] Releases and CI setup
Great! Thanks a lot :) I added a few comments, but nothing very important. On Thu, 3 May 2018 at 12:55 Guillaume Smet wrote: > On Mon, Apr 30, 2018 at 6:09 PM, Guillaume Smet > wrote: > > > So I would say: > > - either we fix the issue we have with all the branches being tested for > > each commit that we discussed numerous times > > > > Here it is: > > https://github.com/gsmet/github-plugin/commit/03328fcf8ecd9117d1c8c2a9b967efa111d6be46 > > Unit tests are OK and I added new ones. We would need to test it in the > wild. > > It's not intrusive at all and rather clean (there's a small hack related to > BranchSpec pattern matching but it's due to a bug in an underlying > library). The class GitHubRepositoryBranchSpecContributor that might look a > bit obscure is just an adaptation of the original Jenkins class filtering > on repository name only. > > Comments welcome, I would like to test it on our CI at some point. > > -- > Guillaume > ___ > hibernate-dev mailing list > hibernate-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev > -- Yoann Rodiere yo...@hibernate.org / yrodi...@redhat.com Software Engineer Hibernate NoORM team ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] Releases and CI setup
Very cool! Feel free to build it and deploy on ci.hibernate.org ? On 3 May 2018 at 11:53, Guillaume Smet wrote: > On Mon, Apr 30, 2018 at 6:09 PM, Guillaume Smet > wrote: > >> So I would say: >> - either we fix the issue we have with all the branches being tested for >> each commit that we discussed numerous times >> > > Here it is: > https://github.com/gsmet/github-plugin/commit/03328fcf8ecd9117d1c8c2a9b967efa111d6be46 > > Unit tests are OK and I added new ones. We would need to test it in the > wild. > > It's not intrusive at all and rather clean (there's a small hack related to > BranchSpec pattern matching but it's due to a bug in an underlying > library). The class GitHubRepositoryBranchSpecContributor that might look a > bit obscure is just an adaptation of the original Jenkins class filtering > on repository name only. > > Comments welcome, I would like to test it on our CI at some point. > > -- > Guillaume > ___ > hibernate-dev mailing list > hibernate-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] HSEARCH-3000 Pick Jigsaw Automatic Module names for all published modules
Since the names are about to be set in stone, and there has been some
discussion off list, I'd like to talk a bit about the Elasticsearch/Lucene
module names...
In the Search 6 architecture, these modules really provide a "backend",
i.e. a component that actually implements indexing/querying over multiple
indexes, but operates "behind" the engine (the engine uses the backend).
Essentially the data flow is:
Indexing: User => Mapper (user APIs) => Engine => Backend => Disk/network
Querying: User <= Mapper (user APIs) <= Engine <= Backend <= Disk/network
We discussed it a bit with Sanne (and with Guillaume) and agreed that
"backend" was the right word for this part of our architecture.
Now, the discussion is about module names, not component names.
Still, there are several reasons because of which I'd rather use
"org.hibernate.search.backend.":
- It is closer to how we will probably refer to the module on a
day-to-day basis: I don't know about you, but I'll probably talk about "the
Elasticsearch backend" rather than "Elasticsearch indexing" or "the
Elasticsearch module".
- It is generic enough to cover all of the tasks we delegate to
Elasticsearch/Lucene.
"org.hibernate.search.indexing.",
on the other hand, feels like it's only about indexing, and not querying,
maybe not even about defining a schema.
- It is consistent with our architecture and APIs in Search 6:
- The main component (and entry point) in those modules will be an
object implementing BackendImplementor (SPI)
- We will actually expose "backends" as part of our Java APIs, for
instance to allow users to retrieve the Elasticsearch client, or to query
some backend-scoped metadata.
- We will also use the word "backend" in the user configuration,
where the user will have to define a backend, its type and its options
using the "hibernate.search.backends..*" properties
- As opposed to the "prefix-less" naming scheme
("org.hibernate.search."), it makes it easy to infer
what the module actually provides to users: the actual implementation
behind Hibernate Search, even without extensive knowledge about Hibernate
Search.
- When listing all the modules of Hibernate Search, it makes it easy
to spot which module is about what, and which modules provide the same
thing using a different technology:
org.hibernate.search.engine
org.hibernate.search.orm
org.hibernate.search.clustering.jms
org.hibernate.search.clustering.jgroups
org.hibernate.search.backend.elasticsearch
org.hibernate.search.backend.lucene
vs:
org.hibernate.search.engine
org.hibernate.search.orm
org.hibernate.search.jms
org.hibernate.search.jgroups
org.hibernate.search.elasticsearch
org.hibernate.search.lucene
- When the module name appears in some error message (for instance
when a user starts experimenting with Java 9), it will also make it clear
which part of Hibernate Search is misconfigured. And that may be
important
to some users, since not all the people working on an application
know the details of every part of the application...
Of course, in the meantime in Search 5, there will be some confusion due to
the fact "backend" has a different meaning in Search 5.
If you are worried about that, I'm not opposed to slightly altering the
module names between Search 5 and Search 6: users will have to make changes
when migrating anyway.
In particular, a lot of APIs will change, some class may be moved to
different packages to comply the "no split package" constraint of Java 9,
Maven coordinates will change to use the "org.hibernate.search" group ID
instead of "org.hibernate", and people using the Lucene backend will have
to depend on a new module (both in their Maven dependencies and in their
Java 9 module dependencies).
So we could just use "org.hibernate.elasticsearch" in Search 5, and switch
to "org.hibernate.backend.elasticsearch" in Search 6. It's not even a big
deal, since the Elastisearch module is openly advertised as experimental.
On Tue, 13 Feb 2018 at 22:51 Sanne Grinovero wrote:
> On 13 February 2018 at 11:48, Guillaume Smet
> wrote:
> > On Tue, Feb 13, 2018 at 11:59 AM, Yoann Rodiere
> wrote:
> >>
> >> > - org.hibernate.search.jms-support
> >>
> >> Not sure it's a valid name (aren't hyphens forbidden in package names,
> and
> >> aren't module names constrained by the same rules?), but apart from that
> >> it
> >> looks good. Maybe "jms_support" instead, if "jms-support" is not
> allowed.
> >> I would like to find a name for what the JMS and JGroups modules provide
> >> in
> >> Hibernate Search 6 though, something less misleading than "backend".
> >> Something like "work routing" or "clustering support" or "distribution
> >> support" or whatever. Would you be ok with changing the module name in
> 6?
> >> If not, maybe we have to think about it now... Note th
Re: [hibernate-dev] HSEARCH-3000 Pick Jigsaw Automatic Module names for all published modules
On Thu, May 3, 2018 at 2:17 PM, Yoann Rodiere wrote: > Still, there are several reasons because of which I'd rather use > "org.hibernate.search.backend.": > >- It is closer to how we will probably refer to the module on a >day-to-day basis: I don't know about you, but I'll probably talk about "the >Elasticsearch backend" rather than "Elasticsearch indexing" or "the >Elasticsearch module". > > So, you might remember that I was the one proposing "indexing.elasticsearch" and this is the main argument that convinced me. > >- It is consistent with our architecture and APIs in Search 6: > - The main component (and entry point) in those modules will be an > object implementing BackendImplementor (SPI) > - We will actually expose "backends" as part of our Java APIs, for > instance to allow users to retrieve the Elasticsearch client, or to > query > some backend-scoped metadata. > - We will also use the word "backend" in the user configuration, > where the user will have to define a backend, its type and its options > using the "hibernate.search.backends..*" properties > > + this one. We can use whatever name we want for Search 5 but I think the "backend" choice is pretty obvious for Search 6. -- Guillaume ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] Releases and CI setup
On Thu, May 3, 2018 at 1:17 PM, Sanne Grinovero wrote: > Very cool! > > Feel free to build it and deploy on ci.hibernate.org ? > I deployed it on ci.hibernate.org just now. It would be nice if you could check which jobs get triggered when you push something to master/a specific branch. -- Guillaume ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] Releases and CI setup
On Thu, May 3, 2018 at 2:47 PM, Guillaume Smet wrote: > On Thu, May 3, 2018 at 1:17 PM, Sanne Grinovero > wrote: > >> Very cool! >> >> Feel free to build it and deploy on ci.hibernate.org ? >> > > I deployed it on ci.hibernate.org just now. > > It would be nice if you could check which jobs get triggered when you push > something to master/a specific branch. > I did a test with HV, deploying something to master and then to 6.0 and it works as expected \o/. Please refrain from updating the GitHub plugin (it has a snapshot version with my login as a suffix). If you encounter any surprising behavior, feel free to ping me. -- Guillaume ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] HSEARCH-3000 Pick Jigsaw Automatic Module names for all published modules
Ok, I surrender. so it's going to be `org.hibernate.search.backend.elasticsearch`. Even if it's not ideal in the context of Hibernate Search 5.x, IMO long term stability is more important for people. For the record I still think this tendency to over-qualify things and add many prefixes should be put under control. I totally understand it makes sense to "us" but please make an effort to look at how people use things: for users this is just "the ES stuff", but whatever they won't care about the prefix anyway :) Remember this though: when you'll want to include include more than just the backend bits in this module you will have resistance against changing the name. Thanks, Sanne On 3 May 2018 at 13:26, Guillaume Smet wrote: > On Thu, May 3, 2018 at 2:17 PM, Yoann Rodiere wrote: >> >> Still, there are several reasons because of which I'd rather use >> "org.hibernate.search.backend.": >> >> It is closer to how we will probably refer to the module on a day-to-day >> basis: I don't know about you, but I'll probably talk about "the >> Elasticsearch backend" rather than "Elasticsearch indexing" or "the >> Elasticsearch module". > > > So, you might remember that I was the one proposing "indexing.elasticsearch" > and this is the main argument that convinced me. > >> >> It is consistent with our architecture and APIs in Search 6: >> >> The main component (and entry point) in those modules will be an object >> implementing BackendImplementor (SPI) >> We will actually expose "backends" as part of our Java APIs, for instance >> to allow users to retrieve the Elasticsearch client, or to query some >> backend-scoped metadata. >> We will also use the word "backend" in the user configuration, where the >> user will have to define a backend, its type and its options using the >> "hibernate.search.backends..*" properties > > > + this one. > > We can use whatever name we want for Search 5 but I think the "backend" > choice is pretty obvious for Search 6. > > -- > Guillaume > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] Releases and CI setup
Awesome job! On 3 May 2018 at 14:04, Guillaume Smet wrote: > On Thu, May 3, 2018 at 2:47 PM, Guillaume Smet > wrote: >> >> On Thu, May 3, 2018 at 1:17 PM, Sanne Grinovero >> wrote: >>> >>> Very cool! >>> >>> Feel free to build it and deploy on ci.hibernate.org ? >> >> >> I deployed it on ci.hibernate.org just now. >> >> It would be nice if you could check which jobs get triggered when you push >> something to master/a specific branch. > > > I did a test with HV, deploying something to master and then to 6.0 and it > works as expected \o/. > > Please refrain from updating the GitHub plugin (it has a snapshot version > with my login as a suffix). > > If you encounter any surprising behavior, feel free to ping me. > > -- > Guillaume > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] SingleTableEntityPersister memory footprint in 5.3
I just looked at that thread and it's comparing 3.6 to 5.3.CR2... On Thu, May 3, 2018 at 4:17 AM, Vlad Mihalcea wrote: > I asked because someone might have an idea about some change that was doe > in 5.3 that might affect it. > For the moment, I haven't started doing anything about it. > > Vlad > > On Thu, May 3, 2018 at 2:01 PM, Guillaume Smet > wrote: > > > On Thu, May 3, 2018 at 9:35 AM, Vlad Mihalcea > > wrote: > > > >> I think it's worth investigating the cause and see how we could address > it > >> in the next release. > >> > > > > Sure, something looks fishy there. > > > > Will you do it or do you expect someone else to do it? It's not very > clear > > from your email. > > > > -- > > Guillaume > > > > > ___ > hibernate-dev mailing list > hibernate-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] SingleTableEntityPersister memory footprint in 5.3
FYI, those batch loaders were updated to use "load plans" in 4.3. I suspect that might have something to do with it. Please let me know if I should look into this. On Thu, May 3, 2018 at 10:54 AM, Gail Badner wrote: > I just looked at that thread and it's comparing 3.6 to 5.3.CR2... > > On Thu, May 3, 2018 at 4:17 AM, Vlad Mihalcea > wrote: > >> I asked because someone might have an idea about some change that was doe >> in 5.3 that might affect it. >> For the moment, I haven't started doing anything about it. >> >> Vlad >> >> On Thu, May 3, 2018 at 2:01 PM, Guillaume Smet >> wrote: >> >> > On Thu, May 3, 2018 at 9:35 AM, Vlad Mihalcea >> > wrote: >> > >> >> I think it's worth investigating the cause and see how we could >> address it >> >> in the next release. >> >> >> > >> > Sure, something looks fishy there. >> > >> > Will you do it or do you expect someone else to do it? It's not very >> clear >> > from your email. >> > >> > -- >> > Guillaume >> > >> > >> ___ >> hibernate-dev mailing list >> hibernate-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hibernate-dev >> > > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] SingleTableEntityPersister memory footprint in 5.3
Yeah, we have been having a chat about this on HipChat On Thu, May 3, 2018 at 12:54 PM Gail Badner wrote: > I just looked at that thread and it's comparing 3.6 to 5.3.CR2... > > On Thu, May 3, 2018 at 4:17 AM, Vlad Mihalcea > wrote: > > > I asked because someone might have an idea about some change that was doe > > in 5.3 that might affect it. > > For the moment, I haven't started doing anything about it. > > > > Vlad > > > > On Thu, May 3, 2018 at 2:01 PM, Guillaume Smet > > > wrote: > > > > > On Thu, May 3, 2018 at 9:35 AM, Vlad Mihalcea > > > > wrote: > > > > > >> I think it's worth investigating the cause and see how we could > address > > it > > >> in the next release. > > >> > > > > > > Sure, something looks fishy there. > > > > > > Will you do it or do you expect someone else to do it? It's not very > > clear > > > from your email. > > > > > > -- > > > Guillaume > > > > > > > > ___ > > hibernate-dev mailing list > > hibernate-dev@lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hibernate-dev > > > ___ > hibernate-dev mailing list > hibernate-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] SingleTableEntityPersister memory footprint in 5.3
OK, let me know if I should jump in... On Thu, May 3, 2018 at 10:59 AM, Steve Ebersole wrote: > Yeah, we have been having a chat about this on HipChat > > On Thu, May 3, 2018 at 12:54 PM Gail Badner wrote: > >> I just looked at that thread and it's comparing 3.6 to 5.3.CR2... >> >> On Thu, May 3, 2018 at 4:17 AM, Vlad Mihalcea >> wrote: >> >> > I asked because someone might have an idea about some change that was >> doe >> > in 5.3 that might affect it. >> > For the moment, I haven't started doing anything about it. >> > >> > Vlad >> > >> > On Thu, May 3, 2018 at 2:01 PM, Guillaume Smet < >> guillaume.s...@gmail.com> >> > wrote: >> > >> > > On Thu, May 3, 2018 at 9:35 AM, Vlad Mihalcea < >> mihalcea.v...@gmail.com> >> > > wrote: >> > > >> > >> I think it's worth investigating the cause and see how we could >> address >> > it >> > >> in the next release. >> > >> >> > > >> > > Sure, something looks fishy there. >> > > >> > > Will you do it or do you expect someone else to do it? It's not very >> > clear >> > > from your email. >> > > >> > > -- >> > > Guillaume >> > > >> > > >> > ___ >> > hibernate-dev mailing list >> > hibernate-dev@lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/hibernate-dev >> > >> ___ >> hibernate-dev mailing list >> hibernate-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hibernate-dev >> > ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
Re: [hibernate-dev] SingleTableEntityPersister memory footprint in 5.3
On 3 May 2018 at 19:33, Gail Badner wrote: > OK, let me know if I should jump in... Thanks Gail, I think you can relax for now :) - knowing that 4.3 introduced the load plans explains a good deal of the difference - Guillaume is checking details too - the Red Hat performance team is going to check general allocation with the usual benchmarks (not specifically this issue, but still reassuring enough) > > On Thu, May 3, 2018 at 10:59 AM, Steve Ebersole wrote: > >> Yeah, we have been having a chat about this on HipChat >> >> On Thu, May 3, 2018 at 12:54 PM Gail Badner wrote: >> >>> I just looked at that thread and it's comparing 3.6 to 5.3.CR2... >>> >>> On Thu, May 3, 2018 at 4:17 AM, Vlad Mihalcea >>> wrote: >>> >>> > I asked because someone might have an idea about some change that was >>> doe >>> > in 5.3 that might affect it. >>> > For the moment, I haven't started doing anything about it. >>> > >>> > Vlad >>> > >>> > On Thu, May 3, 2018 at 2:01 PM, Guillaume Smet < >>> guillaume.s...@gmail.com> >>> > wrote: >>> > >>> > > On Thu, May 3, 2018 at 9:35 AM, Vlad Mihalcea < >>> mihalcea.v...@gmail.com> >>> > > wrote: >>> > > >>> > >> I think it's worth investigating the cause and see how we could >>> address >>> > it >>> > >> in the next release. >>> > >> >>> > > >>> > > Sure, something looks fishy there. >>> > > >>> > > Will you do it or do you expect someone else to do it? It's not very >>> > clear >>> > > from your email. >>> > > >>> > > -- >>> > > Guillaume >>> > > >>> > > >>> > ___ >>> > hibernate-dev mailing list >>> > hibernate-dev@lists.jboss.org >>> > https://lists.jboss.org/mailman/listinfo/hibernate-dev >>> > >>> ___ >>> hibernate-dev mailing list >>> hibernate-dev@lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/hibernate-dev >>> >> > ___ > hibernate-dev mailing list > hibernate-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev ___ hibernate-dev mailing list hibernate-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev
