I did heap analysis today. It is quite certain that the problem is not related with component auto-scanning, it is related with the memory leak introduced by Spring AOP. I'm now working on a solution to it.
Kelven On 2/21/13 3:02 PM, "Kelven Yang" <kelven.y...@citrix.com> wrote: >Rohit, > >I'll checkout out your work about using XML instead of @Component. >However, I'm not sure if the memory leak issue is related with auto-scan >at all. I'm now doing a heap analysis and will update with the result > >Kelven > > > >On 2/21/13 7:02 AM, "Rohit Yadav" <bhais...@apache.org> wrote: > >>Okay Chip, posted a comment on the issue. >> >>Hi Kelven, you want to check my fix. Thanks. >> >>Regards. >> >>On Thu, Feb 21, 2013 at 8:17 PM, Chip Childers >><chip.child...@sungard.com> wrote: >>> >>> Can you coordinate with Kelvin on this? He's got CLOUDSTACK-1276, and >>> had previously mentioned this exact same thing (using the xml file vs. >>> annotations). >>> >>> On Thu, Feb 21, 2013 at 07:26:42PM +0530, Rohit Yadav wrote: >>>> Just to add, we will have separate xmls for every kind of profile, I >>>> can make every plugin have its own spring xml which is copied and >>>> injected to the web.xml comments, suggestions, flames? >>>> >>>> Possible future experiments: >>>> - Spring xmls artefacts based >>>> - Have a way to fix AOP, don't add pointcuts on all classes, methods >>>> - Make DI framework replaceable in CloudStack, I would go for Guice ;) >>>> >>>> Regards. >>>> >>>> On Thu, Feb 21, 2013 at 7:22 PM, Rohit Yadav <bhais...@apache.org> >>>>wrote: >>>> > I'm experimenting with removing all @Components and moving beans >>>>into >>>> > an xml that could be read by spring to do DI. >>>> > This decreases the load time and >>>> > >>>> > Errors: >>>> > - Abstract classes annotated as spring components >>>> > - Annotations for project which are not included in either build >>>> > profiles: usage, simulator, oss (default) and nonoss >>>> > - Some project don't even build or are not part of any build >>>> > artefacts, looks like the @Component annotation was done in a hurry >>>>:) >>>> > - Interfaces being annotated as spring components >>>> > - Prasanna, Abhi and I spoke with Spring gurus Vijay V and Vaibhav >>>>on >>>> > how to fix this bloat, so in applicationContext.xml we do some aop >>>> > stuff to have transaction stuff we get from @DB annotation and this >>>> > needs fixing as it's applying the following for all classes and >>>> > methods (dude bloat, run! :) >>>> > >>>> > <aop:config proxy-target-class="true"> >>>> > <aop:aspect id="dbContextBuilder" >>>>ref="transactionContextBuilder"> >>>> > <aop:pointcut id="captureAnyMethod" >>>> > expression="execution(* *(..))" >>>> > /> >>>> > >>>> > Methodology and results: >>>> > I wrote a python program that would go through the whole source code >>>> > and fix all java files (except test, awsapi) who have @Component and >>>> > create bean rules in an xml file. Next I had to manually fix few >>>> > things (see error above) and I got the mgmt server running, the load >>>> > time had improved a lot (little faster now) after disabling >>>> > autoscanning and just using the xml generated. I see the benefit of >>>> > just using the xml and not having spring as a build time dependency >>>> > (we're using @Component in our classes so :). >>>> > >>>> > This was an experiment mostly because a lot of folks are having >>>>spring >>>> > memory issues, especially Prasanna's devcloud-ci cannot run and >>>>check >>>> > our commits as memory kills dom0. So, if no one objects may I go >>>>ahead >>>> > and commit it? For me the jvm took about 650MB real memory on osx. >>>> > >>>> > Right now I made it for all build profiles, nonoss being the >>>>superset, >>>> > my experiment can be checked out from here: >>>> > https://github.com/bhaisaab/cloudstack/tree/spring-annotation-to-xml >>>> > >>>> > Regards. >>>> > >>>> > The python program, in case anyone wants to replicate or is curious: >>>> > (defaulter is just a file containing result of: file . | grep java$ >>>>> >>>> > defaulter) >>>> > f = open('defaulter', 'r') >>>> > files = f.read().split('\n')[:-1] >>>> > files = filter(lambda x: not x.startswith('./awsapi') and >>>> > x.find('/test/') == -1, files) >>>> > xmldata = "" >>>> > bean = """ <bean id="%s" class="%s"/>\n""" >>>> > xml = """<!-- >>>> > Licensed to the Apache Software Foundation (ASF) under one >>>> > or more contributor license agreements. See the NOTICE file >>>> > distributed with this work for additional information >>>> > regarding copyright ownership. The ASF licenses this file >>>> > to you under the Apache License, Version 2.0 (the >>>> > "License"); you may not use this file except in compliance >>>> > with the License. You may obtain a copy of the License at >>>> > >>>> > http://www.apache.org/licenses/LICENSE-2.0 >>>> > >>>> > Unless required by applicable law or agreed to in writing, >>>> > software distributed under the License is distributed on an >>>> > "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY >>>> > KIND, either express or implied. See the License for the >>>> > specific language governing permissions and limitations >>>> > under the License. >>>> > --> >>>> > <beans xmlns="http://www.springframework.org/schema/beans"; >>>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>>> > xmlns:context="http://www.springframework.org/schema/context"; >>>> > xmlns:tx="http://www.springframework.org/schema/tx"; >>>> > xmlns:aop="http://www.springframework.org/schema/aop"; >>>> > xsi:schemaLocation="http://www.springframework.org/schema/beans >>>> > >>>> > http://www.springframework.org/schema/beans/spring-beans-3.0.xsd >>>> > http://www.springframework.org/schema/tx >>>> > >>>>http://www.springframework.org/schema/tx/spring-tx-3.0.xsd >>>> > http://www.springframework.org/schema/aop >>>> > >>>> > http://www.springframework.org/schema/aop/spring-aop-3.0.xsd >>>> > http://www.springframework.org/schema/context >>>> > >>>> > >>>>http://www.springframework.org/schema/context/spring-context-3.0.xsd";> >>>> > >>>> > %s >>>> > </beans> >>>> > """ >>>> > misses = [] >>>> > for file in files: >>>> > name = file.split('/')[-1].split('.')[0] >>>> > name = name[0].lower() + name[1:] >>>> > pkgname = "" >>>> > idx = 0 >>>> > paths = file.split('/')[1:] >>>> > paths[-1] = paths[-1].split('.')[0] >>>> > for str in paths: >>>> > if str == 'org' or str == 'com': >>>> > pkgname = '.'.join(paths[idx:]) >>>> > idx += 1 >>>> > f = open(file, 'r') >>>> > data = f.read() >>>> > f.close() >>>> > idx = data.find('@Component') >>>> > if idx == -1: >>>> > misses.append(file) >>>> > continue >>>> > string = '@Component' >>>> > if data[len(string)+idx] == '(': >>>> > endidx = data.find(')', len(string)+idx) >>>> > name = >>>>data[len(string)+idx+1:endidx].split('=')[-1].replace('"', '') >>>> > name = name.replace("'", '') >>>> > data = data[:idx] + data[endidx+2:] >>>> > else: >>>> > data = data.replace(string, '') >>>> > data = data.replace('import >>>> > org.springframework.stereotype.Component;\n', '') >>>> > f = open(file, 'w') >>>> > f.write(data) >>>> > f.close() >>>> > print name, pkgname >>>> > xmldata += bean % (name, pkgname) >>>> > xmldata += "\n" >>>> > >>>> > xml = xml % xmldata >>>> > f = open('./client/tomcatconf/defaultComponentContext.xml.in', 'w') >>>> > f.write(xml) >>>> > f.close() >>>> > >>>> > f = open('misses', 'w') >>>> > f.write('\n'.join(misses)) >>>> > f.close() >>>> >
