Just some heads up. Ther are a bunch of issues with parallelism in the tests. Many tests use the default ports for transports. Many tests create a jmx endpoint on the default port. Many tests use default directory locations for persistence
There is a bunch of work here. Peek at: https://issues.apache.org/jira/browse/AMQ-5552 The intent is to pull a few representative test into a profile to get get a fast turn around with sanity check on a full build. It is a step in the right direction, but will never be a substitute for the full -Dactivemq.tests=all run On 25 February 2015 at 01:36, Kevin Burton <[email protected]> wrote: > Yes. Agreed that they look like integration tests. More than a minute is > more of an integration test. > > What I’ll do is come up with a proposal once I have our internal code > testing with parallelism. > > My ideal is to use the maven dependency graph and possibly something like > codeship’s new ParallelCI to build the dependency graph in parallel. > > If a specific module’s tests take too long you can split them into N > modules, each with a few tests. > > With the right among of parallelism I bet you could get the tests down to > 30 minutes … maximum. > > Kevin > > On Tue, Feb 24, 2015 at 3:31 PM, Hadrian Zbarcea <[email protected]> wrote: > >> Kevin, >> >> Go for it. >> >> Yes, out tests need a lot of love. Actually most of the tests are not >> really unit tests, but more like integration tests. >> Helping with the spring (as in season) cleanup would be immensely >> appreciated. >> >> Hadrian >> >> >> >> On 02/24/2015 02:18 PM, Kevin Burton wrote: >> >>> Ah. OK… I assume continuous integration isn’t a goal. I’m upset if my >>> testing takes more than 20 minutes :-P >>> >>> I bet that this would be taken down to an hour if they were parallelized. >>> But of course someone needs to take the time to get that done. >>> >>> I might be allocating some of our internal dev resources to getting our >>> build to parallelize across containers. If this generalizable to other >>> projects I’ll OSS it.. Would be great to get better testing feedback. >>> >>> Right now I think my patches are ready to be merged but I’d feel more >>> comfortable suggesting it if I knew the tests worked. >>> >>> Should I propose it now and if there are any bugs we can fix them later? >>> >>> Kevin >>> >>> On Tue, Feb 24, 2015 at 10:54 AM, Timothy Bish <[email protected]> >>> wrote: >>> >>> On 02/24/2015 01:22 PM, Kevin Burton wrote: >>>> >>>> oh geez… seriously though. can you give me an estimate? >>>>> >>>>> Assuming 30 seconds each I’m assuming about 11 hours. >>>>> >>>>> That's probably not to far off, usually just let them run over night >>>> for >>>> the all profile. >>>> >>>> >>>> http://activemq.apache.org/junit-reports.html >>>>> https://hudson.apache.org/hudson/job/ActiveMQ/ >>>>> Hudson is down for ActiveMQ. >>>>> >>>>> Also, is there a disk space requirement for the tests? I have 10GB free >>>>> or >>>>> so and I’m getting TONS of these: >>>>> >>>>> I don't think anyone has looked into the min required free for the >>>> tests >>>> to run. >>>> >>>> >>>> >>>> java.lang.RuntimeException: Failed to start provided job scheduler >>>>> store: >>>>> JobSchedulerStore:activemq-data >>>>> >>>>> at java.io.RandomAccessFile.seek(Native Method) >>>>> >>>>> at >>>>> org.apache.activemq.util.RecoverableRandomAccessFile.seek( >>>>> RecoverableRandomAccessFile.java:384) >>>>> >>>>> at >>>>> org.apache.activemq.store.kahadb.disk.page.PageFile. >>>>> readPage(PageFile.java:877) >>>>> >>>>> at >>>>> org.apache.activemq.store.kahadb.disk.page.Transaction. >>>>> load(Transaction.java:427) >>>>> >>>>> at >>>>> org.apache.activemq.store.kahadb.disk.page.Transaction. >>>>> load(Transaction.java:377) >>>>> >>>>> at >>>>> org.apache.activemq.store.kahadb.disk.index.BTreeIndex. >>>>> load(BTreeIndex.java:159) >>>>> >>>>> at >>>>> org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl$ >>>>> MetaData.load(JobSchedulerStoreImpl.java:90) >>>>> >>>>> at >>>>> org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl$3. >>>>> execute(JobSchedulerStoreImpl.java:277) >>>>> >>>>> at >>>>> org.apache.activemq.store.kahadb.disk.page.Transaction. >>>>> execute(Transaction.java:779) >>>>> >>>>> at >>>>> org.apache.activemq.store.kahadb.scheduler. >>>>> JobSchedulerStoreImpl.doStart( >>>>> JobSchedulerStoreImpl.java:261) >>>>> >>>>> at org.apache.activemq.util.ServiceSupport.start( >>>>> ServiceSupport.java:55) >>>>> >>>>> at >>>>> org.apache.activemq.broker.BrokerService.setJobSchedulerStore( >>>>> BrokerService.java:1891) >>>>> >>>>> at >>>>> org.apache.activemq.transport.stomp.StompTestSupport. >>>>> createBroker(StompTestSupport.java:174) >>>>> >>>>> at >>>>> org.apache.activemq.transport.stomp.StompTestSupport. >>>>> startBroker(StompTestSupport.java:112) >>>>> >>>>> at >>>>> org.apache.activemq.transport.stomp.StompTestSupport.setUp( >>>>> StompTestSupport.java:94) >>>>> >>>>> at >>>>> org.apache.activemq.transport.stomp.Stomp12Test.setUp( >>>>> Stomp12Test.java:41) >>>>> >>>>> at >>>>> org.apache.activemq.transport.stomp.Stomp12SslAuthTest. >>>>> setUp(Stomp12SslAuthTest.java:38) >>>>> >>>>> On Tue, Feb 24, 2015 at 10:09 AM, Timothy Bish <[email protected]> >>>>> wrote: >>>>> >>>>> On 02/24/2015 01:01 PM, Kevin Burton wrote: >>>>> >>>>>> How long do the tests usually take? >>>>>> >>>>>>> An eternity + 1hr >>>>>>> >>>>>> I’m looking at 45 minutes right now >>>>>> >>>>>> before I gave up… I think part of it was that the box I was testing on >>>>>>> was >>>>>>> virtualized and didn’t have enough resources. >>>>>>> >>>>>>> I tried to parallelize the tests (-T 8 with maven) but I got other >>>>>>> errors >>>>>>> so I assume the ports are singletons. >>>>>>> >>>>>>> They won't be happy if you run them parallel. >>>>>>> >>>>>> >>>>>> >>>>>> On Tue, Feb 24, 2015 at 8:03 AM, Gary Tully <[email protected]> >>>>>> >>>>>>> wrote: >>>>>>> >>>>>>> if there are any test failures - try to run them individually >>>>>>> >>>>>>> -Dtest=a,b etc. There may be an issue with a full test run, but all >>>>>>>> of >>>>>>>> the tests that are enabled should work. I know there are some issues >>>>>>>> with jdbc tests that hang or fail due to previous runs no cleaning >>>>>>>> up, >>>>>>>> but that should be the most of it. I got a bunch of full test runs >>>>>>>> before the 5.11 release if that is any help. >>>>>>>> >>>>>>>> On 23 February 2015 at 20:38, Kevin Burton <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>> OK. This is ready to go and I have a patch branch: >>>>>>>> >>>>>>>>> https://issues.apache.org/jira/browse/AMQ-5609 >>>>>>>>> >>>>>>>>> I’m stuck at the moment though because tests don’t pass. But it was >>>>>>>>> failing tests before so I don’t think it has anything to do with my >>>>>>>>> changes. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, Feb 22, 2015 at 11:11 PM, Kevin Burton <[email protected]> >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>> Actually, is the lock even needed here? Why would it be? if we’re >>>>>>>> >>>>>>>>> *removing* a subscription, why does it care if we possibly ALSO >>>>>>>>>> remove >>>>>>>>>> a >>>>>>>>>> separate / isolated queue before/after the subscription is removed. >>>>>>>>>> >>>>>>>>>> I think this is redundant and can be removed. Maybe I’m wrong >>>>>>>>>> though. >>>>>>>>>> >>>>>>>>>> I looked at all the callers and none were associated with queues. >>>>>>>>>> >>>>>>>>>> On Sun, Feb 22, 2015 at 11:07 PM, Kevin Burton <[email protected] >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>> So I have some working/theoretical code that should resolve this. >>>>>>>>> >>>>>>>>> It acquires a lock *per* ActiveMQDestination, this way there is no >>>>>>>>>> >>>>>>>>>>> lock >>>>>>>>>>> contention. >>>>>>>>>>> >>>>>>>>>>> But here’s where I’m stuck. >>>>>>>>>>> >>>>>>>>>>> @Override >>>>>>>>>>> >>>>>>>>>>> public void removeSubscription(ConnectionContext context, >>>>>>>>>>> >>>>>>>>>>>> RemoveSubscriptionInfo info) throws Exception { >>>>>>>>>>>> inactiveDestinationsPurgeLock.readLock().lock(); >>>>>>>>>>>> try { >>>>>>>>>>>> topicRegion.removeSubscription(context, info); >>>>>>>>>>>> } finally { >>>>>>>>>>>> inactiveDestinationsPurgeLock. >>>>>>>>>>>> readLock().unlock(); >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> .. this is in RegionBroker. >>>>>>>>>>>> >>>>>>>>>>> There is no ActiveMQDestination involved here so I’m not sure the >>>>>>>>>>> best >>>>>>>>>>> way to resolve this. >>>>>>>>>>> >>>>>>>>>>> Any advice? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sun, Feb 22, 2015 at 8:11 PM, Kevin Burton <[email protected] >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>> Yes. That was my thinking too.. that just replacing the >>>>>>>>> >>>>>>>>> CopyOnWriteArraySet >>>>>>>>>> with something more performance would solve the issue. >>>>>>>>>> This would also improve queue creation time as well as queue >>>>>>>>>> deletion >>>>>>>>>> >>>>>>>>>>> time. >>>>>>>>>>>> >>>>>>>>>>>> What I think I”m going to do in the mean time is: >>>>>>>>>>>> >>>>>>>>>>>> - implement a granular lock based on queue name… I am going to >>>>>>>>>>>> use >>>>>>>>>>>> an >>>>>>>>>>>> interface so we can replace the implementation later. >>>>>>>>>>>> >>>>>>>>>>>> - implement timing for the purge thread so it tracks how long it >>>>>>>>>>>> takes >>>>>>>>>>>> to remove a queue but also how long the entire loop takes. >>>>>>>>>>>> >>>>>>>>>>>> I’ll do this on a branch so it should be easy to merge. >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Feb 22, 2015 at 7:40 PM, Tim Bain <[email protected] >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>> A decent amount of the time is being spent calling remove() on >>>>>>>>>> various >>>>>>>>>> >>>>>>>>>>> array-backed collections. Those data structures might be >>>>>>>>>>> >>>>>>>>>> inappropriate >>>>>>>>>> >>>>>>>>>>> for >>>>>>>>>>> >>>>>>>>>> the number of destinations you're running, since array-backed >>>>>>>>>> >>>>>>>>>>> collections >>>>>>>>>>>>> tend to have add/remove operations that are O(N); some >>>>>>>>>>>>> improvement >>>>>>>>>>>>> >>>>>>>>>>>>> might >>>>>>>>>>>>> >>>>>>>>>>>> come from something as simple as moving to a ConcurrentHashSet >>>>>>>>>>> >>>>>>>>>> instead >>>>>>>>>> >>>>>>>>>>> of a >>>>>>>>>>> >>>>>>>>>> CopyOnWriteArraySet, for example. (Or it might make performance >>>>>>>>>> >>>>>>>>>>> worse >>>>>>>>>>>>> >>>>>>>>>>>> because of other aspects of how those collections are used; >>>>>>>>>>> people >>>>>>>>>>> >>>>>>>>>> other >>>>>>>>>> >>>>>>>>>>> than me would be in a better position to evaluate the full range >>>>>>>>>>> of >>>>>>>>>>> >>>>>>>>>> performance requirements for those collections.) >>>>>>>>>> >>>>>>>>>>> Scheduler.cancel() also takes an alarming amount of time for what >>>>>>>>>>>>> >>>>>>>>>>>>> looks >>>>>>>>>>>>> >>>>>>>>>>>> like a really simple method ( >>>>>>>>>>> >>>>>>>>>>>> http://grepcode.com/file/repo1.maven.org/maven2/org. >>>>>>>>>>>>> >>>>>>>>>>>>> apache.activemq/activemq-all/5.10.0/org/apache/activemq/ >>>>>>>>>>>> >>>>>>>>>>> thread/Scheduler.java#Scheduler.cancel%28java.lang.Runnable%29 >>>>>>>> >>>>>>>> ). >>>>>>>> >>>>>>>>> On Sun, Feb 22, 2015 at 7:56 PM, Kevin Burton <[email protected]> >>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Here’s a jprofiler view with the advisory support enabled if >>>>>>>>>>>>> you’re >>>>>>>>>>>>> >>>>>>>>>>>>> curious. >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://i.imgur.com/I1jesZz.jpg >>>>>>>>>>>>>> >>>>>>>>>>>>>> I’m not familiar with the internals of ActiveMQ enough to have >>>>>>>>>>>>>> any >>>>>>>>>>>>>> >>>>>>>>>>>>>> obvious >>>>>>>>>>>>>> >>>>>>>>>>>>> optimization ideas. >>>>>>>>>>>>> >>>>>>>>>>>>>> One other idea I had (which would require a ton of refactoring >>>>>>>>>>>>>> I >>>>>>>>>>>>>> >>>>>>>>>>>>>> think) >>>>>>>>>>>>>> >>>>>>>>>>>>> would be to potentially bulk delete all the queues at once. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, Feb 22, 2015 at 6:42 PM, Kevin Burton < >>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>> And spending some more time in jprofiler, looks like 20% of >>>>>>>>>>>>> this >>>>>>>>>>>>> >>>>>>>>>>>>>> is >>>>>>>>>>>>>> >>>>>>>>>>>>>> do to >>>>>>>>>>>>> >>>>>>>>>>>> schedulerSupport and the other 80% of this is do to >>>>>>>>>> >>>>>>>>>>> advisorySupport. >>>>>>>>>>>>>> >>>>>>>>>>>>>> if I set both to false the total runtime of my tests drops in >>>>>>>>>>>>> >>>>>>>>>>>> half… >>>>>>>>>> >>>>>>>>>>> and >>>>>>>>>>>>> >>>>>>>>>>>> the latencies fall from >>>>>>>>>> >>>>>>>>>>> max create producer latency: 10,176 ms >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> max create message on existing producer and consumer: 2 ms >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> … to >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> max create producer latency: 1 ms >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> max create message on existing producer and consumer: 1 ms >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> and this isn’t without fixing the purge background lock. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So the question now is what the heck is the advisory support >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> doing >>>>>>>>>>>>>>> >>>>>>>>>>>>>> that >>>>>>>>>>>>> >>>>>>>>>>>> can result in such massive performance overhead. >>>>>>>>>> >>>>>>>>>>> … and I think advisorySupport is enabled by default so that’s >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> problematic >>>>>>>>>>>>>>> >>>>>>>>>>>>>> as well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, Feb 22, 2015 at 4:45 PM, Kevin Burton < >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>> OK. Loaded up JProfiler and confirmed that it’s not LevelDB. >>>>>>>>>> >>>>>>>>>>> This is a non-persistent broker I’m testing on. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Looks like it’s spending all it’s time in >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> CopyOnWriteArrayList.remove >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> and >>>>>>>>>>>>>> >>>>>>>>>>>>>> Timer.purge… >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Which is hopeful because this is ALL due to ActiveMQ internals >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> in >>>>>>>>>>>>>> >>>>>>>>>>>>> theory LevelDB should perform well if we improve the >>>>>>>>>> >>>>>>>>>>> performance of >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ActiveMQ internals and fix this lock bug. >>>>>>>>>>>>>> >>>>>>>>>>>>> Which would rock! >>>>>>>>>> >>>>>>>>>>> It should ALSO make queue creation faster. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sun, Feb 22, 2015 at 4:10 PM, Kevin Burton < >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Feb 22, 2015 at 3:30 PM, Tim Bain < >>>>>>>>>> >>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>> So if LevelDB cleanup during removeDestination() is the >>>>>>>>>> >>>>>>>>>>> presumed >>>>>>>>>>>>>>>> culprit, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> can we spin off the LevelDB cleanup work into a separate >>>>>>>>>> >>>>>>>>>>> thread >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> (better: a >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> task object to be serviced by a ThreadPool so you can avoid a >>>>>>>>>> >>>>>>>>>>> fork >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> bomb >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> if >>>>>>>>>>>>>> >>>>>>>>>>>>>>> we remove many destinations at once) so the call to >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> removeDestination() >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>> return quickly and LevelDB can do its record-keeping in the >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> background >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> without blocking message-processing? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Would that be possible? If the delete is pending on >>>>>>>>>>>>>>> ActiveMQ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> there is >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>> >>>>>>>>>>>>>> race where a producer could re-create it unless the lock is >>>>>>>>>>>>>> >>>>>>>>>>>>>>> held. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Though I guess if you dispatched to the GC thread WITH the >>>>>>>>>>>>>>> lock >>>>>>>>>>>>>>> >>>>>>>>>>>>>> still >>>>>>>>>> >>>>>>>>>>> held you would be ok but I think if we use the existing purge >>>>>>>>>>>>>>> thread >>>>>>>>>>>>>>> then >>>>>>>>>>>>>>> >>>>>>>>>>>>>> we’re fine. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> OK. I think I’m wrong about LevelDB being the issue. To be >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> fair I >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wasn’t 100% certain before but I should have specified. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Our current production broker is running with >>>>>>>>>> >>>>>>>>>>> persistent=false.. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> .and I >>>>>>>>>>>>>>> >>>>>>>>>>>>>> just re-ran the tests without disk persistence and it has the >>>>>>>>>> >>>>>>>>>>> same >>>>>>>>>>>>>>> problem. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> So the main issue how is why the heck is ActiveMQ taking SO >>>>>>>>>> >>>>>>>>>>> LONG >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> to GC >>>>>>>>>>>>>>> >>>>>>>>>>>>>> a >>>>>>>>>> >>>>>>>>>>> queue. It’s taking about 100ms which is an insane amount of >>>>>>>>>>>>>> >>>>>>>>>>>>>>> time >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> considering this is done all in memory. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Kevin >>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Founder/CEO Spinn3r.com >>>>>>>>>>>>>>>>> Location: *San Francisco, CA* >>>>>>>>>>>>>>>>> blog: http://burtonator.wordpress.com >>>>>>>>>>>>>>>>> … or check out my Google+ profile >>>>>>>>>>>>>>>>> <https://plus.google.com/102718274791889610666/posts> >>>>>>>>>>>>>>>>> <http://spinn3r.com> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Founder/CEO Spinn3r.com >>>>>>>>>>>>>>>> Location: *San Francisco, CA* >>>>>>>>>>>>>>>> blog: http://burtonator.wordpress.com >>>>>>>>>>>>>>>> … or check out my Google+ profile >>>>>>>>>>>>>>>> <https://plus.google.com/102718274791889610666/posts> >>>>>>>>>>>>>>>> <http://spinn3r.com> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Founder/CEO Spinn3r.com >>>>>>>>>>>>>>> Location: *San Francisco, CA* >>>>>>>>>>>>>>> blog: http://burtonator.wordpress.com >>>>>>>>>>>>>>> … or check out my Google+ profile >>>>>>>>>>>>>>> <https://plus.google.com/102718274791889610666/posts> >>>>>>>>>>>>>>> <http://spinn3r.com> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>> Founder/CEO Spinn3r.com >>>>>>>>>>>>>> Location: *San Francisco, CA* >>>>>>>>>>>>>> blog: http://burtonator.wordpress.com >>>>>>>>>>>>>> … or check out my Google+ profile >>>>>>>>>>>>>> <https://plus.google.com/102718274791889610666/posts> >>>>>>>>>>>>>> <http://spinn3r.com> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>> Founder/CEO Spinn3r.com >>>>>>>>>>>> Location: *San Francisco, CA* >>>>>>>>>>>> blog: http://burtonator.wordpress.com >>>>>>>>>>>> … or check out my Google+ profile >>>>>>>>>>>> <https://plus.google.com/102718274791889610666/posts> >>>>>>>>>>>> <http://spinn3r.com> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>> Founder/CEO Spinn3r.com >>>>>>>>>>> Location: *San Francisco, CA* >>>>>>>>>>> blog: http://burtonator.wordpress.com >>>>>>>>>>> … or check out my Google+ profile >>>>>>>>>>> <https://plus.google.com/102718274791889610666/posts> >>>>>>>>>>> <http://spinn3r.com> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>> Founder/CEO Spinn3r.com >>>>>>>>>> Location: *San Francisco, CA* >>>>>>>>>> blog: http://burtonator.wordpress.com >>>>>>>>>> … or check out my Google+ profile >>>>>>>>>> <https://plus.google.com/102718274791889610666/posts> >>>>>>>>>> <http://spinn3r.com> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>> Founder/CEO Spinn3r.com >>>>>>>>> Location: *San Francisco, CA* >>>>>>>>> blog: http://burtonator.wordpress.com >>>>>>>>> … or check out my Google+ profile >>>>>>>>> <https://plus.google.com/102718274791889610666/posts> >>>>>>>>> <http://spinn3r.com> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>> >>>>>> Tim Bish >>>>>> Sr Software Engineer | RedHat Inc. >>>>>> [email protected] | www.redhat.com >>>>>> skype: tabish121 | twitter: @tabish121 >>>>>> blog: http://timbish.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>>> -- >>>> Tim Bish >>>> Sr Software Engineer | RedHat Inc. >>>> [email protected] | www.redhat.com >>>> skype: tabish121 | twitter: @tabish121 >>>> blog: http://timbish.blogspot.com/ >>>> >>>> >>>> >>> >> > > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com>
