Hi Thomas.
The start and stop scripts are actually in the rc folder called assp.dat
start.dat and stop.dat.
They must have been current when I first set things up I've moved
everything over to the Debian script and it didn't take long for both ASSP
instances to do exactly the same as before so these have nothing to do with
it I'm afraid.
All the best,
Colin.
On Wed, Oct 5, 2016 at 12:38 PM, cw <colin.war...@gmail.com> wrote:
> Hi Thomas,
>
> The init scripts and start/stop scripts must be hanging around from an
> earlier version. The start script pretty much calls /usr/bin/perl
> /usr/local/assp/assp.pl /usr/local/assp
>
> I don't think the stop script is ever actually used - there is no
> reference to it anywhere in the config and my external monitoring just
> calls the start script. I'll definitely look at updating them.
>
> ASSP shuts itself down in these circumstances as follows:
>
> 2016-10-05 12:27:57 [Main_Thread] Error: Main_Thread is unable to transfer
> connection to any worker within 120 seconds - restart ASSP!
> 2016-10-05 12:27:57 [Main_Thread] Initializing shutdown sequence
> 2016-10-05 12:27:57 [Shutdown] Info: removing all SMTP and Proxy listeners
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 6 - QUIT
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 3 - QUIT
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 4 - QUIT
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 5 - QUIT
> 2016-10-05 12:27:57 [Worker_4] Worker_4 has active connections. Will wait
> until all connections are finished but max 45 seconds!
> 2016-10-05 12:27:57 [Worker_3] Worker_3 has active connections. Will wait
> until all connections are finished but max 45 seconds!
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 7 - QUIT
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 1 - QUIT
> 2016-10-05 12:27:57 [Worker_1] Worker_1 has active connections. Will wait
> until all connections are finished but max 45 seconds!
> 2016-10-05 12:27:57 [Worker_7] Worker_7 has active connections. Will wait
> until all connections are finished but max 45 seconds!
> 2016-10-05 12:27:57 [Shutdown] Tell Worker 2 - QUIT
> 2016-10-05 12:28:47 [Shutdown] Error: at least one of the SMTP workers has
> not finished work within 50 seconds
> 2016-10-05 12:28:47 [Shutdown] Closing all databases
> 2016-10-05 12:28:47 [Shutdown] Info: removing all WEB listeners
> 2016-10-05 12:28:47 [Shutdown] Info: shutdown reason was: restarting
> 2016-10-05 12:28:47 [Shutdown] ASSP finished work
>
> So no unclean kill or anything like that. I can't remember the last time
> that ASSP locked up completely and had to be forcibly terminated.
>
> On Wed, Oct 5, 2016 at 12:14 PM, cw <colin.war...@gmail.com> wrote:
>
>> Thanks.
>>
>> I've had both servers come up against unable to detect any running worker
>> since clearing out all the files suggested. So I'm getting 16279 running
>> now.
>>
>> I noticed the startup with those files removed was really quick, starting
>> back up the second time took several minutes so presumably reading those
>> files during startup takes a little while versus creating them fresh.
>>
>> I've caught one already and traced it through the new logs:
>>
>> 016-10-05 11:47:38 [Main_Thread] Info: Main_Thread got connection request
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread looks up the best
>> Worker for new connection - 73
>> 2016-10-05 11:47:38 [Main_Thread] Info: try to interrupt worker Worker_3
>> (12) for new connection
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread interrupted Worker_3
>> (12) to submit the connection
>> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait (max 30 s)
>> for the answer of Worker_3 which handles 12 sockets
>> 2016-10-05 11:47:41 [Worker_3] SC-Time Worker_3: 0.0462169647216797
>> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 got connection from
>> MainThread - 73/73
>> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed Main_Thread - 170
>> 2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by interrupted
>> Worker_3 in 2.972 seconds - got (ok)
>> 2016-10-05 11:47:41 [Worker_3] Connected: session:7FF3AA6E9A28
>> 109.168.50.75:41612 > 92.63.138.65:25 > 127.0.0.1:125
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.103252172470093
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0278699398040771
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 [SMTP Reply] 220
>> mail2.smtphost.co.uk ESMTP Exim 4.86_2 Ubuntu Wed, 05 Oct 2016 11:47:41
>> +0100
>> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3: 0.0423040390014648
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 SC-Time Worker_3:
>> 0.0296478271484375
>> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
>> session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
>> 2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by interrupted
>> Worker_3 in 31.940 seconds - got (ok)
>>
>> In this case, it looks like the connection ended without any actual data.
>> There is lots of activity reported by Worker_3 in between 11:47:43 and
>> 11:48:13 but this all pertains to other connections that were already in
>> progress.
>>
>> That is a bit different to the earlier one where a message was received
>> and the message completed within the 30s window.
>>
>> I'm not seeing anything to help me figure out why though and I don't want
>> to simply post a big excerpt of the maillog.txt.
>>
>> On Wed, Oct 5, 2016 at 11:02 AM, Thomas Eckardt <
>> thomas.ecka...@thockar.com> wrote:
>>
>>> >I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
>>> it
>>> hasn't made its way live yet.
>>>
>>> Sorry, my background CVS sync was not running - update is done.
>>>
>>> Thomas
>>>
>>>
>>>
>>>
>>> Von: cw <colin.war...@gmail.com>
>>> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
>>> Datum: 05.10.2016 11:48
>>> Betreff: Re: [Assp-test] unable to detect any running worker
>>>
>>>
>>>
>>> Thank you Thomas.
>>>
>>> useDB4IntCache - already set to off
>>> I've set WorkerLog to diagnostic and done the other steps.
>>> I don't have anything in CorrectASSPcfg.pm so as part of the
>>> troubleshooting I have previously deleted it and downloaded a fresh copy
>>> from SourceForge.
>>>
>>> I have to say that ASSP started up within seconds after clearing those
>>> files out - it normally takes several minutes and has always done.
>>>
>>> I looked at SF but only see 16275 in test (updated 3 days ago) so maybe
>>> it
>>> hasn't made its way live yet.
>>>
>>> On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
>>> <thomas.ecka...@thockar.com>
>>> wrote:
>>>
>>> > I've provided an updated assp.pl (2.5.4 16279) in CVS /test. This
>>> version
>>> > shows some more information, if 'WorkerLog' is set to diagnostic.
>>> >
>>> > Thomas
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Von: cw <colin.war...@gmail.com>
>>> > An: ASSP development mailing list <assp-test@lists.sourceforge.net
>>> >
>>> > Datum: 05.10.2016 10:10
>>> > Betreff: Re: [Assp-test] unable to detect any running worker
>>> >
>>> >
>>> >
>>> > Hi Thomas,
>>> >
>>> > Thanks for chipping in. All modules are installed by running the latest
>>> > mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in it so it
>>> requires
>>> > a minor edit to the code to get it to build. So that module was
>>> installed
>>> > by switching to /root/.cpan/build/Crypt-GOST-x-x-x and running:
>>> > make clean
>>> > perl Makefile.PL
>>> > make
>>> > make test
>>> > make install
>>> >
>>> > I run cpan-outdated -p|cpanm from time to time in order to keep modules
>>> up
>>> > to date as well so things don't stay stuck on old versions.
>>> >
>>> > Everything that is installed now is a completely fresh build as the
>>> > Upgrade
>>> > to 16.04 replaced perl 5.18 with 5.22 and all the modules therefore had
>>> to
>>> > be installed from scratch.
>>> >
>>> > I bypassed the issue by truncating the tables and setting up the users
>>> > again. With there only being one user it was easier to do that than
>>> muck
>>> > about with it - especially with the other bigger issue at hand.
>>> >
>>> > On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
>>> > <thomas.ecka...@thockar.com>
>>> > wrote:
>>> >
>>> > > >to replace the corrupted encrypted
>>> > > strings with the correct values
>>> > >
>>> > > Was the 'Crypt::GOST' module from the SF download page at the old
>>> assp
>>> > > instance?
>>> > > If it was, did you install the 'Crypt::GOST' module from the SF
>>> download
>>> > > page, before you started the new assp instance?
>>> > >
>>> > > https://sourceforge.net/projects/assp/files/ASSP%20V2%
>>> > > 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
>>> > >
>>> > > Thomas
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > Von: cw <colin.war...@gmail.com>
>>> > > An: ASSP development mailing list
>>> <assp-test@lists.sourceforge.net>
>>> > > Datum: 05.10.2016 09:52
>>> > > Betreff: Re: [Assp-test] unable to detect any running worker
>>> > >
>>> > >
>>> > >
>>> > > Cheers for the reply.
>>> > >
>>> > > I don't think backups are an option seen as I've moved completely
>>> from
>>> > > Ubuntu 14.04 to Ubuntu 16.04.
>>> > >
>>> > > Also, this issue has been around for months. It was causing a handful
>>> of
>>> > > shutdowns a week with the occasional spat of more frequent shutdowns.
>>> It
>>> > > is
>>> > > entirely possible that the errors are behaviour related and nothing
>>> to
>>> > do
>>> > > with the upgrade and that current email behaviour is triggering it
>>> big
>>> > > style.
>>> > >
>>> > > I'm not convinced though, unfortunately I'm not convinced of anything
>>> > else
>>> > > hence not having much to go on. I can't see any consistencies in the
>>> > > behaviour leading up to the events.
>>> > >
>>> > > I don't think it is database related. It happened on one of the mail
>>> > > servers during the upgrade to 16.04 when ASSP had just started but I
>>> had
>>> > > not yet got into the web interface to replace the corrupted encrypted
>>> > > strings with the correct values so all database connections were in
>>> > error.
>>> > >
>>> > > The problem has already started again this morning so I can see this
>>> > being
>>> > > another fun day that either leads to a fix or having to put something
>>> > else
>>> > > in place.
>>> > >
>>> > > On Wed, Oct 5, 2016 at 4:34 AM, K Post <nntp.p...@gmail.com> wrote:
>>> > >
>>> > > > I've been reading here, but I haven't had anything to suggest. All
>>> > > seems
>>> > > > quite odd if it was working prior to upgrading and downgrading
>>> didn't
>>> > > work.
>>> > > >
>>> > > >
>>> > > > Could you spin up a backup of the installation after copying the
>>> > current
>>> > > > data? Sure you'd have an older corpus, but I'd think you could add
>>> > the
>>> > > new
>>> > > > files if necessary, manually replace whitelist etc.
>>> > > >
>>> > > >
>>> > > > On Tue, Oct 4, 2016 at 5:48 PM, cw <colin.war...@gmail.com> wrote:
>>> > > >
>>> > > > > Further development on this today, very little.
>>> > > > > I have moved both servers onto Ubuntu 16.04 LTS which means going
>>> > from
>>> > > > perl
>>> > > > > 5.18 to 5.22 and rebuilding all perl modules from scratch.
>>> > > > >
>>> > > > > The admin user db did not work after the upgrade so I had to
>>> empty
>>> > the
>>> > > > > tables before it would come back online.
>>> > > > >
>>> > > > > I'm still getting delayed emails and assp shutting down telling
>>> me
>>> > it
>>> > > is
>>> > > > > unable
>>> > > > > to detect any running worker.
>>> > > > >
>>> > > > > If this goes on much longer the MD will pull the plug and we'll
>>> end
>>> > up
>>> > > > > moving to a third party solution which is not something I want
>>> but
>>> > if
>>> > > I
>>> > > > > can't fix it I can't defend it :/
>>> > > > >
>>> > > > > ------------------------------------------------------------
>>> > > > > ------------------
>>> > > > > Check out the vibrant tech community on one of the world's most
>>> > > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > > > > _______________________________________________
>>> > > > > Assp-test mailing list
>>> > > > > Assp-test@lists.sourceforge.net
>>> > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> > > > >
>>> > > > >
>>> > > >
>>> > > > ------------------------------------------------------------
>>> > > > ------------------
>>> > > > Check out the vibrant tech community on one of the world's most
>>> > > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > > > _______________________________________________
>>> > > > Assp-test mailing list
>>> > > > Assp-test@lists.sourceforge.net
>>> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> > > >
>>> > > >
>>> > > ------------------------------------------------------------
>>> > > ------------------
>>> > > Check out the vibrant tech community on one of the world's most
>>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > > _______________________________________________
>>> > > Assp-test mailing list
>>> > > Assp-test@lists.sourceforge.net
>>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > DISCLAIMER:
>>> > > *******************************************************
>>> > > This email and any files transmitted with it may be confidential,
>>> > legally
>>> > > privileged and protected in law and are intended solely for the use
>>> of
>>> > the
>>> > >
>>> > > individual to whom it is addressed.
>>> > > This email was multiple times scanned for viruses. There should be no
>>> > > known virus in this email!
>>> > > *******************************************************
>>> > >
>>> > >
>>> > > ------------------------------------------------------------
>>> > > ------------------
>>> > > Check out the vibrant tech community on one of the world's most
>>> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > > _______________________________________________
>>> > > Assp-test mailing list
>>> > > Assp-test@lists.sourceforge.net
>>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> > >
>>> > >
>>> > ------------------------------------------------------------
>>> > ------------------
>>> > Check out the vibrant tech community on one of the world's most
>>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > _______________________________________________
>>> > Assp-test mailing list
>>> > Assp-test@lists.sourceforge.net
>>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> >
>>> >
>>> >
>>> >
>>> > DISCLAIMER:
>>> > *******************************************************
>>> > This email and any files transmitted with it may be confidential,
>>> legally
>>> > privileged and protected in law and are intended solely for the use of
>>> the
>>> >
>>> > individual to whom it is addressed.
>>> > This email was multiple times scanned for viruses. There should be no
>>> > known virus in this email!
>>> > *******************************************************
>>> >
>>> >
>>> > ------------------------------------------------------------
>>> > ------------------
>>> > Check out the vibrant tech community on one of the world's most
>>> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> > _______________________________________________
>>> > Assp-test mailing list
>>> > Assp-test@lists.sourceforge.net
>>> > https://lists.sourceforge.net/lists/listinfo/assp-test
>>> >
>>> >
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Assp-test mailing list
>>> Assp-test@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>>
>>>
>>>
>>>
>>> DISCLAIMER:
>>> *******************************************************
>>> This email and any files transmitted with it may be confidential, legally
>>> privileged and protected in law and are intended solely for the use of
>>> the
>>>
>>> individual to whom it is addressed.
>>> This email was multiple times scanned for viruses. There should be no
>>> known virus in this email!
>>> *******************************************************
>>>
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Assp-test mailing list
>>> Assp-test@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>>
>>>
>>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
