Awesome, good to hear it's been useful. :) Even better if you can
improve it further.
--Ere
dc...@prosentient.com.au kirjoitti 21.4.2021 klo 3.46:
Thanks for that optimization. My restart time has gone from 2 minutes 45
seconds to 1 minute 30 seconds. Much better.
Now I'm going to try the preload and see if I can get it down to 45 seconds or
better...
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
-----Original Message-----
From: Koha-devel <koha-devel-boun...@lists.koha-community.org> On Behalf Of
dc...@prosentient.com.au
Sent: Wednesday, 21 April 2021 10:28 AM
To: 'Ere Maijala' <ere.maij...@helsinki.fi>; koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Optimizing Starman startup
Hi Ere,
Thanks for your reply. 24700 looks much better. I'll look at backporting it
locally.
Although I'm looking at JSON::Validator::OpenAPI::Mojolicious at
https://metacpan.org/pod/release/JHTHORSEN/Mojolicious-Plugin-OpenAPI-2.19/lib/JSON/Validator/OpenAPI/Mojolicious.pm
and it says "Do not use this module directly. Use Mojolicious::Plugin::OpenAPI instead."
I notice that you're using the "bundle" method. Do we really need that there? Why don't
we just load the JSON using the JSON module, merge with the plugin spec files, and then pass it to
the OpenAPI plugin? Shouldn't the plugin take care of the $ref replacement?
Hmm... I didn't realize until now that the OpenAPI plugin was doing a validate
behind the scenes. That's tricky.
At a glance, we might be able to pre-load the app into the Starman master
process pre-fork. There are warnings about doing that with open database
connections, so we'd need to review plack.psgi, but a quick glance suggests it
might be OK. (Alternatively, I have wondered about running the REST API as a
separate process apart from Starman using hypnotoad. According to
https://docs.mojolicious.org/Mojolicious/Guides/Cookbook, Mojo::Server::Prefork
preloads the application in the manager/master process, and Hypnotoad is based
off that, so that would help.)
It does seem like changes to the OpenAPI plugin would be needed for caching.
I'm going to try backporting your change and try pre-loading and see how far
that gets me.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
-----Original Message-----
From: Koha-devel <koha-devel-boun...@lists.koha-community.org> On Behalf Of Ere
Maijala
Sent: Tuesday, 20 April 2021 4:48 PM
To: koha-devel@lists.koha-community.org
Subject: Re: [Koha-devel] Optimizing Starman startup
Hi,
I did some work on improving it here:
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24700
That shaved a good bit of time from it, but it's still a heavy operation, and
it would make sense to
1.) avoid doing it too often
2.) cache the results and avoid doing it if results are cached
If you could address the first one, that'd go a long way. I'm afraid the second
one would require changes to the OpenAPI plugin to support caching.
--Ere
dc...@prosentient.com.au kirjoitti 20.4.2021 klo 6.15:
Hi all,
Do you despair when you see the following periodically in “top” when a
starman worker is recreated ?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
9529 my-koha 20 0 460108 197212 17172 R 100.0 0.4 0:03.41
/usr/share/koha/api/v1/app.pl
Or the following in top when you install koha-common package or
restart the koha-common service?
11101 1-koha 20 0 447232 193320 16076 R 10.6 0.4 0:09.09
/usr/share/koha/api/v1/app.pl
11168 1-koha 20 0 447240 193264 16056 R 10.6 0.4 0:08.72
/usr/share/koha/api/v1/app.pl
11306 2-koha 20 0 447220 193148 16000 R 10.6 0.4 0:08.07
/usr/share/koha/api/v1/app.pl
11543 2-koha 20 0 447232 193036 15828 R 10.6 0.4 0:07.07
/usr/share/koha/api/v1/app.pl
11784 3-koha 20 0 441536 189664 16172 R 10.6 0.4 0:06.04
/usr/share/koha/api/v1/app.pl
11830 3-koha 20 0 439548 187212 15748 R 10.6 0.4 0:05.82
/usr/share/koha/api/v1/app.pl
11831 4-koha 20 0 438620 186344 15748 R 10.6 0.4 0:05.81
/usr/share/koha/api/v1/app.pl
11853 4-koha 20 0 437680 185672 16000 R 10.6 0.4 0:05.79
/usr/share/koha/api/v1/app.pl
Well, I still have a lot of investigation left to do, but I notice 1
place that a lot of time taken is here (per worker):
my $validator = JSON::Validator::OpenAPI::Mojolicious->new;
$validator->load_and_validate_schema(
$self->home->rel_file("api/v1/swagger/swagger.json"),
{
allow_invalid_ref => 1,
}
);
push @{$self->routes->namespaces}, 'Koha::Plugin';
my $spec = $validator->schema->data;
$self->plugin(
'Koha::REST::Plugin::PluginRoutes' => {
spec => $spec,
validator => $validator
}
);
$self->plugin(
OpenAPI => {
spec => $spec,
route =>
$self->routes->under('/api/v1')->to('Auth#under'),
allow_invalid_ref =>
1, # required by our spec because $ref directly
under
# Paths-, Parameters-, Definitions- & Info-object
# is not allowed by the OpenAPI specification.
}
);
Anyone have ideas for improving this? Do we have to validate the
schema every time? Can we move the schema validation into a different
module and preload it into Starman using the -M flag so that it’s done
1 time per Starman master instance rather than 1 time per Starman worker
instance?
I find “/usr/share/koha/api/v1/app.pl” to be the bane of deployments,
as it puts a massive load on a server, when you have multiple Koha
instances on the server.
David Cook
Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia
Office: 02 9212 0899
Online: 02 8005 0595
_______________________________________________
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/ git :
https://git.koha-community.org/ bugs :
https://bugs.koha-community.org/
--
Ere Maijala
Kansalliskirjasto / The National Library of Finland
_______________________________________________
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/ git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/
_______________________________________________
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/ git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/
--
Ere Maijala
Kansalliskirjasto / The National Library of Finland
_______________________________________________
Koha-devel mailing list
Koha-devel@lists.koha-community.org
https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-devel
website : https://www.koha-community.org/
git : https://git.koha-community.org/
bugs : https://bugs.koha-community.org/
