Your message dated Sun, 22 Feb 2004 16:26:43 +0100
with message-id <[EMAIL PROTECTED]>
and subject line Developer.php performance (+patch)
has caused the attached Bug report to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what I am
talking about this indicates a serious mail system misconfiguration
somewhere. Please contact me immediately.)
Debian bug tracking system administrator
(administrator, Debian Bugs database)
--------------------------------------
Received: (at submit) by bugs.debian.org; 18 Feb 2004 18:30:12 +0000
>From [EMAIL PROTECTED] Wed Feb 18 10:30:12 2004
Return-path: <[EMAIL PROTECTED]>
Received: from a-eskwadraat.nl [131.211.34.218] (mail)
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian))
id 1AtWSZ-0008HK-00; Wed, 18 Feb 2004 10:30:11 -0800
Received: from jeroen by A-Eskwadraat.nl with local (Exim 3.35 #1 (Debian))
id 1AtWST-0007bi-00; Wed, 18 Feb 2004 19:30:05 +0100
Date: Wed, 18 Feb 2004 19:30:05 +0100
To: Jeroen van Wolffelaar <[EMAIL PROTECTED]>
Cc: debian-qa@lists.debian.org, [EMAIL PROTECTED]
Subject: Re: Developer.php performance (+patch)
Message-ID: <[EMAIL PROTECTED]>
References: <[EMAIL PROTECTED]>
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="tcC6YSqBgqqkz7Sb"
Content-Disposition: inline
In-Reply-To: <[EMAIL PROTECTED]>
User-Agent: Mutt/1.3.28i
X-Habeas-SWE-1: winter into spring
X-Habeas-SWE-2: brightly anticipated
X-Habeas-SWE-3: like Habeas SWE (tm)
X-Habeas-SWE-4: Copyright 2002 Habeas (tm)
X-Habeas-SWE-5: Sender Warranted Email (SWE) (tm). The sender of this
X-Habeas-SWE-6: email in exchange for a license for this Habeas
X-Habeas-SWE-7: warrant mark warrants that this is a Habeas Compliant
X-Habeas-SWE-8: Message (HCM) and not spam. Please report use of this
X-Habeas-SWE-9: mark in spam to <http://www.habeas.com/report/>.
From: Jeroen van Wolffelaar <[EMAIL PROTECTED]>
Delivered-To: [EMAIL PROTECTED]
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_02_18
(1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-7.8 required=4.0 tests=HABEAS_SWE,HAS_PACKAGE,
HTML_10_20,HTML_MESSAGE autolearn=no
version=2.60-bugs.debian.org_2004_02_18
X-Spam-Level:
--tcC6YSqBgqqkz7Sb
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Package: qa.debian.org
Severity: minor
Tags: patch
On Tue, Feb 17, 2004 at 03:29:56PM +0100, Jeroen van Wolffelaar wrote:
> Hi,
>
> developer.php is quite slow sometimes, especially on people with much
> packages, you can notice delays, it even timeouts sometimes.
Thanks to Martin Michlmayr, I was able to test developer.php myself.
Find attached a patch with these changes:
- Fix all PHP warning that were in the script (but surpressed by the
server)
- Read bug information using a tree lookup (db2) rather than linear
search
- Fix a off-by-one mistake and a missing $, causing respectively a
warning and wrong comaintainer-behaviour (These were easy to catch
with warnings enabled)
- Add a 'Page generated in' item at the bottom
And a 'convert-bugs-to-db' script, to be run (after fixing paths)
immediately after bugs.txt is updated (new db is moved over old, no race
conditions).
Note: if for some reason qa.debian.org isn't running woody (where I
tested it on), you might need to change db2 to db3 or db4, I worked
around the current bug in php4 (but you do need an uptodate php4 for it
to work).
While with the current implementation the developer.php page for
[EMAIL PROTECTED] fails with a 30sec timelimit timeout, that same
page takes only 0.1 seconds with this updated implementation.
Test URL: http://jeroen.a-eskwadraat.nl/misc/qa/developer.php
(not updated with new info, unless someone can get me new extract &
bugs.txt files on the required intervals).
A developer.php page with freetext developer name still takes about
700ms constant time, I plan to look after that later, but currently other
things on my TODO list have more priority.
An improvement that shouldn't be hard, is making developer.php able to
generate a package-table based on other keys that the maintainer, like
based on task, or on a certain package with all his dependencies, stuff
like that. IMHO quite useful tracking down stuff.
--Jeroen
--
Jeroen van Wolffelaar
[EMAIL PROTECTED] (also for Jabber & MSN; ICQ: 33944357)
http://Jeroen.A-Eskwadraat.nl
--tcC6YSqBgqqkz7Sb
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="performance.patch"
Index: developer.wml
===================================================================
RCS file: /cvs/qa/wml/developer.wml,v
retrieving revision 1.49
diff -u -r1.49 developer.wml
--- developer.wml 20 Nov 2003 14:09:21 -0000 1.49
+++ developer.wml 18 Feb 2004 18:11:49 -0000
@@ -28,6 +28,7 @@
</style>
<?
+$time = microtime();
$branch = array ("main",
"contrib",
@@ -322,7 +323,7 @@
}
}
$global_count -= $result;
- }elseif(comaint == "only") {
+ }elseif($comaint == "only") {
$result = 0;
for ($i = 0; $i < strlen($packages); $i++) {
if ($packages[$i] == '#') {
@@ -410,18 +411,15 @@
$priority = $results[6];
$section = $results[7];
- $fp = fopen("$prefix/bugs.txt", "r");
- $bug_list = explode(" ", "0(0) 0(0) 0(0) 0(0)");
- $temp = preg_replace("/\+/", "_", $package);
- while(!feof($fp)) {
- $line = fgets($fp, "512");
- $line = preg_replace("/\+/", "_", $line);
-
if(preg_match("/^$temp\s+(\d+\(\d+\)\s\d+\(\d+\)\s\d+\(\d+\)\s\d+\(\d+\))/",
$line, $results)) {
- $bug_list = explode(" ", $results[1]);
- break;
- }
+ // make $bugsdb link persistent during this pagerequest, we don't lock, so
+ // this doesn't matter at all
+ static $bugsdb;
+ if (!$bugsdb) $bugsdb = dba_open("$prefix/bugs.db", 'r-', 'db2');
+
+ if (!($bug_list = dba_fetch($package, $bugsdb))) {
+ $bug_list = '0(0) 0(0) 0(0) 0(0)';
}
- fclose($fp);
+ $bug_list = explode(' ', $bug_list);
if(strcmp($version_stable, "-") == 0 || !debcheckavailable('stable',
$package)) {
$stable_dep = "<small>-</small>";
@@ -499,12 +497,12 @@
}
}
- if ($background == 1) {
+ if (@$background) {
print '<tr align="center">';
- $background=0;
+ $background=false;
}else{
print '<tr align="center" bgcolor="#dcdcdc">';
- $background=1;
+ $background=true;
}
?>
<td width="15%">
@@ -529,8 +527,8 @@
$bugs_all = "http://bugs.debian.org/cgi-bin/pkgreport.cgi?src=";.
rawurlencode($package);
$bug_count = $bug_list[0] + $bug_list[1] + $bug_list[2] + $bug_list[3];
$real_bug_count = 0;
- for ($i = 0; $i <= count($bug_list); $i++) {
- preg_match("/\d+\((\d+)\)/", $bug_list[$i], $count_array);
+ foreach ($bug_list as $bug_item) {
+ preg_match("/\d+\((\d+)\)/", $bug_item, $count_array);
$real_bug_count += $count_array[1];
}
if($bug_count == 0) { ?>
@@ -607,7 +605,7 @@
<td>
<small>
<?
- if($all == 1 || preg_match("/non-free/", $branch) ||
(strcmp($version_unstable, "-") == 0)) {
+ if(@$all || preg_match("/non-free/", $branch) || (strcmp($version_unstable,
"-") == 0)) {
print "-";
$all = 0;
}else{
@@ -670,10 +668,10 @@
<?
}
-if($comaint == "") {
+if([EMAIL PROTECTED]) {
$comaint = "yes";
}
-if($excuse != "") {
+if(@$excuse) {
$char = substr($excuse, 0, 1);
$excuse = preg_replace("/ /", "+", $excuse);
if(file_exists("$prefix/excuses/$char/$excuse")) {
@@ -683,7 +681,7 @@
}else{
print "<h2>No excuse for $excuse</h2>";
}
-}else if($gpg_key != "") {
+}else if (@$gpg_key) {
$contents = file("$prefix/extract");
$printed = 0;
@@ -715,7 +713,7 @@
}
printBackLink();
-}else if($login != "") {
+}else if(@$login) {
$fp = fopen("$prefix/extract", "r");
$contents = fread($fp, filesize ("$prefix/extract"));
@@ -753,7 +751,7 @@
}
fclose($fp);
printBackLink();
-}else if ($package != "") {
+}else if (@$package) {
$fp = fopen("$prefix/extract", "r");
$found = 0;
if(preg_match("/\+\+/", $package)) {
@@ -796,7 +794,7 @@
printBackLink();
}else{
print_form();
- if($all) {
+ if(@$all) {
print_all();
}else{
print_query_all();
@@ -818,5 +816,10 @@
General information: <? print $general; ?><br>
Bug information: <? print $bugs; ?><br>
General information is updated once a day.<br>
-Bug information is updated every 15 minutes.
+Bug information is updated every 15 minutes.<br>
+<?
+list($micro,$sec) = explode(' ', microtime());
+list($smicro,$ssec) = explode(' ', $time);
+echo "Time needed to generate page: ".($sec-$ssec+$micro-$smicro)."s";
+?>
</small></p>
--tcC6YSqBgqqkz7Sb
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=convert-bugs-to-db
#!/usr/bin/php4 -q
<?
$db = dba_open('bugs.db.new', 'n-', 'db2')
or die("Opening bugs.db failed");
$fp = fopen("bugs.txt", "r");
while ($line = fgets($fp, 512)) {
list($package, $bugdata) = preg_split('/\\s+/', $line, 2);
$bugdata = trim($bugdata);
dba_insert($package, $bugdata, $db)
or die("Insert for $package failed");
}
dba_optimize($db)
or die("Optimize failed");
dba_close($db);
rename('bugs.db.new', 'bugs.db');
--tcC6YSqBgqqkz7Sb--
---------------------------------------
Received: (at 233564-done) by bugs.debian.org; 22 Feb 2004 15:27:19 +0000
>From [EMAIL PROTECTED] Sun Feb 22 07:27:19 2004
Return-path: <[EMAIL PROTECTED]>
Received: from home.genibel.org (tass) [81.56.84.10]
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian))
id 1AuvVm-0001o2-00; Sun, 22 Feb 2004 07:27:19 -0800
Received: from igor by tass with local (Exim 4.30)
id 1AuvVD-0008DW-HK; Sun, 22 Feb 2004 16:26:43 +0100
Date: Sun, 22 Feb 2004 16:26:43 +0100
From: Igor Genibel <[EMAIL PROTECTED]>
To: Jeroen van Wolffelaar <[EMAIL PROTECTED]>
Cc: debian-qa@lists.debian.org, [EMAIL PROTECTED]
Subject: Re: Developer.php performance (+patch)
Message-ID: <[EMAIL PROTECTED]>
Mail-Followup-To: Jeroen van Wolffelaar <[EMAIL PROTECTED]>,
debian-qa@lists.debian.org, [EMAIL PROTECTED]
References: <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
protocol="application/pgp-signature"; boundary="3uo+9/B/ebqu+fSQ"
Content-Disposition: inline
In-Reply-To: <[EMAIL PROTECTED]>
Organisation: TuxFamily free hosting for free people
X-URL: http:/www.genibel.org/
User-Agent: Mutt/1.5.5.1+cvs20040105i
Delivered-To: [EMAIL PROTECTED]
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_02_22
(1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=0.0 required=4.0 tests=none autolearn=no
version=2.60-bugs.debian.org_2004_02_22
X-Spam-Level:
--3uo+9/B/ebqu+fSQ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
* Jeroen van Wolffelaar <[EMAIL PROTECTED]> [2004-02-18 19:30:05 +0100]:
I applied the patch and modified backend scripts in order to have these
functionalities into DDPO.
This really increase the performances and permits users that have a lot
of package to display correctly their page without timeouts.
Thanks a lot Jeroen ;)
--=20
Igor Genibel=20
http://www.answare.fr/ [EMAIL PROTECTED]
http://www.tuxfamily.org/ [EMAIL PROTECTED]
http://people.debian.org/~igenibel/ [EMAIL PROTECTED]
GPG: 1024D/1648F6C8
--3uo+9/B/ebqu+fSQ
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFAOMoz+xgdMBZI9sgRAsYZAKCTLaZVbL2ct9kFD7oWqZSidwtQrgCfWl0F
0Qq6t7miFdus9mdVs0v/yaw=
=OuE8
-----END PGP SIGNATURE-----
--3uo+9/B/ebqu+fSQ--
