Re: libbarclay-java needs update, libpicard-java runtime tests by pigx-scrnaseq

[Pierre Gruet]

Hi Steffen,

Le 11/09/2020 à 15:27, Steffen Möller a écrit :
> Hello again,
> 
> [...]
> 
> and the error prevails when minimizing the invocation as to
> 
> 
> ~/git/med-team/pigx/pigx-scrnaseq$ /usr/bin/java -XX:ParallelGCThreads=1
> -Xmx4G -Xss16M -Djava.io.tmpdir=/tmp -cp $PICARD_CLASSPATH -j
> ar /usr/share/java/picard.jar FastqToSam  
> Error: Unable to initialize main class picard.cmdline.PicardCommandLine
> 
> while the following (without the -jar) is working (borrowed from the
> picard-tools script):
> 
> 
> export
> PICARD_CLASSPATH=/usr/share/java/picard.jar:/usr/share/java/htsjdk.jar:/usr/share/java/guava
> .jar:/usr/lib/jvm/default-java/lib/tools.jar:/usr/share/java/commons-lang3.jar:/usr/share/java/gkl.jar:/usr/share/java/gatk-native-bindings.jar:/usr/share/java/barc
> lay.jar
> 
> ~/git/med-team/pigx/pigx-scrnaseq$ /usr/bin/java -XX:ParallelGCThreads=1
> -Xmx4G -Xss16M -Djava.io.tmpdir=/tmp -cp $PICARD_CLASSPATH  p
> icard.cmdline.PicardCommandLine FastqToSam
> 
> With a quick confirmation from
> https://stackoverflow.com/questions/15930782/call-java-jar-myfile-jar-with-additional-classpath-option
> it seems like some work is needed on the Manifest that accompanies our
> libpicard-java package.
> 
> I'll read up how to do that properly in Debian, so our .jar is
> compatible with what the community apparently expects. @Andreas, if this
> is something close to your routine, please jump in.
>

I have just looked at picard-tools and I saw that, yesterday, you
modified libpicard-java.manifest. I suspect this fixed the test-case you
submitted above. Can you confirm?
Let me know if the classpath of libpicard-java needs further investigation.
> 
> Best,
> 
> Steffen
> 
> 
Best,
Pierre

Re: Autopkgtest failure with new version of kaptive - may be related to new version of biopython

[Étienne Mollier]

Hi Andreas,

Andreas Tille, on 2020-09-13 08:39:20 +0200:
> the new version (0.7.3) of kaptive[1] fails its autopkgtest:
> 
> + kaptive.py -a exact_match.fasta -k 
> /usr/share/kaptive/reference_database/Klebsiella_k_locus_primary_reference.gbk
>  -o output/test
> /usr/lib/python3/dist-packages/Bio/GenBank/__init__.py:1287: 
> BiopythonParserWarning: The NCBI states double-quote characters like " should 
> be escaped as "" (two double - quotes), but here it was not: 
> 'undecaprenyl-phosphate galactose phosphotransferase" glucose-1-phosphate 
> transferase'
>   warnings.warn(
> Error: tblastn crashed!
> 
> 
> Interestingly also the old version shows currently an issue in
> testing on amd64[2]:
> 
> + kaptive.py -a exact_match.fasta -k 
> /usr/share/kaptive/reference_database/Klebsiella_k_locus_primary_reference.gbk
>  -o output/test
> Error: tblastn crashed!
> 
> 
> But this seems to be kind of a random failure since unstable as well as
> arm64 and ppc64el in testing are currently OK[3].

Thanks for the notice, I tried running the test in several
architectures as well.  After running a few times autopkgtests,
I see the intermittent error indeed, at least on amd64, and
i386.  Other architectures I tested (arm64, riscv64) didn't show
this issue so far; I'm virtualizing them, so maybe Qemu
interferes.

My kernel traps a general protection fault in libblast.so during
tblastn execution:

$ sudo dmesg
[...]
[1741071.802876] traps: tblastn[3579372] general protection fault 
ip:7ff551327dde sp:7ffdb8072790 error:0 in libblast.so[7ff55130f000+63000]

Looking up into straces, it seems the condition occurs when the
program frees its memory before handing back the hand to the
system, so maybe something in the taste of a double free while
threads are not well aligned.  I'm not sure yet of the exact
reproducer with the tblastn command, here is what I record in my
`ps` output; some of the files referenced are temporary
artifacts obtained from former steps of the script:

/usr/bin/tblastn -db_gencode 11 -seg no -db 
/tmp/autopkgtest.DOVsFP/autopkgtest_tmp/exact_match.fasta -query 
/tmp/autopkgtest.DOVsFP/autopkgtest_tmp/output/kaptive_temp_3600033_520819/temp_gene_seqs.fasta
 -num_threads 4 -outfmt 6 qseqid sseqid qstart qend sstart send evalue bitscore 
length pident qlen qseq sseq

I'm also near finishing packaging Biopython 1.78, so that update
might add a bit more entropy into these testings.

Kind Regards,
-- 
Étienne Mollier 
Old rsa/3072: 5ab1 4edf 63bb ccff 8b54  2fa9 59da 56fe fff3 882d
New rsa/4096: 8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/5, please excuse my verbosity.


signature.asc
Description: PGP signature

[RFS] survivor

[Nilesh Patra]

Hi,
I've packaged survivor - this is  a NEW package which builds
in a clean chroot and is lintian clean with passing autoopkgtests.
My changes are pushed to the team-repo here[1].

SInce this needs to build in the Debug sub-directory instead of the main
directory. I took help
from Steffen's way of packaging hnswlib, thanks Steffen!
Needs review and sponsorship.

[1]: https://salsa.debian.org/med-team/survivor

Thanks and regards
Nilesh

Re: makeblastdb 2.10.x on 32-bit architectures

[Étienne Mollier]

Control: tag -1 reopen

Greetings,

Étienne Mollier, on 2020-09-08 21:10:18 +0200:
> Aaron M. Ucko, on 2020-09-07 21:55:45 -0400:
> > Étienne Mollier  writes:
> > > Thanks for your work on this topic.  I attempted to remove the
> > > patches to address #960756 in Biopython.  While the problem is
> > > fixed on i386 without having to fall back on version 4, I hit a
> > > very similar issue on amd64 this time.
> > 
> > Oops, I accidentally introduced a regression on 64-bit architectures;
> > fixed in -3 just now.  (An intended formal clarification misfired.)
> 
> No problem, I have done my fair share of oopses here and there,
> that could happen to anybody.  I confirm I haven't seen the bug
> reappear on amd64 or i386 with the new version.  The patch for
> working the test around could now be safely dropped.
> 
> > Thank you very much for pointing it out!
> 
> You're most welcome, glad you could solve it so quickly!
> 
> Have a nice day,  :)

While running the testsuite of the python-biopython package
impending update, I noticed the autopkgtest set back to do the
run with `makeblastdb -version 5` failed in arm64, so the
regression looks still there in version 2.10.0-3, at least on
arm64 architecture.

This issue doesn't seem caught by the test suite of ncbi-blast+,
so it might be worth adding this test.  In the meanwhile, I
wrapped up a small patch for the python-biopython test suite to
fall back to version 4 on a selection of architectures.

Thanks,
-- 
Étienne Mollier 
Old rsa/3072: 5ab1 4edf 63bb ccff 8b54  2fa9 59da 56fe fff3 882d
New rsa/4096: 8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/8, please excuse my verbosity.


signature.asc
Description: PGP signature

Re: libbarclay-java needs update, libpicard-java runtime tests by pigx-scrnaseq

[Steffen Möller]

Hi Pierre,

On 13.09.20 11:25, Pierre Gruet wrote:
> Hi Steffen,
>
> Le 11/09/2020 à 15:27, Steffen Möller a écrit :
>> Hello again,
>>
>> [...]
>>
>> and the error prevails when minimizing the invocation as to
>>
>>
>> ~/git/med-team/pigx/pigx-scrnaseq$ /usr/bin/java -XX:ParallelGCThreads=1
>> -Xmx4G -Xss16M -Djava.io.tmpdir=/tmp -cp $PICARD_CLASSPATH -j
>> ar /usr/share/java/picard.jar FastqToSam  
>> Error: Unable to initialize main class picard.cmdline.PicardCommandLine
>>
>> while the following (without the -jar) is working (borrowed from the
>> picard-tools script):
>>
>>
>> export
>> PICARD_CLASSPATH=/usr/share/java/picard.jar:/usr/share/java/htsjdk.jar:/usr/share/java/guava
>> .jar:/usr/lib/jvm/default-java/lib/tools.jar:/usr/share/java/commons-lang3.jar:/usr/share/java/gkl.jar:/usr/share/java/gatk-native-bindings.jar:/usr/share/java/barc
>> lay.jar
>>
>> ~/git/med-team/pigx/pigx-scrnaseq$ /usr/bin/java -XX:ParallelGCThreads=1
>> -Xmx4G -Xss16M -Djava.io.tmpdir=/tmp -cp $PICARD_CLASSPATH  p
>> icard.cmdline.PicardCommandLine FastqToSam
>>
>> With a quick confirmation from
>> https://stackoverflow.com/questions/15930782/call-java-jar-myfile-jar-with-additional-classpath-option
>> it seems like some work is needed on the Manifest that accompanies our
>> libpicard-java package.
>>
>> I'll read up how to do that properly in Debian, so our .jar is
>> compatible with what the community apparently expects. @Andreas, if this
>> is something close to your routine, please jump in.
>>
> I have just looked at picard-tools and I saw that, yesterday, you
> modified libpicard-java.manifest. I suspect this fixed the test-case you
> submitted above. Can you confirm?
> Let me know if the classpath of libpicard-java needs further investigation.

Sorry, yes, I forgot to forward this to the list. It works now and I
also added a smallish test invoking that picard.jar in d/rules

Thank you for chasing this up, Pierre, much appreciated. I somehow
thought that this classpath-fruit hangs low enough for me reach if I
just stretch myself a bit.

Best,

Steffen

Re: [MoM] ampl-netlib-solvers

[Andrei Rozanski]

Hi Andreas,


I made the modifications that you have suggested.

Not sure about the copyright warning on lintian. I have tried 
scan-copyrights and grepping README. No success.


Many thanks!


Best

AndreiR


On 9/12/20 10:49 PM, Andreas Tille wrote:

Hi Andrei,

On Sat, Sep 12, 2020 at 08:44:40PM +0200, Andrei Rozanski wrote:

Thanks for that.

You are welcome.
  

I have compiled it in the past without issues making no changes in cflags.

The only reason I did the patch was a sed line in
https://github.com/ghackebeil/gjh_asl_json/blob/master/Thirdparty/get.ASL

Probably those cflags would not make huge difference in the end.

In Debian it is good to propagate CFLAGS (and other compiler flags into
upstream Makefile.  I fixed the patch to accomplish this and have
standard build flags (which are generated from the hardening option
for instance) as well as the recommended flags.

The resulting package does not yet feature any build results which is as
far as I see a development library.  That means the package name should
be probably

 libampl-netlib-solvers-dev

Please make sure the *.a file that is build will be installed.  I'd
recomment to have a look for instance into

 
https://salsa.debian.org/med-team/libvcflib/-/blob/master/debian/libvcflib-dev.install

(well, there is a *.so file installed but the point is that it is installed
in DEB_HOST_MULTIARCH).  You also need to add dh-exec to Build-Depends.
Usually also *.h files need to be installed.

Regarding lintian warnings:  I think you should simply remove the
doc-base file.  There is no relevant documentation to register and this
template is not needed.  Probably also the debian/tests dir can be
removed.

Kind regards

Andreas.

Re: [RFS] survivor

[Steffen Möller]

;) Thank you for your praise, Nilesh. Should should possibly also
mention that I do not really like what I did and hope for an (official)
easier approach towards it.

@Andreas, leave this one for me to sponsor.

Best,

Steffen

On 13.09.20 14:06, Nilesh Patra wrote:
> Hi,
> I've packaged survivor - this is  a NEW package which builds
> in a clean chroot and is lintian clean with passing autoopkgtests.
> My changes are pushed to the team-repo here[1].
>
> SInce this needs to build in the Debug sub-directory instead of the
> main directory. I took help
> from Steffen's way of packaging hnswlib, thanks Steffen!
> Needs review and sponsorship.
>
> [1]: https://salsa.debian.org/med-team/survivor
> 
>
> Thanks and regards
> Nilesh

[RFS] python-biopython 1.78+dfsg-1

[Étienne Mollier]

Greetings,

I pulled Biopython 1.78 into Salsa, and tried to make sure the
continuous integration pipeline went to green.  The package
seems to be in almost good condition to reach Sid to me.  As
always, I am not against a rereading, I wouldn't be surprised to
learn I missed points while proceeding to the update.  :)

I have the following lintian warning, which I am not sure what
to do with.  I believe the purpose of the tag is to highligh
shared objects pointing outside /usr/lib initially, but in this
case, it is a DTD specification:

 W: breakout-link
usr/lib/python3/dist-packages/Bio/Entrez/DTDs/mathml2.dtd
-> usr/share/xml/schema/w3c/mathml/dtd/mathml2.dtd 

Looking into the change log, this link originates from #672407,
so I am rather tempted to lintian-override the warning, but I
would like to know your view on that matter.

Otherwise, the package is one `routine-update -f` away of being
ready for upload.

Kind Regards,
-- 
Étienne Mollier 
Old rsa/3072: 5ab1 4edf 63bb ccff 8b54  2fa9 59da 56fe fff3 882d
New rsa/4096: 8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
Sent from /dev/pts/3, please excuse my verbosity.


signature.asc
Description: PGP signature

Re: [RFS] survivor

[Steffen Möller]

Uploaded. Some smallish changes - ignore.

A nitty-gritty one - you have already placed a debian-tag, which now
does not point to the exact version that was uploaded to the archive.
Near miss, though.

Thank you for this package and your other efforts in the "Genome
Structure" tab.

Steffen

On 13.09.20 19:04, Steffen Möller wrote:
> ;) Thank you for your praise, Nilesh. Should should possibly also
> mention that I do not really like what I did and hope for an (official)
> easier approach towards it.
>
> @Andreas, leave this one for me to sponsor.
>
> Best,
>
> Steffen
>
> On 13.09.20 14:06, Nilesh Patra wrote:
>> Hi,
>> I've packaged survivor - this is  a NEW package which builds
>> in a clean chroot and is lintian clean with passing autoopkgtests.
>> My changes are pushed to the team-repo here[1].
>>
>> SInce this needs to build in the Debug sub-directory instead of the
>> main directory. I took help
>> from Steffen's way of packaging hnswlib, thanks Steffen!
>> Needs review and sponsorship.
>>
>> [1]: https://salsa.debian.org/med-team/survivor
>> 
>>
>> Thanks and regards
>> Nilesh

Re: Problem when trying to upgrade htsjdk

[Pierre Gruet]

Hi Andreas, hi everyone,

Le 11/09/2020 à 09:08, Andreas Tille a écrit :
> Hi,
> 
> I tried to upgrade htsjdk[1] but unfortunately I got some build time
> test errors.  It would be great if somebody could have a look.
>

I have looked and tried to build, which succeeded. Then I looked at the
log and noticed, to sum up, 4 points:

- many tests output to stdout or stderr because they are verbose but
they succeed : no problem with them;

- the test
htsjdk.samtools.SAMSequenceDictionaryTest.testMergeDictionaries[7]
outputs an error, but it's OK because it is expected to raise an Exception;

- the test
htsjdk.samtools.CRAMReferencelessTest.testReadCRAMWithEmbeddedReference
outputs several messages like

Ignoring SAM validation error: ERROR::INVALID_ALIGNMENT_START:Read name
20FUKAAXX100202:2:1:20271:61529, Mate Alignment start (748) must be
<= reference sequence length (200) on reference 20

I guess this is bad, but don't know what to do with it;

- the tests htsjdk.samtools.SAMFileWriterFactoryTest.testMakeWriter*
output lines like
Ignoring SAM validation error:
ERROR::HEADER_RECORD_MISSING_REQUIRED_TAG:File
/work/testMakeWriterPathbam, Error parsing SAM header. @RG line missing
SM tag. Line: @RG   ID:1

I don't know either what to do with it.

Does anyone have some experience with htsjdk to guide me? Inspecting the
inputs of the tests is hard because many files are binary.

> 
> Kind regards
> 
> Andreas.
> 
> 

Best regards,
Pierre

Re: makeblastdb 2.10.x on 32-bit architectures

[Aaron M. Ucko]

Étienne Mollier  writes:

> regression looks still there in version 2.10.0-3, at least on
> arm64 architecture.

How can I reproduce it?  AFAICT python-biopython's autopkgtest boils
down to the upstream test suite, which runs fine as part of a build of
revision 15084ca (immediately preceding your patch to drop back to
version 4 on selected architectures) on the porterbox amdahl.  Perhaps
system limits come into play in some other environments, but then I
expect they would have done the same on 2.10.0-1.  Also, the telltale
warning

  
/<>/c++/include/objtools/blast/seqdb_writer/writedb_lmdb.hpp:52:59:
 warning: integer overflow in expression of type ‘int’ results in ‘-647710720’ 
[-Woverflow]

occurs only in the logs for 2.10.0-2.

-- 
Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org)
http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?a...@monk.mit.edu

Re: Autopkgtest failure with new version of kaptive - may be related to new version of biopython

[Aaron M. Ucko]

Étienne Mollier  writes:

> My kernel traps a general protection fault in libblast.so during
> tblastn execution:

There's a new upstream release (2.10.1) that might conceivably help
here, but as noted in another thread, the update won't be entirely
trivial -- there's at least one x86ism I'll need to conditionalize
appropriately to avoid FTBFS errors on other architectures.

-- 
Aaron M. Ucko, KB1CJC (amu at alum.mit.edu, ucko at debian.org)
http://www.mit.edu/~amu/ | http://stuff.mit.edu/cgi/finger/?a...@monk.mit.edu

New Mail list debian-ai available

[Mo Zhou]

Hi folks,

Just to inform you that I've applied for a new mailing list named
debian-ai, focusing on AI and some related acceleration libraries:

  https://lists.debian.org/debian-ai/

Which is expected to be the main discussion channel for the deep learning
team (https://salsa.debian.org/deeplearning-team) and the ROCm team
(https://salsa.debian.org/rocm-team) since discussions are archived.

Apart from that, I think I'm going to use this mailing list as the
maintainer mail address for Debian Deep Learning Team (replacing the
alioth address of the Debian Science Team). That means you may want to
subscribe to this new list in order to receive information about the
ml/dl develpment of debian in the future.

Re: New Mail list debian-ai available

[Norbert Preining]

Hi Mo,

>   https://lists.debian.org/debian-ai/

Great to hear that it is accepted. I am already subscribed.

> Which is expected to be the main discussion channel for the deep learning
> team (https://salsa.debian.org/deeplearning-team) and the ROCm team
> (https://salsa.debian.org/rocm-team) since discussions are archived.

Fully agreed.

> Apart from that, I think I'm going to use this mailing list as the
> maintainer mail address for Debian Deep Learning Team (replacing the

And I think the same should be done for the rocm packages.

Thanks a lot

Norbert

--
PREINING Norbert  https://www.preining.info
Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev
GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13

Re: Autopkgtest failure with new version of kaptive - may be related to new version of biopython

[Andreas Tille]

Hi Étienne,

On Sun, Sep 13, 2020 at 12:58:41PM +0200, Étienne Mollier wrote:
> > But this seems to be kind of a random failure since unstable as well as
> > arm64 and ppc64el in testing are currently OK[3].
> 
> Thanks for the notice, I tried running the test in several
> architectures as well.  After running a few times autopkgtests,
> I see the intermittent error indeed, at least on amd64, and
> i386.  Other architectures I tested (arm64, riscv64) didn't show
> this issue so far; I'm virtualizing them, so maybe Qemu
> interferes.
> 
> My kernel traps a general protection fault in libblast.so during
> tblastn execution:
> 
>   $ sudo dmesg
>   [...]
>   [1741071.802876] traps: tblastn[3579372] general protection fault 
> ip:7ff551327dde sp:7ffdb8072790 error:0 in libblast.so[7ff55130f000+63000]
> 
> Looking up into straces, it seems the condition occurs when the
> program frees its memory before handing back the hand to the
> system, so maybe something in the taste of a double free while
> threads are not well aligned.  I'm not sure yet of the exact
> reproducer with the tblastn command, here is what I record in my
> `ps` output; some of the files referenced are temporary
> artifacts obtained from former steps of the script:
> 
>   /usr/bin/tblastn -db_gencode 11 -seg no -db 
> /tmp/autopkgtest.DOVsFP/autopkgtest_tmp/exact_match.fasta -query 
> /tmp/autopkgtest.DOVsFP/autopkgtest_tmp/output/kaptive_temp_3600033_520819/temp_gene_seqs.fasta
>  -num_threads 4 -outfmt 6 qseqid sseqid qstart qend sstart send evalue 
> bitscore length pident qlen qseq sseq

H, this all sounds complicated.  I explicitly put Aaron in To since
he is an expert in the blast code and might have some idea.  From my gut
feeling we seem to have face some issue inside either blast or biopython  
which became visible by chance when running the kaptive test suite.

> I'm also near finishing packaging Biopython 1.78, so that update
> might add a bit more entropy into these testings.

Thumbs up for working on Biopython 1.78 - lets see what side effect it
will have on the issue above.

Kind regards

Andreas.

-- 
http://fam-tille.de