Re: File search progress: database review and question on triggers

[Hartmut Goebel]

Am 13.08.20 um 12:04 schrieb Pierre Neidhardt:
> SQLite pattern search queries are extremely fast (<0.1s) and cover all
> examples named so far:
>
> - exact basename match
> - partial path match
> - pattern match (e.g. "/include/%foo%")


For comparison: These are the options Debian Package search
 supports:

  * paths ending with the keyword  -> e.g. file-extensions
  * packages that contain files named like this -> basename
  * packages that contain files whose names contain the keyword  ->
LIKE  %%

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: File search progress: database review and question on triggers

[Hartmut Goebel]

Am 11.08.20 um 14:35 schrieb Pierre Neidhardt:
> Unlike Nix, we would like to do more than just index executable files.
> Indeed, it's very useful to know where to find, say, a C header, a .so
> library, a TeXlive .sty file, etc.

+1

Most of the time I'm searching for non-executable files.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: merge wip-haskell?

[Ricardo Wurmus]

Ricardo Wurmus  writes:

> Timothy Sample  writes:
>
>> I just pushed “wip-haskell-updates-2” which integrates my work from
>> .  I left the original branch intact
>> to make it easy to compare.
>
> I rebased this on top of “master” and pushed it as “wip-haskell”.  I
> updated my changes to pandoc and pandoc-citeproc so that the statically
> linked variants are now separate packages.
>
> The evaluations on ci.guix.gnu.org failed due to a few minor bugs in the
> extra-directories code (missing “match” case for inputs with declared
> outputs, and a missing list wrapper in one package).  I fixed up that
> commit and now wait for ci.guix.gnu.org to start building things:
>
> https://ci.guix.gnu.org/jobset/wip-haskell

I just merged “wip-haskell” into “master” (after merging “master” into
“wip-haskell”).

-- 
Ricardo

Re: Linux-libre 5.8 and beyond

[Mark H Weaver]

Hi Alexandre,

Alexandre Oliva  wrote:
> On Aug 12, 2020, Mark H Weaver  wrote:
>
 It may be useful for users with newer hardware devices, which are
 not yet well supported by the latest stable release, to use an
 arbitrary commit from either Linus' mainline git repository or some
 other subsystem tree.
>>> 
>>> The cleaning up scripts are version-specific and won't work on an
>>> "arbitrary commit from Linus's mainline git repository" (i.e., someone
>>> wanting to get today's most recent commit going into 5.9.) The scripts
>>> would fall over and die in such a scenario,
>
>> Okay, perhaps this was wishful thinking on my part.
>
> Yup.  If you ran a deblob-check in verify mode on the resulting
> tarballs, you'd see how error-prone this is.  You'd at least stop
> non-Free code from silently sneaking in and finding its way into running
> on users' machines.  That's the *least* someone who runs the
> deblob-scripts on their own should do to smoke-test the result WRT
> *known* freedom issues.

What is this "verify mode" that you're referring to, and where is it
documented?  The word "verify" does not occur in either of the deblob
scripts that I know about, namely "deblob-" and "deblob-check".
The string "verif" occurs a few times, but nothing related to the script
functionality.  I don't see anything like a verification mode mentioned
in the options documented at the top of those two scripts.

For the record, it was not my intent to skip any automated checking
provided by these scripts.  If we're running the scripts in a suboptimal
way, please tell me a better way.

FYI, right now we're simply running the main 'deblob-' script
with no arguments in the unpacked Linux source directory, with the
corresponding 'deblob-check' script in $PATH and $PYTHON pointing to
python 2.x.  If 'deblob-' exits abnormally or with a non-zero
result, the Guix build process fails.

Last I checked, 'deblob-check' was certainly being run by
'deblob-' as a subprocess, because I had to make several
substitutions of hard-coded paths before it would work in Guix
(e.g. /bin/sed and /usr/bin/python).

>> I had hoped that the deblob scripts would typically mostly work, even
>> if they weren't able to do a comprehensive cleaning.
>
> I'd honestly hope for a much higher standard than that for a
> FSDG-compliant distro, especially one that carries the GNU mark.

As I wrote below:

>> I would oppose adding such a partly-cleaned kernel to Guix itself,

With this in mind, your accusation above is not relevant to Guix.

Above, I was talking about my hope to enable users, *on their own
machines* and using *their own private build recipes*, to make a
best-effort deblobbing of a non-standard kernel variant that they need
to use for whatever reason.  If they aren't provided with that option,
the obvious alternative (which I expect 99% of such users would do
anyway) is to simply run a fully-blobbed kernel instead.

> But you don't!  That's what you get when you jump the gun and use
> outdated cleaning up scripts, without waiting for us to verify,
> update and release them for a newer version.

Here you are conflating two substantially different scenarios:

(1) Attempting to use your deblob scripts on a newer kernel that almost
certainly includes many new drivers and blobs that aren't detected
by your scripts.  That's the case that I said I would oppose for
inclusion in Guix.

(2) Using the deblob scripts made for 5.4.57 on a 5.4.58 kernel in order
to apply security fixes more quickly, and where the probability of
uncleaned new blobs is quite low.

>> but I wanted to enable users who need to use some other branch of
>> Linux on their own systems to make a best-effort cleaning.
>
> Besides the likelihood of something going wrong, that seems like a
> backwards goal for a distro that is not expected to as much as point
> users at a non-Free package.

It's *not* a goal for Guix, and it wasn't even my motivation for
teaching Guix to run the Linux-libre deblob scripts.  It's just
something that, on a whim, I chose to include in my list of possible
advantages to having such functionality, nothing more.

> I'm sure that's not what you intend, but this arrangement, plus your
> mention of hurriedly getting releases out, adds up to an incentive to
> disable the deblobbing so as to get a faster build.

I don't understand how you reached this conclusion.  As far as I can
tell, changing Guix to run the deblob scripts made *no* difference to
what someone would have to do to ask Guix to build fully-blobbed kernel.

> I hope you'll agree that this is undesirable.

Agreed.

>> In my experience, the deblob scripts are very rarely changed after the
>> first few point releases of a stable release series.
>
> My personal experience tells me otherwise.  5.7 had only one update at
> .8; 5.6, at .6 and .16; 5.5, at .3, .11 and .19; 5.4, at .14, .18, .27,
> .34 and .44; 5.3, at .4 and .11; 5.2 at .1, .3 and .11; 5.1 at .2, .18
> and .20; 5.0 at

Re: File search progress: database review and question on triggers

[Arun Isaac]

Hi Pierre,

I tried the wip-filesearch branch. Nice work! :-)
persist-all-local-packages takes around 350 seconds on my machine (slow
machine with spinning disk) and the database is 50 MB. Some other
comments follow.

- Maybe, we shouldn't index hidden files, particularly all the .xxx-real
  files created by our wrap phases.

- You should use SQL prepared statements with sqlite-prepare,
  sqlite-bind, etc. That would correctly handle escaping special
  characters in the search string. Currently, searching for
  "transmission-gtk", "libm.so", etc. errors out.

- Searching for "git perl5" works as expected, but searching for "git
  perl" returns no results. I think this is due to the tokenizer used by
  the full text search indexer. The tokenizer sees the word "perl5" as
  one indivisible token and does not realize that "perl" is a prefix of
  "perl5". Unfortunately, I think this is a fundamental problem with FTS
  -- one that can only be fixed by using simple LIKE patterns. FTS is
  meant for natural language search where this kind of thing would be
  normal.

- I guess you are only indexing local packages now, but will include all
  packages later by some means.

Cheers!


signature.asc
Description: PGP signature

Re: File search progress: database review and question on triggers

[Bengt Richter]

Hi Hartmut, et al

On +2020-08-15 14:47:12 +0200, Hartmut Goebel wrote:
> Am 13.08.20 um 12:04 schrieb Pierre Neidhardt:
> > SQLite pattern search queries are extremely fast (<0.1s) and cover all
> > examples named so far:
> >
> > - exact basename match
> > - partial path match
> > - pattern match (e.g. "/include/%foo%")
> 
> 
> For comparison: These are the options Debian Package search
>  supports:
> 
>   * paths ending with the keyword  -> e.g. file-extensions
>   * packages that contain files named like this -> basename
>   * packages that contain files whose names contain the keyword  ->
> LIKE  %%
>

If you are on debian, have you tried
dpkg -l '*your*globbed*name*here*'
or
dpkg -l '*'|egrep your-regex-here
or the former, preserving header and excluding non-installed, e.g.,?
dpkg -l '*your*globbed*name*here*'|grep -v ^un

> -- 
> Regards
> Hartmut Goebel
> 
> | Hartmut Goebel  | h.goe...@crazy-compilers.com   |
> | www.crazy-compilers.com | compilers which you thought are impossible |
> 
> 
-- 
Regards,
Bengt Richter

Re: Linux-libre 5.8 and beyond

[Mark H Weaver]

Hi Alexandre,

I thought about it some more, and I've changed my mind on one point:
I've decided that for future kernel updates, in order to eliminate the
risk of unintentionally allowing blobs into Guix, I will either wait for
Linux-libre to publish updated deblob scripts, or else I will manually
check for new blobs.

 Thanks,
   Mark