On Thu, 14 May 2020, Albretch Mueller wrote:
The thing is that I have to call, say sha256sum, on millions of files
Probably debian admin people dealing with packaging have to deal with
the same kinds of issues.
For checksums, mtree(8) from package mtree-netbsd might be worth a look.
+======================================================================+
mtree-netbsd - Validates directory tree against specification
The mtree utility compares a file hierarchy against a specification,
creates a specification for a file hierarchy, or modifies a
specification. This specification can be controlled by the user, but
typically includes file/directory/symlink names, ownership
information, permission bits, and so forth. It may optionally also
include various hashes, such as SHA-256 or MD5.
.
This mtree utility can understand its own files, as well as those
generated by the FreeBSD mtree (in Debian as fmtree in
freebsd-buildutils and freebsd-glue) and bsdtar/libarchive.
Homepage:
http://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc/pkgtools/mtree/README.html
+======================================================================+
Good luck with your project.
Also, on Fri, 15 May 2020, Albretch Mueller wrote:
On 5/14/20, Nicolas George <geo...@nsup.org> wrote:
The question was not how to find the files, the formulation of the
question indicates that Albretch has that covered.
Yeah, my problem is not finding the files per se. I have them or
could have them easily listed.
The thing is that when you work on copora research you have to get
fairly complicated answers from millions of text "as fast as possible"
and you have to make sure that your baseline hasn't been changed.
I will have to play (again) with the options that you have given me
and by the way I said sha256sum as an example in the typical case you
would run "file" and two hashes on each file and that would take
forever a user's machine.
lbrtchx
--
The day will come | Last words, August Spies (1855--1887).
When our silence will be | Hanged, by the U.S. state of Illinois,
More powerful than | alongside fellow journalists
The voices you strangle today | Adolf Fischer and Albert Parsons.
