babelia
I restarted working on my personal search engine. It used to be called culturia [0] with too many planned features. At some point, I called it asylum [1] and focused on personal knowledge base aspects and the last iteration was called gotofish [2] [0] https://framagit.org/a-guile-mind/culturia [1] https://framagit.org/a-guile-mind/culturia.next [2] https://git.sr.ht/~amz3/guile-gotofish I learned much from all this projects. In particular, I learned that it will be a long long long project, even if I focus only on "personal search engine" line of work. The last iteration, gotofish, was not too bad even if it has bitrot. Based on my research and practical experiment, it seems very clear that there is no workaround the use of map-reduce, that might be known as n-par-for-each [3]. [3] https://www.gnu.org/software/guile/manual/html_node/Parallel-Forms.html#index-n_002dpar_002dfor_002deach I made a prototype similar to that n-par-for-each, except it works with guile-fibers, is asynchronous and works with a shared pool of threads instead of spawning N threads for each incoming query like gotofish does. Related blog post: https://hyper.dev/blog/on-the-road-to-babelia.html If you want to help or discuss those matters, do not hesitate to reply to this message. Cheers, Amirouche ~ amz3 ~ https://hyper.dev
Re: babelia
Le sam. 16 nov. 2019 à 11:06, Amirouche Boubekki a écrit : > > I restarted working on my personal search engine. > > It used to be called culturia [0] with too many planned features. At > some point, I called it asylum [1] and focused on personal knowledge > base aspects and the last iteration was called gotofish [2] > > [0] https://framagit.org/a-guile-mind/culturia > [1] https://framagit.org/a-guile-mind/culturia.next > [2] https://git.sr.ht/~amz3/guile-gotofish > > I learned much from all this projects. In particular, I learned that > it will be a long long long project, even if I focus only on "personal > search engine" line of work. > > The last iteration, gotofish, was not too bad even if it has bitrot. > Based on my research and practical experiment, it seems very clear > that there is no workaround the use of map-reduce, that might be known > as n-par-for-each [3]. > > [3] > https://www.gnu.org/software/guile/manual/html_node/Parallel-Forms.html#index-n_002dpar_002dfor_002deach > > I made a prototype similar to that n-par-for-each, except it works > with guile-fibers, is asynchronous and works with a shared pool of > threads instead of spawning N threads for each incoming query like > gotofish does. > > Related blog post: https://hyper.dev/blog/on-the-road-to-babelia.html > > If you want to help or discuss those matters, do not hesitate to reply > to this message. I forgot to add that there is several big-ish tasks that can be tackled in parallel (see the above blog post). In particular, a parser for wet or warc files, see https://en.wikipedia.org/wiki/Web_ARChive. This is the most common format of the output of crawlers e.g. http://commoncrawl.org/
Re: babelia
Hi Amirouche, For the firefox driver you might get a good start from skewer-mode: https://github.com/skeeto/skewer-mode Best wishes, Arne > Related blog post: https://hyper.dev/blog/on-the-road-to-babelia.html -- Unpolitisch sein heißt politisch sein ohne es zu merken signature.asc Description: PGP signature
Re: State of lilypond with Guile 2?
Am Do., 14. Nov. 2019 um 22:15 Uhr schrieb Arne Babenhauserheide : > So with your patches Lilypond works with Guile 2.0.14. Well, the mentioned zip contains several patches from several people. Including some of your suggestions. ;) My own are sometimes more (rude) workarounds than fixes. > Is there a chance > that you could get them into a new guile-v2-work-rev2 branch or such? I hesitate to checkin patches reverting upstream work. Better to fix the underlying problem, ofcourse. In general I'd prefer to wait for official guile3 release. Any idea when it will happen? > And maybe post the commands you use to build and test it? Nothing surprising. >From lilypond-git-repo's master I checkout a local branch and throw-in all those patches. Then I follow our "Contributor's guide", i.e. sh autogen.sh --noconfigure mkdir -p build/ cd build/ ../configure --enable-guile2 make For the docs (if wished) make doc For testings make test-baseline and make check Ofcourse `configure --enable-guile2´ will search for the system-installed guile2-version. One could point to other guile and guile-config... Cheers, Harm > > Best wishes, > Arne > -- > Unpolitisch sein > heißt politisch sein > ohne es zu merken
