Re: /usr/src and /usr/ports not git directories ?
On Thu, Jan 23, 2025 at 06:37:21AM +0100, Steffen Nurpmeso wrote: > > What i mean seems to be that these guys already have the knowledge > how to access the object database without any fancy and featureful > and fat environment, so if FreeBSD would do that then possibly > a simple from scratch checkout -- and nothing else -- would be > possible with only shell tools and a small binary that fetches the > blobs? Maybe even only possible with a very specific tree, you > know? As in a readily prepared recipe (could be compressed and > reside in the .git directory) that can be walked by the shell > script(s) and accessed by the super minimal blob fetcher? > The format is well documented: https://git-scm.com/book/en/v2/Git-Internals-Git-Objects You basically have three types of objects: commits, trees and blobs. Everything has a hash and is compressed with zlib. If you really wanted to, it is not too difficult to write something that does a checkout. But there are already enough git alternatives out there that offer the necessary functionality that have more suitable licenses. -- Best regards, Daniel
Re: HEADS UP: NFS changes coming into CURRENT early February
On Tue, Jan 21, 2025 at 10:27 PM Gleb Smirnoff wrote: > > CAUTION: This email originated from outside of the University of Guelph. Do > not click links or open attachments unless you recognize the sender and know > the content is safe. If in doubt, forward suspicious emails to > ith...@uoguelph.ca. > > > Hi, > > TLDR version: > users of NFS with Kerberos (e.g. running gssd(8)) as well as users of NFS with > TLS (e.g. running rpc.tlsclntd(8) or rpc.tlsservd(8)) as well as users of > network lock manager (e.g. having 'options NFSLOCKD' and running rpcbind(8)) > are affected. You would need to recompile & reinstall both the world and the > kernel together. Of course this is what you'd normally do when you track > FreeBSD CURRENT, but better be warned. I will post hashes of the specific > revisions that break API/ABI when they are pushed. > > Longer version: > last year I tried to check-in a new implementation of unix(4) SOCK_STREAM and > SOCK_SEQPACKET in d80a97def9a1, but was forced to back it out due to several > kernel side abusers of a unix(4) socket. The most difficult ones are the NFS > related RPC services, that act as RPC clients talking to an RPC servers in > userland. Since it is impossible to fully emulate a userland process > connection to a unix(4) socket they need to work with the socket internal > structures bypassing all the normal KPIs and conventions. Of course they > didn't tolerate the new implementation that totally eliminated intermediate > buffer on the sending side. > > While the original motivation for the upcoming changes is the fact that I want > to go forward with the new unix/stream and unix/seqpacket, I also tried to > make > kernel to userland RPC better. You judge if I succeeded or not :) Here are > some highlights: > > - Code footprint both in kernel clients and in userland daemons is reduced. > Example: gssd:1 file changed, 5 insertions(+), 64 deletions(-) >kgssapi: 1 file changed, 26 insertions(+), 78 deletions(-) > 4 files changed, 1 insertion(+), 11 deletions(-) > - You can easily see all RPC calls from kernel to userland with genl(1): > # genl monitor rpcnl > - The new transport is multithreaded in kernel by default, so kernel clients > can send a bunch of RPCs without any serialization and if the userland > figures out how to parallelize their execution, such parallelization would > happen. Note: new rpc.tlsservd(8) will use threads. > - One ad-hoc single program syscall is removed - gssd_syscall. Note: > rpctls syscall remains, but I have some ideas on how to improve that, too. > Not at this step though. > - All sleeps of kernel RPC calls are now in single place, and they all have > timeouts. I believe NFS services are now much more resilient to hangs. > A deadlock when NFS kernel thread is blocked on unix socket buffer, and > the socket can't go away because its application is blocked in some other > syscall is no longer possible. > > The code is posted on phabricator, reviews D48547 through D48552. > Reviewers are very welcome! > > I share my branch on Github. It is usually rebased on today's CURRENT: > > https://github.com/glebius/FreeBSD/commits/gss-netlink/ > > Early testers are very welcome! Unfortunately it looks like I won't be able to test this until after it is committed to main. Since there are library changes, etc, it appears that it will need a "make buildworld". On the laptop I currently have running it, this will take about a week, if it finishes. (I usually do "make buildworld"s on the universe machines, but since all I currently have is flakey wifi, I don't think that is practical either.) Once there is a snapshot of main that has it, I can download and test that. I will try and take a look at the stuff in phabricator, but given the size of it and my lack of knowledge w.r.t. netlink, I doubt I'll have much to say about it. Hopefully someone else can do some review/testing? rick > > -- > Gleb Smirnoff >
Re: [9fans] /usr/src and /usr/ports not git directories ?
Bakul Shah wrote on Date: Fri, 24 Jan 2025 00:15:04 UTC : > [-9fans, +freebsd-current as 9fans adds a reply-to: 9fans line] > > > On Jan 23, 2025, at 3:53 PM, Warner Losh wrote: > > > > I fail to see how putting code in the kernel is better than just using got > > for the few people that are alergic to git. Even if it is only 1000 lines > > in plan 9, but likely more in FreeBSD and by the way not yet ported to > > FreeBSD. We know got can't crash the system and is small enough to not > > matter, even if it isn't in the base today. > > > May I suggest: > - always ship the *commit hash* for any release or snapshot with its base.txz For stable/* and main builds via PkgBase: I'll note that, for PkgBase, the commit hashes for the "base" vs. the "kernel" need not be identical from what I've seen. In other words, the separate packages for /usr/src/sys/ vs. for the rest of /usr/src/ need not be based on the exact-same commit hash. Releases avoid that via the explicit control of when the releng/*.* content is updated: no update happens between and teh result happens to always match because of that (if I understand right). This seems to be something new for any official FreeBSD builds of stable/* and main . > - src.txz as now (or add commit hash) See above if a stable/* or main context might involve PkgBase build instead. > - this is enough to download a repo (1-deep or whatever), bare if src.txz was > also unpacked. > - add a simple script to download as above. > - people can install whatever git client they want for further work. > > git9 doesn't require any kernel code but on freebsd you'd have to > use plan9port. It is far simpler but has a different interface. === Mark Millard marklmi at yahoo.com
Re: [9fans] /usr/src and /usr/ports not git directories ?
> On Jan 23, 2025, at 7:30 PM, Mark Millard wrote: > > Bakul Shah wrote on > Date: Fri, 24 Jan 2025 00:15:04 UTC : > >> [-9fans, +freebsd-current as 9fans adds a reply-to: 9fans line] >> >>> On Jan 23, 2025, at 3:53 PM, Warner Losh wrote: >>> >>> I fail to see how putting code in the kernel is better than just using got >>> for the few people that are alergic to git. Even if it is only 1000 lines >>> in plan 9, but likely more in FreeBSD and by the way not yet ported to >>> FreeBSD. We know got can't crash the system and is small enough to not >>> matter, even if it isn't in the base today. >> >> >> May I suggest: >> - always ship the *commit hash* for any release or snapshot with its base.txz > > For stable/* and main builds via PkgBase: > > I'll note that, for PkgBase, the commit hashes for the "base" vs. the "kernel" > need not be identical from what I've seen. In other words, the separate > packages for /usr/src/sys/ vs. for the rest of /usr/src/ need not be > based on the exact-same commit hash. News to me. Since there is one repo for /usr/src and /usr/src/sys, this would mess things up. Why have more or less two identical repos. In the quick hack script below I assume one. This will need more work, this was just a PoC to show what I mean. For an existing local repo the script can be enhanced to fetch to a later commit hash (or one can just use git). #!/bin/sh cmd=`basename $0` case .$2 in .) echo Usage: $cmd url commit-hash; exit 1;; esac if [ -e .git ]; then echo $cmd: .git already exists; exit 1; fi # check if there are any other files if [ ! -z "$(ls -A)" ] ; then echo $cmd: directory must be empty exit 1 fi git init git remote add origin $1 git fetch --depth=1 origin $2 || exit 1 git checkout FETCH_HEAD > > Releases avoid that via the explicit control of when the releng/*.* content is > updated: no update happens between and teh result happens to always match > because of that (if I understand right). > > This seems to be something new for any official FreeBSD builds of stable/* and > main . > >> - src.txz as now (or add commit hash) > > See above if a stable/* or main context might involve PkgBase > build instead. > >> - this is enough to download a repo (1-deep or whatever), bare if src.txz >> was also unpacked. >> - add a simple script to download as above. >> - people can install whatever git client they want for further work. >> >> git9 doesn't require any kernel code but on freebsd you'd have to >> use plan9port. It is far simpler but has a different interface. > > === > Mark Millard > marklmi at yahoo.com >
Re: [9fans] /usr/src and /usr/ports not git directories ?
[-9fans, +freebsd-current as 9fans adds a reply-to: 9fans line] > On Jan 23, 2025, at 3:53 PM, Warner Losh wrote: > > I fail to see how putting code in the kernel is better than just using got > for the few people that are alergic to git. Even if it is only 1000 lines in > plan 9, but likely more in FreeBSD and by the way not yet ported to FreeBSD. > We know got can't crash the system and is small enough to not matter, even if > it isn't in the base today. May I suggest: - always ship the *commit hash* for any release or snapshot with its base.txz - src.txz as now (or add commit hash) - this is enough to download a repo (1-deep or whatever), bare if src.txz was also unpacked. - add a simple script to download as above. - people can install whatever git client they want for further work. git9 doesn't require any kernel code but on freebsd you'd have to use plan9port. It is far simpler but has a different interface.
Re: [9fans] /usr/src and /usr/ports not git directories ?
Hello. Jacob Moody wrote in : |On 1/22/25 23:37, Steffen Nurpmeso wrote: |> Tomoaki AOKI wrote in |> <20250123074932.170b7959bca443a934599...@dec.sakura.ne.jp>: |>|On Thu, 23 Jan 2025 07:40:50 +0900 |>|Tomoaki AOKI wrote: |>|> On Wed, 22 Jan 2025 14:31:56 -0700 |>|> Warner Losh wrote: |>|>> On Wed, Jan 22, 2025, 2:11 PM Tomek CEDRO wrote: |>|>>> On Wed, Jan 22, 2025 at 5:49 PM Gleb Smirnoff \ |>| On Wed, Jan 22, 2025 at 09:30:09AM -0700, Warner Losh wrote: |>| W> I think what you want is |>| W> |>| W> % git clone https://github.com/freebsd/freebsd-src --depth=1 -b |>|>>> releng/14.2 |>| W> --single-branch fred --bare ... |>| W> % git checkout releng/14.2 |> ... |>| I really like this plan. So before checkout .git is 337M and after |>|>>> checkout |>| it is 975M, which is made by actual sources + .git. |> ... |>|>>> One question, would that imply git in base? :-) |>|>> |>|>> Imply? No. ... |> The Plan9/9front people have -- btw -- created a Plan9-style |> interface to git, which is funny it seems (filesystem plus minimal That is, an implementation of their own that is capable to interact with normal git packs and objects (i think). |> programs using that, most are rc shell scripts. I track their |> repo via "normal" git (git://git.9front.org/plan9front/plan9front) |> without problems since June 2021 says the file system. ... |> What i mean seems to be that these guys already have the knowledge |> how to access the object database without any fancy and featureful |> and fat environment, so if FreeBSD would do that then possibly |> a simple from scratch checkout -- and nothing else -- would be |> possible with only shell tools and a small binary that fetches the |> blobs? Maybe even only possible with a very specific tree, you |> know? As in a readily prepared recipe (could be compressed and |> reside in the .git directory) that can be walked by the shell |> script(s) and accessed by the super minimal blob fetcher? | |Only getting bits of this conversation it's hard to infer exactly what |the goal is. Is the issue wanting to use git in a way that would require |it to be present within the base system, but encountering issues with doing |so due to licenses? The git present in 9front is MIT licensed, and \ |has already |been used in part for OpenBSD's game of trees (got). I imagine taking \ I think they tend to keep the base system more "clean" as in the past, making use of pkg and ports even for tools they effectively need, ie, all developers necessarily need git, but it is not part of the base system as cvs as been in the past, for example. License, they try to go BSD etc, and replace GPL etc software as BSD etc software can replace it. There is GPL still, for example diff(1). |whatever |pieces are required to bootstrap what you want from git9 would be sensible. |Git9 lacks some things (shallow clones for one), but is largely an entirely |usable interface to git, so I imagine it would have everything you \ |need if not |very close to what you need. That seems like a path forward, if people \ |don't |mind doing the port work. I myself was only thinking of really only a simplemost program that works like "git show" aka "git cat-file". Ie, they have mtree(8) or could otherwise create a snapshot of the checked out tree at a specific git revision, noting access rights etc (though this normally is not stored except for X bit in git i think), symlinks etc etc for one. And then walking this list of files and only fetching the file content from the (single, aggressively gc'd) git object .pack / .rev / .idx combo, cat(1)ing it into the fs, you know. So "nothing but a simple fetch-the-blob-data" thing. Like this no git(1) is needed at all to checkout a work tree from a repository with no checkout at all / that has checked out an all empty branch. |If you're looking for more technical details of how git9 works Ori \ |did a write up |for the IWP9 last year[0], which goes in to pretty good detail. | |[0] http://10e.iwp9.org/10iwp9proceedings.pdf Yes, i know; and thanks i have downloaded and read most of the stuff. I was only hoping for a quick statement on how complicated the envisioned program would really be, from a person who went down that road on its own. Or even, best, a "hey i have code which could be used for this with not much effort", you know.. --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) | |In Fall and Winter, feel "The Dropbear Bard"s pint(er). | |The banded bear |without a care, |Banged on himself for e'er and e'er | |Farewell, dear collar bear
Re: /usr/src and /usr/ports not git directories ?
Daniel Tameling wrote in : |On Thu, Jan 23, 2025 at 06:37:21AM +0100, Steffen Nurpmeso wrote: |> What i mean seems to be that these guys already have the knowledge |> how to access the object database without any fancy and featureful |> and fat environment, so if FreeBSD would do that then possibly |> a simple from scratch checkout -- and nothing else -- would be |> possible with only shell tools and a small binary that fetches the |> blobs? Maybe even only possible with a very specific tree, you |> know? As in a readily prepared recipe (could be compressed and |> reside in the .git directory) that can be walked by the shell |> script(s) and accessed by the super minimal blob fetcher? | |The format is well documented: https://git-scm.com/book/en/v2/Git-Intern\ |als-Git-Objects I usually look into the Documentation/ of the source tree, there it was in the past. |You basically have three types of objects: commits, trees and blobs. |Everything has a hash and is compressed with zlib. If you really |wanted to, it is not too difficult to write something that does a |checkout. But there are already enough git alternatives out there that |offer the necessary functionality that have more suitable licenses. The complications usually arise as you go is my experience. And someone would have to go to make this happen. Why not ask the Plan9 aka 9front people and dance with Glenda? --steffen | |Der Kragenbaer,The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) | |In Fall and Winter, feel "The Dropbear Bard"s pint(er). | |The banded bear |without a care, |Banged on himself for e'er and e'er | |Farewell, dear collar bear
