Re: listing all modules compiled into a kernel instance
On 2011-03-01 2:13 PM, John Baldwin wrote: On 2011-03-01 5:00 AM, John Baldwin wrote: Maybe ucom doesn't appear because it doesn't have a DRIVER_MODULE() declaration (because it isn't a driver). Yes, that would explain it. I can explicitly include ucom in a kernel by adding "device ucom" in the configuration file, in which case it would call DRIVER_MODULE(), right? That would then make it appear in the "kldstat -v" list? So why is it a driver when it's done explicitly, but not a driver when done implicitly? That makes no sense to me since the functionality doesn't change. IMHO, this is a bug that needs to be fixed, not just for ucom but any implicitly included driver. No, the _source_ code of device ucom has to explicitly say "I am a module named 'foo'" using a DECLARE_MODULE() macro (or another macro such as DRIVER_MODULE() that invokes DECLARE_MODULE()). The 'device ucom' in a config file does not generate this, that is just an instruction that config(8) uses when looking in sys/conf/files to see which C source files to include in the kernel build. My wording was unclear. I do understand that it's the source code rather than the configuration file that invokes the macro. The argument I am making is that no matter how the ucom source code ends up being compiled into the kernel, the end result is that the ucom device is functionally present and available at run time. As such, it makes no sense to me that one can discover it's presence/availability with "kldstat -v" _only_ when compiled in as a consequence of a "device ucom" statement. As a user I care about accurate reporting when I query for information and currently "kldstat -v" cannot be relied upon. I shouldn't have to concern myself with what mechanism caused ucom to be included, but only that it was. Moreover, I suggest that for all practical purposes, a module is a module by virtue of its behaviour, not because DECLARE_MODULE() was invoked. Thus my assertion that this is a bug. Until it is fixed, please tell me how I can reliably query an existing kernel for a list of its functional modules/drivers. --Carl ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
vm.phys_free
Can someone explain exactly what phys_free is telling me, I understand the three pools (DEFAULT, DIRECT and CACHE) and 13 orders of pages. But what does lcnt represent? [root@sn12 ~]# sysctl vm.phys_free vm.phys_free: FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 | POOL 2 ---- -- -- -- -- -- -- 12 ( 16384K) | 12 | 0 | 6 11 ( 8192K) | 15 | 0 | 26 10 ( 4096K) | 39 | 0 | 38 9 ( 2048K) | 49 | 2 | 72 8 ( 1024K) | 29 | 1 | 137 7 ( 512K) | 41 | 1 | 231 6 ( 256K) | 74 | 0 | 332 5 ( 128K) | 127 | 1 | 448 4 (64K) | 149 | 4 | 725 3 (32K) | 278 | 13 | 980 2 (16K) | 388 | 36 |1290 1 ( 8K) | 213 | 200 |1831 0 ( 4K) | 0 | 627 |2865 Dr. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: listing all modules compiled into a kernel instance
On Thursday, March 03, 2011 3:03:02 am Carl wrote: > On 2011-03-01 2:13 PM, John Baldwin wrote: > >> On 2011-03-01 5:00 AM, John Baldwin wrote: > Maybe ucom doesn't appear because it doesn't have a DRIVER_MODULE() > declaration (because it isn't a driver). > >>> > >>> Yes, that would explain it. > >> > >> I can explicitly include ucom in a kernel by adding "device ucom" in the > >> configuration file, in which case it would call DRIVER_MODULE(), right? > >> That would then make it appear in the "kldstat -v" list? So why is it a > >> driver when it's done explicitly, but not a driver when done implicitly? > >> That makes no sense to me since the functionality doesn't change. IMHO, > >> this is a bug that needs to be fixed, not just for ucom but any > >> implicitly included driver. > > > > No, the _source_ code of device ucom has to explicitly say "I am a module > > named 'foo'" using a DECLARE_MODULE() macro (or another macro such as > > DRIVER_MODULE() that invokes DECLARE_MODULE()). The 'device ucom' in a > > config > > file does not generate this, that is just an instruction that config(8) uses > > when looking in sys/conf/files to see which C source files to include in the > > kernel build. > > My wording was unclear. I do understand that it's the source code rather > than the configuration file that invokes the macro. The argument I am > making is that no matter how the ucom source code ends up being compiled > into the kernel, the end result is that the ucom device is functionally > present and available at run time. As such, it makes no sense to me that > one can discover it's presence/availability with "kldstat -v" _only_ > when compiled in as a consequence of a "device ucom" statement. As a > user I care about accurate reporting when I query for information and > currently "kldstat -v" cannot be relied upon. I shouldn't have to > concern myself with what mechanism caused ucom to be included, but only > that it was. Moreover, I suggest that for all practical purposes, a > module is a module by virtue of its behaviour, not because > DECLARE_MODULE() was invoked. Thus my assertion that this is a bug. Ah, but your assertion is what is wrong. There is no 'apic' module for 'device apic' for example. Also, a single 'device foo' might enable multiple "modules" (e.g. if foo supports devices on both PCI and ISA buses, you will have foo/pci and foo/isa modules). A module != a kld. A kld file may contain zero or more modules. Most kld's include at least one module. > Until it is fixed, please tell me how I can reliably query an existing > kernel for a list of its functional modules/drivers. There are ways to query multiple things about the kernel, but they are more specific than a nebulous "module" concept: - kldstat lists the kld's currently loaded - kldstat -v lists the declared modules in all of the kld's - lsvfs lists the filesystems currently available - all new-bus device drivers end up in the kldstat -v output as 'driver/parent', but this does not work for devices that are actually support libraries shared by other drivers (e.g. ucom). ucom is a bit special as it isn't an actual driver, it's a library of routines shared by various USB serial drivers: u3g, uark, ubsa, uftdi, etc. Those are the "real" drivers that one would want to test for. By itself 'device ucom' doesn't buy you anything. 'device ucom' is probably dubious as if you put 'device u3g' in your kernel config, the kernel will automatically include the USB serial driver library routines. If you 'kldload u3g.ko' it will automatically load 'ucom.ko' as a dependency, so an explicit 'device ucom' is generally not needed. There is no 'device uether' for the common USB ethernet routines shared by all the USB ethernet drivers (though there is a uether.ko), and 'device ucom' should probably be removed. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Puzzled about VFS sysctl OIDs -- signed vs. unsigned
I'm using a little shell script to capture selected sysctl OID values periodically, in an attempt to get a better idea how the resources of a system are being used during a long-running (usually measured in hours), mission-critical workload. In the process of testing this, I've seen some of the VFS sysctl OIDs (in particular) report negative values ... when the description looks to me as if the OID in question is intended to be a monotonically increasing counter. For example: > sysctl -d vfs.getnewbufcalls vfs.getnewbufcalls: Number of calls to getnewbuf > sysctl vfs.getnewbufcalls vfs.getnewbufcalls: -348909432 Examining sys/kern/vfs_bio.c, the definition of vfs.getnewbufcalls appears to be: ... static int getnewbufcalls; SYSCTL_INT(_vfs, OID_AUTO, getnewbufcalls, CTLFLAG_RW, &getnewbufcalls, 0, "Number of calls to getnewbuf"); ... Many of the other OIDs defined nearby are also SYSCTL_INT (or SYSCTL_LONG), vs. SYSCTL_UINT (or SYSCTL_ULONG), and the corresponding variables are defined as static int (or static long) vs. static u_int (or static u_long). Is this both correct and reasonable? If so, how should I interpret such negative values? [GSoC project, anyone?] Thanks! Peace, david -- David H. Wolfskill da...@catwhisker.org Depriving a girl or boy of an opportunity for education is evil. See http://www.catwhisker.org/~david/publickey.gpg for my public key. pgpNidQUlZ8TR.pgp Description: PGP signature
Re: listing all modules compiled into a kernel instance
On 2011-03-03 7:08 AM, John Baldwin wrote: Ah, but your assertion is what is wrong. There is no 'apic' module for 'device apic' for example. Also, a single 'device foo' might enable multiple "modules" (e.g. if foo supports devices on both PCI and ISA buses, you will have foo/pci and foo/isa modules). A module != a kld. A kld file may contain zero or more modules. Most kld's include at least one module. Until it is fixed, please tell me how I can reliably query an existing kernel for a list of its functional modules/drivers. There are ways to query multiple things about the kernel, but they are more specific than a nebulous "module" concept: - kldstat lists the kld's currently loaded - kldstat -v lists the declared modules in all of the kld's - lsvfs lists the filesystems currently available - all new-bus device drivers end up in the kldstat -v output as 'driver/parent', but this does not work for devices that are actually support libraries shared by other drivers (e.g. ucom). ucom is a bit special as it isn't an actual driver, it's a library of routines shared by various USB serial drivers: u3g, uark, ubsa, uftdi, etc. Those are the "real" drivers that one would want to test for. By itself 'device ucom' doesn't buy you anything. 'device ucom' is probably dubious as if you put 'device u3g' in your kernel config, the kernel will automatically include the USB serial driver library routines. If you 'kldload u3g.ko' it will automatically load 'ucom.ko' as a dependency, so an explicit 'device ucom' is generally not needed. There is no 'device uether' for the common USB ethernet routines shared by all the USB ethernet drivers (though there is a uether.ko), and 'device ucom' should probably be removed. Thanks for the great explanation, John. What prompted this thread was me wanting to know which *.ko files corresponded to functionality already included in the kernel. And ucom became a point of focus, in part, because no less than 13 different man pages specify that "device ucom" is required in the configuration file despite the fact that the GENERIC kernel has no such statement and contains ucom. Since the man pages are therefore in error, I've already provided HPS with a patch that perhaps he will use to correct the man pages. --Carl ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Puzzled about VFS sysctl OIDs -- signed vs. unsigned
On Thu, Mar 3, 2011 at 11:49 AM, David Wolfskill wrote: > I'm using a little shell script to capture selected sysctl OID > values periodically, in an attempt to get a better idea how the > resources of a system are being used during a long-running (usually > measured in hours), mission-critical workload. > > In the process of testing this, I've seen some of the VFS sysctl > OIDs (in particular) report negative values ... when the description > looks to me as if the OID in question is intended to be a monotonically > increasing counter. > > For example: > >> sysctl -d vfs.getnewbufcalls > vfs.getnewbufcalls: Number of calls to getnewbuf >> sysctl vfs.getnewbufcalls > vfs.getnewbufcalls: -348909432 > > Examining sys/kern/vfs_bio.c, the definition of vfs.getnewbufcalls > appears to be: > > ... > static int getnewbufcalls; > SYSCTL_INT(_vfs, OID_AUTO, getnewbufcalls, CTLFLAG_RW, &getnewbufcalls, 0, > "Number of calls to getnewbuf"); > ... > > Many of the other OIDs defined nearby are also SYSCTL_INT (or > SYSCTL_LONG), vs. SYSCTL_UINT (or SYSCTL_ULONG), and the corresponding > variables are defined as static int (or static long) vs. static u_int > (or static u_long). > > Is this both correct and reasonable? If so, how should I interpret such > negative values? > > [GSoC project, anyone?] > > Thanks! > > Peace, > david The following initiative may factor heavily into any decision to change sysctl declarations at this point: http://www.freebsd.org/news/status/report-2010-10-2010-12.html#SYSCTL-Type-Safety I don't pretend to fully understand the reasons or impact of this project, but I have to imagine that at least the scope is similar to a proposed GSoC project (probably bigger). -Brandon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Puzzled about VFS sysctl OIDs -- signed vs. unsigned
On Thu, Mar 3, 2011 at 1:03 PM, Brandon Gooch wrote: > On Thu, Mar 3, 2011 at 11:49 AM, David Wolfskill wrote: >> I'm using a little shell script to capture selected sysctl OID >> values periodically, in an attempt to get a better idea how the >> resources of a system are being used during a long-running (usually >> measured in hours), mission-critical workload. >> >> In the process of testing this, I've seen some of the VFS sysctl >> OIDs (in particular) report negative values ... when the description >> looks to me as if the OID in question is intended to be a monotonically >> increasing counter. >> >> For example: >> >>> sysctl -d vfs.getnewbufcalls >> vfs.getnewbufcalls: Number of calls to getnewbuf >>> sysctl vfs.getnewbufcalls >> vfs.getnewbufcalls: -348909432 >> >> Examining sys/kern/vfs_bio.c, the definition of vfs.getnewbufcalls >> appears to be: >> >> ... >> static int getnewbufcalls; >> SYSCTL_INT(_vfs, OID_AUTO, getnewbufcalls, CTLFLAG_RW, &getnewbufcalls, 0, >> "Number of calls to getnewbuf"); >> ... >> >> Many of the other OIDs defined nearby are also SYSCTL_INT (or >> SYSCTL_LONG), vs. SYSCTL_UINT (or SYSCTL_ULONG), and the corresponding >> variables are defined as static int (or static long) vs. static u_int >> (or static u_long). >> >> Is this both correct and reasonable? If so, how should I interpret such >> negative values? >> >> [GSoC project, anyone?] >> >> Thanks! >> >> Peace, >> david > > The following initiative may factor heavily into any decision to > change sysctl declarations at this point: > > http://www.freebsd.org/news/status/report-2010-10-2010-12.html#SYSCTL-Type-Safety The intent of the type-safety is to make sure that the types assumed for the kernel's sysctl handler match the type of the variable. This project won't fix issues where a signed type is being used and the value wraps (at least, I presume that's what happened in this case?) Thanks, matthew ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Puzzled about VFS sysctl OIDs -- signed vs. unsigned
On Thu, Mar 3, 2011 at 3:34 PM, Matthew Fleming wrote: > On Thu, Mar 3, 2011 at 1:03 PM, Brandon Gooch > wrote: >> On Thu, Mar 3, 2011 at 11:49 AM, David Wolfskill >> wrote: >>> I'm using a little shell script to capture selected sysctl OID >>> values periodically, in an attempt to get a better idea how the >>> resources of a system are being used during a long-running (usually >>> measured in hours), mission-critical workload. >>> >>> In the process of testing this, I've seen some of the VFS sysctl >>> OIDs (in particular) report negative values ... when the description >>> looks to me as if the OID in question is intended to be a monotonically >>> increasing counter. >>> >>> For example: >>> sysctl -d vfs.getnewbufcalls >>> vfs.getnewbufcalls: Number of calls to getnewbuf sysctl vfs.getnewbufcalls >>> vfs.getnewbufcalls: -348909432 >>> >>> Examining sys/kern/vfs_bio.c, the definition of vfs.getnewbufcalls >>> appears to be: >>> >>> ... >>> static int getnewbufcalls; >>> SYSCTL_INT(_vfs, OID_AUTO, getnewbufcalls, CTLFLAG_RW, &getnewbufcalls, 0, >>> "Number of calls to getnewbuf"); >>> ... >>> >>> Many of the other OIDs defined nearby are also SYSCTL_INT (or >>> SYSCTL_LONG), vs. SYSCTL_UINT (or SYSCTL_ULONG), and the corresponding >>> variables are defined as static int (or static long) vs. static u_int >>> (or static u_long). >>> >>> Is this both correct and reasonable? If so, how should I interpret such >>> negative values? >>> >>> [GSoC project, anyone?] >>> >>> Thanks! >>> >>> Peace, >>> david >> >> The following initiative may factor heavily into any decision to >> change sysctl declarations at this point: >> >> http://www.freebsd.org/news/status/report-2010-10-2010-12.html#SYSCTL-Type-Safety > > The intent of the type-safety is to make sure that the types assumed > for the kernel's sysctl handler match the type of the variable. This > project won't fix issues where a signed type is being used and the > value wraps (at least, I presume that's what happened in this case?) > > Thanks, > matthew Yes, it's wrapping. I wonder, would an audit of the SYCTL_* types be of general use to FreeBSD? I haven't ran into these issues myself, but I can see where this could become more of a problem with the OS in general as larger, heavier loads are placed on general-purpose type systems; where FreeBSD has been used in a product, I assume that the companies engineers, such as those at Isilon, have applied local patches where necessary. Y'know, I think this could be a good GSoC project after all... -Brandon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Puzzled about VFS sysctl OIDs -- signed vs. unsigned
On Thu, Mar 3, 2011 at 2:03 PM, Brandon Gooch wrote: > On Thu, Mar 3, 2011 at 3:34 PM, Matthew Fleming wrote: >> On Thu, Mar 3, 2011 at 1:03 PM, Brandon Gooch >> wrote: >>> On Thu, Mar 3, 2011 at 11:49 AM, David Wolfskill >>> wrote: I'm using a little shell script to capture selected sysctl OID values periodically, in an attempt to get a better idea how the resources of a system are being used during a long-running (usually measured in hours), mission-critical workload. In the process of testing this, I've seen some of the VFS sysctl OIDs (in particular) report negative values ... when the description looks to me as if the OID in question is intended to be a monotonically increasing counter. For example: > sysctl -d vfs.getnewbufcalls vfs.getnewbufcalls: Number of calls to getnewbuf > sysctl vfs.getnewbufcalls vfs.getnewbufcalls: -348909432 Examining sys/kern/vfs_bio.c, the definition of vfs.getnewbufcalls appears to be: ... static int getnewbufcalls; SYSCTL_INT(_vfs, OID_AUTO, getnewbufcalls, CTLFLAG_RW, &getnewbufcalls, 0, "Number of calls to getnewbuf"); ... Many of the other OIDs defined nearby are also SYSCTL_INT (or SYSCTL_LONG), vs. SYSCTL_UINT (or SYSCTL_ULONG), and the corresponding variables are defined as static int (or static long) vs. static u_int (or static u_long). Is this both correct and reasonable? If so, how should I interpret such negative values? [GSoC project, anyone?] Thanks! Peace, david >>> >>> The following initiative may factor heavily into any decision to >>> change sysctl declarations at this point: >>> >>> http://www.freebsd.org/news/status/report-2010-10-2010-12.html#SYSCTL-Type-Safety >> >> The intent of the type-safety is to make sure that the types assumed >> for the kernel's sysctl handler match the type of the variable. This >> project won't fix issues where a signed type is being used and the >> value wraps (at least, I presume that's what happened in this case?) >> >> Thanks, >> matthew > > Yes, it's wrapping. I wonder, would an audit of the SYCTL_* types be > of general use to FreeBSD? I haven't ran into these issues myself, but > I can see where this could become more of a problem with the OS in > general as larger, heavier loads are placed on general-purpose type > systems; where FreeBSD has been used in a product, I assume that the > companies engineers, such as those at Isilon, have applied local > patches where necessary. Y'know, I think this could be a good GSoC > project after all... Yes and no. :-) The problem with changing the types is that it can be an ABI change. See the function sysctl_bufspace() in vfs_bio.c which stands on its head to output an int size if that's what the caller expects. This problem is mitigated (or made worse, depending on your POV) by a commit I plan to make when $WORK is slightly less insane, to add a new sysctl handler that will copy out 4 bytes if that's what the caller expects and there's no overflow, or even perhaps if there is, even if the kernel type is 8 bytes. The upside of this new handler is that, like bufspace, it preserves ABI for applications that thought they knew the size. The downside is that (1) there's lots of design options, like copying out 4 bytes even if this truncates the data versus reporting the error, and (2) such a change means that broken applications don't know they're broken, if the FreeBSD type changes. I hope this explains the issues. All that said, personally I'm in favor of having the kernel have the right types, and fixing broken apps as they're found. This isn't the most friendly stance for third-party vendors, though. Cheers, matthew ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
