Hi, On 09/24/2018 01:19 AM, Adam Borowski wrote: > On Sun, Sep 23, 2018 at 11:54:12PM +0200, Hans van Kranenburg wrote: >> Two examples have been added, which use the new code. I would appreciate >> extra testing. Please try them and see if the reported numbers make sense: >> >> space_calculator.py >> ------------------- >> Best to be initially described as a CLI version of the well-known >> webbased btrfs space calculator by Hugo. ;] Throw a few disk sizes at >> it, choose data and metadata profile and see how much space you would >> get to store actual data. >> >> See commit message "Add example to calculate usable and wasted space" >> for example output. >> >> show_usage.py >> ------------- >> The contents of the old show_usage.py example that simply showed a list >> of block groups are replaced with a detailed usage report of an existing >> filesystem. > > I wonder, perhaps at least some of the examples could be elevated to > commands meant to be run by end-user? Ie, installing them to /usr/bin/, > dropping the extension? They'd probably need less generic names, though.
Some of the examples are very useful, and I keep using them frequently. That's actually also the reason that I for now just have copied the examples/ to /usr/share/doc/python3-btrfs/examples for the Debian package, so that they're easily available on all systems that I work on. Currently the examples collection is serving a few purposes. It's my poor mans testing framework, which covers all functionality of the lib. It displays all the things that you can do. There's a rich git commit message history on them, which I plan to transform into documentation and tutorial stuff later. So, yes, a bunch of the things are quite useful actually. The new show_usage and space_calculator are examples of things that are possible which start to ascend the small thingies on debugging level. So what would be candidates to be promoted to 'official' utils? 0) Ah, btrfs-heatmap Yeah, that's the thing it all started with. I started writing all of the code to be able to debug why my filesystems were allocating raw disk space all the time and not reusing the free already allocated space. But, that one is already done. https://github.com/knorrie/btrfs-heatmap/ 1) Custom btrfs balance If really needed (and luckily, the need for it is mostly removed after solving the -o ssd issues) I always use balance_least_used.py instead of regular btrfs balance. I think it totally makes sense to do the analysis of what blockgroups to feed to balance in what order in user space. I also used another custom script to feed block groups with highly fragmented free space to balance to try repairing filesystems that had been using the cluster data extent allocator. That's not in examples, but when you combine show_free_space_fragmentation with parts of balance_least_used, you get the idea. The best example I can think of here is a program that uses the new usage information to find out how to feed block groups to balance to actually get a balanced filesystem with minimal amount of wasted raw space, and then do exactly that in the quickest way possible while providing interesting progress information, instead of just brute force rewriting all of the data and having no idea what's actually happening. 2) Advanced usage reporting Something like the new show_usage, but hey, when using python with some batteries included, I guess we can relatively easily do a nice html or pdf output with pie and bar charts which provide the user with information about the filesystem. Just having users run that when they're asking for help on IRC and share the result would be nice. :o) 3) The space calculator Yup, obviously. 4) Maybe show_orphan_cleaner_progress I use that one now and then to get a live view on mass-removal of subvolumes (backup snapshot expiry), but it's very close to a debug tool. Or maybe I'm already spoiled and used to it now, and I don't realize any more how frustrating it must be to see disk IO and cpu go all places and have no idea about what btrfs is doing. 5) So much more... So... the examples are just basic test coverage. There is so much more that can be done. And yes, to be able to write a small thingie that uses the lib, you already have to know a lot about btrfs. -> That's why I started writing the tutorial. And yes, when promoting things like the new show_usage example to programs that are easily available, users will probably start parsing the output of them with sed and awk which is a total abomination and the absolute opposite of the purpose of the library. So be it. Let it go. :D "The code never bothered me any way". The interesting question that remains is where the result should go. btrfs-heatmap is a thing of its own now, but it's a bit of the "show case" example using the lib, with its own collection of documentation and even possibility to script it again. Shipping the 'binaries' in the python3-btrfs package wouldn't be the right thing, so where should they go? apt-get install btrfs-moar-utils-yolo? Or should btrfs-progs start to use this to accelerate improvement for providing a richer collection of useful progs for things that are not on essential level (like, you won't need them inside initramfs, so they can use python)? -- Hans van Kranenburg
