Re: [gentoo-user] asciidoc Fetched file: asciidoc-9.0.5.tar.gz VERIFY FAILED!
On Sun, 28 Feb 2021 22:29:49 -0600, Dale wrote: > I'm not sure what is going on with the servers but when I switched to > one of them, recent updates didn't even exist. I just did a KDE and I > think Firefox and neither seemed to be in the tree from those servers. > Thing is, I didn't note which server it was. If someone notices that > their syncing isn't working right, may want to switch and find one that > is up to date. Alternatively, switch to syncing from github and you'll always be as up to date as possible - and it's much faster. % cat /etc/portage/repos.conf/gentoo.conf [DEFAULT] main-repo = gentoo [gentoo] priority = 20 location = /var/portage sync-type = git sync-uri = https://github.com/gentoo-mirror/gentoo auto-sync = yes -- Neil Bothwick Windows Error #09: Game Over. Exiting Windows. pgpx0PpamTJLT.pgp Description: OpenPGP digital signature
[gentoo-user] [OT] Saving an image as black and white
I've got a bunch of scans, let's assume they're text documents. And they're rather big ... I want to email them. How on earth do I convert them to TRUE b&w documents? At the moment they are jpegs that weigh in at 3MB, and I guess they're using about 5 bytes to store all the colour, luminance, whatever, per pixel. But actually, there's only ONE BIT of information there - whether that pixel is black or white. I'm using imagemagick, but so far all my attempts to strip out the surplus information have resulted in INcreasing the file size ??? So basically, how do I save an image as "one bit per pixel" like you'd think you'd send to a B&W printer? Even at 300dpi, I make that 300*300/8 ~= 10KB/in^2 or 800KB of uncompressed info for a page of A4, not 3MB. Cheers, Wol
Re: [gentoo-user] [OT] Saving an image as black and white
On March 1, 2021 12:50:35 PM GMT+01:00, Wols Lists wrote: >I've got a bunch of scans, let's assume they're text documents. And >they're rather big ... I want to email them. > >How on earth do I convert them to TRUE b&w documents? At the moment they >are jpegs that weigh in at 3MB, and I guess they're using about 5 bytes >to store all the colour, luminance, whatever, per pixel. But actually, >there's only ONE BIT of information there - whether that pixel is black >or white. > >I'm using imagemagick, but so far all my attempts to strip out the >surplus information have resulted in INcreasing the file size ??? > >So basically, how do I save an image as "one bit per pixel" like you'd >think you'd send to a B&W printer? > >Even at 300dpi, I make that 300*300/8 ~= 10KB/in^2 or 800KB of >uncompressed info for a page of A4, not 3MB. > >Cheers, >Wol > Have you tried an optical character recognition software like Tesseract[1]? 1. https://github.com/tesseract-ocr/tesseract -- Hund
Re: [gentoo-user] Re: [OT] Saving an image as black and white
On 01/03/21 12:11, (Nuno Silva) wrote: > On 2021-03-01, Wols Lists wrote: > >> I've got a bunch of scans, let's assume they're text documents. And >> they're rather big ... I want to email them. >> >> How on earth do I convert them to TRUE b&w documents? At the moment they >> are jpegs that weigh in at 3MB, and I guess they're using about 5 bytes >> to store all the colour, luminance, whatever, per pixel. But actually, >> there's only ONE BIT of information there - whether that pixel is black >> or white. >> >> I'm using imagemagick, but so far all my attempts to strip out the >> surplus information have resulted in INcreasing the file size ??? >> >> So basically, how do I save an image as "one bit per pixel" like you'd >> think you'd send to a B&W printer? >> >> Even at 300dpi, I make that 300*300/8 ~= 10KB/in^2 or 800KB of >> uncompressed info for a page of A4, not 3MB. >> >> Cheers, >> Wol > > Somebody else might have a better suggestion, or perhaps a better > understanding of the JPEG format and of what needs to be tuned, but, for > example: > > convert origin.jpg -threshold 70% -monochrome result.jpg > > (And adjust the "-threshold percent" if needed. It might be that you > don't need thresholding at all, but if you do, it apparently must go > before "-monochrome".) > > (Depending on the receiving end, you could also explore other > formats. Here, if the scanned document can be stored in monochrome, I > usually use djvu.) > Thanks but no, I've already tried that. It makes matters worse! I've messed about with the scanner, so it is now creating 800KB images, but I don't want to rescan everything I've done. The problem is that it is clearly saving the images as greyscale, not as black&white. And when I search for help, what I want is swamped by all the false positives for greyscale. Oh - and for Nuno - sorry tesseract is no use, they are NOT text. That's why I used the word "assume" - to make it clear that I want a 1-bit/pixel palette, not a 5-byte/pixel greyscale. Cheers, Wol
Re: [gentoo-user] Re: [OT] Saving an image as black and white
save/convert to pdf - use gs from ghostscrpit to convert them (I use ebook for the target) which gives 10-20x reduction in size with only a small reduction in quality - perfect for emailing. I dont have the actual command string but I originally found the suggestion via google. BillK On 1/3/21 9:17 pm, Wols Lists wrote: > On 01/03/21 12:11, (Nuno Silva) wrote: >> On 2021-03-01, Wols Lists wrote: >> >>> I've got a bunch of scans, let's assume they're text documents. And >>> they're rather big ... I want to email them. >>> >>> How on earth do I convert them to TRUE b&w documents? At the moment they >>> are jpegs that weigh in at 3MB, and I guess they're using about 5 bytes >>> to store all the colour, luminance, whatever, per pixel. But actually, >>> there's only ONE BIT of information there - whether that pixel is black >>> or white. >>> >>> I'm using imagemagick, but so far all my attempts to strip out the >>> surplus information have resulted in INcreasing the file size ??? >>> >>> So basically, how do I save an image as "one bit per pixel" like you'd >>> think you'd send to a B&W printer? >>> >>> Even at 300dpi, I make that 300*300/8 ~= 10KB/in^2 or 800KB of >>> uncompressed info for a page of A4, not 3MB. >>> >>> Cheers, >>> Wol >> Somebody else might have a better suggestion, or perhaps a better >> understanding of the JPEG format and of what needs to be tuned, but, for >> example: >> >> convert origin.jpg -threshold 70% -monochrome result.jpg >> >> (And adjust the "-threshold percent" if needed. It might be that you >> don't need thresholding at all, but if you do, it apparently must go >> before "-monochrome".) >> >> (Depending on the receiving end, you could also explore other >> formats. Here, if the scanned document can be stored in monochrome, I >> usually use djvu.) >> > Thanks but no, I've already tried that. It makes matters worse! > > I've messed about with the scanner, so it is now creating 800KB images, > but I don't want to rescan everything I've done. > > The problem is that it is clearly saving the images as greyscale, not as > black&white. And when I search for help, what I want is swamped by all > the false positives for greyscale. > > Oh - and for Nuno - sorry tesseract is no use, they are NOT text. That's > why I used the word "assume" - to make it clear that I want a > 1-bit/pixel palette, not a 5-byte/pixel greyscale. > > Cheers, > Wol >
Re: [gentoo-user] [OT] Saving an image as black and white
On Mon, 1 Mar 2021 11:50:35 +, Wols Lists wrote: > I've got a bunch of scans, let's assume they're text documents. And > they're rather big ... I want to email them. > > How on earth do I convert them to TRUE b&w documents? At the moment they > are jpegs that weigh in at 3MB, and I guess they're using about 5 bytes > to store all the colour, luminance, whatever, per pixel. But actually, > there's only ONE BIT of information there - whether that pixel is black > or white. > > I'm using imagemagick, but so far all my attempts to strip out the > surplus information have resulted in INcreasing the file size ??? > > So basically, how do I save an image as "one bit per pixel" like you'd > think you'd send to a B&W printer? $ convert input.jpg -threshold 50% output.png should do it, you may need to play with the threshold setting. The file command reports the output file as being "1-bit grayscale". You can also use -monochrome but that will produce a dithered image, that's probably not what you want judging by your description. -- Neil Bothwick If we aren't supposed to eat animals, why are they made of meat? pgp3M42KfH1ou.pgp Description: OpenPGP digital signature
Re: [gentoo-user] [OT] Saving an image as black and white
On Mon, Mar 1, 2021 at 8:48 AM Neil Bothwick wrote: > > should do it, you may need to play with the threshold setting. The file > command reports the output file as being "1-bit grayscale". > > You can also use -monochrome but that will produce a dithered image, > that's probably not what you want judging by your description. Keep in mind that your starting image might not be 1-bit. You might be scanning in greyscale, which is probably 8-bit. Nothing wrong with converting to 1-bit, but in that case you would be throwing away detail. If you plan to do any processing of the file you might want to do that before throwing out the detail. You also may or may not want the threshold to be 50%. Also, as some are starting to hit on, jpeg may or may not be an ideal format depending on what you're scanning. It was designed for photographs, and it doesn't really cope well with sharp edges unless you use very high quality levels. I don't want to offer too much advice beyond that as I don't really deal with document scanning at any kind of scale where I get concerned with this stuff - defaults are almost always fine for me. I'm sure the right format and process would depend a bit on what you intend to do with the files. -- Rich
Re: [gentoo-user] [OT] Saving an image as black and white
On 01/03/21 13:48, Neil Bothwick wrote: > On Mon, 1 Mar 2021 11:50:35 +, Wols Lists wrote: > >> I've got a bunch of scans, let's assume they're text documents. And >> they're rather big ... I want to email them. >> >> How on earth do I convert them to TRUE b&w documents? At the moment they >> are jpegs that weigh in at 3MB, and I guess they're using about 5 bytes >> to store all the colour, luminance, whatever, per pixel. But actually, >> there's only ONE BIT of information there - whether that pixel is black >> or white. >> >> I'm using imagemagick, but so far all my attempts to strip out the >> surplus information have resulted in INcreasing the file size ??? >> >> So basically, how do I save an image as "one bit per pixel" like you'd >> think you'd send to a B&W printer? > > $ convert input.jpg -threshold 50% output.png > > should do it, you may need to play with the threshold setting. The file > command reports the output file as being "1-bit grayscale". > > You can also use -monochrome but that will produce a dithered image, > that's probably not what you want judging by your description. > > FINALLY! Thanks, that worked! Okay, I also adjusted the dpi because the original scan was 600 and I've reduced it to 300, but this has reduced the file size from 3MB to 180KB. Dunno why, but everything I was trying was INcreasing the file size :-( And the png does make a massive difference - the same command with jpg output is 1.7MB - so why is my scanner chucking out 800KB jpegs if I set it correctly? Cheers, Wol
Re: [gentoo-user] asciidoc Fetched file: asciidoc-9.0.5.tar.gz VERIFY FAILED!
On 3/1/21 3:43 AM, Neil Bothwick wrote: On Sun, 28 Feb 2021 22:29:49 -0600, Dale wrote: I'm not sure what is going on with the servers but when I switched to one of them, recent updates didn't even exist. I just did a KDE and I think Firefox and neither seemed to be in the tree from those servers. Thing is, I didn't note which server it was. If someone notices that their syncing isn't working right, may want to switch and find one that is up to date. Alternatively, switch to syncing from github and you'll always be as up to date as possible - and it's much faster. % cat /etc/portage/repos.conf/gentoo.conf [DEFAULT] main-repo = gentoo [gentoo] priority = 20 location = /var/portage sync-type = git sync-uri = https://github.com/gentoo-mirror/gentoo auto-sync = yes Syncing won't help until the ebuilds are fixed, and the bug does not say that has happened. The problem is not (yet) syncing your portage tree, it is with the mirrors, some of which have the old tarballs and some of which have the new tarballs. If the ebuild gets updated, and you pull one of the old tarballs, you will have essentially the same errors trying to emerge. You need an ebuild and Manifest that match the tarball you are trying to use.
Re: [gentoo-user] asciidoc Fetched file: asciidoc-9.0.5.tar.gz VERIFY FAILED!
On Mon, 1 Mar 2021 10:45:32 -0500, Jack wrote: > > Alternatively, switch to syncing from github and you'll always be as > > up to date as possible - and it's much faster. > Syncing won't help until the ebuilds are fixed, and the bug does not > say that has happened. The problem is not (yet) syncing your portage > tree, it is with the mirrors, some of which have the old tarballs and > some of which have the new tarballs. So it's the source mirrors that are out of date rather than the rsync mirrors? > If the ebuild gets updated, and > you pull one of the old tarballs, you will have essentially the same > errors trying to emerge. You need an ebuild and Manifest that match > the tarball you are trying to use. This sort of issue should only last an hour or so, after which time everything should be in sync. if it is still there the next day, something is wrong with the mirror you are using. -- Neil Bothwick Top Oxymorons Number 11: Terribly pleased pgpEQT2JVK9fM.pgp Description: OpenPGP digital signature
[gentoo-user] Re: [OT] Saving an image as black and white
On 2021-03-01, Wols Lists wrote: > I've got a bunch of scans, let's assume they're text documents. And > they're rather big ... I want to email them. > > How on earth do I convert them to TRUE b&w documents? At the moment they > are jpegs that weigh in at 3MB, and I guess they're using about 5 bytes > to store all the colour, luminance, whatever, per pixel. But actually, > there's only ONE BIT of information there - whether that pixel is black > or white. > > I'm using imagemagick, but so far all my attempts to strip out the > surplus information have resulted in INcreasing the file size ??? > > So basically, how do I save an image as "one bit per pixel" like you'd > think you'd send to a B&W printer? > > Even at 300dpi, I make that 300*300/8 ~= 10KB/in^2 or 800KB of > uncompressed info for a page of A4, not 3MB. > > Cheers, > Wol Somebody else might have a better suggestion, or perhaps a better understanding of the JPEG format and of what needs to be tuned, but, for example: convert origin.jpg -threshold 70% -monochrome result.jpg (And adjust the "-threshold percent" if needed. It might be that you don't need thresholding at all, but if you do, it apparently must go before "-monochrome".) (Depending on the receiving end, you could also explore other formats. Here, if the scanned document can be stored in monochrome, I usually use djvu.) -- Nuno Silva
Re: [gentoo-user] asciidoc Fetched file: asciidoc-9.0.5.tar.gz VERIFY FAILED!
On Mon, Mar 1, 2021 at 11:03 AM Neil Bothwick wrote: > > On Mon, 1 Mar 2021 10:45:32 -0500, Jack wrote: > > > > Alternatively, switch to syncing from github and you'll always be as > > > up to date as possible - and it's much faster. > > > Syncing won't help until the ebuilds are fixed, and the bug does not > > say that has happened. The problem is not (yet) syncing your portage > > tree, it is with the mirrors, some of which have the old tarballs and > > some of which have the new tarballs. > > So it's the source mirrors that are out of date rather than the > rsync mirrors? > Not exactly. There are three files here: 1. The ebuild/mainfest - which uses hash/size of an old version of the source. 2. The distfile mirrors - which contain the corresponding old version of the source. 3. The upstream SRC_URI - which contains a newer version of the source. Nothing has been fixed as of the time of sending this email, so you can sync your repo all you want and it will do no good. If you use a Gentoo distfile mirror you should be fine, because you're fetching an old version of the source, which matches what the ebuild is expecting. However, if you don't use a Gentoo distfile mirror then you're directly fetching the upstream source, which DOESN'T match the ebuild, and so you'll get an error. The Gentoo mirrors are going to reject any attempts to fetch the newer source, because they will only mirror a file if they match the Gentoo manifest. Usually these bugs do get fixed quickly, but there were changes in the upstream source files, so the maintainer probably wants to do some testing before updating the manifests. That is a GOOD thing and that is exactly why we have those manifests in the first place. They protect you, the user, in case upstream goes and changes a file out for any reason - Portage will reject this as this is not the file that was used by the package maintainer when doing all their QA activities. The biggest risk is malware getting snuck into an upstream file, but in this case it sounds like upstream changed something without versioning it. At the very least the maintainer is going to want to make sure it still works - you can't necessarily expect the same ebuild to work if upstream changes the sources. -- Rich
[gentoo-user] Re: [OT] Saving an image as black and white
On 2021-03-01, Wols Lists wrote: > On 01/03/21 12:11, (Nuno Silva) wrote: >> On 2021-03-01, Wols Lists wrote: >> >>> I've got a bunch of scans, let's assume they're text documents. And >>> they're rather big ... I want to email them. >>> >>> How on earth do I convert them to TRUE b&w documents? At the moment they >>> are jpegs that weigh in at 3MB, and I guess they're using about 5 bytes >>> to store all the colour, luminance, whatever, per pixel. But actually, >>> there's only ONE BIT of information there - whether that pixel is black >>> or white. >>> >>> I'm using imagemagick, but so far all my attempts to strip out the >>> surplus information have resulted in INcreasing the file size ??? >>> >>> So basically, how do I save an image as "one bit per pixel" like you'd >>> think you'd send to a B&W printer? >>> >>> Even at 300dpi, I make that 300*300/8 ~= 10KB/in^2 or 800KB of >>> uncompressed info for a page of A4, not 3MB. >>> >>> Cheers, >>> Wol >> >> Somebody else might have a better suggestion, or perhaps a better >> understanding of the JPEG format and of what needs to be tuned, but, for >> example: >> >> convert origin.jpg -threshold 70% -monochrome result.jpg >> >> (And adjust the "-threshold percent" if needed. It might be that you >> don't need thresholding at all, but if you do, it apparently must go >> before "-monochrome".) >> >> (Depending on the receiving end, you could also explore other >> formats. Here, if the scanned document can be stored in monochrome, I >> usually use djvu.) >> > Thanks but no, I've already tried that. It makes matters worse! > > I've messed about with the scanner, so it is now creating 800KB images, > but I don't want to rescan everything I've done. > > The problem is that it is clearly saving the images as greyscale, not as > black&white. And when I search for help, what I want is swamped by all > the false positives for greyscale. > > Oh - and for Nuno - sorry tesseract is no use, they are NOT text. That's > why I used the word "assume" - to make it clear that I want a > 1-bit/pixel palette, not a 5-byte/pixel greyscale. > > Cheers, > Wol Sorry, my bad - I was checking the file sizes, but I didn't notice the larger one was the new, "monochrome" version. More coffee needed, it seems. -- Nuno Silva
Re: [gentoo-user] [OT] Saving an image as black and white
On Mon, Mar 1, 2021 at 10:54 AM Wols Lists wrote: > > And the png does make a massive difference - the same command with jpg > output is 1.7MB - so why is my scanner chucking out 800KB jpegs if I set > it correctly? jpeg quality is adjustable. You can output a jpeg file of almost any size. Software less geared towards image editing may not actually let you set the quality level, but the software IS using one. So, two programs could output the same file at different sizes. The smaller you make the file, the lower the quality. This does have diminishing returns - as you approach maximum quality you increase the size greatly with very little difference in visual quality. Of course, if you try to convert that 1.7MB jpeg into a 30kb jpeg, you'll probably notice the difference. This is why this is a meme: http://needsmorejpeg.com/ -- Rich
[gentoo-user] zfs repair needed (due to fingers being faster than brain)
HI, Gentooers! So, I typed dd if=/dev/zero of=/dev/sd, and despite hitting ctrl-c quite quickly, zeroed out some portion of the initial part of a disk. Which did this to my zfs raidz3 array: NAME STATE READ WRITE CKSUM zfs DEGRADED 0 0 0 raidz3-0 DEGRADED 0 0 0 ata-HGST_HUS724030ALE640_PK1234P8JJJVKP ONLINE 0 0 0 ata-HGST_HUS724030ALE640_PK1234P8JJP3AP ONLINE 0 0 0 ata-ST4000NM0033-9ZM170_Z1Z80P4C ONLINE 0 0 0 ata-ST4000NM0033-9ZM170_Z1ZAZ8F1 ONLINE 0 0 0 14296253848142792483 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-ST4000NM0033-9ZM170_Z1ZAZDJ0-part1 ata-ST4000NM0033-9ZM170_Z1Z80KG0 ONLINE 0 0 0 Could have been worse. I do have backups, and it is raid3, so all I've injured is my pride, but I do want to fix things.I'd appreciate some guidance before I attempt doing this - I have no experience at it myself. The steps I envision are 1) zpool offline zfs 14296253848142792483 (What's that number?) 2) do something to repair the damaged disk 3) zpool online zfs Right now, the device name for the damaged disk is /dev/sda. Gdisk says this about it: Caution: invalid main GPT header, but valid backup; regenerating main header from backup! Warning: Invalid CRC on main header data; loaded backup partition table. Warning! Main and backup partition tables differ! Use the 'c' and 'e' options on the recovery & transformation menu to examine the two tables. Warning! Main partition table CRC mismatch! Loaded backup partition table instead of main partition table! Warning! One or more CRCs don't match. You should repair the disk! Main header: ERROR Backup header: OK Main partition table: ERROR Backup partition table: OK Partition table scan: MBR: not present BSD: not present APM: not present GPT: damaged Found invalid MBR and corrupt GPT. What do you want to do? (Using the GPT MAY permit recovery of GPT data.) 1 - Use current GPT 2 - Create blank GPT Your answer: ( I haven't given one yet) I'm not exactly sure what this is telling me. But I'm guessing it means that the main partition table is gone, but there's a good backup. In addition, some, but not all disk id info is gone: 1) /dev/disk/by-id still shows ata-ST4000NM0033-9ZM170_Z1ZAZDJ0 (the damaged disk) but none of its former partitions 2) /dev/disk/by-partlabel shows entries for the undamaged disks in the pool, but not the damaged one 3) /dev/disk/by-partuuid similar to /dev/disk/by-partlabel 4) /dev/disk/by-uuid does not show the damaged disk This particular disk is from a batch of 4 I bought with the same make and specification and very similar ids (/dev/disk/by-id). Can I repair this disk by copying something off one of those other disks onto this one? Is repair just repartitioning - as in the Gentoo handbook? Is it as simple as running gdisk and typing 1 to accept gdisk's attempt at recovering the gpt? Is running gdisk's recovery and transformation facilities the way to go (the b option looks like it's made for exactly this situation)? Anybody experienced at this and willing to guide me? Thanks, John Blinka
Re: [gentoo-user] zfs repair needed (due to fingers being faster than brain)
Firstly, I'll say I'm not experienced, but knowing a fair bit about raid and recovering corrupted arrays ... On 01/03/2021 22:25, John Blinka wrote: HI, Gentooers! So, I typed dd if=/dev/zero of=/dev/sd, and despite hitting ctrl-c quite quickly, zeroed out some portion of the initial part of a disk. Which did this to my zfs raidz3 array: NAME STATE READ WRITE CKSUM zfs DEGRADED 0 0 0 raidz3-0 DEGRADED 0 0 0 ata-HGST_HUS724030ALE640_PK1234P8JJJVKP ONLINE 0 0 0 ata-HGST_HUS724030ALE640_PK1234P8JJP3AP ONLINE 0 0 0 ata-ST4000NM0033-9ZM170_Z1Z80P4C ONLINE 0 0 0 ata-ST4000NM0033-9ZM170_Z1ZAZ8F1 ONLINE 0 0 0 14296253848142792483 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-ST4000NM0033-9ZM170_Z1ZAZDJ0-part1 ata-ST4000NM0033-9ZM170_Z1Z80KG0 ONLINE 0 0 0 Could have been worse. I do have backups, and it is raid3, so all I've injured is my pride, but I do want to fix things.I'd appreciate some guidance before I attempt doing this - I have no experience at it myself. The steps I envision are 1) zpool offline zfs 14296253848142792483 (What's that number?) 2) do something to repair the damaged disk 3) zpool online zfs Right now, the device name for the damaged disk is /dev/sda. Gdisk says this about it: Caution: invalid main GPT header, but valid backup; regenerating main header from backup! The GPT table is stored at least twice, this is telling you the primary copy is trashed, but the backup seems okay ... Warning: Invalid CRC on main header data; loaded backup partition table. Warning! Main and backup partition tables differ! Use the 'c' and 'e' options on the recovery & transformation menu to examine the two tables. Warning! Main partition table CRC mismatch! Loaded backup partition table instead of main partition table! Warning! One or more CRCs don't match. You should repair the disk! Main header: ERROR Backup header: OK Main partition table: ERROR Backup partition table: OK Partition table scan: MBR: not present BSD: not present APM: not present GPT: damaged Found invalid MBR and corrupt GPT. What do you want to do? (Using the GPT MAY permit recovery of GPT data.) 1 - Use current GPT 2 - Create blank GPT Your answer: ( I haven't given one yet) I'm not exactly sure what this is telling me. But I'm guessing it means that the main partition table is gone, but there's a good backup. Yup. I don't understand that prompt, but I THINK it's saying that if you do choose choice 1, it will recover your partition table for you. In addition, some, but not all disk id info is gone: 1) /dev/disk/by-id still shows ata-ST4000NM0033-9ZM170_Z1ZAZDJ0 (the damaged disk) but none of its former partitions Because this is the disk, and you've damaged the contents, so this is completely unaffected. 2) /dev/disk/by-partlabel shows entries for the undamaged disks in the pool, but not the damaged one 3) /dev/disk/by-partuuid similar to /dev/disk/by-partlabel For both of these, "part" is short for partition, and you've just trashed them ... 4) /dev/disk/by-uuid does not show the damaged disk Because the uuid is part of the partition table. This particular disk is from a batch of 4 I bought with the same make and specification and very similar ids (/dev/disk/by-id). Can I repair this disk by copying something off one of those other disks onto this one? GOD NO! You'll start copying uuids, so they'll no longer be unique, and things really will be broken! Is repair just repartitioning - as in the Gentoo handbook? Is it as simple as running gdisk and typing 1 to accept gdisk's attempt at recovering the gpt? Is running gdisk's recovery and transformation facilities the way to go (the b option looks like it's made for exactly this situation)? Anybody experienced at this and willing to guide me? Make sure that option 1 really does recover the GPT, then use it. Of course, the question then becomes what further damage will rear its head. You need to make sure that your raid 3 array can recover from a corrupt disk. THIS IS IMPORTANT. If you tried to recover an md-raid-5 array from this situation you'd almost certainly trash it completely. Actually, if your setup is raid, I'd just blow out the trashed disk completely. Take it out of your system, replace it, and let zfs repair itself onto the new disk. You can then zero out the old disk and it's now a spare. Just be careful here, because I don't know what zfs does, but btrfs by default mirrors metadata but not data, so with that you'd think a mirrored filesystem could repair itself but it can't ... if you want to repair the filesystem without rebuilding from scratch, you need to
Re: [gentoo-user] zfs repair needed (due to fingers being faster than brain)
On 3/1/21 3:25 PM, John Blinka wrote: HI, Gentooers! Hi, So, I typed dd if=/dev/zero of=/dev/sd, and despite hitting ctrl-c quite quickly, zeroed out some portion of the initial part of a disk. Which did this to my zfs raidz3 array: OOPS!!! NAME STATE READ WRITE CKSUM zfs DEGRADED 0 0 0 raidz3-0 DEGRADED 0 0 0 ata-HGST_HUS724030ALE640_PK1234P8JJJVKP ONLINE 0 0 0 ata-HGST_HUS724030ALE640_PK1234P8JJP3AP ONLINE 0 0 0 ata-ST4000NM0033-9ZM170_Z1Z80P4C ONLINE 0 0 0 ata-ST4000NM0033-9ZM170_Z1ZAZ8F1 ONLINE 0 0 0 14296253848142792483 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-ST4000NM0033-9ZM170_Z1ZAZDJ0-part1 ata-ST4000NM0033-9ZM170_Z1Z80KG0 ONLINE 0 0 0 Okay. So the pool is online and the data is accessible. That's actually better than I originally thought. -- I thought you had accidentally damaged part of the ZFS partition that existed on a single disk. -- I've been able to repair this with minimal data loss (zeros) with Oracle's help on Solaris in the past. Aside: My understanding is that ZFS stores multiple copies of it's metadata on the disk (assuming single disk) and that it is possible to recover a pool if any one (or maybe two for consistency checks) are viable. Though doing so is further into the weeds than you normally want to be. Could have been worse. I do have backups, and it is raid3, so all I've injured is my pride, but I do want to fix things.I'd appreciate some guidance before I attempt doing this - I have no experience at it myself. First, your pool / it's raidz3 is only 'DEGRADED', which means that the data is still accessible. 'OFFLINE' would be more problematic. The steps I envision are 1) zpool offline zfs 14296253848142792483 (What's that number?) I'm guessing it's an internal ZFS serial number. You will probably need to reference it. I see no reason to take the pool offline. 2) do something to repair the damaged disk I don't think you need to do anything at the individual disk level yet. 3) zpool online zfs I think you can fix this with the pool online. Right now, the device name for the damaged disk is /dev/sda. Gdisk says this about it: Caution: invalid main GPT header, This is to be expected. but valid backup; regenerating main header from backup! This looks promising. Warning: Invalid CRC on main header data; loaded backup partition table. Warning! Main and backup partition tables differ! Use the 'c' and 'e' options on the recovery & transformation menu to examine the two tables. I'm assuming that the main partition table is at the start of the disk and that it's what got wiped out. So I'd think that you can look at the 'c' and 'e' options on the recovery & transformation menu for options to repair the main partition table. Warning! Main partition table CRC mismatch! Loaded backup partition table instead of main partition table! I know. Thank you for using the backup partition table. Warning! One or more CRCs don't match. You should repair the disk! I'm guessing that this is a direct result of the dd oops. I would want more evidence to support it being a larger problem. The CRC may be calculated over a partially zeroed chunk of disk. (Chunk because I don't know what term is best here and I want to avoid implying anything specific or incorrectly.) Main header: ERROR Backup header: OK Main partition table: ERROR Backup partition table: OK ACK Partition table scan: MBR: not present BSD: not present APM: not present GPT: damaged Found invalid MBR and corrupt GPT. What do you want to do? (Using the GPT MAY permit recovery of GPT data.) 1 - Use current GPT 2 - Create blank GPT Your answer: ( I haven't given one yet) I'd assume #1, Use current GPT. I'm not exactly sure what this is telling me. But I'm guessing it means that the main partition table is gone, but there's a good backup. That's my interpretation too. It jives with the description of what happened. In addition, some, but not all disk id info is gone: 1) /dev/disk/by-id still shows ata-ST4000NM0033-9ZM170_Z1ZAZDJ0 (the damaged disk) but none of its former partitions The disk ID still being there may be a symptom / side effect of when udev creates the links. I would expect it to not be there post-reboot. Well, maybe. The disk serial number is independent of any data on the disk. Partitions by ID would probably be gone post reboot (or eject and re-insertion). 2) /dev/disk/by-partlabel shows entries for the undamaged disks in the pool, but not the damaged one Okay. That means that udev is recognizing the change faster than I would have expected. That
