[OT] Re: [gentoo-user] NAS and replacing with larger drives

[Peter Humphrey]

On Friday, 9 December 2022 10:34:00 GMT I wrote:

> in the 1970s the national grid was monitored and analysed with a Ferranti
> Argus 500 machine with 24KB RAM and a 2MB disk. It was common for
> American visitors to believe that was just driving the control engineers'
> displays, and where was the main computer?

Er... There was no RAM in those days, not of the type we know today. In fact 
it was 2-microsecond core store. Each tiny ferromagnetic toroid was threaded 
with one X wire, one Y wire and (I think it was) a sync pulse wire. A 
remarkable labour of love to build such a thing.

-- 
Regards,
Peter.

Re: [gentoo-user] NAS and replacing with larger drives

[Mark Knecht]

On Sat, Dec 10, 2022 at 9:35 PM Dale  wrote:
>
> Mark Knecht wrote:
> > 
> > >
> > > I got a old rig I can use.  I actually burned OpenNAS, TrueNAS or
> > FreeNAS on a USB stick.  I can't recall which one I put on it tho.  I
> > downloaded all three.  lol  If you know that one is better than the
> > others, feel free to share.  Also, I'd like to keep using LVM if I
> > can.  If nothing else, I already got the data on the drives and won't
> > have to reformat and copy again.  It took almost 100 hours to copy to
> > the new 16TB drive.  Using LVM would make that easier, and faster.
> > >
> > > I'll have to work with what I got for now but I really like the
> > Raspberry option for its size and good options to upgrade later.  I'll
> > just make do with something else until that option is doable.  Maybe
> > it won't be to long.
> > >
> > > Dale
> > >
> > > :-)  :-)
> >
> > TrueNAS Core. It's the free one. Works great. Very stable, but it is
> > BSD, not Linux so you'll be frustrated sometimes. None the less it
> > works very well.
>
>
> Well, I booted it and it is FreeNAS.  I got it on a USB stick tho.
> Well, I put the installer on one stick and then installed on a second
> stick.  Kinda odd but I get it.  I also noticed it is BSD based.  I
> played with BSD once before.  One thing I can say, it's secure.  Big time.
>

I'm not clear exactly but FreeNAS _BECAME_ TrueNAS Core and TrueNAS
(all 3 versions) are the ones being worked on.

Installing from USB is pretty standard. Installing to a USB flash drive
is not unheard of in home NAS servers but be careful of machine
placement because people talk about USB sockets being unreliable
long term. I'm sure you'll figure it out, but make sure you're using
TrueNAS Core.

> I see it uses ZFS or something.  No mention of LVM.  I figured that.  Oh
> well.

I see LVM as something that belongs on your machine, not your NAS
device. Your LVM volumes will just be directories on the NAS. You will
make your pools as large as you can afford and the NAS will just store
your data. You don't really need to worry about that much. My NAS
stores backups from 3 different machine, but all the backup data
is in a single ZFS RAID1 pool located in directories which macth the
name of the machine that wrote them.

>
> If I can't hammer FreeNAS into shape, I'll try TrueNAS next.  If it
> works, that's fine too.  ;-)

My input for the third time. Move to TrueNAS Core. That's the one
that is being developed and getting support.

Mark

Re: [gentoo-user] NAS and replacing with larger drives

[Dale]

Mark Knecht wrote:
>
>
> On Sat, Dec 10, 2022 at 9:35 PM Dale  > wrote:
> >
> > Mark Knecht wrote:
> > > 
> > > >
> > > > I got a old rig I can use.  I actually burned OpenNAS, TrueNAS or
> > > FreeNAS on a USB stick.  I can't recall which one I put on it tho.  I
> > > downloaded all three.  lol  If you know that one is better than the
> > > others, feel free to share.  Also, I'd like to keep using LVM if I
> > > can.  If nothing else, I already got the data on the drives and won't
> > > have to reformat and copy again.  It took almost 100 hours to copy to
> > > the new 16TB drive.  Using LVM would make that easier, and faster.
> > > >
> > > > I'll have to work with what I got for now but I really like the
> > > Raspberry option for its size and good options to upgrade later.  I'll
> > > just make do with something else until that option is doable.  Maybe
> > > it won't be to long.
> > > >
> > > > Dale
> > > >
> > > > :-)  :-)
> > >
> > > TrueNAS Core. It's the free one. Works great. Very stable, but it is
> > > BSD, not Linux so you'll be frustrated sometimes. None the less it
> > > works very well.
> >
> >
> > Well, I booted it and it is FreeNAS.  I got it on a USB stick tho.
> > Well, I put the installer on one stick and then installed on a second
> > stick.  Kinda odd but I get it.  I also noticed it is BSD based.  I
> > played with BSD once before.  One thing I can say, it's secure.  Big
> time.
> >
>
> I'm not clear exactly but FreeNAS _BECAME_ TrueNAS Core and TrueNAS
> (all 3 versions) are the ones being worked on.
>
> Installing from USB is pretty standard. Installing to a USB flash drive
> is not unheard of in home NAS servers but be careful of machine
> placement because people talk about USB sockets being unreliable
> long term. I'm sure you'll figure it out, but make sure you're using
> TrueNAS Core. 
>
> > I see it uses ZFS or something.  No mention of LVM.  I figured that.  Oh
> > well.
>
> I see LVM as something that belongs on your machine, not your NAS
> device. Your LVM volumes will just be directories on the NAS. You will
> make your pools as large as you can afford and the NAS will just store
> your data. You don't really need to worry about that much. My NAS
> stores backups from 3 different machine, but all the backup data
> is in a single ZFS RAID1 pool located in directories which macth the 
> name of the machine that wrote them.
>  
> >
> > If I can't hammer FreeNAS into shape, I'll try TrueNAS next.  If it
> > works, that's fine too.  ;-)
>
> My input for the third time. Move to TrueNAS Core. That's the one
> that is being developed and getting support.
>  
> Mark


I think I'm going to switch.  I need to start over anyway.  I set up a
user account and a large pool but while I can mount it, I can't put
anything in it yet.  I get a permission error.  I likely missed a step
or something.  Starting over will help correct that.  lol 

By the way, when I got it installed, it did update to a newer version. 
I didn't look to see if it was dated in any way but updates seem to be
available for FreeNAS.  I dunno. 

Thanks for the info.

Dale

:-)  :-) 

Re: [gentoo-user] NAS and replacing with larger drives

[Mark Knecht]

On Sun, Dec 11, 2022 at 8:01 AM Dale  wrote:

> I think I'm going to switch.  I need to start over anyway.  I set up a
user account and a large pool but while I can mount it, I can't put
anything in it yet.  I get a permission error.  I likely missed a step or
something.  Starting over will help correct that.  lol
>
> By the way, when I got it installed, it did update to a newer version.  I
didn't look to see if it was dated in any way but updates seem to be
available for FreeNAS.  I dunno.
>
> Thanks for the info.
>
> Dale
>
> :-)  :-)

At this point I think you're wise to just plug around in it for a little
while. Learn it a little bit. Build a few pools and get used to how it
works. It's a bit different than Linux.

In my case everything is NFS mounts and NFS exports work differently on
BSD. Assume you have a pool:

/mnt/MyPool/mark/Backups

and under that you want to have 3 directories exported to different
machines for backups, so

/mnt/MyPool/mark/Backups/science
/mnt/MyPool/mark/Backups/sciene2
/mnt/MyPool/mark/Backups/StellarMate

where each user machine has a place to put things, and hence you can find
it, but no LVM, it's just a big pool of storage. Note there are all the
standard problems about permissions when you first set these directories
up, like making sure you own them, that they are writable, etc.

In Linux NFS I would likely export all three separately, while in TrueNAS
BSD I export

/mnt/MyPool/mark --alldirs

If you cared about science mucking with science2's backups there are ways
to stop that, but I don't care because each machine on my network has a
bash scripts that points it where I want it to go:

mark@science2:~$ cat ./bin/DoTrueNAS
#rsync -avx -n --port=873
--exclude={000_NOT_BACKED_UP,RIPS,.cache,.nv,'google-chrome*',DiskImages,Current}
/home/mark mark@truenas1:/mnt/MyPool/mark/Backups/science2/.

rsync -avx --port=873
--exclude={000_NOT_BACKED_UP,RIPS,.cache,.nv,'google-chrome*',DiskImages,Current}
/home/mark mark@truenas1:/mnt/MyPool/mark/Backups/science2/.
mark@science2:~$

where the first one is a test config and the second is a real transfer.
Because it's rsync if something doesn't finish then I can pick up again
with little time lost.

Also, I think there are ways for you to build complex pools like a RAID0
from your 6TB and 8TB drives, and then a RAID1 using the RAID0 and your
14TB drive but I've never tried it because mine don't have enough drive
slots for that.

Also, turn on compression. It saves me between 15-20% so 14TB becomes 16TB
storage. YMMV. Video files don't compress, at least not much. Data files
generally do.

Hope this helps. I think you'll find TrueNAS fun actually but there is a
learning curve. I've used it for about a year and barely scratched the
surface.

Good luck,
Mark

Re: [gentoo-user] Docker installation issues

[Mansour Al Akeel]

This is what I have on my machine. I can see the ANSWER SECTION in reversed
order

localhost ~ # dig registry-1.docker.io

; <<>> DiG 9.16.33 <<>> registry-1.docker.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10324
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;registry-1.docker.io. IN A

;; ANSWER SECTION:
registry-1.docker.io. 51 IN A 3.216.34.172
registry-1.docker.io. 51 IN A 44.205.64.79
registry-1.docker.io. 51 IN A 34.205.13.154

;; AUTHORITY SECTION:
docker.io. 171984 IN NS ns-1168.awsdns-18.org.
docker.io. 171984 IN NS ns-1827.awsdns-36.co.uk.
docker.io. 171984 IN NS ns-421.awsdns-52.com.
docker.io. 171984 IN NS ns-513.awsdns-00.net.

;; Query time: 15 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Sun Dec 11 11:06:33 EST 2022
;; MSG SIZE  rcvd: 237



On Sun, Dec 11, 2022 at 12:59 AM Andreas Fink  wrote:

> On Sat, 10 Dec 2022 15:51:17 -0500
> Mansour Al Akeel  wrote:
>
> > Andreas,
> >
> > Thank you very much. In fact I didn't go that deep yet, and not sure if I
> > should. I just found that the url is not accessible even from a browser.
> > Googling a bit, tells me there is no clear URL for docker-registry and
> > possibly this one is outdated. I will continue looking around to confirm
> > what the current default repo should be. If you have any suggestions,
> > please let me know.
> >
> >
> >  localhost in ~
> > ○ → curl -k -v https://registry-1.docker.io/v2/
> > *   Trying 34.228.211.243:443...
> >
> > * connect to 34.228.211.243 port 443 failed: Connection timed out
> > * Failed to connect to registry-1.docker.io port 443 after 129401 ms:
> > Couldn't connect to server
> > * Closing connection 0
> > curl: (28) Failed to connect to registry-1.docker.io port 443 after
> 129401
> > ms: Couldn't connect to server
>
>
> You have a a strange DNS resolution. The IP address 34.227.211.243
> seems wrong. Here is what I see when I look at the DNS records:
> andreas@localhost ~$ dig registry-1.docker.io
>
> ; <<>> DiG 9.16.33 <<>> registry-1.docker.io
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11419
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 4096
> ;; QUESTION SECTION:
> ;registry-1.docker.io.  IN  A
>
> ;; ANSWER SECTION:
> registry-1.docker.io.   13  IN  A   34.205.13.154
> registry-1.docker.io.   13  IN  A   44.205.64.79
> registry-1.docker.io.   13  IN  A   3.216.34.172
>
> ;; Query time: 10 msec
> ;; SERVER: 79.143.183.251#53(79.143.183.251)
> ;; WHEN: Sun Dec 11 06:56:50 CET 2022
> ;; MSG SIZE  rcvd: 97
>
> I am not sure
> Not sure where you get the wrong IP from, but it is a DNS issue.
>
>

Re: [gentoo-user] NAS and replacing with larger drives

[Frank Steinmetzger]

Am Sun, Dec 11, 2022 at 08:44:42AM -0700 schrieb Mark Knecht:

> Also, I think there are ways for you to build complex pools like a RAID0
> from your 6TB and 8TB drives, and then a RAID1 using the RAID0 and your
> 14TB drive but I've never tried it because mine don't have enough drive
> slots for that.

After a longer fruitless search on the interwebs (I ddidn’t want to start up
my NAS just to check this) I finally found the right search keywords and
found a reddit thread about that. And it even throws LVM into the
discussion. ^^
https://www.reddit.com/r/zfs/comments/fitc73/raidz_with_nested_vdevs/

Also :
“Here's a definitive answer from the man page for zpool.

Virtual devices cannot be nested, so a mirror or raidz virtual device can
only contain files or disks. Mirrors of mirrors (or other combinations) are
not allowed.”


I would advise against a JBOD pool anyways. Because if one drive dies, the
whole JBOD is gone. That goes for ZFS and probably for LVM, too (though I am
not sure how writes are distributed across JBOD disks). If the goal is
redundancy, you could buy a second drive to match the size of an existing
one and build a mirror. If redundancy is not a goal, then use the drives
separately like you do now. If one fails, then only its content is gone (or
even just the files sitting on the broken sector).

> Also, turn on compression. It saves me between 15-20% so 14TB becomes 16TB
> storage. YMMV. Video files don't compress, at least not much. Data files
> generally do.

It doesn’t hurt to switch it on, especially with lzo. But with video, the
benefit will be negligible. When storing a block of data (a “record” in ZFS
speak), it is passed through the compressor and only if the compression gain
is above a given threshold (10 % methinks), the block is written to disk
with compression.

What is more relevant in filesystems for big files (i.e. videos): set the
record size to 1 MB. The default is 64 kB, IIRC. Each record requires one
block of metadata (which includes the record checksum). So bigger records →
fewer meta blocks → better storage efficiency.

If you use big records for small files, then efficiency goes down a little.
It’s a similar (but a little more complicated) principle as when you write a
100 byte text file to a file system that uses 4 kB clusters. That file will
still use up 4 kB on disk.

The record size can be set per-dataset. So in your pool you could create a
dataset with a smaller record size for office documents, images and music,
and another dataset just for videos.

> Hope this helps. I think you'll find TrueNAS fun actually but there is a
> learning curve. I've used it for about a year and barely scratched the
> surface.

The main reason for me why I would wanna use it as opposed to a standard
Gentoo install: the OOTB web interface to manage all sorts of accounts,
access and permissions under one nice hood.

-- 

Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

A fermata comes to the doctor: “I can’t hold it any longer...”


signature.asc
Description: PGP signature

Re: [gentoo-user] NAS and replacing with larger drives

[Dale]

Frank Steinmetzger wrote:
> Am Sun, Dec 11, 2022 at 08:44:42AM -0700 schrieb Mark Knecht:
>
>> Also, I think there are ways for you to build complex pools like a RAID0
>> from your 6TB and 8TB drives, and then a RAID1 using the RAID0 and your
>> 14TB drive but I've never tried it because mine don't have enough drive
>> slots for that.
> After a longer fruitless search on the interwebs (I ddidn’t want to start up
> my NAS just to check this) I finally found the right search keywords and
> found a reddit thread about that. And it even throws LVM into the
> discussion. ^^
> https://www.reddit.com/r/zfs/comments/fitc73/raidz_with_nested_vdevs/
>
> Also :
> “Here's a definitive answer from the man page for zpool.
>
> Virtual devices cannot be nested, so a mirror or raidz virtual device can
> only contain files or disks. Mirrors of mirrors (or other combinations) are
> not allowed.”
>
>
> I would advise against a JBOD pool anyways. Because if one drive dies, the
> whole JBOD is gone. That goes for ZFS and probably for LVM, too (though I am
> not sure how writes are distributed across JBOD disks). If the goal is
> redundancy, you could buy a second drive to match the size of an existing
> one and build a mirror. If redundancy is not a goal, then use the drives
> separately like you do now. If one fails, then only its content is gone (or
> even just the files sitting on the broken sector).
>
>> Also, turn on compression. It saves me between 15-20% so 14TB becomes 16TB
>> storage. YMMV. Video files don't compress, at least not much. Data files
>> generally do.
> It doesn’t hurt to switch it on, especially with lzo. But with video, the
> benefit will be negligible. When storing a block of data (a “record” in ZFS
> speak), it is passed through the compressor and only if the compression gain
> is above a given threshold (10 % methinks), the block is written to disk
> with compression.
>
> What is more relevant in filesystems for big files (i.e. videos): set the
> record size to 1 MB. The default is 64 kB, IIRC. Each record requires one
> block of metadata (which includes the record checksum). So bigger records →
> fewer meta blocks → better storage efficiency.
>
> If you use big records for small files, then efficiency goes down a little.
> It’s a similar (but a little more complicated) principle as when you write a
> 100 byte text file to a file system that uses 4 kB clusters. That file will
> still use up 4 kB on disk.
>
> The record size can be set per-dataset. So in your pool you could create a
> dataset with a smaller record size for office documents, images and music,
> and another dataset just for videos.
>
>> Hope this helps. I think you'll find TrueNAS fun actually but there is a
>> learning curve. I've used it for about a year and barely scratched the
>> surface.
> The main reason for me why I would wanna use it as opposed to a standard
> Gentoo install: the OOTB web interface to manage all sorts of accounts,
> access and permissions under one nice hood.
>


Interesting info.  Since this is a duplicate copy already, I'm not to
worried about RAID stuff.  I'd rather have two separate backups myself. 
Store them in different places for even more safety.  Still, one of
these days.  ;-) 

I'm still getting this error. 

root@fireball ~ # mkdir /mnt/Backup/Videos
mkdir: cannot create directory ‘/mnt/Backup/Videos’: Permission denied
root@fireball ~ #

I've tried every permission in every place I can find.  I have a user
set up, set permissions under storage and pools, all the little
directory thingys under it too.  It mounts so I'd think I got everything
set up correctly in the Share section.  I actually followed a guide and
I don't think I missed anything.  Still, I can't write anything to the
thing.  It mounts fine, even shows it is mounted rw.  I can't find
anything wrong on the puter or NAS ends.  This reminds me of the last
time I played with BSD.  It works so different, it just throws a person
upside down and shakes them. 

I did switch to TrueNAS tho.  Thing is, I got the same result from both,
exact same error.  Can't be a bug.  It has to be me.  No idea what I
missed but I bet I did.  lol  Just gonna keep banging on it.  :-D  I do
like the GUI part.  That's pretty neat. 

Dale

:-) :-) 

Re: [gentoo-user] NAS and replacing with larger drives

[Dale]

Dale wrote:
>
> Interesting info.  Since this is a duplicate copy already, I'm not to
> worried about RAID stuff.  I'd rather have two separate backups myself. 
> Store them in different places for even more safety.  Still, one of
> these days.  ;-) 
>
> I'm still getting this error. 
>
> root@fireball ~ # mkdir /mnt/Backup/Videos
> mkdir: cannot create directory ‘/mnt/Backup/Videos’: Permission denied
> root@fireball ~ #
>
> I've tried every permission in every place I can find.  I have a user
> set up, set permissions under storage and pools, all the little
> directory thingys under it too.  It mounts so I'd think I got everything
> set up correctly in the Share section.  I actually followed a guide and
> I don't think I missed anything.  Still, I can't write anything to the
> thing.  It mounts fine, even shows it is mounted rw.  I can't find
> anything wrong on the puter or NAS ends.  This reminds me of the last
> time I played with BSD.  It works so different, it just throws a person
> upside down and shakes them. 
>
> I did switch to TrueNAS tho.  Thing is, I got the same result from both,
> exact same error.  Can't be a bug.  It has to be me.  No idea what I
> missed but I bet I did.  lol  Just gonna keep banging on it.  :-D  I do
> like the GUI part.  That's pretty neat. 
>
> Dale
>
> :-) :-) 
>


Update.  I thought about just using Dolphin to play with the directory
that's actually on the NAS.  I could create folders and files just fine
with Dolphin.  Well, isn't that interesting.  Finally I found a comment
with google that gave me a clue.  I needed to change the options I use
with rsync plus it appears on my computer, I need to su to dale, same
user as on NAS.  Now it works.  I can rsync my files over. 

Then I noticed something else.  The network card in the NAS box, it's a
old 100MB card.  Has anyone ever poured cold molasses before?  Does
waiting on leap year sound familiar?  ROFL  Just saying. 

Hey, at least I got the silly thing to work, slowly, but works.  :/

Dale

:-)  :-) 

[gentoo-user] media-libs/babl-0.1.96-r1 compile dies during emerge @world

[Walter Dnes]

  The subject line sums it up.  stable 64-bit Gentoo.  My rsync_excludes
block the following...

app-emacs
app-leechcraft
app-mobilephone
app-pda
app-xemacs
dev-dotnet
dev-embedded
dev-haskell
dev-java
dev-qt
dev-ros
dev-ruby
java-virtuals
kde-apps
kde-base
kde-frameworks
kde-misc
kde-plasma
lxde-base
lxqt-base
mate-base
mate-extra
net-p2p
ros-meta
sec-policy
www-apache
xfce-base
xfce-extra

  I don't think they should make any difference, but included for
completeness.  File-attached is the gzipped build log...

-- 
I've seen things, you people wouldn't believe; Gopher, Netscape with
frames, the first Browser Wars.  Searching for pages with AltaVista,
pop-up windows self-replicating, trying to uninstall RealPlayer.  All
those moments, will be lost in time like tears in rain... time to die.


buildlog.txt.gz
Description: application/gzip