On Mon, Dec 20, 2021 at 1:52 PM Mark Knecht <[email protected]> wrote: > > I've recently built 2 TrueNAS file servers. The first (and main) unit > runs all the time and serves to backup my home user machines. > Generally speaking I (currently) put data onto it using rsync but it > also has an NFS mount that serves as a location for my Raspberry Pi to > store duplicate copies of astrophotography pictures live as they come > off the DSLR in the middle of the night. > > ... > > The thing is that the ZIL is only used for synchronous writes and I > don't know whether anything I'm doing to back up my user machines, > which currently is just rsync commands, is synchronous or could be > made synchronous, and I do not know if the NFS writes from the R_Pi > are synchronous or could be made so. >
Disclaimer: some of this stuff is a bit arcane and the documentation isn't very great, so I could be missing a nuance somewhere. First, one of your options is to set sync=always on the zfs dataset, if synchronous behavior is strongly desired. That will force ALL writes at the filesystem level to be synchronous. It will of course also normally kill performance but the ZIL may very well save you if your SSD performs adequately. This still only applies at the filesystem level, which may be an issue with NFS (read on). I'm not sure how exactly you're using rsync from the description above (rsyncd, directly client access, etc). In any case I don't think rsync has any kind of option to force synchronous behavior. I'm not sure if manually running a sync on the server after using rsync will use the ZIL or not. If you're using sync=always then that should cover rsync no matter how you're doing it. Nfs is a little different as both the server-side and client-side have possible asynchronous behavior. By default the nfs client is asynchronous, so caching can happen on the client before the file is even sent to the server. This can be disabled with the mount option sync on the client side. That will force all data to be sent to the server immediately. Any nfs server or filesystem settings on the server side will not have any impact if the client doesn't transmit the data to the server. The server also has a sync setting which defaults to on, and it additionally has another layer of caching on top of that which can be disabled with no_wdelay on the export. Those server-side settings probably delay anything getting to the filesystem and so they would have precedence over any filesystem-level settings. As you can see you need to use a bit of a kill-it-with-fire approach to get synchronous behavior, as it traditionally performs so poorly that everybody takes steps to try to prevent it from happening. I'll also note that the main thing synchronous behavior protects you from is unclean shutdown of the server. It has no bearing on what happens if a client goes down uncleanly. If you don't expect server crashes it may not provide much benefit. If you're using ZIL you should consider having the ZIL mirrored, as any loss of the ZIL devices will otherwise cause data loss. Use of the ZIL is also going to create wear on your SSD so consider that and your overall disk load before setting sync=always on the dataset. Since the setting is at the dataset level you could have multiple mountpoints and have a different sync policy for each. The default is normal POSIX behavior which only syncs when requested (sync, fsync, O_SYNC, etc). -- Rich
