On 24 Jan 2021, at 12:07, Michael Peddemors via mailop wrote:
+1 Customers using NFS is perfectly fine, and scales well.
But do yourself a favour.. not all storage appliances are the same.
Go NetApp (amazing how reasonable cost you can get them now) if you
can. Seen it too many times where people 'experiment' with this and
get themselves in trouble.
Also, for email storage, we still like NFS 3, had some early issues
with NFS 4, might be time for us to experiment with it again, but for
email..
As someone that did such an experiment and sustained growth from 10Ks to
millions of subscribers. We tested a few NFS solutions (then EMC²,
Hitachi + Sun, NetApp, BlueArc).
* Do stay with NetApp. Back in the day we settled with redundant heads
(960s if memory serves well) with FC drives. Dollar for dollar, this
outperformed their competition. Their pre-, post-sales engineering and
support was truly knowledgeable as well.
* Maildir+ — I actually wrote a specialized MDA that encoded the
length and relevant dates in the filename, which allowed us to avoid
using the stat() NFS call. Our setup ran lock-free with no issues, while
allowing a noticeable throughput increase due to the very large number
of files involved.
* Local delivery happened directly from the SMTP servers, which mounted
the remote filesystems RW.
* Most email access happened via POP on dedicated servers, with a
dedicated load-balancing switch at the front. We also provided IMAP,
which was used for webmail exclusively. Dovecot would likely change this
substantially, as it can do its own service routing.
* Separate your users in a few filesystems. Try to keep them below a few
TBs. We used MD5 to spread out users through the possible filesystems
automatically. This strategy tended to spread the load surprisingly
evenly.
* At the time, we worked with NFSv3. v4 wasn't a widely usable option
for some reason I cannot recall. This might no longer hold true for your
scenario.
* Switches are very important. Make sure they can sustain a continuous
stream of jumbo frames back to back. The NetApps were actually capable
of saturating their 1000BaseT interfaces routinely, and at some point
our switches would just not be able to keep up. Nowadays, you would get
10G or the largest interfaces you could buy, but switching needs to be
well thought.
With the above, we received visits from prospective NetApp customers
twice during my tenure. They were very keen to show their customers how
our boxes actually performed ~20% more IOPS than the spec sheet claimed.
Best regards
-lem
_______________________________________________
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop
