Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-29 Thread Adrian Klaver
On 11/29/2016 01:15 AM, Thomas Güttler wrote: Am 28.11.2016 um 16:01 schrieb Adrian Klaver: On 11/28/2016 06:28 AM, Thomas Güttler wrote: Hi, PostgreSQL is rock solid and one of the most reliable parts of our toolchain. Thank you Up to now, we don't store files in PostgreSQL. I was tol

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-29 Thread Adrian Klaver
On 11/29/2016 01:50 AM, Thomas Güttler wrote: Am 29.11.2016 um 01:52 schrieb Mike Sofen: From: Thomas Güttler Sent: Monday, November 28, 2016 6:28 AM ...I have 2.3TBytes of files. File count is 17M Since we already store our structured data in postgres, I think about storing the files in P

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-29 Thread Jacob Bunk Nielsen
Thomas Güttler writes: > I have 2.3TBytes of files. File count is 17M > > Up to now we use rsync (via rsnapshot) to backup our data. Isn't putting those files into your database going to make any sort of maintanance on your database cumbersome? How big is your database currently? Is it worth gro

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-29 Thread Stuart Bishop
On 29 November 2016 at 16:50, Thomas Güttler wrote: > > > Am 29.11.2016 um 01:52 schrieb Mike Sofen: > >> From: Thomas Güttler Sent: Monday, November 28, 2016 6:28 AM >> >> ...I have 2.3TBytes of files. File count is 17M >> >> Since we already store our structured data in postgres, I think abou

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-29 Thread Jerome Wagner
2 other options that you may want to look at : - cephfs This has nothing to do with postgres but is a distributed filesystem handling very large amount of files (thinks next generation NFS) I haven't tried it myself yet but they reached a "stable" milestone regarding the distributed fs. cf https:

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-29 Thread Thomas Güttler
Am 29.11.2016 um 01:52 schrieb Mike Sofen: From: Thomas Güttler Sent: Monday, November 28, 2016 6:28 AM ...I have 2.3TBytes of files. File count is 17M Since we already store our structured data in postgres, I think about storing the files in PostgreSQL, too. Is it feasible to store file

We reached the limit of inotify. Was: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-29 Thread Thomas Güttler
Am 28.11.2016 um 17:43 schrieb Daniel Verite: Thomas Güttler wrote: Up to now we use rsync (via rsnapshot) to backup our data. But it takes longer and longer for rsync to detect the changes. Rsync checks many files. But daily only very few files really change. More than 99.9% don't.

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-29 Thread Thomas Güttler
Am 28.11.2016 um 16:01 schrieb Adrian Klaver: On 11/28/2016 06:28 AM, Thomas Güttler wrote: Hi, PostgreSQL is rock solid and one of the most reliable parts of our toolchain. Thank you Up to now, we don't store files in PostgreSQL. I was told, that you must not do this But this was

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-28 Thread Mike Sofen
From: Thomas Güttler Sent: Monday, November 28, 2016 6:28 AM ...I have 2.3TBytes of files. File count is 17M Since we already store our structured data in postgres, I think about storing the files in PostgreSQL, too. Is it feasible to store file in PostgreSQL? --- I am doing something

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-28 Thread
Thomas Güttler wrote: > > Up to now, we don't store files in PostgreSQL. > I was told, that you must not do this But this was 20 years ago. > I have 2.3TBytes of files. File count is 17M > Up to now we use rsync (via rsnapshot) to backup our data. > But it takes longer and longer for rsync t

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-28 Thread Eduardo Morras
On Mon, 28 Nov 2016 15:28:28 +0100 Thomas Güttler wrote: > Hi, > > Up to now, we don't store files in PostgreSQL. > > I was told, that you must not do this But this was 20 years ago. > > > I have 2.3TBytes of files. File count is 17M > > Up to now we use rsync (via rsnapshot) to backup

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-28 Thread Daniel Verite
Thomas Güttler wrote: > Up to now we use rsync (via rsnapshot) to backup our data. > > But it takes longer and longer for rsync to detect > the changes. Rsync checks many files. But daily only > very few files really change. More than 99.9% don't. lsyncd+rsync has worked nicely for me on

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-28 Thread Chris Travers
On Mon, Nov 28, 2016 at 3:28 PM, Thomas Güttler < guettl...@thomas-guettler.de> wrote: > Hi, > > PostgreSQL is rock solid and one of the most reliable parts of our > toolchain. > >Thank you > > Up to now, we don't store files in PostgreSQL. > > I was told, that you must not do this But th

Re: [GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-28 Thread Adrian Klaver
On 11/28/2016 06:28 AM, Thomas Güttler wrote: Hi, PostgreSQL is rock solid and one of the most reliable parts of our toolchain. Thank you Up to now, we don't store files in PostgreSQL. I was told, that you must not do this But this was 20 years ago. I have 2.3TBytes of files. File c

[GENERAL] Storing files: 2.3TBytes, 17M file count

2016-11-28 Thread Thomas Güttler
Hi, PostgreSQL is rock solid and one of the most reliable parts of our toolchain. Thank you Up to now, we don't store files in PostgreSQL. I was told, that you must not do this But this was 20 years ago. I have 2.3TBytes of files. File count is 17M Up to now we use rsync (via rsnaps