I ran some more tests on this, and found that there is a temp file being
stored in a dir /tmp/pspp* and that file is where all the space is going
while its writing the actual csv.
I tested pspp-convert as Ben P suggested and can confirm that it doesn't
suffer this same issue, just using the space
I can confirm this same behavior exists on non docker environments. I went back
to my trusty dev vm running in virtual box and see the same behavior. Here is
the config of the vm:
deploy@app1[local]:~$ uname -a
Linux app1 4.4.0-169-generic #198-Ubuntu SMP Tue Nov 12 10:38:00 UTC 2019
x86_64 x86
Once the conversion is complete the space is returned so its not a long term
problem - only during the conversion. This became an issue because in
kubernetes you control your resources much more tightly and that’s why this was
highlighted.
I’m not sure there is anything special about the SAV fi
I will give that a try and see how the behavior is - if its better, maybe
adding the /CELLS=LABEL is a feature that can be added.
Watch this space!
Cheers
Dave
On Dec 4, 2019, 11:09 AM -0600, Ben Pfaff , wrote:
> That *is* higher than I would expect.
>
> Do you see less disk activity if you use th
Alternatively, you could hack up a small perl program, using the pspp.pm
module to read the file and write it out however you wish. That would
also be far more optimal that using pspp to do the task.
On Wed, Dec 04, 2019 at 09:09:34AM -0800, Ben Pfaff wrote:
That *is* higher than I would ex
I'm curious to see what the devs say. I think they use Debian, but I
don't know about docker.
So is the excessive disk space used and then returned and when pspp is
done, so only 150MB are consumed? Or is it that many GB of storage seem
to disappear (so maybe the file shows a CSV file size of 150M
That *is* higher than I would expect.
Do you see less disk activity if you use the "pspp-convert" program?
It does not have the exact feature you want (in particular the
/CELLS=LABELS part) but it is better optimized in general for that
particular task.
On Wed, Dec 4, 2019 at 4:42 AM Dave Trollop
Hi Alan,
Sorry, yes I forgot to mention this is linux, Debian GNU/Linux 9
Linux e1e6db1d8408 4.9.184-linuxkit #1 SMP Tue Jul 2 22:58:16 UTC 2019 x86_64
GNU/Linux
I’ve reproduced this behavior in kubernetes and outside kubernetes in a raw
docker container so its not kubernetes specific but may b
Wow, that's a lot. Do you mean that 7GB of space are needed (for, I
guess temporary files)? And you did not observe that previously?
Maybe the devs are familiar with kubernetes; I only know the name. Can
you describe the environment (e.g., OS)? And pspp version? How many
conversions have you obser
