Re: [pgAdmin4][patch][runtime]: RM#2829, RM#2491 - pgAdmin4 crashes while saving CSV data from Query tool

2017-11-17 Thread Neel Patel 
Hi Dave,

On Thu, Nov 16, 2017 at 8:13 PM, Dave Page  wrote:

> Hi
>
> On Thu, Nov 16, 2017 at 1:47 PM, Neel Patel 
> wrote:
>
>> Hi,
>>
>> Adding information.
>>
>> With this patch, RM#2715 should also be resolved.
>>
>> Thanks,
>> Neel Patel
>>
>> On Thu, Nov 16, 2017 at 7:01 PM, Neel Patel 
>> wrote:
>>
>>> Hi,
>>>
>>> I am able to reproduce the crash while downloading and save data in CSV
>>> file from query tool.
>>>
>>> Please find attached updated patch with below changes after reading Qt
>>> documentation.
>>>
>>>- Added new signal "readyRead". As per the Qt documentation, this
>>>signal will be emitted when data is ready from IO channel for large 
>>> amount
>>>of data transfer between server and client.
>>>- Ready read and DownloadInProgress signal is very quick in call so
>>>we should not do large operation inside that slot because for downloading
>>>big data it may possible of frequent call of those signals which may 
>>> cause
>>>the crash or missing data to write inside the file so removed unnecessary
>>>logic from that slot.
>>>- Fixed the crash while opening IODevice with NULL handle.
>>>
>>> With above changes, I have tested with same data as earlier and it is
>>> working as expected without crashing the application.
>>>
>>> Do review it and let me know for comments.
>>>
>>
> My first test was on Mac using Qt 5.8 with webkit, and when I attempted to
> download the results from a 300K row query, it basically hung showing the
> Downloading File progress indicator. It let me cancel it and carried on
> working, but hung again the next time I tried the CSV download. Each time
> it seems to download some data - from the same query I've seen 2.8MB, 5.1MB
> and 1.5MB.
>

We are using "*downloadProgress*"  signal to update the progress bar and
here we are getting "readData" as bytes but "totalData" is unknown so we
received "totalData" as -1 because Qt doesn't know the total size of the
file we are getting from server but when we open "pgAdmin4" website in
another link and try to download some binary or source at that time we are
getting "totalData" as known size and we are displaying the progress bar
correctly.

Here, Qt sending the signal "downloadProgress" with unknown totalData size
as -1 so we update the progress bar with 100% so from user point of view it
looks like hang but it is downloading and writing to file at back ground.
if we apply the same patch in windows then it display running progress bar
so user can know something happening at back end side.

I think it is same as browser - if we download big size file from server
then it keep on downloading the file without displaying total size. What
should we do here ? Should we create one custom progress bar dialog with
moving cursor or something other so that user can know that activity is
doing at backend ? Thoughts ?



> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

Re: [pgAdmin4][patch][runtime]: RM#2829, RM#2491 - pgAdmin4 crashes while saving CSV data from Query tool

2017-11-17 Thread Dave Page 
On Fri, Nov 17, 2017 at 8:45 AM, Neel Patel 
wrote:

> Hi Dave,
>
> On Thu, Nov 16, 2017 at 8:13 PM, Dave Page  wrote:
>
>> Hi
>>
>> On Thu, Nov 16, 2017 at 1:47 PM, Neel Patel 
>> wrote:
>>
>>> Hi,
>>>
>>> Adding information.
>>>
>>> With this patch, RM#2715 should also be resolved.
>>>
>>> Thanks,
>>> Neel Patel
>>>
>>> On Thu, Nov 16, 2017 at 7:01 PM, Neel Patel >> > wrote:
>>>
 Hi,

 I am able to reproduce the crash while downloading and save data in CSV
 file from query tool.

 Please find attached updated patch with below changes after reading Qt
 documentation.

- Added new signal "readyRead". As per the Qt documentation, this
signal will be emitted when data is ready from IO channel for large 
 amount
of data transfer between server and client.
- Ready read and DownloadInProgress signal is very quick in call so
we should not do large operation inside that slot because for 
 downloading
big data it may possible of frequent call of those signals which may 
 cause
the crash or missing data to write inside the file so removed 
 unnecessary
logic from that slot.
- Fixed the crash while opening IODevice with NULL handle.

 With above changes, I have tested with same data as earlier and it is
 working as expected without crashing the application.

 Do review it and let me know for comments.

>>>
>> My first test was on Mac using Qt 5.8 with webkit, and when I attempted
>> to download the results from a 300K row query, it basically hung showing
>> the Downloading File progress indicator. It let me cancel it and carried on
>> working, but hung again the next time I tried the CSV download. Each time
>> it seems to download some data - from the same query I've seen 2.8MB, 5.1MB
>> and 1.5MB.
>>
>
> We are using "*downloadProgress*"  signal to update the progress bar and
> here we are getting "readData" as bytes but "totalData" is unknown so we
> received "totalData" as -1 because Qt doesn't know the total size of the
> file we are getting from server but when we open "pgAdmin4" website in
> another link and try to download some binary or source at that time we are
> getting "totalData" as known size and we are displaying the progress bar
> correctly.
>
> Here, Qt sending the signal "downloadProgress" with unknown totalData size
> as -1 so we update the progress bar with 100% so from user point of view it
> looks like hang but it is downloading and writing to file at back ground.
> if we apply the same patch in windows then it display running progress bar
> so user can know something happening at back end side.
>
> I think it is same as browser - if we download big size file from server
> then it keep on downloading the file without displaying total size. What
> should we do here ? Should we create one custom progress bar dialog with
> moving cursor or something other so that user can know that activity is
> doing at backend ? Thoughts ?
>

It is misleading, but it *is* also hanging. 10+ minutes to download a few
megs of data from a server on the same machine is not just a misleading
progress indicator :-)


-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Next release

2017-11-17 Thread Dave Page 
Hi

The entire point of this patch was to allow an older version of pgAdmin to
run against a newer version of the database without throwing errors like
this. Of course, it'll only work if we're careful to ensure we don't make
any backward-incompatible changes, but adding columns such as this change
does should not cause issues.

On Fri, Nov 17, 2017 at 3:48 AM, Murtuza Zabuawala <
murtuza.zabuaw...@enterprisedb.com> wrote:

> Hi Dave,
>
> There is no programmatic way of fixing the issue if you have already
> deleted the new migration file without downgrading via alembic downgrade
> command.
> The only way I know is to manually update 'alembic_version' table in
> pgAdmin4.db to current revision which is 'ef590e979b0d'
>
> ​-- Murtuza​
>
>
> On Thu, Nov 16, 2017 at 6:58 PM, Dave Page  wrote:
>
>>
>>
>> On Fri, Aug 25, 2017 at 9:34 AM, Dave Page  wrote:
>>
>>>
>>>
>>> On Fri, Aug 25, 2017 at 9:28 AM, Harshal Dhumal <
>>> harshal.dhu...@enterprisedb.com> wrote:
>>>
 One more thing that this will only work for future pgAdmin4 versions

>>>
>>>
>>> Yeah :-(
>>>
>>
>> Hmm, can you check this please? I'm getting the error below when using a
>> newer DB (that I tested an upgrade with) with an older code version:
>>
>> Traceback (most recent call last):
>>   File "/Users/dpage/git/pgadmin4/web/pgAdmin4.py", line 67, in 
>> app = create_app()
>>   File "/Users/dpage/git/pgadmin4/web/pgadmin/__init__.py", line 303, in
>> create_app
>> db_upgrade(app)
>>   File "/Users/dpage/git/pgadmin4/web/pgadmin/setup/db_upgrade.py", line
>> 25, in db_upgrade
>> flask_migrate.upgrade(migration_folder)
>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
>> ges/flask_migrate/__init__.py", line 244, in upgrade
>> command.upgrade(config, revision, sql=sql, tag=tag)
>>   File 
>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/command.py",
>> line 254, in upgrade
>> script.run_env()
>>   File 
>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
>> line 425, in run_env
>> util.load_python_file(self.dir, 'env.py')
>>   File 
>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/util/pyfiles.py",
>> line 81, in load_python_file
>> module = load_module_py(module_id, path)
>>   File 
>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/util/compat.py",
>> line 141, in load_module_py
>> mod = imp.load_source(module_id, path, fp)
>>   File "/Users/dpage/git/pgadmin4/web/pgadmin/setup/../../migrations/env.py",
>> line 94, in 
>> run_migrations_online()
>>   File "/Users/dpage/git/pgadmin4/web/pgadmin/setup/../../migrations/env.py",
>> line 87, in run_migrations_online
>> context.run_migrations()
>>   File "", line 8, in run_migrations
>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
>> ges/alembic/runtime/environment.py", line 836, in run_migrations
>> self.get_context().run_migrations(**kw)
>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
>> ges/alembic/runtime/migration.py", line 321, in run_migrations
>> for step in self._migrations_fn(heads, self):
>>   File 
>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/command.py",
>> line 243, in upgrade
>> return script._upgrade_revs(revision, rev)
>>   File 
>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
>> line 338, in _upgrade_revs
>> for script in reversed(list(revs))
>>   File 
>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py",
>> line 35, in __exit__
>> self.gen.throw(type, value, traceback)
>>   File 
>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
>> line 174, in _catch_revision_errors
>> compat.raise_from_cause(util.CommandError(resolution))
>>   File 
>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/util/compat.py",
>> line 205, in raise_from_cause
>> reraise(type(exception), exception, tb=exc_tb)
>>   File 
>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
>> line 143, in _catch_revision_errors
>> yield
>>   File 
>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
>> line 334, in _upgrade_revs
>> revs = list(revs)
>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
>> ges/alembic/script/revision.py", line 645, in _iterate_revisions
>> requested_lowers = self.get_revisions(lower)
>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
>> ges/alembic/script/revision.py", line 299, in get_revisions
>> return sum([self.get_revisions(id_elem) for id_elem in id_], ())
>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
>> ges/alembic/script/revision.py", line 304, in get_revisions
>> for rev_id in resolved_id)
>>   File "/U

Re: [pgAdmin4][patch][runtime]: RM#2829, RM#2491 - pgAdmin4 crashes while saving CSV data from Query tool

2017-11-17 Thread Neel Patel 
Hi Dave,

On Fri, Nov 17, 2017 at 2:42 PM, Dave Page  wrote:

>
>
> On Fri, Nov 17, 2017 at 8:45 AM, Neel Patel 
> wrote:
>
>> Hi Dave,
>>
>> On Thu, Nov 16, 2017 at 8:13 PM, Dave Page  wrote:
>>
>>> Hi
>>>
>>> On Thu, Nov 16, 2017 at 1:47 PM, Neel Patel >> > wrote:
>>>
 Hi,

 Adding information.

 With this patch, RM#2715 should also be resolved.

 Thanks,
 Neel Patel

 On Thu, Nov 16, 2017 at 7:01 PM, Neel Patel <
 neel.pa...@enterprisedb.com> wrote:

> Hi,
>
> I am able to reproduce the crash while downloading and save data in
> CSV file from query tool.
>
> Please find attached updated patch with below changes after reading Qt
> documentation.
>
>- Added new signal "readyRead". As per the Qt documentation, this
>signal will be emitted when data is ready from IO channel for large 
> amount
>of data transfer between server and client.
>- Ready read and DownloadInProgress signal is very quick in call
>so we should not do large operation inside that slot because for
>downloading big data it may possible of frequent call of those signals
>which may cause the crash or missing data to write inside the file so
>removed unnecessary logic from that slot.
>- Fixed the crash while opening IODevice with NULL handle.
>
> With above changes, I have tested with same data as earlier and it is
> working as expected without crashing the application.
>
> Do review it and let me know for comments.
>

>>> My first test was on Mac using Qt 5.8 with webkit, and when I attempted
>>> to download the results from a 300K row query, it basically hung showing
>>> the Downloading File progress indicator. It let me cancel it and carried on
>>> working, but hung again the next time I tried the CSV download. Each time
>>> it seems to download some data - from the same query I've seen 2.8MB, 5.1MB
>>> and 1.5MB.
>>>
>>
>> We are using "*downloadProgress*"  signal to update the progress bar and
>> here we are getting "readData" as bytes but "totalData" is unknown so we
>> received "totalData" as -1 because Qt doesn't know the total size of the
>> file we are getting from server but when we open "pgAdmin4" website in
>> another link and try to download some binary or source at that time we are
>> getting "totalData" as known size and we are displaying the progress bar
>> correctly.
>>
>> Here, Qt sending the signal "downloadProgress" with unknown totalData
>> size as -1 so we update the progress bar with 100% so from user point of
>> view it looks like hang but it is downloading and writing to file at back
>> ground. if we apply the same patch in windows then it display running
>> progress bar so user can know something happening at back end side.
>>
>> I think it is same as browser - if we download big size file from server
>> then it keep on downloading the file without displaying total size. What
>> should we do here ? Should we create one custom progress bar dialog with
>> moving cursor or something other so that user can know that activity is
>> doing at backend ? Thoughts ?
>>
>
> It is misleading, but it *is* also hanging. 10+ minutes to download a few
> megs of data from a server on the same machine is not just a misleading
> progress indicator :-)
>
Can you please share some table definition with no of rows with ~table size
? I have tried but not able to reproduce the issue.

>
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

Re: [pgAdmin4][patch][runtime]: RM#2829, RM#2491 - pgAdmin4 crashes while saving CSV data from Query tool

2017-11-17 Thread Dave Page 
On Fri, Nov 17, 2017 at 9:36 AM, Neel Patel 
wrote:

> Hi Dave,
>
> On Fri, Nov 17, 2017 at 2:42 PM, Dave Page  wrote:
>
>>
>>
>> On Fri, Nov 17, 2017 at 8:45 AM, Neel Patel 
>> wrote:
>>
>>> Hi Dave,
>>>
>>> On Thu, Nov 16, 2017 at 8:13 PM, Dave Page  wrote:
>>>
 Hi

 On Thu, Nov 16, 2017 at 1:47 PM, Neel Patel <
 neel.pa...@enterprisedb.com> wrote:

> Hi,
>
> Adding information.
>
> With this patch, RM#2715 should also be resolved.
>
> Thanks,
> Neel Patel
>
> On Thu, Nov 16, 2017 at 7:01 PM, Neel Patel <
> neel.pa...@enterprisedb.com> wrote:
>
>> Hi,
>>
>> I am able to reproduce the crash while downloading and save data in
>> CSV file from query tool.
>>
>> Please find attached updated patch with below changes after reading
>> Qt documentation.
>>
>>- Added new signal "readyRead". As per the Qt documentation, this
>>signal will be emitted when data is ready from IO channel for large 
>> amount
>>of data transfer between server and client.
>>- Ready read and DownloadInProgress signal is very quick in call
>>so we should not do large operation inside that slot because for
>>downloading big data it may possible of frequent call of those signals
>>which may cause the crash or missing data to write inside the file so
>>removed unnecessary logic from that slot.
>>- Fixed the crash while opening IODevice with NULL handle.
>>
>> With above changes, I have tested with same data as earlier and it is
>> working as expected without crashing the application.
>>
>> Do review it and let me know for comments.
>>
>
 My first test was on Mac using Qt 5.8 with webkit, and when I attempted
 to download the results from a 300K row query, it basically hung showing
 the Downloading File progress indicator. It let me cancel it and carried on
 working, but hung again the next time I tried the CSV download. Each time
 it seems to download some data - from the same query I've seen 2.8MB, 5.1MB
 and 1.5MB.

>>>
>>> We are using "*downloadProgress*"  signal to update the progress bar
>>> and here we are getting "readData" as bytes but "totalData" is unknown so
>>> we received "totalData" as -1 because Qt doesn't know the total size of the
>>> file we are getting from server but when we open "pgAdmin4" website in
>>> another link and try to download some binary or source at that time we are
>>> getting "totalData" as known size and we are displaying the progress bar
>>> correctly.
>>>
>>> Here, Qt sending the signal "downloadProgress" with unknown totalData
>>> size as -1 so we update the progress bar with 100% so from user point of
>>> view it looks like hang but it is downloading and writing to file at back
>>> ground. if we apply the same patch in windows then it display running
>>> progress bar so user can know something happening at back end side.
>>>
>>> I think it is same as browser - if we download big size file from server
>>> then it keep on downloading the file without displaying total size. What
>>> should we do here ? Should we create one custom progress bar dialog with
>>> moving cursor or something other so that user can know that activity is
>>> doing at backend ? Thoughts ?
>>>
>>
>> It is misleading, but it *is* also hanging. 10+ minutes to download a few
>> megs of data from a server on the same machine is not just a misleading
>> progress indicator :-)
>>
> Can you please share some table definition with no of rows with ~table
> size ? I have tried but not able to reproduce the issue.
>

It was simply (on PG 9.4):

SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid
UNION ALL
SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid
UNION ALL
SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid
UNION ALL
...

Unioned enough times to get a shade under 300K rows.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [pgAdmin4][patch][runtime]: RM#2829, RM#2491 - pgAdmin4 crashes while saving CSV data from Query tool

2017-11-17 Thread Neel Patel 
Hi Dave,

On Fri, Nov 17, 2017 at 3:09 PM, Dave Page  wrote:

>
>
> On Fri, Nov 17, 2017 at 9:36 AM, Neel Patel 
> wrote:
>
>> Hi Dave,
>>
>> On Fri, Nov 17, 2017 at 2:42 PM, Dave Page  wrote:
>>
>>>
>>>
>>> On Fri, Nov 17, 2017 at 8:45 AM, Neel Patel >> > wrote:
>>>
 Hi Dave,

 On Thu, Nov 16, 2017 at 8:13 PM, Dave Page  wrote:

> Hi
>
> On Thu, Nov 16, 2017 at 1:47 PM, Neel Patel <
> neel.pa...@enterprisedb.com> wrote:
>
>> Hi,
>>
>> Adding information.
>>
>> With this patch, RM#2715 should also be resolved.
>>
>> Thanks,
>> Neel Patel
>>
>> On Thu, Nov 16, 2017 at 7:01 PM, Neel Patel <
>> neel.pa...@enterprisedb.com> wrote:
>>
>>> Hi,
>>>
>>> I am able to reproduce the crash while downloading and save data in
>>> CSV file from query tool.
>>>
>>> Please find attached updated patch with below changes after reading
>>> Qt documentation.
>>>
>>>- Added new signal "readyRead". As per the Qt documentation,
>>>this signal will be emitted when data is ready from IO channel for 
>>> large
>>>amount of data transfer between server and client.
>>>- Ready read and DownloadInProgress signal is very quick in call
>>>so we should not do large operation inside that slot because for
>>>downloading big data it may possible of frequent call of those 
>>> signals
>>>which may cause the crash or missing data to write inside the file so
>>>removed unnecessary logic from that slot.
>>>- Fixed the crash while opening IODevice with NULL handle.
>>>
>>> With above changes, I have tested with same data as earlier and it
>>> is working as expected without crashing the application.
>>>
>>> Do review it and let me know for comments.
>>>
>>
> My first test was on Mac using Qt 5.8 with webkit, and when I
> attempted to download the results from a 300K row query, it basically hung
> showing the Downloading File progress indicator. It let me cancel it and
> carried on working, but hung again the next time I tried the CSV download.
> Each time it seems to download some data - from the same query I've seen
> 2.8MB, 5.1MB and 1.5MB.
>

 We are using "*downloadProgress*"  signal to update the progress bar
 and here we are getting "readData" as bytes but "totalData" is unknown so
 we received "totalData" as -1 because Qt doesn't know the total size of the
 file we are getting from server but when we open "pgAdmin4" website in
 another link and try to download some binary or source at that time we are
 getting "totalData" as known size and we are displaying the progress bar
 correctly.

 Here, Qt sending the signal "downloadProgress" with unknown totalData
 size as -1 so we update the progress bar with 100% so from user point of
 view it looks like hang but it is downloading and writing to file at back
 ground. if we apply the same patch in windows then it display running
 progress bar so user can know something happening at back end side.

 I think it is same as browser - if we download big size file from
 server then it keep on downloading the file without displaying total size.
 What should we do here ? Should we create one custom progress bar dialog
 with moving cursor or something other so that user can know that activity
 is doing at backend ? Thoughts ?

>>>
>>> It is misleading, but it *is* also hanging. 10+ minutes to download a
>>> few megs of data from a server on the same machine is not just a misleading
>>> progress indicator :-)
>>>
>> Can you please share some table definition with no of rows with ~table
>> size ? I have tried but not able to reproduce the issue.
>>
>
> It was simply (on PG 9.4):
>
> SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid
> UNION ALL
> SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid
> UNION ALL
> SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid
> UNION ALL
> ...
>
> Unioned enough times to get a shade under 300K rows.
>

Thanks. I will check and keep you updated.

>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

Re: Next release

2017-11-17 Thread Surinder Kumar 
Hi

On Fri, Nov 17, 2017 at 2:46 PM, Dave Page  wrote:

> Hi
>
> The entire point of this patch was to allow an older version of pgAdmin to
> run against a newer version of the database without throwing errors like
> this. Of course, it'll only work if we're careful to ensure we don't make
> any backward-incompatible changes, but adding columns such as this change
> does should not cause issues.
>
Since this patch was commited in pgAdmin4.2.0 where it executes
`db_upgrade(app)`
​ method defined​
in `pgadmin/__init__.py​`
​if and only if current schema version is greater than schema version entry
present in version table. Now this code(to check version) is not present in
1.5v and it always goes to run ' db_upgrade(app)' and thus it fails.

The solution is to comment the line ​
`db_upgrade(app)`
​ in ​o
lder version of pgAdmin so that it will not run migration against newer
version of database and
thus the newer database will work with older code.

>
>
> On Fri, Nov 17, 2017 at 3:48 AM, Murtuza Zabuawala <
> murtuza.zabuaw...@enterprisedb.com> wrote:
>
>> Hi Dave,
>>
>> There is no programmatic way of fixing the issue if you have already
>> deleted the new migration file without downgrading via alembic downgrade
>> command.
>> The only way I know is to manually update 'alembic_version' table in
>> pgAdmin4.db to current revision which is 'ef590e979b0d'
>>
>> ​-- Murtuza​
>>
>>
>> On Thu, Nov 16, 2017 at 6:58 PM, Dave Page  wrote:
>>
>>>
>>>
>>> On Fri, Aug 25, 2017 at 9:34 AM, Dave Page  wrote:
>>>


 On Fri, Aug 25, 2017 at 9:28 AM, Harshal Dhumal <
 harshal.dhu...@enterprisedb.com> wrote:

> One more thing that this will only work for future pgAdmin4 versions
>


 Yeah :-(

>>>
>>> Hmm, can you check this please? I'm getting the error below when using a
>>> newer DB (that I tested an upgrade with) with an older code version:
>>>
>>> Traceback (most recent call last):
>>>   File "/Users/dpage/git/pgadmin4/web/pgAdmin4.py", line 67, in 
>>> app = create_app()
>>>   File "/Users/dpage/git/pgadmin4/web/pgadmin/__init__.py", line 303,
>>> in create_app
>>> db_upgrade(app)
>>>   File "/Users/dpage/git/pgadmin4/web/pgadmin/setup/db_upgrade.py",
>>> line 25, in db_upgrade
>>> flask_migrate.upgrade(migration_folder)
>>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
>>> ges/flask_migrate/__init__.py", line 244, in upgrade
>>> command.upgrade(config, revision, sql=sql, tag=tag)
>>>   File 
>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/command.py",
>>> line 254, in upgrade
>>> script.run_env()
>>>   File 
>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
>>> line 425, in run_env
>>> util.load_python_file(self.dir, 'env.py')
>>>   File 
>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/util/pyfiles.py",
>>> line 81, in load_python_file
>>> module = load_module_py(module_id, path)
>>>   File 
>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/util/compat.py",
>>> line 141, in load_module_py
>>> mod = imp.load_source(module_id, path, fp)
>>>   File 
>>> "/Users/dpage/git/pgadmin4/web/pgadmin/setup/../../migrations/env.py",
>>> line 94, in 
>>> run_migrations_online()
>>>   File 
>>> "/Users/dpage/git/pgadmin4/web/pgadmin/setup/../../migrations/env.py",
>>> line 87, in run_migrations_online
>>> context.run_migrations()
>>>   File "", line 8, in run_migrations
>>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
>>> ges/alembic/runtime/environment.py", line 836, in run_migrations
>>> self.get_context().run_migrations(**kw)
>>>   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
>>> ges/alembic/runtime/migration.py", line 321, in run_migrations
>>> for step in self._migrations_fn(heads, self):
>>>   File 
>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/command.py",
>>> line 243, in upgrade
>>> return script._upgrade_revs(revision, rev)
>>>   File 
>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
>>> line 338, in _upgrade_revs
>>> for script in reversed(list(revs))
>>>   File 
>>> "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py",
>>> line 35, in __exit__
>>> self.gen.throw(type, value, traceback)
>>>   File 
>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
>>> line 174, in _catch_revision_errors
>>> compat.raise_from_cause(util.CommandError(resolution))
>>>   File 
>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/util/compat.py",
>>> line 205, in raise_from_cause
>>> reraise(type(exception), exception, tb=exc_tb)
>>>   File 
>>> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
>>> line 143, in _catch_revision_errors
>>>

Re: Next release

2017-11-17 Thread Dave Page 
On Fri, Nov 17, 2017 at 10:04 AM, Surinder Kumar <
surinder.ku...@enterprisedb.com> wrote:

> Hi
>
> On Fri, Nov 17, 2017 at 2:46 PM, Dave Page  wrote:
>
>> Hi
>>
>> The entire point of this patch was to allow an older version of pgAdmin
>> to run against a newer version of the database without throwing errors like
>> this. Of course, it'll only work if we're careful to ensure we don't make
>> any backward-incompatible changes, but adding columns such as this change
>> does should not cause issues.
>>
> Since this patch was commited in pgAdmin4.2.0 where it executes
> `db_upgrade(app)`
> ​ method defined​
> in `pgadmin/__init__.py​`
> ​if and only if current schema version is greater than schema version
> entry present in version table. Now this code(to check version) is not
> present in 1.5v and it always goes to run ' db_upgrade(app)' and thus it
> fails.
>
> The solution is to comment the line ​
> `db_upgrade(app)`
> ​ in ​o
> lder version of pgAdmin so that it will not run migration against newer
> version of database and
> thus the newer database will work with older code.
>

I'm not using pgAdmin < 2.0. I was working from head, tested the patch,
then reverted it.



>
>>
>> On Fri, Nov 17, 2017 at 3:48 AM, Murtuza Zabuawala <
>> murtuza.zabuaw...@enterprisedb.com> wrote:
>>
>>> Hi Dave,
>>>
>>> There is no programmatic way of fixing the issue if you have already
>>> deleted the new migration file without downgrading via alembic downgrade
>>> command.
>>> The only way I know is to manually update 'alembic_version' table in
>>> pgAdmin4.db to current revision which is 'ef590e979b0d'
>>>
>>> ​-- Murtuza​
>>>
>>>
>>> On Thu, Nov 16, 2017 at 6:58 PM, Dave Page  wrote:
>>>


 On Fri, Aug 25, 2017 at 9:34 AM, Dave Page  wrote:

>
>
> On Fri, Aug 25, 2017 at 9:28 AM, Harshal Dhumal <
> harshal.dhu...@enterprisedb.com> wrote:
>
>> One more thing that this will only work for future pgAdmin4 versions
>>
>
>
> Yeah :-(
>

 Hmm, can you check this please? I'm getting the error below when using
 a newer DB (that I tested an upgrade with) with an older code version:

 Traceback (most recent call last):
   File "/Users/dpage/git/pgadmin4/web/pgAdmin4.py", line 67, in
 
 app = create_app()
   File "/Users/dpage/git/pgadmin4/web/pgadmin/__init__.py", line 303,
 in create_app
 db_upgrade(app)
   File "/Users/dpage/git/pgadmin4/web/pgadmin/setup/db_upgrade.py",
 line 25, in db_upgrade
 flask_migrate.upgrade(migration_folder)
   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
 ges/flask_migrate/__init__.py", line 244, in upgrade
 command.upgrade(config, revision, sql=sql, tag=tag)
   File 
 "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/command.py",
 line 254, in upgrade
 script.run_env()
   File 
 "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
 line 425, in run_env
 util.load_python_file(self.dir, 'env.py')
   File 
 "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/util/pyfiles.py",
 line 81, in load_python_file
 module = load_module_py(module_id, path)
   File 
 "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/util/compat.py",
 line 141, in load_module_py
 mod = imp.load_source(module_id, path, fp)
   File 
 "/Users/dpage/git/pgadmin4/web/pgadmin/setup/../../migrations/env.py",
 line 94, in 
 run_migrations_online()
   File 
 "/Users/dpage/git/pgadmin4/web/pgadmin/setup/../../migrations/env.py",
 line 87, in run_migrations_online
 context.run_migrations()
   File "", line 8, in run_migrations
   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
 ges/alembic/runtime/environment.py", line 836, in run_migrations
 self.get_context().run_migrations(**kw)
   File "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packa
 ges/alembic/runtime/migration.py", line 321, in run_migrations
 for step in self._migrations_fn(heads, self):
   File 
 "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/command.py",
 line 243, in upgrade
 return script._upgrade_revs(revision, rev)
   File 
 "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
 line 338, in _upgrade_revs
 for script in reversed(list(revs))
   File 
 "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py",
 line 35, in __exit__
 self.gen.throw(type, value, traceback)
   File 
 "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/alembic/script/base.py",
 line 174, in _catch_revision_errors
 compat.raise_from_cause(util.CommandError(resolution))
   File

Re: [pgAdmin4][Patch]: To make error message uniform for Create schema action

2017-11-17 Thread Murtuza Zabuawala 
Hi,

PFA updated.

On Thu, Nov 16, 2017 at 5:12 PM, Dave Page  wrote:

>
>
> On Tue, Nov 14, 2017 at 5:56 PM, Murtuza Zabuawala <
> murtuza.zabuaw...@enterprisedb.com> wrote:
>
>> Hi,
>>
>> PFA minor patch to make error message uniform for 'Create schema' in both
>> query tool and create schema dialog.
>> RM#2094
>>
>
> The error message is definitely an improvement, but the dialogue title
> needs work:
>
> Error saving properties: GONE
>
> Why does it have : GONE on the end? That seems unhelpful.
>
​That's coming from ajax statusText
, I
thought it was intended.

>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
diff --git 
a/web/pgadmin/browser/server_groups/servers/databases/schemas/__init__.py 
b/web/pgadmin/browser/server_groups/servers/databases/schemas/__init__.py
index 04d2432..49c953c 100644
--- a/web/pgadmin/browser/server_groups/servers/databases/schemas/__init__.py
+++ b/web/pgadmin/browser/server_groups/servers/databases/schemas/__init__.py
@@ -580,8 +580,7 @@ It may have been removed by another user.
 return make_json_response(
 status=410,
 success=0,
-errormsg=res + '\n' +
- 'Operation failed while running create statement'
+errormsg=res
 )
 
 # we need oid to to add object in tree at browser,
diff --git a/web/pgadmin/browser/static/js/node.js 
b/web/pgadmin/browser/static/js/node.js
index 9a5b894..2beadf9 100644
--- a/web/pgadmin/browser/static/js/node.js
+++ b/web/pgadmin/browser/static/js/node.js
@@ -1153,10 +1153,8 @@ define(
   error: function(m, jqxhr) {
 Alertify.pgNotifier(
   "error", jqxhr,
-  S(
-gettext("Error saving properties: %s")
-).sprintf(jqxhr.statusText).value()
-  );
+  gettext("Error saving properties")
+);
 
 // Hide progress cursor
 $('.obj_properties').removeClass('show_progress');
diff --git a/web/pgadmin/tools/grant_wizard/static/js/grant_wizard.js 
b/web/pgadmin/tools/grant_wizard/static/js/grant_wizard.js
index 61b0eda..4898f7f 100644
--- a/web/pgadmin/tools/grant_wizard/static/js/grant_wizard.js
+++ b/web/pgadmin/tools/grant_wizard/static/js/grant_wizard.js
@@ -1089,10 +1089,8 @@ define([
 error: function(m, jqxhr) {
   alertify.pgNotifier(
 "error", jqxhr,
-S(
-  gettext("Error saving properties: %s")
-  ).sprintf(jqxhr.statusText).value()
-);
+gettext("Error saving properties")
+  );
 
 // Release wizard objects
 self.releaseObjects();

[Patch]: Installer: pgAdmin4 should not install in special character path

2017-11-17 Thread Paresh More 
Hello Team,

I have made three changes for installer generation scripts.

1) Modified Make-MinGW.bat to support yarn and nodejs validation in the
script and also QT 5.9.1 support.

2) Bug fix for RM1365.
Currently string datatypes used in Inno tool does not support regular
expression, so added a user function which will do path validation and it
would be work fine for inno unicode setup.

3) Restructured  installer script so that it should support unicode inno
and for RM1365 as well i.e Inno Setup Installer(unicode) also updated
README file for the same.


Please note - Installer path validation is invoked two times

1) Inno tool internally validate's path which user program do not have
access to it , so section [MESSAGE] error string message will be used to
show error message string.

2) From the code section, so section [CustomMessages] error message will be
used.

For both (1) and (2) common error message will be used.


Attached is the patch


-- 

Thanks & Regards

*Paresh More*

[image: NEW-EDB-logo-4c]

Pune, India.
www.enterprisedb.com


RM1365.patch
Description: Binary data

[pgAdmin4][Patch]: Add support for time without time zone

2017-11-17 Thread Murtuza Zabuawala 
Hi,

PFA minor patch to add support for time without time zone.
RM#2881

--
Regards,
Murtuza Zabuawala
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
diff --git a/web/pgadmin/utils/driver/psycopg2/__init__.py 
b/web/pgadmin/utils/driver/psycopg2/__init__.py
index eabe3b0..2027aca 100644
--- a/web/pgadmin/utils/driver/psycopg2/__init__.py
+++ b/web/pgadmin/utils/driver/psycopg2/__init__.py
@@ -81,8 +81,8 @@ pg_types_to_string_type = psycopg2.extensions.new_type(
 # date, timestamp, timestamptz, bigint, double precision
 1700, 1082, 1114, 1184, 20, 701,
 
-# real
-700
+# real, time without time zone
+700, 1083, 1183
 ),
 'TYPECAST_TO_STRING', psycopg2.STRING
 )

Re: [pgAdmin4][patch][runtime]: RM#2829, RM#2491 - pgAdmin4 crashes while saving CSV data from Query tool

2017-11-17 Thread Neel Patel 
Hi Dave,

I have tried for 350K rows with query given by you but not able to
reproduce the hang issue.
I have put some debug statement in Qt Signal's handler. Can you please
apply this debug patch and provide me console output if possible ?

Thanks,
Neel Patel

On Fri, Nov 17, 2017 at 3:13 PM, Neel Patel 
wrote:

> Hi Dave,
>
> On Fri, Nov 17, 2017 at 3:09 PM, Dave Page  wrote:
>
>>
>>
>> On Fri, Nov 17, 2017 at 9:36 AM, Neel Patel 
>> wrote:
>>
>>> Hi Dave,
>>>
>>> On Fri, Nov 17, 2017 at 2:42 PM, Dave Page  wrote:
>>>


 On Fri, Nov 17, 2017 at 8:45 AM, Neel Patel <
 neel.pa...@enterprisedb.com> wrote:

> Hi Dave,
>
> On Thu, Nov 16, 2017 at 8:13 PM, Dave Page  wrote:
>
>> Hi
>>
>> On Thu, Nov 16, 2017 at 1:47 PM, Neel Patel <
>> neel.pa...@enterprisedb.com> wrote:
>>
>>> Hi,
>>>
>>> Adding information.
>>>
>>> With this patch, RM#2715 should also be resolved.
>>>
>>> Thanks,
>>> Neel Patel
>>>
>>> On Thu, Nov 16, 2017 at 7:01 PM, Neel Patel <
>>> neel.pa...@enterprisedb.com> wrote:
>>>
 Hi,

 I am able to reproduce the crash while downloading and save data in
 CSV file from query tool.

 Please find attached updated patch with below changes after reading
 Qt documentation.

- Added new signal "readyRead". As per the Qt documentation,
this signal will be emitted when data is ready from IO channel for 
 large
amount of data transfer between server and client.
- Ready read and DownloadInProgress signal is very quick in
call so we should not do large operation inside that slot because 
 for
downloading big data it may possible of frequent call of those 
 signals
which may cause the crash or missing data to write inside the file 
 so
removed unnecessary logic from that slot.
- Fixed the crash while opening IODevice with NULL handle.

 With above changes, I have tested with same data as earlier and it
 is working as expected without crashing the application.

 Do review it and let me know for comments.

>>>
>> My first test was on Mac using Qt 5.8 with webkit, and when I
>> attempted to download the results from a 300K row query, it basically 
>> hung
>> showing the Downloading File progress indicator. It let me cancel it and
>> carried on working, but hung again the next time I tried the CSV 
>> download.
>> Each time it seems to download some data - from the same query I've seen
>> 2.8MB, 5.1MB and 1.5MB.
>>
>
> We are using "*downloadProgress*"  signal to update the progress bar
> and here we are getting "readData" as bytes but "totalData" is unknown so
> we received "totalData" as -1 because Qt doesn't know the total size of 
> the
> file we are getting from server but when we open "pgAdmin4" website in
> another link and try to download some binary or source at that time we are
> getting "totalData" as known size and we are displaying the progress bar
> correctly.
>
> Here, Qt sending the signal "downloadProgress" with unknown totalData
> size as -1 so we update the progress bar with 100% so from user point of
> view it looks like hang but it is downloading and writing to file at back
> ground. if we apply the same patch in windows then it display running
> progress bar so user can know something happening at back end side.
>
> I think it is same as browser - if we download big size file from
> server then it keep on downloading the file without displaying total size.
> What should we do here ? Should we create one custom progress bar dialog
> with moving cursor or something other so that user can know that activity
> is doing at backend ? Thoughts ?
>

 It is misleading, but it *is* also hanging. 10+ minutes to download a
 few megs of data from a server on the same machine is not just a misleading
 progress indicator :-)

>>> Can you please share some table definition with no of rows with ~table
>>> size ? I have tried but not able to reproduce the issue.
>>>
>>
>> It was simply (on PG 9.4):
>>
>> SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid
>> UNION ALL
>> SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid
>> UNION ALL
>> SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid
>> UNION ALL
>> ...
>>
>> Unioned enough times to get a shade under 300K rows.
>>
>
> Thanks. I will check and keep you updated.
>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>
>


CSV_Download_debug.patch
Description: Binary data