Hi all,
in my project, I'm traversing through a huge directory with millions of files,
parse them and write the results into a Postgres database.
Unfortunately, I'm facing erros when I'm using multiprocessing.Pool with more
than one processes.
This is how I'm inserting new values into the database:
obj, created = MyModel.objects.get_or_create(foo=foo, defaults={'foo': foo,}) #
foo is unique
And this is the error message I'm getting:
[ERROR] no results to fetch
Traceback (most recent call last):
File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py",
line 97, in inner
return func(*args, **kwargs)
psycopg2.ProgrammingError: no results to fetch
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "~/develop/myApp/web/service/ais.py", line 94, in save
db_foo = utils.create_foo(obj.get("foo"), obj.get("source"))
File "~/develop/myApp/web/utils.py", line 85, in create_foo
obj = MyModel.objects.get(foo=foo)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py",
line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py",
line 411, in get
num = len(clone)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py",
line 258, in __len__
self._fetch_all()
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py",
line 1261, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py",
line 57, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch,
chunk_size=self.chunk_size)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py",
line 1177, in execute_sql
return list(result)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py",
line 1576, in cursor_iter
for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py",
line 1576, in <lambda>
for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py",
line 97, in inner
return func(*args, **kwargs)
File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py",
line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py",
line 97, in inner
return func(*args, **kwargs)
django.db.utils.ProgrammingError: no results to fetch
[ERROR] error with status PGRES_TUPLES_OK and no message from the libpq
Traceback (most recent call last):
File "~/develop/myApp/web/utils.py", line 85, in create_foo
obj = MyModel.objects.get(foo=foo)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py",
line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py",
line 415, in get
raise self.model.DoesNotExist(
web.models.MyModel.DoesNotExist: MyModel matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py",
line 86, in _execute
return self.cursor.execute(sql, params)
psycopg2.DatabaseError: error with status PGRES_TUPLES_OK and no message from
the libpq
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "~/develop/myApp/web/service/ais.py", line 94, in save
db_foo = utils.create_foo(obj.get("foo"), obj.get("source"))
File "~/develop/myApp/web/utils.py", line 89, in create_foo
obj = MyModel.objects.create(foo=foo)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py",
line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py",
line 433, in create
obj.save(force_insert=True, using=self.db)
File "~/develop/myApp/web/models.py", line 34, in save
return super().save(*args, **kwargs)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py",
line 745, in save
self.save_base(using=using, force_insert=force_insert,
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py",
line 782, in save_base
updated = self._save_table(
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py",
line 887, in _save_table
results = self._do_insert(cls._base_manager, using, fields,
returning_fields, raw)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py",
line 924, in _do_insert
return manager._insert(
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py",
line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py",
line 1204, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py",
line 1384, in execute_sql
cursor.execute(sql, params)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py",
line 68, in execute
return self._execute_with_wrappers(sql, params, many=False,
executor=self._execute)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py",
line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py",
line 86, in _execute
return self.cursor.execute(sql, params)
File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py",
line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File
"~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py",
line 86, in _execute
return self.cursor.execute(sql, params)
django.db.utils.DatabaseError: error with status PGRES_TUPLES_OK and no message
from the libpq
---
Some additional info:
- This does not happen with sqlite
- I'm pretty sure it has something to do with multithreading, since I
don't get this error with Pool(processes=1) and the higher the
processes-parameter
is, the more often does this error occur.
- Versions:
- Django==3.0.3
- psycopg2-binary==2.8.4
Is there any chance I can get this working with ~8 threads? Is there
something wrong with how I insert the data?
Unfortunately, I have several models that depend on each other (although I
get this error with just inserting the first - didn't even get to the other
models), so bulk-insert is not an option.
>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/d32a2dfd-4d27-43a0-9f2d-446e1d6302aa%40googlegroups.com.
