And yes, it is now sync'd. Thanks again!
Brian On Fri, May 15, 2020 at 2:55 PM Brian Dickson <[email protected]> wrote: > Yes, it's still running, but I'm not getting the error messages I was > seeing before. > I expect it to work completely now. > > Thanks, everyone, especially Milton! > > (Also, you may want to approve his patch. :-) ) > > Brian > > On Fri, May 15, 2020 at 2:42 PM Chris Caputo <[email protected]> wrote: > >> Any better with Milton's 'self.start_time = datetime.utcnow()' change? >> >> Thanks, >> Chris >> >> On Fri, 15 May 2020, Brian Dickson wrote: >> > On Fri, May 15, 2020 at 12:57 PM Brian Dickson < >> [email protected]> wrote: >> > The local database is empty. I'll make sure that's the case just >> to be safe. >> > >> > # peeringdb --version >> > >> > peeringdb 1.0.0 >> > >> > >> > # django-admin --version >> > >> > 2.2.12 >> > >> > >> > # pip freeze >> > >> > asgiref==3.2.7 >> > >> > certifi==2020.4.5.1 >> > >> > cfu==1.5.0 >> > >> > chardet==3.0.4 >> > >> > click==7.1.2 >> > >> > Django==2.2.12 >> > >> > django-countries==6.1.2 >> > >> > django-handleref==0.5.0 >> > >> > django-inet==0.4.0 >> > >> > django-peeringdb==2.0.0 >> > >> > future==0.18.2 >> > >> > idna==2.9 >> > >> > munge==1.0.0 >> > >> > mysqlclient==1.4.4 >> > >> > peeringdb==1.0.0 >> > >> > PyMySQL==0.9.3 >> > >> > pytz==2020.1 >> > >> > PyYAML==5.3.1 >> > >> > requests==2.23.0 >> > >> > six==1.14.0 >> > >> > sqlparse==0.3.1 >> > >> > twentyc.rpc==0.4.0 >> > >> > urllib3==1.25.9 >> > >> > >> > >> > Here is the result when running the sync on a clean empty local >> database: >> > >> > bash-3.2# peeringdb drop-tables >> > >> > bash-3.2# peeringdb sync >> > >> > Syncing to https://www.peeringdb.com/api >> > >> > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc >> > >> > Fetching & updating all: org >> > >> > Updates to be processed: 18439 >> > >> > Ignoring object updated after sync began: (org-6864) >> > >> > Ignoring object updated after sync began: (org-13687) >> > >> > Ignoring object updated after sync began: (org-23582) >> > >> > Ignoring object updated after sync began: (org-26092) >> > >> > Ignoring object updated after sync began: (org-26094) >> > >> > Ignoring object updated after sync began: (org-26095) >> > >> > Ignoring object updated after sync began: (org-26096) >> > >> > Ignoring object updated after sync began: (org-26097) >> > >> > Ignoring object updated after sync began: (org-26098) >> > >> > Fetching & updating all: fac >> > >> > Updates to be processed: 3714 >> > >> > Ignoring object updated after sync began: (fac-3634) >> > >> > Ignoring object updated after sync began: (fac-8262) >> > >> > Ignoring object updated after sync began: (fac-8582) >> > >> > Ignoring object updated after sync began: (fac-8583) >> > >> > Ignoring object updated after sync began: (fac-8584) >> > >> > Ignoring object updated after sync began: (fac-8585) >> > >> > Ignoring object updated after sync began: (fac-8586) >> > >> > Ignoring object updated after sync began: (fac-8587) >> > >> > Ignoring object updated after sync began: (fac-8588) >> > >> > Ignoring object updated after sync began: (fac-8591) >> > >> > Ignoring object updated after sync began: (fac-8592) >> > >> > Ignoring object updated after sync began: (fac-8593) >> > >> > Ignoring object updated after sync began: (fac-8594) >> > >> > Ignoring object updated after sync began: (fac-8595) >> > >> > Ignoring object updated after sync began: (fac-8597) >> > >> > Ignoring object updated after sync began: (fac-8598) >> > >> > Ignoring object updated after sync began: (fac-8599) >> > >> > Ignoring object updated after sync began: (fac-8600) >> > >> > Ignoring object updated after sync began: (fac-8601) >> > >> > Ignoring object updated after sync began: (fac-8602) >> > >> > Ignoring object updated after sync began: (fac-8603) >> > >> > Fetching & updating all: net >> > >> > Updates to be processed: 18845 >> > >> > Ignoring object updated after sync began: (net-2913) >> > >> > Ignoring object updated after sync began: (net-3087) >> > >> > Ignoring object updated after sync began: (net-4823) >> > >> > Ignoring object updated after sync began: (net-5442) >> > >> > Ignoring object updated after sync began: (net-7482) >> > >> > Ignoring object updated after sync began: (net-9435) >> > >> > Ignoring object updated after sync began: (net-13681) >> > >> > Ignoring object updated after sync began: (net-14816) >> > >> > Ignoring object updated after sync began: (net-19500) >> > >> > Ignoring object updated after sync began: (net-20575) >> > >> > Ignoring object updated after sync began: (net-21292) >> > >> > Ignoring object updated after sync began: (net-21916) >> > >> > Ignoring object updated after sync began: (net-23222) >> > >> > Ignoring object updated after sync began: (net-23266) >> > >> > Ignoring object updated after sync began: (net-23280) >> > >> > Ignoring object updated after sync began: (net-23283) >> > >> > Ignoring object updated after sync began: (net-23284) >> > >> > Ignoring object updated after sync began: (net-23285) >> > >> > Traceback (most recent call last): >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", >> line 8, in <module> >> > >> > sys.exit(main()) >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", >> line 62, in main >> > >> > return handler(config=cfg, **vars(options)) >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", >> line 20, in _wrapped >> > >> > r = func(*a, **k) >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", >> line 219, in handle >> > >> > client.update_all(rs) >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", >> line 66, in update_all >> > >> > self._atomic_update(lambda: ctx.sync_resource(r, since=since)) >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", >> line 78, in _atomic_update >> > >> > sync_func() >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", >> line 66, in <lambda> >> > >> > self._atomic_update(lambda: ctx.sync_resource(r, since=since)) >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", >> line 65, in _wrapped >> > >> > return loop.run_until_complete(func(*a, **k)) >> > >> > File >> "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", >> line 488, in run_until_complete >> > >> > return future.result() >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", >> line 41, in _wrapped >> > >> > item = gen.send(r) >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", >> line 287, in sync_row >> > >> > B.clean(obj) >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend.py", >> line 145, in clean >> > >> > obj.full_clean() >> > >> > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", >> line 1203, in full_clean >> > >> > raise ValidationError(errors) >> > >> > django.core.exceptions.ValidationError: {'org': ['organization instance >> with id 13687 does not exist.']} >> > >> > >> > >> > On Fri, May 15, 2020 at 10:55 AM Chris Caputo <[email protected]> >> wrote: >> > Hi. Does restarting with a fresh/empty local database help at >> all? >> > >> > Also, what does the following show?: >> > >> > peeringdb --version >> > django-admin --version >> > pip freeze >> > >> > Thanks, >> > Chris >> > >> > On Fri, 15 May 2020, Brian Dickson wrote: >> > > Arnold Nipper said to send to this list. >> > > >> > > (This is kind of urgent for me, my goal is to get a snapshot of >> the data, so any alternative workaround would be helpful in the >> > > meantime.) >> > > >> > > I'm in the process of setting up a local copy, and have run >> into a problem after getting the peeringdb-py stuff set up. >> > > >> > > When I do peeringdb sync (with the correct mysql database UTF8 >> stuff and everything,), I am currently getting a persistent error >> > > which aborts the sync with no data in the tables: >> > > >> > > (Apologies in advance for the long error output text.) >> > > Syncing to https://www.peeringdb.com/api >> > > Updating resources: org fac net ix ixfac ixlan ixpfx netfac >> netixlan poc >> > > Fetching & updating all: org >> > > Updates to be processed: 5 >> > > Ignoring object updated after sync began: (org-26015) >> > > Ignoring object updated after sync began: (org-26052) >> > > Ignoring object updated after sync began: (org-17918) >> > > Ignoring object updated after sync began: (org-26053) >> > > Fetching & updating all: fac >> > > Updates to be processed: 0 >> > > Fetching & updating all: net >> > > Updates to be processed: 18800 >> > > Ignoring object updated after sync began: (net-1356) >> > > Ignoring object updated after sync began: (net-3684) >> > > Ignoring object updated after sync began: (net-7924) >> > > Ignoring object updated after sync began: (net-10733) >> > > Ignoring object updated after sync began: (net-13084) >> > > Ignoring object updated after sync began: (net-14581) >> > > Ignoring object updated after sync began: (net-15702) >> > > Traceback (most recent call last): >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", >> > > line 84, in _execute >> > > return self.cursor.execute(sql, params) >> > > File >> > > >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", >> line >> > > 71, in execute >> > > return self.cursor.execute(query, args) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", >> line 209, in >> > > execute >> > > res = self._query(query) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", >> line 315, in >> > > _query >> > > db.query(q) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py", >> line >> > > 226, in query >> > > _mysql.connection.query(self, query) >> > > MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or >> update a child row: a foreign key constraint fails >> > > (`peeringdb`.`peeringdb_network`, CONSTRAINT >> `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY >> > > (`org_id`) REFERENCES `peeringdb_organization` (`id`))') >> > > >> > > The above exception was the direct cause of the following >> exception: >> > > >> > > Traceback (most recent call last): >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", >> line 8, in <module> >> > > sys.exit(main()) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py", >> line 62, in >> > > main >> > > return handler(config=cfg, **vars(options)) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", >> line 20, >> > > in _wrapped >> > > r = func(*a, **k) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py", >> line 219, >> > > in handle >> > > client.update_all(rs) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", >> line 66, >> > > in update_all >> > > self._atomic_update(lambda: ctx.sync_resource(r, >> since=since)) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", >> line 78, >> > > in _atomic_update >> > > sync_func() >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", >> line 66, >> > > in <lambda> >> > > self._atomic_update(lambda: ctx.sync_resource(r, >> since=since)) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", >> line >> > > 65, in _wrapped >> > > return loop.run_until_complete(func(*a, **k)) >> > > File >> "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", >> line 488, in >> > > run_until_complete >> > > return future.result() >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py", >> line >> > > 41, in _wrapped >> > > item = gen.send(r) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py", >> line 274, >> > > in sync_row >> > > B.save(obj) >> > > >> File"/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend >> > > .py", line 148, in save >> > > obj.save() >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", >> line >> > > 741, in save >> > > force_update=force_update, update_fields=update_fields) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", >> line >> > > 779, in save_base >> > > force_update, using, update_fields, >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", >> line >> > > 870, in _save_table >> > > result = self._do_insert(cls._base_manager, using, fields, >> update_pk, raw) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py", >> line >> > > 908, in _do_insert >> > > using=using, raw=raw) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/manager.py", >> > > line 82, in manager_method >> > > return getattr(self.get_queryset(), name)(*args, **kwargs) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/query.py", >> line >> > > 1186, in _insert >> > > return >> query.get_compiler(using=using).execute_sql(return_id) >> > > File >> > > >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/sql/compiler.py", >> line >> > > 1375, in execute_sql >> > > cursor.execute(sql, params) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", >> > > line 67, in execute >> > > return self._execute_with_wrappers(sql, params, many=False, >> executor=self._execute) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", >> > > line 76, in _execute_with_wrappers >> > > return executor(sql, params, many, context) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", >> > > line 84, in _execute >> > > return self.cursor.execute(sql, params) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/utils.py", >> line 89, in >> > > __exit__ >> > > raise dj_exc_value.with_traceback(traceback) from exc_value >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py", >> > > line 84, in _execute >> > > return self.cursor.execute(sql, params) >> > > File >> > > >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py", >> line >> > > 71, in execute >> > > return self.cursor.execute(query, args) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", >> line 209, in >> > > execute >> > > res = self._query(query) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py", >> line 315, in >> > > _query >> > > db.query(q) >> > > File >> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py", >> line >> > > 226, in query >> > > _mysql.connection.query(self, query) >> > > django.db.utils.IntegrityError: (1452, 'Cannot add or update a >> child row: a foreign key constraint fails >> > > (`peeringdb`.`peeringdb_network`, CONSTRAINT >> `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY >> > > (`org_id`) REFERENCES `peeringdb_organization` (`id`))') >> > > >> > > Please let me know if this is a problem on my end, or a problem >> with the data in the master db itself... >> > > >> > > If there is something I should do to fix this on my end, any >> advice would be greatly appreciated. >> > > (If there is any other way to grab a copy of the tables in >> mysql format, that's all I'm really trying to get, just a snapshot of >> > > the current peeringdb database to process with some perl >> scripts I have.) >> > > >> > > Thanks in advance, >> > > >> > > Brian Dickson >> > > >> > > >> > > >> > >> > >> > > >
_______________________________________________ Pdb-tech mailing list [email protected] https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
