I have finally re-run my entire automated migration process from start to finish.
After running condensediffs *before* the upgrade, I was then able to run condensediffs after the upgrade as well, all without any warnings or errors. Perhaps you should add a step to the upgrade guide, advising Administrators to rebuild their search index, and also condense diffs? So I seem to have found a work-around to the problem - in case anyone else runs into it. Eric. On Saturday, May 14, 2016 at 1:40:06 PM UTC-7, [email protected] wrote: > > I re-ran the migration, and before doing anything, set the DEBUG flag to > True before fetching any diffs. > > I can confirm that it fails on *every* diff, not just random ones here and > there. > > Running with DEBUG = False, the warning doesn't throw an exception, but I > do see an entry in the log: > WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124: > > Warning: Invalid utf8 character string: '81FE48' > > After I've viewed the diffs of a review request once, the warnings stop > appearing (consistent with the data being migrated to compressed form.). > Switching back to DEBUG = True, and I no longer see failures for the review > request diffs that I looked at while DEBUG = False > > Trying a different approach, I ran "rb-site manage ______ -- > condensediffs", and that also generated the same warnings. Here's a sample: > > WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124: > > Warning: Invalid utf8 character string: 'D76700' > > return self.cursor.execute(query, args) > > > WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124: > > Warning: Invalid utf8 character string: 'A9C813' > > return self.cursor.execute(query, args) > > > ... > > > This generated 169000+ lines of output, corresponding to 56435 individual > warning messages out of a total of 76032 diff files condensed. > > > This gave me an idea. > > > I ran "rb-site ... condensediffs" before the upgrade, then once again > after the upgrade. > > > Problem went away. > > > Weird. Do you want me to try to find out more, and if so, what? > > > Eric. > > > > On Friday, May 13, 2016 at 1:49:53 PM UTC-7, Christian Hammond wrote: >> >> Hi Eric, >> >> Hmm, we'll need to look into that. Is there a way you'd be able to send >> us the diff for that? (I can help you find it.) We will need a copy in >> order to diagnose this. We can sign an NDA for it. >> >> Christian >> >> >> On Friday, May 13, 2016, eric via reviewboard <[email protected]> >> wrote: >> >>> After I migrated my server to 2.5.4, I'm seeing a weird error. I >>> restarted both memcached and apache2, and then browse to a specific review >>> request. >>> >>> Then I click on the "Diff" tab. (After I turned on DEBUG = True in the >>> settings_local.py file) I see this instead of diffs. >>> >>> Traceback (most recent call last): >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/views.py", >>> >>> line 275, in get >>> response = renderer.render_to_response(request) >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py", >>> >>> line 56, in render_to_response >>> return HttpResponse(self.render_to_string(request)) >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py", >>> >>> line 74, in render_to_string >>> large_data=True) >>> File >>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>> >>> line 295, in cache_memoize >>> compress_large_data)) >>> File >>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>> >>> line 249, in cache_memoize_iter >>> items = items_or_callable() >>> File >>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>> >>> line 292, in <lambda> >>> lambda: [lookup_callable()], >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py", >>> >>> line 73, in <lambda> >>> lambda: self.render_to_string_uncached(request), >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py", >>> >>> line 87, in render_to_string_uncached >>> request=request) >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/diffutils.py", >>> >>> line 429, in populate_diff_chunks >>> chunks = list(generator.get_chunks()) >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py", >>> >>> line 756, in get_chunks >>> for chunk in super(DiffChunkGenerator, self).get_chunks(cache_key): >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py", >>> >>> line 107, in get_chunks >>> large_data=True) >>> File >>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>> >>> line 295, in cache_memoize >>> compress_large_data)) >>> File >>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>> >>> line 249, in cache_memoize_iter >>> items = items_or_callable() >>> File >>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>> >>> line 292, in <lambda> >>> lambda: [lookup_callable()], >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py", >>> >>> line 106, in <lambda> >>> lambda: list(self.get_chunks_uncached()), >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py", >>> >>> line 763, in get_chunks_uncached >>> new = get_patched_file(old, self.filediff, self.request) >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/diffutils.py", >>> >>> line 230, in get_patched_file >>> diff = tool.normalize_patch(filediff.diff, filediff.source_file, >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py", >>> >>> line 218, in _get_diff >>> self._migrate_diff_data() >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py", >>> >>> line 421, in _migrate_diff_data >>> diff_hash_is_new = self._set_diff(self.legacy_diff_hash.binary) >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py", >>> >>> line 225, in _set_diff >>> RawFileDiffData.objects.get_or_create_from_data(diff) >>> File >>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/managers.py", >>> >>> line 345, in get_or_create_from_data >>> 'compression': compression, >>> File "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", >>> line 154, in get_or_create >>> return self.get_queryset().get_or_create(**kwargs) >>> File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", >>> line 383, in get_or_create >>> obj.save(force_insert=True, using=self.db) >>> File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", >>> line 545, in save >>> force_update=force_update, update_fields=update_fields) >>> File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", >>> line 573, in save_base >>> updated = self._save_table(raw, cls, force_insert, force_update, >>> using, update_fields) >>> File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", >>> line 654, in _save_table >>> result = self._do_insert(cls._base_manager, using, fields, >>> update_pk, raw) >>> File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", >>> line 687, in _do_insert >>> using=using, raw=raw) >>> File "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", >>> line 232, in _insert >>> return insert_query(self.model, objs, fields, **kwargs) >>> File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", >>> line 1514, in insert_query >>> return query.get_compiler(using=using).execute_sql(return_id) >>> File >>> "/usr/lib64/python2.7/site-packages/django/db/models/sql/compiler.py", line >>> 903, in execute_sql >>> cursor.execute(sql, params) >>> File >>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/log/middleware.py", >>> >>> line 32, in execute >>> return self.cursor.execute(sql, params) >>> File >>> "/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py", line >>> 124, in execute >>> return self.cursor.execute(query, args) >>> File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line >>> 207, in execute >>> if not self._defer_warnings: self._warning_check() >>> File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line >>> 117, in _warning_check >>> warn(w[-1], self.Warning, 3) >>> Warning: Invalid utf8 character string: '890600' >>> >>> >>> Note that if I set DEBUG = False, then I see the diffs in the browser, >>> but I still see warnings in the log file. I'm nervous that migrating my >>> production system forward to 2.5.4 will corrupt data. So now I'm holding >>> off. >>> >>> Looking back through the stack trace, I see there's some logic in there >>> about "_migrate_diff_data" - code which didn't exist in 2.0.X. >>> >>> I suspect this is a manifestation of this bug: >>> https://hellosplat.com/s/beanbag/tickets/4007/ >>> >>> What can I do? >>> >>> Eric. >>> >>> >>> -- >>> Supercharge your Review Board with Power Pack: >>> https://www.reviewboard.org/powerpack/ >>> Want us to host Review Board for you? Check out RBCommons: >>> https://rbcommons.com/ >>> Happy user? Let us know! https://www.reviewboard.org/users/ >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "reviewboard" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> -- >> Christian Hammond >> President/CEO of Beanbag <https://www.beanbaginc.com/> >> Makers of Review Board <https://www.reviewboard.org/> >> >> -- Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/ Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/ Happy user? Let us know! https://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups "reviewboard" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
