Hi Eric, I'd still really love to get a copy of those entries from your database so I can replicate the problem. I simply can't make it happen here, and have no idea why you'd be seeing what you're seeing. It should not be necessary to run condensediffs prior to upgrading.
Christian -- Christian Hammond President/CEO of Beanbag <https://www.beanbaginc.com/> Makers of Review Board <https://www.reviewboard.org/> On Sun, May 15, 2016 at 1:11 PM, eric via reviewboard < [email protected]> wrote: > I have finally re-run my entire automated migration process from start to > finish. > > After running condensediffs *before* the upgrade, I was then able to run > condensediffs after the upgrade as well, all without any warnings or errors. > > Perhaps you should add a step to the upgrade guide, advising > Administrators to rebuild their search index, and also condense diffs? > > So I seem to have found a work-around to the problem - in case anyone else > runs into it. > > Eric. > > > On Saturday, May 14, 2016 at 1:40:06 PM UTC-7, [email protected] wrote: >> >> I re-ran the migration, and before doing anything, set the DEBUG flag to >> True before fetching any diffs. >> >> I can confirm that it fails on *every* diff, not just random ones here >> and there. >> >> Running with DEBUG = False, the warning doesn't throw an exception, but I >> do see an entry in the log: >> WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124: >> Warning: Invalid utf8 character string: '81FE48' >> >> After I've viewed the diffs of a review request once, the warnings stop >> appearing (consistent with the data being migrated to compressed form.). >> Switching back to DEBUG = True, and I no longer see failures for the review >> request diffs that I looked at while DEBUG = False >> >> Trying a different approach, I ran "rb-site manage ______ -- >> condensediffs", and that also generated the same warnings. Here's a sample: >> >> WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124: >> Warning: Invalid utf8 character string: 'D76700' >> >> return self.cursor.execute(query, args) >> >> >> WARNING:py.warnings:/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py:124: >> Warning: Invalid utf8 character string: 'A9C813' >> >> return self.cursor.execute(query, args) >> >> >> ... >> >> >> This generated 169000+ lines of output, corresponding to 56435 individual >> warning messages out of a total of 76032 diff files condensed. >> >> >> This gave me an idea. >> >> >> I ran "rb-site ... condensediffs" before the upgrade, then once again >> after the upgrade. >> >> >> Problem went away. >> >> >> Weird. Do you want me to try to find out more, and if so, what? >> >> >> Eric. >> >> >> >> On Friday, May 13, 2016 at 1:49:53 PM UTC-7, Christian Hammond wrote: >>> >>> Hi Eric, >>> >>> Hmm, we'll need to look into that. Is there a way you'd be able to send >>> us the diff for that? (I can help you find it.) We will need a copy in >>> order to diagnose this. We can sign an NDA for it. >>> >>> Christian >>> >>> >>> On Friday, May 13, 2016, eric via reviewboard <[email protected]> >>> wrote: >>> >>>> After I migrated my server to 2.5.4, I'm seeing a weird error. I >>>> restarted both memcached and apache2, and then browse to a specific review >>>> request. >>>> >>>> Then I click on the "Diff" tab. (After I turned on DEBUG = True in the >>>> settings_local.py file) I see this instead of diffs. >>>> >>>> Traceback (most recent call last): >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/views.py", >>>> line 275, in get >>>> response = renderer.render_to_response(request) >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py", >>>> line 56, in render_to_response >>>> return HttpResponse(self.render_to_string(request)) >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py", >>>> line 74, in render_to_string >>>> large_data=True) >>>> File >>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>>> line 295, in cache_memoize >>>> compress_large_data)) >>>> File >>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>>> line 249, in cache_memoize_iter >>>> items = items_or_callable() >>>> File >>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>>> line 292, in <lambda> >>>> lambda: [lookup_callable()], >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py", >>>> line 73, in <lambda> >>>> lambda: self.render_to_string_uncached(request), >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/renderers.py", >>>> line 87, in render_to_string_uncached >>>> request=request) >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/diffutils.py", >>>> line 429, in populate_diff_chunks >>>> chunks = list(generator.get_chunks()) >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py", >>>> line 756, in get_chunks >>>> for chunk in super(DiffChunkGenerator, self).get_chunks(cache_key): >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py", >>>> line 107, in get_chunks >>>> large_data=True) >>>> File >>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>>> line 295, in cache_memoize >>>> compress_large_data)) >>>> File >>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>>> line 249, in cache_memoize_iter >>>> items = items_or_callable() >>>> File >>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/cache/backend.py", >>>> line 292, in <lambda> >>>> lambda: [lookup_callable()], >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py", >>>> line 106, in <lambda> >>>> lambda: list(self.get_chunks_uncached()), >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/chunk_generator.py", >>>> line 763, in get_chunks_uncached >>>> new = get_patched_file(old, self.filediff, self.request) >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/diffutils.py", >>>> line 230, in get_patched_file >>>> diff = tool.normalize_patch(filediff.diff, filediff.source_file, >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py", >>>> line 218, in _get_diff >>>> self._migrate_diff_data() >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py", >>>> line 421, in _migrate_diff_data >>>> diff_hash_is_new = self._set_diff(self.legacy_diff_hash.binary) >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/models.py", >>>> line 225, in _set_diff >>>> RawFileDiffData.objects.get_or_create_from_data(diff) >>>> File >>>> "/usr/lib64/python2.7/site-packages/ReviewBoard-2.5.4-py2.7.egg/reviewboard/diffviewer/managers.py", >>>> line 345, in get_or_create_from_data >>>> 'compression': compression, >>>> File >>>> "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", line 154, >>>> in get_or_create >>>> return self.get_queryset().get_or_create(**kwargs) >>>> File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", >>>> line 383, in get_or_create >>>> obj.save(force_insert=True, using=self.db) >>>> File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", >>>> line 545, in save >>>> force_update=force_update, update_fields=update_fields) >>>> File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", >>>> line 573, in save_base >>>> updated = self._save_table(raw, cls, force_insert, force_update, >>>> using, update_fields) >>>> File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", >>>> line 654, in _save_table >>>> result = self._do_insert(cls._base_manager, using, fields, >>>> update_pk, raw) >>>> File "/usr/lib64/python2.7/site-packages/django/db/models/base.py", >>>> line 687, in _do_insert >>>> using=using, raw=raw) >>>> File >>>> "/usr/lib64/python2.7/site-packages/django/db/models/manager.py", line 232, >>>> in _insert >>>> return insert_query(self.model, objs, fields, **kwargs) >>>> File "/usr/lib64/python2.7/site-packages/django/db/models/query.py", >>>> line 1514, in insert_query >>>> return query.get_compiler(using=using).execute_sql(return_id) >>>> File >>>> "/usr/lib64/python2.7/site-packages/django/db/models/sql/compiler.py", line >>>> 903, in execute_sql >>>> cursor.execute(sql, params) >>>> File >>>> "/usr/lib64/python2.7/site-packages/Djblets-0.9.3-py2.7.egg/djblets/log/middleware.py", >>>> line 32, in execute >>>> return self.cursor.execute(sql, params) >>>> File >>>> "/usr/lib64/python2.7/site-packages/django/db/backends/mysql/base.py", line >>>> 124, in execute >>>> return self.cursor.execute(query, args) >>>> File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line >>>> 207, in execute >>>> if not self._defer_warnings: self._warning_check() >>>> File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line >>>> 117, in _warning_check >>>> warn(w[-1], self.Warning, 3) >>>> Warning: Invalid utf8 character string: '890600' >>>> >>>> >>>> Note that if I set DEBUG = False, then I see the diffs in the browser, >>>> but I still see warnings in the log file. I'm nervous that migrating my >>>> production system forward to 2.5.4 will corrupt data. So now I'm holding >>>> off. >>>> >>>> Looking back through the stack trace, I see there's some logic in there >>>> about "_migrate_diff_data" - code which didn't exist in 2.0.X. >>>> >>>> I suspect this is a manifestation of this bug: >>>> https://hellosplat.com/s/beanbag/tickets/4007/ >>>> >>>> What can I do? >>>> >>>> Eric. >>>> >>>> >>>> -- >>>> Supercharge your Review Board with Power Pack: >>>> https://www.reviewboard.org/powerpack/ >>>> Want us to host Review Board for you? Check out RBCommons: >>>> https://rbcommons.com/ >>>> Happy user? Let us know! https://www.reviewboard.org/users/ >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "reviewboard" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> -- >>> -- >>> Christian Hammond >>> President/CEO of Beanbag <https://www.beanbaginc.com/> >>> Makers of Review Board <https://www.reviewboard.org/> >>> >>> -- > Supercharge your Review Board with Power Pack: > https://www.reviewboard.org/powerpack/ > Want us to host Review Board for you? Check out RBCommons: > https://rbcommons.com/ > Happy user? Let us know! https://www.reviewboard.org/users/ > --- > You received this message because you are subscribed to the Google Groups > "reviewboard" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/ Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/ Happy user? Let us know! https://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups "reviewboard" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
