OK, I made some progress on the quest of solving this issue :) While I was implementing the "not-merge-catalog-if-plural-form-differs" policy, I realized that it would be cumbersome for the users to adapt to the new situation if no additional tools are provided. I switch to a Documentation Driven Development approach to have a better picture of all the needs, and here is what I wrote:
.. _plural-forms: Plural Forms ~~~~~~~~~~~~ Django does not support custom plural forms in po files. As all translation catalogs are merged, only the plural form in the main Django po file (located in ``django/conf/locale/<lang_code>/LC_MESSAGES/django.po``) is considered. Plural forms in all other po files are ignored by the GNU gettext merging process. Therefore, you shouldn't use different plural forms in your project or application po files, as it may lead to unexpected results. To prevent inconsistencies and undesired behaviors in translations, Django will not merge any catalog that contains a different plural form than the main one and will issue a warning about the conflicting catalog. The warning will show both plural forms (the main and the conflicting one) so you can decide which one is to be used. This conflict may arise mostly in two situations: * when the main plural form for a language is updated in Django and your po files were generated with a previous one, or * when including third-party translations with a different plural form. There are two aspects of the plural form to be considered: the number of plural forms (``nplurals``) and the plural equation (``plural=``). For the number of plurals, Django follows the standards provided by `Unicode < http://www.unicode.org/cldr/charts/latest/supplemental/language_plural_rules.html >`_. In the case of an increase in the number of plurals for a language, you will have to provide with more translations in your po files to comply with the updated language definition (it is considered an API change, noted in the Release Notes and unusual). For this, you may use djadmin:`django-admin makemessages --comply-plurals<makemessages>` to fill the missing plurals from your projects' catalogs with the previous ones so you avoid having translation results in the fallback language or the original string - this will produce the same results as before though you may want to update those later for a better expression of the language if it corresponds. If you do not want to comply with the new language definition, you can create a variant for the language with less plural forms, i.e. ``he_SP`` for "Hebrew (Simplified)" with 2 plural forms and ``he`` with 4 variants acting as a fallback (see :ref:`how-django-discovers-translations`). Note that your variant would need to be explicitly set by Django, or in the url, or by your front-end, as browsers won't know about it. For the plural equation (the way of determining which form will be used), given a number of plurals, *one will be an improvement over the other* - as it is assumed that there is one plural equation that can describe correctly all the plurals' cases of a language. Note that the order of the forms can vary depending on the equation, i.e. the cases for the second plural form in one equation may refer to the third one in another. Once you decided which one is better, there are two aspects to consider for resolving this conflict: updating the plural forms of your project's catalogs or update the ones in Django's base catalogs. For aligning your project's catalogs' plural forms with the main one (most common case after a Django upgrade), use djadmin:`django-admin makemessages --update-plural-form<makemessages>`. This will make your catalogs mergeable again with Django's. After this, djadmin:`compile the messages as usual<compilemessages>` and you'll be ready to go. If you choose to use the version different than Django's, edit the main Django po file and then run: * djadmin:`django-admin makemessages --update-plural-form --base LC<makemessages>` to propagate the change to all catalogs bundled with Django, and then * djadmin:`django-admin compilemessages --base LC<compilemessages>` to compile them. This will align all the Django's catalogs with your plural equation. If so, `consider submiting your plural equation as an improvement to Django in a ticket <https://code.djangoproject.com/newticket>`_. .. versionchanged:: 3.1 Handling plural forms as described above was added. It's not tooooo complicated, but not slick at all... not every user may like to take the time to understand the plural form of the catalogs, although I think for most cases, it will be issuing a "makemessages --update-plural-form". This led me evaluate again the Claude proposal of having a dict with the plural form as keys and merge the catalogs there. This had the main problem of no clear resolution if overlapping, but then I realized that if an ordered dict is used, then the merging order can be somehow retained, and for most cases, it will be sufficient to reproduce the precedence order that Django has actually (project translations will come first than Django, like is described in here <https://docs.djangoproject.com/en/2.2/topics/i18n/translation/#how-django-discovers-language-preference>), so this would be a better way. But, then I realized that there is major caveat on this approach, and that is that updates on the plural equation won't reach users' catalogs, because their catalogs will be kept separately one the plural form differs. This would be the case that Shai pointed on Django 2.2 with the incorrect plural equation for HE. People who have generated their catalogs with makemessages on 2.2 will have a wrong plural equation, that once it is fixed on a new release, it won't reach their catalogs because they will be kept apart. This would make things harder for that users (non-hardcore-translators? :), because tracking that error may be hard unless they previously understood all the process. Then, all the warnings should be shown also in this case, and so the tools to align their pfs to the new one, because updates would have to be done "manually" (by running the command). So, the fix won't be in the merging policy (merge|not-merge|dict-merge) but in the warnings raised about different plural forms co-existing and the tools to align them easily (django -> user, user -> django) to get consistent results. The different merging policy would be a way of engaging the user to be more or less proactive with the problem ({"merge": "You may have unwanted results until you fix it", "not-merge": "it won't work until you fix it", "dict-merge": "you will have to take care of the updates for fixing it"}). Also, it would be better to have a system check for this than doing it "on merge", because only the default language will be merged "on start", you will see the warnings later when it is activated. Any thoughts? PS: The "makemessages --comply-plurals" needs to assume that the new pfs were handled by the previous last one to produce the same results, if not, it will prevent only the fallback. On Thu, Nov 28, 2019 at 12:45 AM Matemática A3K <[email protected]> wrote: > > > On Wed, Nov 27, 2019 at 1:40 AM Matemática A3K <[email protected]> > wrote: > >> >> >> On Tue, Nov 26, 2019 at 12:51 PM Shai Berger <[email protected]> wrote: >> >>> On Tue, 26 Nov 2019 12:28:45 +0100 >>> Maciek Olko <[email protected]> wrote: >>> >>> > It looks like Transifex uses [1] Unicode Language Plural Rules [2]. >>> > If they are incorrect for Hebrew, maybe they should be fixed on >>> > Unicode side? >>> > >>> >>> Just for the record, they are indeed wrong -- not in the sense that has >>> sparked this thread (there are indeed 4 forms) but in the rules (the >>> "many" rule should say "2 < n <= 10", not "n % 10 = 0"). I'm looking >>> into fixing it. >>> >> >> Django's? >> >> The problem raised from Django sync'ing with Transifex without any notice >> of plural rules changes. These changes led sometimes to break Django's >> translations, other times third-party translations and other times both. >> >> For what I have understood, this is a script-driven process. This has to >> be changed if Django is going to maintain the plural rules "independently", >> because they will get overwritten in the next sync. >> >> Maintaining the plural rules "independently" should be the best way to >> handle this - IMO - although the project may not have the resources for all >> the languages it provides translations. This would be sync'ing "only >> strings", not plural rules (forms and equations). >> >> Changes in the number of plurals for a language should be included in the >> Release Notes, as it is changing the i18n API. >> >> Changes in the plural equation doesn't seem necessary to me (to include >> in the RN) as they would be for the better - given the number of plurals is >> the same, the new equation should provide an improvement of the results >> with the same input - - well, mentioning them won't hurt anybody but it >> isn't necessary :) >> >> If Django detects a catalog with a different number of plurals than its >> main, then it won't be merged and a warning should be shown (something like >> "Catalog not merged to prevent inconsistencies due to a different number of >> plurals than the main catalog. See >> https://docs.djangoproject.com/en/2.2/topics/i18n/translation/#pluralization";). >> The documentation should contain information on how to increase the number >> of plurals seamlessly of your catalog to match Django's (the most likely >> case). >> >> If an inconsistency is seen due to a plural equation, it should be >> reported as a bug. As it is not sync'ed anymore, it can be fixed for >> Django. The documentation should contain information on how to fix it >> immediately for your project while the fix is done at the "Django level". >> >> This is under the assumptions that: >> - There is one plural equation which can describe entirely the language's >> plurals >> - The policy in >> https://docs.djangoproject.com/en/2.2/topics/i18n/translation/#pluralization >> - final note, is correct >> >> Giving a convenient complete control of locales persistent to upgrades - >> a "collectlocales" command - can be left for later or a third-party package. >> >> > >> Do you think this a viable way to solve the issue? >> > > Claude? Michal? > > Do you still think that supporting multiple unmerged catalogs is the way > to go? Can you elaborate more on this? > > If not, and the plural-forms aren't maintained by Django, then the option > of maintaining the whole locales in the project dir independently should be > made available (collectlocales). > > For maintaining plural forms, it can be done by notifying django-i18n > mailing list of a new PF and if no one complains in a reasonable time > frame, then it goes to the repository. This extra layer can prevent a lot > of errors. > > >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/20191126195144.066b511b.shai%40platonix.com >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CA%2BFDnhKN%3Dvdmv7QDEnTp4EQtkTOZCrWt_fiP%2BsNbTHuvkb1i4g%40mail.gmail.com.
