[RFR] Major localechooser revamp

[Frans Pop]

Hi,

This weekend has been rather dedicated to localechooser. Although there's 
more in the pipeline, I think these changes are ready to be committed.

Before I commit the changes, here's an overview. If anyone sees any problems 
with these changes or has questions, please let me know.

I have done fairly extensive testing while working on this and am confident 
that there is no major breakage. I've tested both normal and expert 
installs and also tested preseeding from the bootprompt.

Any new dialogs have been added without enabling translation so that they 
can be reviewed before translators see them.


The shortlog:
* Factor out two functions
* Fix logic error in code2country function
* Bring main code under state engine to allow backup to earlier questions

  This is the main change. It means that it now possible to back up between
  questions within localechooser.

* Make sure language list is displayed correctly
* Factor out new function refine_locale
* Clean up languagemap script
* Display message when language switch is no longer possible

  This is implemented, but disabled because there seems to be a bug in the
  way cdebconfs sets the translations-dropped template.

* Move explanation of different display levels to the README
* Merge languagelist and languagelist.l10n

  It seemed to me there was very little point in maintaining two separate
  lists for this. Values not needed inside D-I are filtered out by d/rules.

* Identify languages by language code instead of language name

  This is a major technical change. I've added a Choices-C field that
  contains the language code (including country for pt_BR and zh_*).
  The functions locale2langname and locale2langcode have been dropped.
  This change simplifies localechooser a lot and makes some additional
  changes I'm considering possible.

* Fix mistake in sort order in languagelist
* Build language list for the current display level at runtime (#365307)

  Again a major technical change. It means we no longer have 4 different
  templates with different language lists for different frontends, but
  instead a single (gzipped) file with all data needed to build the
  different lists at runtime. Disadvantage is that it takes a little bit
  longer before the first localechooser dialog is displayed.

Included in these changes are various cleanups.

Attached the new version of the languagelist after all the above changes. 
Note that columns 2 and 3 are not included in the version in the package.


The diffstat:
 packages/localechooser/Makefile                |  21 +-
 packages/localechooser/README                  |  20 +
 packages/localechooser/debian/dirs             |   2 +-
 .../debian/localechooser.templates-in          |  52 +--
 packages/localechooser/debian/rules            |   7 +-
 packages/localechooser/languagelist.l10n       |  87 ---     \  Merged
 packages/localechooser/languagelist            | 158 +++---  /
 packages/localechooser/languagemap             |  43 +-
 packages/localechooser/localechooser           | 575 ++++++++++----------
 packages/localechooser/mktemplates.language    | 168 ------  \  Renamed
 packages/localechooser/mklanguagelist.data     |  77 +++     /
 packages/localechooser/mktemplates.country     |   8 +-
 packages/localechooser/mk_shortlist_templates  | 112 ----    \  Renamed
 packages/localechooser/mktemplates.shortlist   | 112 ++++    /
 14 files changed, 634 insertions(+), 808 deletions(-)


In addition to the changes listed above I have prepared one other change 
that needs a bit more work and for which I'd appreciate comments.
This is the addition of warnings when translations are incomplete, based on 
the translation statistics from the new sublevels and using data from the 
script scripts/l10n/calc-translation-stats. I've distinguished five levels 
of "incompleteness" with (hopefully) appropriate warnings.
Please have a look at the attached patch. 

Cheers,
FJP


commit af58b2793a9c5167d7d95d8b399784248a1c5a2c
Author: Frans Pop <[EMAIL PROTECTED]>
Date:   Sat Mar 8 18:27:22 2008 +0100

    Display warning if translation is not complete
    
    Based on a translation status file included in the initrd at build time.
    The severity of the warning depends on how much is untranslated based on
    the D-I sublevels and taking into account what architecture is being
    installed.
    
    Note that incomplete translations in components not in D-I SVN are not
    taken into account.

diff --git a/packages/localechooser/debian/localechooser.templates-in b/packages/localechooser/debian/localechooser.templates-in
index 82a4188..c9b6275 100644
--- a/packages/localechooser/debian/localechooser.templates-in
+++ b/packages/localechooser/debian/localechooser.templates-in
@@ -26,6 +26,68 @@ Description: Language selection no longer possible
  To select a different language you will need to abort this installation
  and reboot the installer.
 
+Template: localechooser/translation/incomplete
+Type: boolean
+Default: false
+Description: Continue the installation in the selected language?
+ The translation for the language you selected is incomplete.
+ .
+ This means that there is a significant chance that some dialogs will be
+ displayed in a different language, in most cases in English.
+ .
+ Unless you have a good understanding of the alternative language,
+ you are recommended to either select a different language or abort
+ the installation.
+
+Template: localechooser/translation/normal_ok
+Type: boolean
+Default: false
+Description: Continue the installation in the selected language?
+ The translation for the language you selected is incomplete.
+ .
+ If you do anything other than a purely default installation, there is a
+ real chance that some dialogs will be displayed in a different language,
+ in most cases in English.
+ .
+ Unless you have a good understanding of the alternative language,
+ you are recommended to either select a different language or abort
+ the installation.
+
+Template: localechooser/translation/partial
+Type: boolean
+Default: false
+Description: Continue the installation in the selected language?
+ The translation for the language you selected is not fully complete.
+ .
+ If you continue the installation in the selected language, most dialogs
+ should be displayed correctly but - especially if you use the more
+ advanced options of the installer - some may be displayed in a different
+ language, in most cases in English.
+ .
+ If you chose not to continue, you will be given the option of selecting
+ a different language, or you can abort the installation.
+
+Template: localechooser/translation/mostly
+Type: boolean
+Default: true
+Description: Continue the installation in the selected language?
+ The translation for the language you selected is not fully complete.
+ .
+ If you continue the installation in the selected language, dialogs
+ should normally be displayed correctly but - especially if you use the
+ more advanced options of the installer - there is a slight chance some
+ may be displayed in a different language, in most cases in English.
+
+Template: localechooser/translation/exceptions
+Type: boolean
+Default: true
+Description: Continue the installation in the selected language?
+ The translation for the language you selected is not fully complete.
+ .
+ The chance that you will actually encounter a dialog that is not
+ translated into the selected language is extremely small, but it cannot
+ be ruled out completely.
+
 Template: languagechooser/languagelist
 Type: select
 Choices-C: ${CODES}
diff --git a/packages/localechooser/debian/rules b/packages/localechooser/debian/rules
index ab29342..0f6c787 100755
--- a/packages/localechooser/debian/rules
+++ b/packages/localechooser/debian/rules
@@ -21,7 +21,7 @@ install: build
 	dh_testroot
 	dh_clean -k
 	dh_installdirs
-	dh_install localechooser languagemap usr/bin
+	dh_install localechooser languagemap translation-check usr/bin
 	mkdir -p debian/$(PACKAGE)/DEBIAN
 	install -m0755 post-base-installer debian/$(PACKAGE)/usr/lib/post-base-installer.d/05localechooser
 	install -m0755 finish-install debian/$(PACKAGE)/usr/lib/finish-install.d/05localechooser
diff --git a/packages/localechooser/localechooser b/packages/localechooser/localechooser
index ef62b88..4653c4d 100755
--- a/packages/localechooser/localechooser
+++ b/packages/localechooser/localechooser
@@ -314,18 +314,46 @@ while :; do
 		db_set "$consoledisplay"  "$CONSOLE"
 		log "Set $consoledisplay = '$CONSOLE'"
 
+		db_set "debconf/language" "$LANGUAGELIST"
+		log "Set debconf/language = '$LANGUAGELIST'"
+
 		[ -f /etc/lsb-release ] && . /etc/lsb-release
 		if [ "$sel_language" ] && [ $LANGUAGE != en ] && \
 		   [ "$X_INSTALLATION_MEDIUM" = "floppy" ]; then
 			db_input high localechooser/no-translations-yet || true
-			db_go || true
 		fi
+		;;
 
-		db_set "debconf/language" "$LANGUAGELIST"
-		log "Set debconf/language = '$LANGUAGELIST'"
+	   12)	# Warn if translation is incomplete
+		gtk_switch_font
+
+		# Display warning for incomplete translations; skip it for
+		# automated installs to prevent a loop if the default is false
+		twarning=""
+		if [ -z "$sel_language" ] && \
+		   db_get debconf/priority && [ "$RET" != critical ] && \
+		   tstatus=$(translation-check "$LANGUAGE"); then
+			case $tstatus in
+			    0)	twarning=localechooser/translation/incomplete ;;
+			    1)	twarning=localechooser/translation/normal_ok ;;
+			    2)	twarning=localechooser/translation/partial ;;
+			    3)	twarning=localechooser/translation/mostly ;;
+			    4)	twarning=localechooser/translation/exceptions ;;
+			esac
+			db_input high $twarning || [ $? -eq 30 ]
+		fi
+		;;
+
+	   13)	# Continue or choose alternative language
+		if [ "$twarning" ]; then
+			if db_get $twarning && [ "$RET" = false ]; then
+				db_reset $twarning
+				STATE=10
+				continue
+			fi
+		fi
 
 		install_lang_specific
-		gtk_switch_font
 
 		STATE=19
 		continue
diff --git a/packages/localechooser/translation-check b/packages/localechooser/translation-check
new file mode 100755
index 0000000..117f924
--- /dev/null
+++ b/packages/localechooser/translation-check
@@ -0,0 +1,49 @@
+#! /bin/sh
+set -e
+
+# Exits without error if translation is _not_ completely up-to-date;
+# with errror if translation is up-to-date or on errors.
+
+if [ "$1" = en ] || [ "$1" = C ]; then
+	exit 9	# up-to-date by definition
+fi
+
+sfile=/usr/share/localechooser/translation-status
+[ -f $sfile ] || exit 1
+
+tstatus=$(grep "^$1:" $sfile | sed "s/.*:[[:space:]]*//")
+[ "$tstatus" ] || exit 1
+
+tlevel=${tstatus% *}
+tcompl=${tstatus#* }
+expr "$tlevel" : "[1-5]" >/dev/null || exit 1
+expr "$tcompl" : "[FMPLU]" >/dev/null || exit 1
+
+arch=$(archdetect)
+case ${arch%/*} in
+    i386|amd64)
+	archlevel=3 ;;
+    arm*|mips*|powerpc|ppc64|sparc)
+	archlevel=4 ;;
+    alpha|hppa|ia64|m68k|s390)
+	archlevel=5 ;;
+    *)
+	archlevel=5 ;;
+esac
+
+if [ "$tcompl" = F ]; then
+	exit 9	# up-to-date
+elif [ $archlevel = 3 ] && [ $tlevel -eq 2 ] && [ $tcompl != M ]; then
+	echo 0	# incomplete
+elif [ $archlevel = 3 ] && [ $tlevel -eq 2 ]; then
+	echo 1	# incomplete, but normal installs mostly OK
+elif [ $tlevel -lt $archlevel ] ||
+     ([ $tlevel -eq $archlevel ] && [ $tcompl = U ]); then
+	echo 0	# incomplete
+elif [ $tlevel -eq $archlevel ] && [ $tcompl != M ]; then
+	echo 2	# partial at wanted level
+elif [ $tlevel -eq $archlevel ]; then
+	echo 3	# mostly complete
+else
+	echo 4	# only exceptions untranslated
+fi

#
# This is the complete list of languages (locales) to choose from.
# langcode;language (en);language 
(orig);supported_environments;countrycode;fallbacklocale;langlist;console-data
sq;Albanian;Shqip;2;AL;sq_AL.UTF-8;;kbd=lat0-sun16(utf8)
am;Amharic;አማርኛ;4;ET;am_ET;;
ar;Arabic;عربي;3;EG;ar_EG.UTF-8;;console-setup
eu;Basque;Euskaraz;1;ES;eu_ES.UTF-8;;kbd=lat0-sun16(utf8)
be;Belarusian;Беларуская;2;BY;be_BY.UTF-8;;console-setup
bn;Bengali;বাংলা;4;BD;bn_BD;;
bs;Bosnian;Bosanski;2;BA;bs_BA.UTF-8;;kbd=Lat2-Terminus16(utf8)
bg;Bulgarian;Български;2;BG;bg_BG.UTF-8;;console-setup
# For C locale, set language to 'en' to make sure questions are "translated"
# to English instead of showing codes.
C;C;No localization;0;;C;en;
ca;Catalan;Català;1;ES;ca_ES.UTF-8;;kbd=lat0-sun16(utf8)
# Special case for Chinese as the two flavours share the same ISO 639 code
# Both will trigger countrychooser. Each will be the backup for the other
# one
zh_CN;Chinese (Simplified);中文(简体);3;CN;zh_CN.UTF-8;zh_CN:zh;
zh_TW;Chinese (Traditional);中文(繁體);3;TW;zh_TW.UTF-8;zh_TW:zh;
hr;Croatian;Hrvatski;2;HR;hr_HR.UTF-8;;kbd=lat2-sun16(utf8)
cs;Czech;Čeština;2;CZ;cs_CZ.UTF-8;;kbd=lat2-sun16(utf8)
da;Danish;Dansk;1;DK;da_DK.UTF-8;;kbd=lat0-sun16(utf8)
nl;Dutch;Nederlands;1;NL;nl_NL.UTF-8;;kbd=lat0-sun16(utf8)
dz;Dzongkha;རྫོང་ཁ།;4;BT;dz_BT;;
en;English;English;0;US;en_US.UTF-8;;kbd=lat0-sun16(utf8)
# The Esperanto locale *is* (or will be as of 1/11/2006) eo.UTF-8
# so no country on purpose. The default country is Antarctica because...
# ...why not..:-)
eo;Esperanto;Esperanto;2;AQ;eo.UTF-8;;console-setup
et;Estonian;Eesti;2;EE;et_EE.UTF-8;;kbd=lat0-sun16(utf8)
fi;Finnish;Suomi;1;FI;fi_FI.UTF-8;;kbd=lat0-sun16(utf8)
fr;French;Français;1;FR;fr_FR.UTF-8;;console-setup
gl;Galician;Galego;1;ES;gl_ES.UTF-8;;kbd=lat0-sun16(utf8)
ka;Georgian;ქართული;4;GE;ka_GE.UTF-8;;console-setup
de;German;Deutsch;1;DE;de_DE.UTF-8;;kbd=lat0-sun16(utf8)
el;Greek;Ελληνικά;2;GR;el_GR.UTF-8;;console-setup
gu;Gujarati;ગુજરાતી;4;IN;gu_IN;;
he;Hebrew;עברית;3;IL;he_IL.UTF-8;;console-setup
hi;Hindi;हिन्दी ;4;IN;hi_IN;;
hu;Hungarian;Magyar;2;HU;hu_HU.UTF-8;;kbd=lat2-sun16(utf8)
#X is;Icelandic;Íslenska;1;IS;is_IS.UTF-8;;kbd=lat9u-16(utf8)
id;Indonesian;Bahasa Indonesia;1;ID;id_ID.UTF-8;;kbd=lat0-sun16(utf8)
#X ga;Irish;Gaeilge;1;IE;ga_IE.UTF-8;;kbd=lat0-sun16(utf8)
it;Italian;Italiano;1;IT;it_IT.UTF-8;;kbd=lat0-sun16(utf8)
ja;Japanese;日本語;3;JP;ja_JP.UTF-8;;
#X kn;Kannada;ಕನ್ನಡ;4;IN;kn_IN;;
#X ks;Kashmiri;कोशुर;4;IN;ks_IN;;
#X kk;Kazakh;Қазақ;2;KZ;kk_KZ.UTF-8;;console-setup
km;Khmer;ខ្មែរ;4;KH;km_KH;;
ko;Korean;한국어;3;KR;ko_KR.UTF-8;;
ku;Kurdish;Kurdî;2;TR;ku_TR.UTF-8;;kbd=Lat15-Terminus16(utf8)
#X lo;Lao;ລາວ;4;LA;lo_LA;;console-setup
lv;Latvian;Latviski;2;LV;lv_LV.UTF-8;;kbd=lat7-14(utf8)
lt;Lithuanian;Lietuviškai;2;LT;lt_LT.UTF-8;;kbd=LatArCyrHeb-16(utf8)
#X mg;Malagasy;Malagasy;1;MG;mg_MG.UTF-8;mg_MG:fr_FR:fr:en;kbd=lat0-sun16(utf8)
#X ms;Malay;Bahasa Malaysia;1;MY;ms_MY.UTF-8;;kbd=lat0-sun16(utf8)
ml;Malayalam;മലയാളഠ;4;IN;ml_IN;;
#X mr;Marathi;मराठी;4;IN;mr_IN;;
mk;Macedonian;Македонски;2;MK;mk_MK.UTF-8;;console-setup
ne;Nepali;नेपाली ;4;NP;ne_NP;;
# The Sami translation is really incomplete. We however keep Sami on request
# of Skolelinux as a kind of reward to them..:-). They need to be able to 
# choose Sami as an option so that the Sami locale is set as default
se_NO;Northern 
Sami;Sámegillii;1;NO;se_NO.UTF-8;se_NO:nb_NO:nb:no_NO:no:nn_NO:nn:da:sv:en;kbd=lat0-sun(utf8)
nb_NO;Norwegian Bokmaal;Norsk 
bokmål;1;NO;nb_NO.UTF-8;nb_NO:nb:no_NO:no:nn_NO:nn:da:sv:en;kbd=lat0-sun16(utf8)
nn_NO;Norwegian Nynorsk;Norsk 
nynorsk;1;NO;nn_NO.UTF-8;nn_NO:nn:no_NO:no:nb_NO:nb:da:sv:en;kbd=lat0-sun16(utf8)
#X fa;Persian;فارسی;3;IR;fa_IR;;console-setup
pl;Polish;Polski;2;PL;pl_PL.UTF-8;;kbd=lat2-sun16(utf8)
pt;Portuguese;Português;1;PT;pt_PT.UTF-8;pt:pt_BR:en;kbd=lat0-sun16(utf8)
pt_BR;Portuguese (Brazil);Português do 
Brasil;1;BR;pt_BR.UTF-8;pt_BR:pt:en;kbd=lat1-16(utf8)
pa;Punjabi (Gurmukhi);ਪੰਜਾਬੀ;4;IN;pa_IN;;
ro;Romanian;Română;2;RO;ro_RO.UTF-8;;kbd=Lat2-Terminus16(utf8)
ru;Russian;Русский;2;RU;ru_RU.UTF-8;;console-setup
#X sa;Sanskrit;संस्कृत;4;IN;sa_IN;;
# Serbian commented for consistency: too incomplete
#X sr;Serbian;Српски;2;CS;[EMAIL PROTECTED];;console-setup
sk;Slovak;Slovenčina;2;SK;sk_SK.UTF-8;;kbd=lat2-sun16(utf8)
sl;Slovenian;Slovenščina;2;SI;sl_SI.UTF-8;;kbd=lat2-sun16(utf8)
es;Spanish;Español;1;ES;es_ES.UTF-8;;kbd=lat0-sun16(utf8)
sv;Swedish;Svenska;1;SE;sv_SE.UTF-8;;kbd=lat0-sun16(utf8)
tl;Tagalog;Tagalog;1;PH;tl_PH.UTF-8;;kbd=lat0-sun16(utf8)
ta;Tamil;தமிழ்;4;IN;ta_IN;;
#X te;Telugu;తెలుగు;4;IN;te_IN;;
th;Thai;ภาษาไทย;4;TH;th_TH.UTF-8;;console-setup
tr;Turkish;Türkçe;2;TR;tr_TR.UTF-8;;kbd=Lat15-Terminus16(utf8)
uk;Ukrainian;Українська;2;UA;uk_UA.UTF-8;;console-setup
#X ur;Urdu;اردو;3;PK;ur_PK.UTF-8;;console-setup
#X [EMAIL PROTECTED];Valencian-Catalan;Valencià-Català;1;ES;[EMAIL 
PROTECTED];;kbd=lat0-sun16(utf8)
vi;Vietnamese;Tiếng Việt;3;VN;vi_VN.UTF-8;;console-setup
#X cy;Welsh;Cymraeg;2;GB;cy_GB.UTF-8;;kbd=iso14.f16(utf8)
wo;Wolof;Wolof;2;SN;wo_SN;wo:fr:en;
#X xh;Xhosa;Xhosa;2;ZA;xh_ZA.UTF-8;;kbd=lat0-sun16(utf8)

signature.asc
Description: This is a digitally signed message part.