Mike Gran <spk...@yahoo.com> writes:
> My bad. Actually, I should have enclosed the 'with-locale' in the
> context of a 'pass-if', which would have caught the exception.
Yes, but at the cost of not running the tests...
>> I can allow make check to complete by changing that line to
>>
>> (false-if-exception (with-locale "en_US.iso88591"
>>
>> but I doubt that's the best fix. Is the "en_US.iso88591" locale
>> actually important for the enclosed tests?
>
> It is important. This is one of the problems with the whole Unicode
> effort. There is no Unicode-capable regex library. The regexp.test
> tries matching all bytes from 0 to 255, and it uses scm_to_locale_string
> to prep the string for dispatch to the libc regex calls and
> scm_from_locale_string to send them back.
>
> If the current locale is C or ASCII, bytes above 127 will cause errors.
> If the current locale is UTF-8, bytes above 127 will be converted into
> multibyte sequences that won't be matched by the regular expression
> being tested. To pass the test in regexp.test, we need to use the
> encoding that matches all of the codepoints 0 to 255 to single byte
> characters, which is ISO-8859-1.
>
> So until a better regex comes along, wrapping regex in an
> 8-bit-clean-friendly locale like Latin-1 is necessary to avoid encoding
> errors when encoding arbitrary 8-bit data like the test does.
>
> The reason why this problem is cropping up now and didn't occur before
> is because the old scm_to_locale_string was just a stub that passed
> 8-bit data through unmodified.
Thanks for explaining; I think I understand now. So then Ludovic's
suggestion of with-latin1-locale should work, shouldn't it?
> This regex library actually can be used with arbitrary Unicode data
> but it takes extra care. UTF-8 can be used as the locale, and, then
> regular expression must be written keeping in mind that each non-ASCII
> character is really a multibyte string.
Can you give an example of what that ("keeping in mind...") means? Is
it being careful with repetition counts (as in "[a-z]{3}"), for
example?
Thanks,
Neil
