On Mon, 6 Dec 2004 08:54:21 -0600, Doran, Michael D <[EMAIL PROTECTED]> wrote: > > One (perhaps large) caveat: as of now all USMARC records are assumed > > to be MARC-8 encoded, and the data within is always run through > > to_utf8/to_marc8 during XML export/import. > > The MARC-21 standard allows for either MARC-8 or UCS/Unicode. Position > 09 in the record leader indicates the character encoding: a "blank" for > MARC-8, and an "a" for UCS/Unicode. Perhaps your patch could test for > this and then only apply the transformation when required. Note: I > believe the leader itself is limited to characters in the ASCII range, > so you wouldn't have to know the encoding of the record prior to parsing > the leader.
Yeah. I've got a new version that takes this into account. The problem is that MARC::Record on modern Perls (post 5.6) doesn't seem to work properly with Unicode encoded records, at least not without some Encode.pm work. It seems to truncate fields containing combining octets in cases where there is a valid LATIN1 (well, current system encoding/locale, actually) version of the character, such as LATIN1 char 0xF8. This is due to modern Perls "helping" you with string encoding. Because of that, I am now "downgrading" all XML Unicode records to MARC8, though there shouldn't be any loss of data. I am now using the Encode module inside ...::XML.pm and ...::SAX.pm to handle this, but until I get everything fully tested I'll continuing to reencode records to MARC8. Older Perls (pre 5.6) should not actually need Encode's help, but it should not hurt in those cases. > > > What that means is that > > the records from the problem below (containing UTF8 directly in the > > data, without an encoding marker) would probably break during export > > to XML. > > The original record from John Hammer did not contain UTF-8, it contained > MARC-8. I believe that the fact that the combining MARC-8 characters > were replaced by a generic replacement character only indicates that the > app he was using to view the data (post processing by MARC::Record) was > using a character set in which hex E5 and F2, encoded as single octets, > were not valid characters in that app's character set. That app's > character set was apparently Unicode (UTF-8) and so E5 and F2 were > replaced by U+FFFD. That's the long way of saying that the patch should > work fine in his case. :-) > I understand. It wasn't that I was trying to solve that particular problem, it just got me thinking about MARC::File::XML. Sorry for any confusion there. I'm using File::XML regularly now, and I'm trying to fix it up. I am glad that the patch should work with those records, though! One last note. I'm rather new to encoding issues as they pertain to MARC8, since they cannot by implicitly handled by Perl, as other encodings can be in some cases. This will be evolving, and I will do my best not to break anything and to follow the MARC standard, but IANAL(ibrarian), so be gentle. ;) Thanks for the pointers, and I'll send more updates here unless everyone would rather I not. :) -- Mike Rylander [EMAIL PROTECTED] GPLS -- PINES Development Database Developer http://open-ils.org > > > -- Michael > > # Michael Doran, Systems Librarian > # University of Texas at Arlington > # 817-272-5326 office > # 817-688-1926 cell > # [EMAIL PROTECTED] > # http://rocky.uta.edu/doran/ > > > -----Original Message----- > > From: Mike Rylander [mailto:[EMAIL PROTECTED] > > Sent: Saturday, December 04, 2004 1:31 PM > > To: [EMAIL PROTECTED] > > Subject: Re: Character sets - kind of solved? > > > > I've run into some record encoding issues myself, though not the > > problem from below. In any case, this got me thinking about the > > current state of MARC::File::XML, specifically that it could not > > handle MARC8 encoded records. > > > > I submitted a patch a while back to hack around this, but that just > > lets us get the MARC records into well formed XML. Basically, it just > > lets you set the encoding on the XML to something that has embedded > > 8-bit characters, like ISO-8859-1, aka LATIN1. > > > > But that is far from optimal, since the data is being misinterpreted. > > So I took a look at using MARC::Charset inside MARC::File::XML, and > > I've got a working patch that correctly transcodes records from > > USMARC(MARC-8) to MARC21slim(UTF8) and back again. > > > > It's attached below, if anyone would be so kind as to test it. If all > > goes well we sould be able to actually use MARC::File::XML in > > production. If you do decide to test it, it requires MARC::Charset. > > > > One (perhaps large) caveat: as of now all USMARC records are assumed > > to be MARC-8 encoded, and the data within is always run through > > to_utf8/to_marc8 during XML export/import. What that means is that > > the records from the problem below (containing UTF8 directly in the > > data, without an encoding marker) would probably break during export > > to XML. > > > > The attached tarball contains a patched XML.pm and SAX.pm. Replace > > your current MARC/File/XML.pm and MARC/File/SAX.pm with those and you > > should be good to go. I've also included the scripts I used to test > > and one of my old MARC8 encoded records. http://redlightgreen.com > > confirms that the illustrators name is properly transcoded. > > > > On Fri, 3 Dec 2004 17:53:32 -0600, Doran, Michael D > > <[EMAIL PROTECTED]> wrote: > > > First off, Ashley's suggestion that the original encoding was likely > > > MARC-8 is correct. The author's Arabic name, > > transliterated into the > > > Latin alphabet, should be "Bis{latin small letter a with > > macron}{latin > > > small letter t with dot below}{latin small letter i with macron}, > > > Mu{latin small letter h with dot below}ammad." I am basing this on > > > MARC-21 records that can be seen in UCLA's online catalog > > [1]. So, if > > > the above name is encoded in MARC-8 then the underlying > > code would match > > > John's original code points [2]: > > > > >> Looking at the name with a hex editor, it gives, with > > hex values > > > in curly brackets, > > > > >> "Bis{e5}a{f2}t{e5}i, Mu{f2}hammad." > > > > > > Then the question becomes: "What happened?" > > > > > > > >> the name now appears as > > > > >> "Bis{ef bf bd}a{ef bf bd}t{ef bf bd}i, Mu{ef bf bd}hammad." > > > > > > The fact that one byte turned into three bytes, suggests > > UTF-8 encoding. > > > And the fact that *both* MARC-8 combining characters (i.e. "e5" and > > > "f2") now appear as the *same* combination of characters > > (i.e. "ef bf > > > bd") suggests that it was not an encoding translation from one coded > > > character set to the equivalent codepoint in another > > character set. If > > > we assume UTF-8 and convert UTF-8 "ef bf bd" to its Unicode > > code point, > > > we get U+FFFD [3]. If we look up U+FFFD we see that it is the > > > "REPLACEMENT CHARACTER" [4]. > > > > > > Since MARC::Record (obviously) would't object to the original MARC-8 > > > character encoding, I'm guessing that sometime *after* > > processing the > > > record with MARC::Record that it was either moved to, or > > viewed in, a > > > client/platform/environment that was not MARC-8 savvy > > (which is pretty > > > much everything) and that the client/platform/environment, not > > > recognizing the hex e5 and f2 as valid character encodings, replaced > > > them with the generic replacement character for that > > > client/platform/environment. > > > > > > So I'm thinking that we can rule out MARC::Record and look closer at > > > what happened to the data subsequent to MARC::Record > > processing. That's > > > my guess anyway, and I'm sticking with it until I hear a > > better story. > > > ;-) > > > > > > [1] UCLA's Voyager ILMS has been upgraded to a Unicode > > version, and is > > > able to display the characters accurately. My assumption > > is that the > > > author in the links below is the one in question. > > > See for example (looking at the title field, rather than > > the underlined > > > author/name field): > > > http://catalog.library.ucla.edu/cgi-bin/Pwebrecon.cgi?bbid=603048 > > > http://catalog.library.ucla.edu/cgi-bin/Pwebrecon.cgi?bbid=603049 > > > http://catalog.library.ucla.edu/cgi-bin/Pwebrecon.cgi?bbid=5053287 > > > http://catalog.library.ucla.edu/cgi-bin/Pwebrecon.cgi?bbid=4490052 > > > > > > [2] In MARC-8, combining diacritic characters precede the base > > > character, and as Ashley pointed out, E5 is "macron" and F2 is "dot > > > below." > > > > > > [3] hex "ef bf bd" = binary "11101111 10111111 10111101" > > > A three-octet UTF-8 character has the format of 1110xxxx 10xxxxxx > > > 10xxxxxx, with the "x" positions being the significant values in > > > determining the Unicode code point. When we concatenate those x > > > position values from the above binary code, we get 1111111111111101, > > > which converted to hex, is FFFD > > > > > > [4] See: > > > > > http://rocky.uta.edu/doran/urdu/search.cgi?char_set=unicode&ch > ar_type=he > > > x&char_value=fffd > > > (or just go to > > http://rocky.uta.edu/doran/urdu/search.cgi and plug > > > in fffd > > > > > > -- Michael > > > > > > # Michael Doran, Systems Librarian > > > # University of Texas at Arlington > > > # 817-272-5326 office > > > # 817-688-1926 cell > > > # [EMAIL PROTECTED] > > > # http://rocky.uta.edu/doran/ > > > > > > > -----Original Message----- > > > > From: Ashley Sanders [mailto:[EMAIL PROTECTED] > > > > Sent: Wednesday, November 24, 2004 2:23 AM > > > > Cc: [EMAIL PROTECTED] > > > > Subject: Re: Character sets > > > > > > > > Ed Summers wrote: > > > > > On Tue, Nov 23, 2004 at 04:10:05PM -0600, John Hammer wrote: > > > > > > > > > >>I have a character problem that I hope someone can help > > me with. In > > > > >>a MARC record I am modifying using MARC::Record, one of > > the names > > > > >>contains letters with diacritics. Looking at the name with a hex > > > editor, > > > > >>it gives, with hex values in curly brackets,"Bis{e5}a{f2}t{e5}i, > > > > >>Mu{f2}hammad." After running through MARC::Record, the name now > > > appears > > > > >>as "Bis{ef bf bd}a{ef bf bd}t{ef bf bd}i, Mu{ef bf bd}hammad." > > > > > > > > > > > > > > > That's pretty odd. Any chance you could send me the > > MARC record? At > > > this > > > > > time MARC::Record does not play nicely with Unicode (UTF8). > > > > > > > > > > http://rt.cpan.org/NoAuth/Bug.html?id=3707 > > > > > > > > It is possible they are MARC-8 characters rather than > > utf-8. In MARC-8 > > > > E5 is "macron" and F2 is "dot below." Is MARC::Record > > trying to treat > > > > than as Unicode when in fact they are MARC-8? > > > > > > > > Ashley. > > > > > > > > -- > > > > Ashley Sanders [EMAIL PROTECTED] > > > > Copac http://copac.ac.uk -- A MIMAS service funded by JISC > > > > > > > > > > > > > -- > > Mike Rylander > > [EMAIL PROTECTED] > > GPLS -- PINES Development > > Database Developer > > http://open-ils.org > > >
