Thanks for the Explaination,
I find it hard to determine the way to store data in different encodings
to store in postgresql, below is the demo of some data :-
INSERT INTO conceptnet_frame
VALUES(3884,'ja','{1}?{2}???????',16,3,2140,NULL,NULL,NULL);
INSERT INTO conceptnet_frame
VALUES(3885,'ja','{1}?{2}??????????',31,3,2140,NULL,NULL,NULL);
INSERT INTO conceptnet_frame
VALUES(3886,'ja','{1}?{2}??????????????',31,3,2140,NULL,NULL,NULL);
INSERT INTO conceptnet_frame
VALUES(3887,'ja','{1}?{2}??????????',31,3,2140,NULL,NULL,NULL);
INSERT INTO conceptnet_frame
VALUES(3888,'ja','{1}????????{2}???????',29,3,2140,NULL,NULL,NULL);
INSERT INTO conceptnet_frame
VALUES(3889,'ja','{1}??????{2}??????',28,3,2140,NULL,NULL,NULL);
INSERT INTO conceptnet_frame
VALUES(3890,'ja','{1}??{2}???????????',14,3,2140,NULL,NULL,NULL);
INSERT INTO conceptnet_frame
VALUES(3891,'ja','{2}????????{1}??????',9,3,2140,NULL,NULL,NULL);
Below link explains all the things :-
http://www.depesz.com/index.php/2010/03/07/error-invalid-byte-sequence-for-encoding/
Above link shows the above encoding schemes is in utf16 format but
postgresql-8.4 doesn't support it.
Is there any way to store data in different encoding in a utf-8 database.
Happy Holidays!
Alban Hertroys wrote:
On 26 Dec 2011, at 8:22, Adarsh Sharma wrote:
Dear all,
I am facing a unique issue when I try to load an sql into a postgresql database
:-
Actually, your issue isn't unique at all. You'll find it reoccurs on this list
regularly, although perhaps less frequent these days.
I faced an issue some days ago & I solved the issue by the below command :
cat backup.sql | recode iso-8859-1..u8 > backup.sql
That command assumes that every string in the sql file is encoded as iso-8859-1
(unless it already is unicode).
But this time the byte sequence changes to Japanese , & I fail to solve the
issue. Please let me know how to solve the issue as typing the error in Google
shows only one link:
( http://blog.e-shell.org/134 )
The above recode command works for the guys in the blog post you linked, as
they were converting a database with Spanish data to UTF-8. They knew what
encoding they were coming from.
In your case, you have a mixed bag of encodings, going all the way from latin
encodings to japanese.
I'm not sure what recode would do to data that's in a different encoding than
the specified source encoding - I expect that it will just assume it's in the
specified source encoding (it cannot know that this isn't the case for a
particular string) and attempt to convert it to UTF-8 _using that encoding_.
Chances are you just converted valid data in a different encoding (than the
source encoding you specified) into garbage in UTF-8... I seem to recall that
if recode runs into problems recoding a string to UTF-8 it will leave it
untouched, but that will NOT happen in all cases. Sometimes it will succeed,
even though the result has no meaning to a human.
That's a nasty problem you ran into, I hope the archives provide the wisdom you
need.
Alban Hertroys
--
Screwing up is an excellent way to attach something to the ceiling.
