On 2/27/2017 8:57 PM, Cirilo Bernardo wrote: > On Tue, Feb 28, 2017 at 1:07 AM, Wayne Stambaugh <stambau...@gmail.com> wrote: >> On 2/26/2017 4:04 PM, Cirilo Bernardo wrote: >>> There is one other way which I found after much digging >>> and it involves a GCC extension. Since we use GCC on >>> Windows this might be acceptable: >>> >>> a. create a derived class of std::ifstream/ofstream. On >>> Windows the derived class will be used while on other >>> OS it will simply be typedef to std::ifstream/ofstream >> >> This seems reasonable to me. >> > > I had a look at the GCC STL implementation and unfortunately > is is impossible for me to implement (a) since I can't accomplish > what I want by deriving std::ifstream/ofstream due to the access > specifiers on the necessary member variables and the fact that > the open() function is not declared virtual. > >>> >>> b. overload open() to use the gcc extension like this: >>> __gnu_cxx::stdio_filebuf buf( _wopen ( utf8_filename, _O_RDONLY ) ); >>> std::istream mystream ( &buf ); >> >> If this is portable, than I'm file with this as well but on the surface >> it looks gcc specific. If that is the case, then I would rather got >> with option a. >> > > Even solution (a), which I now know is not possible, would have been > a gcc-specific hack. > > The solution I'm working on at the moment requires the replacement of > > std::ifstream X; > X.open( filename, ... ); > X.close(); > > with > > OPEN_ISTREAM( X, filename ); > CLOSE_STREAM( X ); > > On builds which are not MinGW the helper macros generate exactly > the same code as before. On MinGW builds, the helper macros > create an extra class which creates an i/ostream and cleans up > where required on destruction. The only caveat in MinGW is that > rather than an explicit ifstream/ofstream the object which is actually > created is an istream/ostream, but this is not a difficult thing to handle. > The good thing about preserving the use of std::iostream is that I > can eliminate some of the locale switching code and simply use > imbue() on the open streams to avoid unintended effects on other > code.
I'm not sure at this point why you wouldn't just use FILE_LINE_READER or wxFFileInputStream which we know both work with utf8 file names. It seems a bit like reinventing the wheel. I realize this doesn't solve the oce issue but for KiCad's file parsing usage, I think it makes more sense. I'm not saying your solution isn't valid, it just seems like unnecessary work. > > I still need to look into how the issue with OCE can be tackled then > see if the devs are willing to make changes. This UTF8 filename > problem has come up on the MinGW list many times and a number > of users had suggested various changes over the years but this > really seems to be a "won't fix" issue. I spent about an hour the other day looking through the opencascade documentation and the oce source code and I couldn't find the code for the file parsers. Could you point me to the source file where the base file parser code lives so I can take a look at it. We can open utf-8 file names just fine in mingw. I don't understand why oce cannot open them on mingw as well. > > If we use anything other than gcc on Windows we can tackle the > other issues then. > > - Cirilo > >>> >>> The destructor must contain code to delete buf since >>> the istream will not delete it on destruction. >>> >>> If that would be acceptable I'll make some test >>> programs and work on a patch set. This solution >>> would also work on OCE and I can talk to the OCE >>> patch team to see what they think of it. >>> >>> - Cirilo >>> >>> >>> On Mon, Feb 27, 2017 at 4:54 AM, Wayne Stambaugh <stambau...@gmail.com> >>> wrote: >>>> On 2/26/2017 1:50 AM, Cirilo Bernardo wrote: >>>>> Hi folks, >>>>> >>>>> This whole thing with UTF-8 filenames in Windows is a disaster. >>>>> What I've found so far: >>>>> >>>>> 1. Regarding OCE: Since OCE 0.17 (OpenCascade 6.8) UTF8 >>>>> filenames have been supported when built with MSVC but >>>>> obviously not with MinGW. >>>>> >>>>> 2. MinGW does not provide any means of transparently using >>>>> UTF-8 filenames. All filenames within the STL *must* be >>>>> char* and MinGW *will* simply pass these on to OpenFileA() >>>>> on Windows resulting in UTF-8 being interpreted as ASCII-8 >>>>> (and who uses ASCII-8 filenames anyway). >>>>> >>>>> So everything hinges on (2). If OCE uses std::stream then >>>>> fixing all issues under Windows is a lost cause. If OCE >>>>> simply plays with FILE* then it can be patched to work >>>>> in MinGW by invoking _wfopen() rather than fopen(). >>>>> >>>>> As for kicad itself, std::stream is used in: >>>>> (a) VRML export >>>>> (b) IDF static library >>>>> (c) Scenegraph dynamic library for 3D plugins >>>>> >>>>> 2 paths forward come to mind and both will involve some work: >>>>> >>>>> (1) Move to the MSVC build system on Windows: this makes >>>>> it possible for us to use Microsoft extensions to STL to deal >>>>> with non-ASCII filename issues. There is no need to dig into >>>>> the OCE code since we know it will work correctly when built >>>>> with MSVC. >>>> >>>> This is not an acceptable solution. It's not portable and would limit >>>> windows builds to using msvc. >>>> >>>>> >>>>> (2) Rework kicad code to play with FILE* (or wxFileStream) >>>>> rather than std::ifstream/ofstream. Although this will fix the >>>>> issues which are confined to kicad's source, it does nothing >>>>> to address the OCE issue. Whether or not OCE in MinGW >>>>> is a lost cause remains to be seen. >>>> >>>> FILE* is how we pretty much do it everywhere else in KiCad with good >>>> results so I don't see any reason not to do it this way with the model >>>> parser code. At least it's portable across all build platforms. >>>> Doesn't oce have a reader function that takes a FILE *? >>>> >>>>> >>>>> One other possibility (but one which I hadn't looked into) >>>>> is to see if the STL implementation within MinGW uses the >>>>> MinGW-CRT. If it does then it may be possible to fix >>>>> everything by ensuring that the MinGW-CRT converts all >>>>> filenames to UTF16 and opens a file using FileOpenW(). >>>>> In all cases this is not a pleasant task. >>>>> >>>>> Any comments/suggestions? >>>>> >>>>> - Cirilo >>>>> >>>>> _______________________________________________ >>>>> Mailing list: https://launchpad.net/~kicad-developers >>>>> Post to : kicad-developers@lists.launchpad.net >>>>> Unsubscribe : https://launchpad.net/~kicad-developers >>>>> More help : https://help.launchpad.net/ListHelp >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Mailing list: https://launchpad.net/~kicad-developers >>>> Post to : kicad-developers@lists.launchpad.net >>>> Unsubscribe : https://launchpad.net/~kicad-developers >>>> More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~kicad-developers Post to : kicad-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~kicad-developers More help : https://help.launchpad.net/ListHelp
