On 2/28/2017 7:07 PM, Cirilo Bernardo wrote: > On Wed, Mar 1, 2017 at 10:33 AM, Wayne Stambaugh <stambau...@gmail.com> wrote: >> On 2/28/2017 4:08 PM, Cirilo Bernardo wrote: >>> On Wed, Mar 1, 2017 at 12:18 AM, Wayne Stambaugh <stambau...@gmail.com> >>> wrote: >>>> On 2/27/2017 8:57 PM, Cirilo Bernardo wrote: >>>>> On Tue, Feb 28, 2017 at 1:07 AM, Wayne Stambaugh <stambau...@gmail.com> >>>>> wrote: >>>>>> On 2/26/2017 4:04 PM, Cirilo Bernardo wrote: >>>>>>> There is one other way which I found after much digging >>>>>>> and it involves a GCC extension. Since we use GCC on >>>>>>> Windows this might be acceptable: >>>>>>> >>>>>>> a. create a derived class of std::ifstream/ofstream. On >>>>>>> Windows the derived class will be used while on other >>>>>>> OS it will simply be typedef to std::ifstream/ofstream >>>>>> >>>>>> This seems reasonable to me. >>>>>> >>>>> >>>>> I had a look at the GCC STL implementation and unfortunately >>>>> is is impossible for me to implement (a) since I can't accomplish >>>>> what I want by deriving std::ifstream/ofstream due to the access >>>>> specifiers on the necessary member variables and the fact that >>>>> the open() function is not declared virtual. >>>>> >>>>>>> >>>>>>> b. overload open() to use the gcc extension like this: >>>>>>> __gnu_cxx::stdio_filebuf buf( _wopen ( utf8_filename, _O_RDONLY ) ); >>>>>>> std::istream mystream ( &buf ); >>>>>> >>>>>> If this is portable, than I'm file with this as well but on the surface >>>>>> it looks gcc specific. If that is the case, then I would rather got >>>>>> with option a. >>>>>> >>>>> >>>>> Even solution (a), which I now know is not possible, would have been >>>>> a gcc-specific hack. >>>>> >>>>> The solution I'm working on at the moment requires the replacement of >>>>> >>>>> std::ifstream X; >>>>> X.open( filename, ... ); >>>>> X.close(); >>>>> >>>>> with >>>>> >>>>> OPEN_ISTREAM( X, filename ); >>>>> CLOSE_STREAM( X ); >>>>> >>>>> On builds which are not MinGW the helper macros generate exactly >>>>> the same code as before. On MinGW builds, the helper macros >>>>> create an extra class which creates an i/ostream and cleans up >>>>> where required on destruction. The only caveat in MinGW is that >>>>> rather than an explicit ifstream/ofstream the object which is actually >>>>> created is an istream/ostream, but this is not a difficult thing to >>>>> handle. >>>>> The good thing about preserving the use of std::iostream is that I >>>>> can eliminate some of the locale switching code and simply use >>>>> imbue() on the open streams to avoid unintended effects on other >>>>> code. >>>> >>>> I'm not sure at this point why you wouldn't just use FILE_LINE_READER or >>>> wxFFileInputStream which we know both work with utf8 file names. It >>>> seems a bit like reinventing the wheel. I realize this doesn't solve >>>> the oce issue but for KiCad's file parsing usage, I think it makes more >>>> sense. I'm not saying your solution isn't valid, it just seems like >>>> unnecessary work. >>>> >>> >>> The std::stream objects have modular localization support which we need >>> to force "C" locale for VRML and IDF output and of course the '<<' and '>>' >>> stream operators. The code which needs to be reworked already uses >>> streams, so this gcc-specific hack is the easiest way to fix the UTF8 >>> issue within KiCad (but of course not for external libs like OCE). This >>> change is already a 1600+ lines patch + a few hundred lines for the >>> additional files. Changing the modules to use FILE_LINE_READER >>> means that we need to perform app-wide locale changes just to input/ >>> output a file since wx does not implement stream locale settings, plus >>> I would need to change the many hundreds (possibly a few thousand) >>> lines with streaming ops. >> >> Please fix it this way. I'm not sure I really like it but it sounds >> like you've programmed yourself into a hole. I try to avoid the stream >> << and >> operators just because of these issues. >> > > Hi Wayne, > > Do you mean use the wxStreams instead of std::stream?
Now I'm confused. If you can fix it without resorting to a gcc-specific hack, that would be my preference but I assumed that you were talking about std::stream. > > - Cirilo > >>> >>> >>>>> >>>>> I still need to look into how the issue with OCE can be tackled then >>>>> see if the devs are willing to make changes. This UTF8 filename >>>>> problem has come up on the MinGW list many times and a number >>>>> of users had suggested various changes over the years but this >>>>> really seems to be a "won't fix" issue. >>>> >>>> I spent about an hour the other day looking through the opencascade >>>> documentation and the oce source code and I couldn't find the code for >>>> the file parsers. Could you point me to the source file where the base >>>> file parser code lives so I can take a look at it. We can open utf-8 >>>> file names just fine in mingw. I don't understand why oce cannot open >>>> them on mingw as well. >>>> >>> >>> The OpenCascade source is easily grepped for FILE, fstream and so on. >>> The vast bulk of OpenCascade modules make use of defined classes to >>> handle I/O in various 'FSD' files, to to find all relevant files: >>> >>> find . -name "*FSD*" >>> >>> You can see the use of "_wopen" in those classes to open filestreams; >>> this is a Microsoft extension which is available in MSVC but not in >>> MinGW since the STL is different. Other directories of interest are: >>> >>> src/STEPControl >>> src/IGESControl >>> src/STEPCAFControl >>> src/IGESCAFControl >> >> Thanks for the info. I'll take a look at it when I can. >> >>> >>> - Cirilo >>> >>> >>>>> >>>>> If we use anything other than gcc on Windows we can tackle the >>>>> other issues then. >>>>> >>>>> - Cirilo >>>>> >>>>>>> >>>>>>> The destructor must contain code to delete buf since >>>>>>> the istream will not delete it on destruction. >>>>>>> >>>>>>> If that would be acceptable I'll make some test >>>>>>> programs and work on a patch set. This solution >>>>>>> would also work on OCE and I can talk to the OCE >>>>>>> patch team to see what they think of it. >>>>>>> >>>>>>> - Cirilo >>>>>>> >>>>>>> >>>>>>> On Mon, Feb 27, 2017 at 4:54 AM, Wayne Stambaugh <stambau...@gmail.com> >>>>>>> wrote: >>>>>>>> On 2/26/2017 1:50 AM, Cirilo Bernardo wrote: >>>>>>>>> Hi folks, >>>>>>>>> >>>>>>>>> This whole thing with UTF-8 filenames in Windows is a disaster. >>>>>>>>> What I've found so far: >>>>>>>>> >>>>>>>>> 1. Regarding OCE: Since OCE 0.17 (OpenCascade 6.8) UTF8 >>>>>>>>> filenames have been supported when built with MSVC but >>>>>>>>> obviously not with MinGW. >>>>>>>>> >>>>>>>>> 2. MinGW does not provide any means of transparently using >>>>>>>>> UTF-8 filenames. All filenames within the STL *must* be >>>>>>>>> char* and MinGW *will* simply pass these on to OpenFileA() >>>>>>>>> on Windows resulting in UTF-8 being interpreted as ASCII-8 >>>>>>>>> (and who uses ASCII-8 filenames anyway). >>>>>>>>> >>>>>>>>> So everything hinges on (2). If OCE uses std::stream then >>>>>>>>> fixing all issues under Windows is a lost cause. If OCE >>>>>>>>> simply plays with FILE* then it can be patched to work >>>>>>>>> in MinGW by invoking _wfopen() rather than fopen(). >>>>>>>>> >>>>>>>>> As for kicad itself, std::stream is used in: >>>>>>>>> (a) VRML export >>>>>>>>> (b) IDF static library >>>>>>>>> (c) Scenegraph dynamic library for 3D plugins >>>>>>>>> >>>>>>>>> 2 paths forward come to mind and both will involve some work: >>>>>>>>> >>>>>>>>> (1) Move to the MSVC build system on Windows: this makes >>>>>>>>> it possible for us to use Microsoft extensions to STL to deal >>>>>>>>> with non-ASCII filename issues. There is no need to dig into >>>>>>>>> the OCE code since we know it will work correctly when built >>>>>>>>> with MSVC. >>>>>>>> >>>>>>>> This is not an acceptable solution. It's not portable and would limit >>>>>>>> windows builds to using msvc. >>>>>>>> >>>>>>>>> >>>>>>>>> (2) Rework kicad code to play with FILE* (or wxFileStream) >>>>>>>>> rather than std::ifstream/ofstream. Although this will fix the >>>>>>>>> issues which are confined to kicad's source, it does nothing >>>>>>>>> to address the OCE issue. Whether or not OCE in MinGW >>>>>>>>> is a lost cause remains to be seen. >>>>>>>> >>>>>>>> FILE* is how we pretty much do it everywhere else in KiCad with good >>>>>>>> results so I don't see any reason not to do it this way with the model >>>>>>>> parser code. At least it's portable across all build platforms. >>>>>>>> Doesn't oce have a reader function that takes a FILE *? >>>>>>>> >>>>>>>>> >>>>>>>>> One other possibility (but one which I hadn't looked into) >>>>>>>>> is to see if the STL implementation within MinGW uses the >>>>>>>>> MinGW-CRT. If it does then it may be possible to fix >>>>>>>>> everything by ensuring that the MinGW-CRT converts all >>>>>>>>> filenames to UTF16 and opens a file using FileOpenW(). >>>>>>>>> In all cases this is not a pleasant task. >>>>>>>>> >>>>>>>>> Any comments/suggestions? >>>>>>>>> >>>>>>>>> - Cirilo >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Mailing list: https://launchpad.net/~kicad-developers >>>>>>>>> Post to : kicad-developers@lists.launchpad.net >>>>>>>>> Unsubscribe : https://launchpad.net/~kicad-developers >>>>>>>>> More help : https://help.launchpad.net/ListHelp >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Mailing list: https://launchpad.net/~kicad-developers >>>>>>>> Post to : kicad-developers@lists.launchpad.net >>>>>>>> Unsubscribe : https://launchpad.net/~kicad-developers >>>>>>>> More help : https://help.launchpad.net/ListHelp >> _______________________________________________ Mailing list: https://launchpad.net/~kicad-developers Post to : kicad-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~kicad-developers More help : https://help.launchpad.net/ListHelp
