On Wed, Mar 1, 2017 at 12:18 AM, Wayne Stambaugh <stambau...@gmail.com> wrote: > On 2/27/2017 8:57 PM, Cirilo Bernardo wrote: >> On Tue, Feb 28, 2017 at 1:07 AM, Wayne Stambaugh <stambau...@gmail.com> >> wrote: >>> On 2/26/2017 4:04 PM, Cirilo Bernardo wrote: >>>> There is one other way which I found after much digging >>>> and it involves a GCC extension. Since we use GCC on >>>> Windows this might be acceptable: >>>> >>>> a. create a derived class of std::ifstream/ofstream. On >>>> Windows the derived class will be used while on other >>>> OS it will simply be typedef to std::ifstream/ofstream >>> >>> This seems reasonable to me. >>> >> >> I had a look at the GCC STL implementation and unfortunately >> is is impossible for me to implement (a) since I can't accomplish >> what I want by deriving std::ifstream/ofstream due to the access >> specifiers on the necessary member variables and the fact that >> the open() function is not declared virtual. >> >>>> >>>> b. overload open() to use the gcc extension like this: >>>> __gnu_cxx::stdio_filebuf buf( _wopen ( utf8_filename, _O_RDONLY ) ); >>>> std::istream mystream ( &buf ); >>> >>> If this is portable, than I'm file with this as well but on the surface >>> it looks gcc specific. If that is the case, then I would rather got >>> with option a. >>> >> >> Even solution (a), which I now know is not possible, would have been >> a gcc-specific hack. >> >> The solution I'm working on at the moment requires the replacement of >> >> std::ifstream X; >> X.open( filename, ... ); >> X.close(); >> >> with >> >> OPEN_ISTREAM( X, filename ); >> CLOSE_STREAM( X ); >> >> On builds which are not MinGW the helper macros generate exactly >> the same code as before. On MinGW builds, the helper macros >> create an extra class which creates an i/ostream and cleans up >> where required on destruction. The only caveat in MinGW is that >> rather than an explicit ifstream/ofstream the object which is actually >> created is an istream/ostream, but this is not a difficult thing to handle. >> The good thing about preserving the use of std::iostream is that I >> can eliminate some of the locale switching code and simply use >> imbue() on the open streams to avoid unintended effects on other >> code. > > I'm not sure at this point why you wouldn't just use FILE_LINE_READER or > wxFFileInputStream which we know both work with utf8 file names. It > seems a bit like reinventing the wheel. I realize this doesn't solve > the oce issue but for KiCad's file parsing usage, I think it makes more > sense. I'm not saying your solution isn't valid, it just seems like > unnecessary work. >
The std::stream objects have modular localization support which we need to force "C" locale for VRML and IDF output and of course the '<<' and '>>' stream operators. The code which needs to be reworked already uses streams, so this gcc-specific hack is the easiest way to fix the UTF8 issue within KiCad (but of course not for external libs like OCE). This change is already a 1600+ lines patch + a few hundred lines for the additional files. Changing the modules to use FILE_LINE_READER means that we need to perform app-wide locale changes just to input/ output a file since wx does not implement stream locale settings, plus I would need to change the many hundreds (possibly a few thousand) lines with streaming ops. >> >> I still need to look into how the issue with OCE can be tackled then >> see if the devs are willing to make changes. This UTF8 filename >> problem has come up on the MinGW list many times and a number >> of users had suggested various changes over the years but this >> really seems to be a "won't fix" issue. > > I spent about an hour the other day looking through the opencascade > documentation and the oce source code and I couldn't find the code for > the file parsers. Could you point me to the source file where the base > file parser code lives so I can take a look at it. We can open utf-8 > file names just fine in mingw. I don't understand why oce cannot open > them on mingw as well. > The OpenCascade source is easily grepped for FILE, fstream and so on. The vast bulk of OpenCascade modules make use of defined classes to handle I/O in various 'FSD' files, to to find all relevant files: find . -name "*FSD*" You can see the use of "_wopen" in those classes to open filestreams; this is a Microsoft extension which is available in MSVC but not in MinGW since the STL is different. Other directories of interest are: src/STEPControl src/IGESControl src/STEPCAFControl src/IGESCAFControl - Cirilo >> >> If we use anything other than gcc on Windows we can tackle the >> other issues then. >> >> - Cirilo >> >>>> >>>> The destructor must contain code to delete buf since >>>> the istream will not delete it on destruction. >>>> >>>> If that would be acceptable I'll make some test >>>> programs and work on a patch set. This solution >>>> would also work on OCE and I can talk to the OCE >>>> patch team to see what they think of it. >>>> >>>> - Cirilo >>>> >>>> >>>> On Mon, Feb 27, 2017 at 4:54 AM, Wayne Stambaugh <stambau...@gmail.com> >>>> wrote: >>>>> On 2/26/2017 1:50 AM, Cirilo Bernardo wrote: >>>>>> Hi folks, >>>>>> >>>>>> This whole thing with UTF-8 filenames in Windows is a disaster. >>>>>> What I've found so far: >>>>>> >>>>>> 1. Regarding OCE: Since OCE 0.17 (OpenCascade 6.8) UTF8 >>>>>> filenames have been supported when built with MSVC but >>>>>> obviously not with MinGW. >>>>>> >>>>>> 2. MinGW does not provide any means of transparently using >>>>>> UTF-8 filenames. All filenames within the STL *must* be >>>>>> char* and MinGW *will* simply pass these on to OpenFileA() >>>>>> on Windows resulting in UTF-8 being interpreted as ASCII-8 >>>>>> (and who uses ASCII-8 filenames anyway). >>>>>> >>>>>> So everything hinges on (2). If OCE uses std::stream then >>>>>> fixing all issues under Windows is a lost cause. If OCE >>>>>> simply plays with FILE* then it can be patched to work >>>>>> in MinGW by invoking _wfopen() rather than fopen(). >>>>>> >>>>>> As for kicad itself, std::stream is used in: >>>>>> (a) VRML export >>>>>> (b) IDF static library >>>>>> (c) Scenegraph dynamic library for 3D plugins >>>>>> >>>>>> 2 paths forward come to mind and both will involve some work: >>>>>> >>>>>> (1) Move to the MSVC build system on Windows: this makes >>>>>> it possible for us to use Microsoft extensions to STL to deal >>>>>> with non-ASCII filename issues. There is no need to dig into >>>>>> the OCE code since we know it will work correctly when built >>>>>> with MSVC. >>>>> >>>>> This is not an acceptable solution. It's not portable and would limit >>>>> windows builds to using msvc. >>>>> >>>>>> >>>>>> (2) Rework kicad code to play with FILE* (or wxFileStream) >>>>>> rather than std::ifstream/ofstream. Although this will fix the >>>>>> issues which are confined to kicad's source, it does nothing >>>>>> to address the OCE issue. Whether or not OCE in MinGW >>>>>> is a lost cause remains to be seen. >>>>> >>>>> FILE* is how we pretty much do it everywhere else in KiCad with good >>>>> results so I don't see any reason not to do it this way with the model >>>>> parser code. At least it's portable across all build platforms. >>>>> Doesn't oce have a reader function that takes a FILE *? >>>>> >>>>>> >>>>>> One other possibility (but one which I hadn't looked into) >>>>>> is to see if the STL implementation within MinGW uses the >>>>>> MinGW-CRT. If it does then it may be possible to fix >>>>>> everything by ensuring that the MinGW-CRT converts all >>>>>> filenames to UTF16 and opens a file using FileOpenW(). >>>>>> In all cases this is not a pleasant task. >>>>>> >>>>>> Any comments/suggestions? >>>>>> >>>>>> - Cirilo >>>>>> >>>>>> _______________________________________________ >>>>>> Mailing list: https://launchpad.net/~kicad-developers >>>>>> Post to : kicad-developers@lists.launchpad.net >>>>>> Unsubscribe : https://launchpad.net/~kicad-developers >>>>>> More help : https://help.launchpad.net/ListHelp >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Mailing list: https://launchpad.net/~kicad-developers >>>>> Post to : kicad-developers@lists.launchpad.net >>>>> Unsubscribe : https://launchpad.net/~kicad-developers >>>>> More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~kicad-developers Post to : kicad-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~kicad-developers More help : https://help.launchpad.net/ListHelp
