Hi all , thanks for the replies.
>> $xls is a scalar you have in memory of the comma separated value data? Yes, correct, and when output direct to $c->response->body , catalyst server falls over with an out of memory error and reboots itself. I had an IRC discussion about this and it was acknowledged as a limitation of Catalyst and the solution was to wrap it up in a IO:File object. >> but how Windows decides how to open files is something I'm not familiar with. >> That depends on the Browser setting per mime type and not the OS. >> In Firefox you can define it in Preferences / Applications. Perhaps, but it appears default behaviour in IE / FF / Opera / Chrome to open CSV as an XLS spread sheet in Excel seems to require that MIME type and works exactly as desired when used. Reading the RFC link it states no MIME has ever officially been registered of CSV but does give the text/csv MIME type, yet as mentioned I didn't get the desired behaviour when used, is there an issue with using the MIME type I have, if it is yielding desired results? >> Maybe write a standalone test and take Catalyst and browser quirks out of >> the picture. I have already done this, I have two SQL wrapper modules one that uses DBI::DBD::ODBC and one that uses Win32::ODBC, I applied it to the same standalone script that produces CSV output, the only difference between the test was one test accessed SQL with the DBI SQL wrapper and one test accessed SQL with the Win32::ODBC SQL wrapper, DBI outputted junk chars, Win32::ODBC didn't. What else should I be doing to test for the culprit of the corruption? >> Also, you are aware that your data will probably be coming back as UCS2 if >> you're using SQL Server right? No, what is UCS2 and is this handled differently in DBI::DBD::ODBC vs Win32::ODBC ? Thanks, Craig ________________________________ From: Anthony Lucas [[email protected]] Sent: 03 July 2013 06:42 To: The elegant MVC web framework Subject: Re: [Catalyst] CSV / UTF-8 / Unicode I don't see how DBI can be corrupting your data. DBD::ODBC, or more specifically the actual driver it is using will be more suspect in that case. In either case, setting DBI's trace flag to something high should provide some insight who's doing what, and you can make sure they are being handled as wide characters. ODBC drivers are renowned for providing incomplete APIs and lying about stuff. Maybe write a standalone test and take Catalyst and browser quirks out of the picture. Also, you are aware that your data will probably be coming back as UCS2 if you're using SQL Server right? On 2 July 2013 17:29, Craig Chant <[email protected]<mailto:[email protected]>> wrote: >> All the above seems overkill. I suspect what you want is closer to this: >> (but see notes below). Tried that, didn’t work, ended up in a long Catalyst discussion where it was worked out that I needed to wrap any XLS output to an IO:FILE handle otherwise Catalyst dies with an “out of memory” error something to do with streaming data support issues in Catalyst so the work round is to wrap the output into an IO:File object. >>Second, be aware that $c->response->content_length(length($xls)); Yes, I was doing the encode then using Length (I did read on perldocs about requesting the length against the octet) , either way, the length was the least of my worries, keeping Catalyst from falling over with ‘Wide Character’ errors, or not getting garbage was my main concern. And yes, the output is CSV not strictly XLS but I have been told and looked it up on the net that 'application/vnd.ms-excel' Is the correct MIME header to pass for CSV that you want MS Excel to open. Of course, if I have the wrong MIME header for CSV -> MS Excel please can you provide the correct one, as it took me a long time to find that one, as the box standard ‘text/csv’ does not work properly when opened in MS Excel. Though as it appears DBI is corrupting my Unicode data, it might be related to that rather than CSV->MS Excel per sae! From: Bill Moseley [mailto:[email protected]<mailto:[email protected]>] Sent: 02 July 2013 16:59 To: The elegant MVC web framework Subject: Re: [Catalyst] CSV / UTF-8 / Unicode On Tue, Jul 2, 2013 at 2:59 AM, Craig Chant <[email protected]<mailto:[email protected]>> wrote: # output header $c->response->content_type('application/vnd.ms-excel'); $c->response->content_length(length($xls)); $c->response->header(Content_Disposition => 'attachment;filename=NBCS_Export.csv'); # create an IO::File for Catalyst use IO::File; my $iof = IO::File->new; $iof->open(\$xls, "r"); $iof->binmode(":encoding(UTF-8)"); # output XLS data $c->response->body($iof); All the above seems overkill. I suspect what you want is closer to this: (but see notes below). $c->response->content_type('text/csv'); $c->response->body($xls); $c->response->header(Content_Disposition => 'attachment;filename=NBCS_Export.csv'); Then with that content type the plugin would encode $xls as utf8 and add ;charset=utf8 (or whatever it is configured to encode as). Notes: First, you are not returning Excel, so the content type is not what you first listed above, right? Second, be aware that $c->response->content_length(length($xls)); could be very wrong. If $xls is really CSV text AND it's decoded then length($xls) is the length in characters, not octets. Don't set the content length. Third, Catalyst::Plugin::Unicode::Encoding, IMO, has some issues. The plugin limits to just these content types. return $c->next::method(@_) unless $c->response->content_type =~ /^text|xml$|javascript$/; Then it does this: $c->response->body( $c->encoding->encode( $body, $CHECK ) ) if ref(\$body) eq 'SCALAR'; Personally, I think the correct approach is to only encode character data -- that is check to see if the utf8 flag is set before calling encode. Maybe limit to the content types listed above, but throw an exception for other content types where the body is a scalar AND has the utf8 flag on. After all, we can only write out octets or else we get the Wide Character error. -- Bill Moseley [email protected]<mailto:[email protected]> This Email and any attachments contain confidential information and is intended solely for the individual to whom it is addressed. If this Email has been misdirected, please notify the author as soon as possible. If you are not the intended recipient you must not disclose, distribute, copy, print or rely on any of the information contained, and all copies must be deleted immediately. Whilst we take reasonable steps to try to identify any software viruses, any attachments to this e-mail may nevertheless contain viruses, which our anti-virus software has failed to identify. You should therefore carry out your own anti-virus checks before opening any documents. HomeLoan Partnership will not accept any liability for damage caused by computer viruses emanating from any attachment or other document supplied with this e-mail. HomeLoan Partnership reserves the right to monitor and archive all e-mail communications through its network. No representative or employee of HomeLoan Partnership has the authority to enter into any contract on behalf of HomeLoan Partnership by email. HomeLoan Partnership is a trading name of H L Partnership Limited, registered in England and Wales with Registration Number 5011722. Registered office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is authorised and regulated by the Financial Conduct Authority. _______________________________________________ List: [email protected]<mailto:[email protected]> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/[email protected]/ Dev site: http://dev.catalyst.perl.org/ This Email and any attachments contain confidential information and is intended solely for the individual to whom it is addressed. If this Email has been misdirected, please notify the author as soon as possible. If you are not the intended recipient you must not disclose, distribute, copy, print or rely on any of the information contained, and all copies must be deleted immediately. Whilst we take reasonable steps to try to identify any software viruses, any attachments to this e-mail may nevertheless contain viruses, which our anti-virus software has failed to identify. You should therefore carry out your own anti-virus checks before opening any documents. HomeLoan Partnership will not accept any liability for damage caused by computer viruses emanating from any attachment or other document supplied with this e-mail. HomeLoan Partnership reserves the right to monitor and archive all e-mail communications through its network. No representative or employee of HomeLoan Partnership has the authority to enter into any contract on behalf of HomeLoan Partnership by email. HomeLoan Partnership is a trading name of H L Partnership Limited, registered in England and Wales with Registration Number 5011722. Registered office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is authorised and regulated by the Financial Conduct Authority.
_______________________________________________ List: [email protected] Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/[email protected]/ Dev site: http://dev.catalyst.perl.org/
