Template Toolkit and UTF8 (was Re: UTF-8 encoding problems under Apache 2 with mod_perl 2.)

[Clinton Gormley]

I realise that you're probably not using template toolkit, but on a
separate but related note:

For those on the list using Template Toolkit, if your templates contain
UTF8, you need to prefix them with a UTF8 BOM for them to be recognised
as UTF8, otherwise TT gets really confused.

See here for more details:
http://template-toolkit.org/pipermail/templates/2004-June/006270.html
http://template-toolkit.org/pipermail/templates/2005-July/007532.html

In other words the first three bytes of the file need to be :
"\x{EF}\x{BB}\x{BF}"

I have a small script which I run on my templates which checks for the
BOM and adds one as necessary:

#!/usr/bin/perl
use strict;
use warnings FATAL => 'all';

our $root = '/PATH/TO/TEMPLATES';
our $bom  = "\x{EF}\x{BB}\x{BF}";

process_dir($root);

sub process_dir {
    my $dir   = shift;
    my @files = glob( $dir . "/*" );
    foreach my $file (@files) {
        if ( -f $file && $file =~ /\.tt$/ ) {
            process_file($file);
        }
        elsif ( -d $file && $file !~ m|/\.svn| ) {
            process_dir($file);
        }
    }
}

sub process_file {
    my $name = my $file = shift;
    $name =~ s/^$root//;
    print sprintf( "Processing : %-50s", $name );
    local ( *FH, $/ );
    open( FH, '<:bytes', $file )
        or die "can't open $file: $!";
    my $a = <FH>;
    close FH;

    my $b = $a;
    $a =~ s/$bom//g;
    $a = $bom . $a;
    if ( $a ne $b ) {
        open( FH, '>:bytes', $file )
            or die "can't write to $file : $!";
        print FH $a;
        close FH
            or die "can't close file $file";
        print " ...Updated\n";
    }
    else {
        print "\n";
    }
}