Hi Paul,
I think I've narrowed the issue to Net::Cmd itself (read_untill_dot() used
by Net::SNMP). Even if we completely forget about writing the file to disk:
print "220 0 " . $Command['1'] . "\r\n";
print @{$Article};
print ".\r\n";
@{$Article} still only prints \n and not \r\n.
When I look at Net::Cmd getline() around line 301
my @buf = split(/\015?\012/, $buf, -1); ## break into lines
$partial = pop @buf;
push(@{${*$cmd}{'net_cmd_lines'}}, map {"$_\n"} @buf);
Isn't this exactly where my issue is coming from? The map should (in my
case) be using \r\n? I changed this quickly just as a test, and my lines
are now terminated correctly with \r\n as I desire, but obviously this has
now broken ALLOT of other things that uses Net::Cmd... Short of using my
own sockets (which I really don't want to do), would there be a way to
change this behaviour of Net::Cmd without having to change the module's
code? I don't see any options o such for it.
--
Chris.
-----Original Message-----
From: Paul Anderson [mailto:wackyvor...@me.com]
Sent: 12 August 2012 08:04
To: Chris Knipe
Cc: beginners@perl.org
Subject: Re: Net::NNTP inconsistent article retrieval
>From here:
http://perldoc.perl.org/utf8.html
I wonder if utf8::upgrade() might do the trick. With write_file() in
File::Slurp, passing it a hash with binmode set to :utf8 will get it to
write UTF8 output to disk. Perl has in general very good internal UTF
support.
An aside: You know that leafnode basically does everything you're trying to
write, right?
--------
Paul Anderson -- VE3HOP
On 2012-08-11, at 5:21 PM, Chris Knipe <sav...@savage.za.org> wrote:
> Hi Paul,
>
> I will be inclined to disagree - it depends on whether or not the content
> was encoded to begin with (perhaps I should not use the term 'binary' as
> such, but rather refer to it as unreadable characters). File::Slurp did
> make my life allot easier now yes, but my problem is still not solved.
> Looking at the packet capture, it is as clear as daylight to me that
either
> Net::NNTP or File::Slurp is replacing all \r\n line breaks with \n. The
> NNTP RFCs require all lines to be terminated by \r\n. I can't simply use
a
> regex to replace all instances of \n to \r\n as it will affect the
> attachments (binary or not, encoded or not). I've attached an screen
> capture out of Wireshark for a single packet (available at
> http://www.savage.za.org/capture.png as well) which is showing my problem
as
> clearly as daylight.
>
> Line 1: 220 0 <message-id>\r\n This is sent by my application, and it
is
> properly terminated as required.
> Line 2 through to 15 original headers as received by my upstream news
> server. The lines WAS previously terminated with \r\n but either
Net::NNTP
> or File::Slurp, changed all the \r\n values to \n
> Line 18 through to next packet is definitely not text anymore, and neither
> is it terminated by \r\n anymore as it was previously...
>
> I am not trying to process, alter, or extract articles in any way, I
simply
> want to download them, store them, and forward them...
>
> --
> Chris.
>
>
>
> -----Original Message-----
> From: Paul Anderson [mailto:wackyvor...@me.com]
> Sent: 11 August 2012 18:17
> To: Chris Knipe
> Cc: beginners@perl.org
> Subject: Re: Net::NNTP inconsistent article retrieval
>
> Umm... Are you aware that binary attachments on usenet aren't actually
*in*
> binary? They're encoded in ASCII using one of a number of different
methods.
> They're just text, until decoded on the receiving end.
>
> I recommend looking into File::Slurp and CHI. CHI basically implements the
> entire caching back-end for you.
>
> Sent from my iPhone
>
> On 2012-08-11, at 5:21 AM, Chris Knipe <sav...@savage.za.org> wrote:
>
>> Hi All,
>>
>> I'm using Net::NNTP to transfer articles from servers. My aim is to
write
>> an NNTP proxy.
>>
>> I obtain the article from my parent news server, write the file to disk,
> and
>> serve the current, as well as future requests for that article from the
>> local file on the disk. My results are very inconsistent, and I suspect
>> that it is related to the \r\n line breaks conflicting with the binary
> data
>> retrieved from the article (a line may include \r\n when it's not
actually
>> meant to indicate the END of a line). I am currently using:
>>
>> local $/ = "\r\n";
>> my $Article = $nntp->article($Command['1']);
>> if ($Article) {
>> # The parent has the article! Let's take it.
>> open FILE, ">:raw", $File or die $!;
>> binmode(FILE);
>> foreach my $line (@$Article) {
>> print FILE $line;
>> }
>> close(FILE);
>> $ReturnStr = "220 0 " . $Command['1'] . "\r\n";
>> open FILE, "<:raw", $File;
>> binmode(FILE);
>> foreach my $line (<FILE>) {
>> $ReturnStr .= $line;
>> }
>> close(FILE);
>> $ReturnStr .= ".\r\n";
>> binmode(STDOUT);
>> print $ReturnStr;
>>
>> For some articles, the above code is absolutely fine and no problems are
>> returned. For others, the binary attachments to the article (regardless
> of
>> type of file), is corrupt, and cannot be opened. The results are also
> very
>> inconsistent, and being binary I'm not exactly sure how to provide
samples
>> of what works and what doesn't. I've analysed packet captures
> excessively,
>> and I am definitely getting all the data correctly, and consistently from
> my
>> parent news server - the problem is related to me writing the file to the
>> local disk, and serving the content of that file from the local disk.
>>
>> Hopefully someone can assist and point me towards the right direction -
>> after spending close to a week on this, I'm ready to pull out my hair!
:-(
>> --
>> Chris.
>>
>>
>>
>> --
>> To unsubscribe, e-mail: beginners-unsubscr...@perl.org
>> For additional commands, e-mail: beginners-h...@perl.org
>> http://learn.perl.org/
>>
>>
> <capture.png>
--
To unsubscribe, e-mail: beginners-unsubscr...@perl.org
For additional commands, e-mail: beginners-h...@perl.org
http://learn.perl.org/
