Hi Sebb
Here are my observations:
1)
When retrieving a file, FTP servers send the transfer status right after they
send the EOF on the data socket, but before the client closes the socket.
This may lead to a race condition:
If we've sent a NOOP not longer than a second ago and want to consume the NOOP
reply, we may receive the transfer status instead. Subsequent call to
completePendingCommand() will consume the NOOP reply.
Maybe it's not a big problem, because we will have either
226-File successfully transferred
or
200 NOOP command successful.
as our last message and a high-level FTPClient method will return true.
2) Pure-FTPd:
During a transmission Pure-FTPd replies to NOOPs immediately with:
500 Unknown command
(perhaps because it only expects ABOR in this state)
At the end of transmission it sends the transmission status (usually 226)
3) BulletProof FTP Server:
During a transmission Pure-FTPd replies to NOOPs immediately with:
200 NOOP command successful.
If there was at least one NOOP and the command is STOR, the transmission status
is not sent.
If there were no NOOPs or the command is RETR, then the status is returned
(usually 226)
4) Most other FTP servers:
NOOP replies are delayed. After a transmission completes, servers send 226
first and the NOOP replies after that.
Here are possible response sequences:
- Pure-FTPd success
500,500,500,500,226
500,500,500,226,200 (in case of race condition)
- Pure-FTPd error (RETR data socket closed by client)
500,500,500,500,150
500,500,500,150,200 (in case of race condition)
- BulletProof FTP Server STOR, success/error (STOR data socket closed by client)
200,200,200,200,---
200,200,200,---,200 (in case of race condition)
226 (when no NOOPs sent)
- BulletProof FTP Server RETR, success
226 (when no NOOPs sent)
200,200,200,200,---
200,200,200,---,200 (in case of race condition)
- BulletProof FTP Server RETR, error (RETR data socket closed by client)
426 (when no NOOPs sent)
200,200,200,200,426
200,200,200,426,200 (in case of race condition)
- Most other FTP servers success
226,200,200,200,200 (no race condition possible)
- Most other FTP servers error (RETR data socket closed by client)
426,200,200,200,200 (no race condition possible)
Conclusion:
If we consume NOOP responses during the transmission, most of the time
completePendingCommand() consumes the right response:
If a server does not support asynchronous NOOPs, then completePendingCommand()
consumes the status and cleanUp() consumes all NOOP responses.
If a server supports asynchronous NOOPs, then all NOOP replies are consumed
before completePendingCommand() is called.
(Except for BulletProof STOR, that forgets to send 226, so
completePendingCommand() fails with SocketTimeoutException)
In case of a race condition (for asyncronous servers):
If transmission succeeds, completePendingCommand() consumes either 226 or "200
NOOP command successful." and return true in either case.
If transmission fails, Util.copyStream() may throw something. If it doesn't,
completePendingCommand() consumes "200 NOOP command successful." and returns
false positive.
This can happen if last data block was sent by client, but never received by
server.
If we delay consuming NOOP responses from asynchronous servers,
completePendingCommand() always consumes the first NOOP response.
In case of Pure-FTPd, it is "500 Unknown command", so the method returns false
negative.
For other asynchronous servers it is "200 NOOP command successful." and a
possible 426 status is eaten by cleanUp(), so the method returns false positive.
On the other hand, completePendingCommand() won't throw SocketTimeoutException
for BulletProof STOR and besides, for servers that don't support async NOOPs we
don't have to
wait in __noop()
I think that what we need to do is to collect (#NOOPs + 1) responses at the end
with a read timeout 10s, filter out 500 and 200 and the remaining status will
be 226 or an
error status.
On 19.10.2017 20:46, Basin Ilya wrote:
> Ok, so I found these two that support it:
> - BulletProof FTP Server
> - Pure-FTPd
>
> Will test further.
>
> On 17.10.2017 18:29, sebb wrote:
>> On 17 October 2017 at 16:01, Basin Ilya <basini...@gmail.com> wrote:
>>> Hi sebb
>>>
>>>> No, because some FTP servers *do* support asynchronous control channels.
>>> Do you know any?
>>
>> I cannot remember the name, but I know I came across at least one when
>> testing.
>>
>> But even if there were currently no such servers, AFAIK it is
>> permitted by the RFCs so NET should not assume there are none.
>>
>>>
>>> On 17.10.2017 17:54, sebb wrote:
>>>> On 17 October 2017 at 12:34, Basin Ilya <basini...@gmail.com> wrote:
>>>>> Hi.
>>>>> I'm using
>>>>> FTPClient.retrieveFileStream()
>>>>>
>>>>> and therefore I need to implement keepalive mechanism by my own.
>>>>
>>>> Not necessarily.
>>>>
>>>> The FTP server does not need the NOOPs.
>>>> They are only needed to deal with routers that detect the inactive
>>>> control channel and decide to drop the connection even though the data
>>>> channel is active.
>>>>
>>>>> I wanted to mimic the implementation from FTPClient.CSL, but then I
>>>>> thought:
>>>>>
>>>>> Most FTP servers don't reply to NOOPs until transmission has finished and
>>>>> yet, sending NOOPs helps to keep them alive.
>>>>
>>>> No, the FTP servers don't need the NOOPs.
>>>> And some do reply before data transmission completes.
>>>>
>>>>> So we can safely assume that no FTP servers support this and let
>>>>> CSL.cleanUp() collect all the responses at the end.
>>>>
>>>> No, because some FTP servers *do* support asynchronous control channels.
>>>>
>>>>> My tests show vsftpd 2.2.2 does not support that and you can send up to
>>>>> 27000 NOOP commands to it until the socket write gets blocked.
>>>>>
>>>>> On the other hand, on most FTP servers for each NOOP keepalive FTPClient
>>>>> waits for 1000 milliseconds for a reply that never comes. This slows
>>>>> things down a little.
>>>>>
>>>>> What bad thing will happen, if we remove __getReplyNoReport() from
>>>>> FTP.__noop() and increment notAcked unconditionally?
>>>>
>>>> Try it and see?
>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>>>>> For additional commands, e-mail: dev-h...@commons.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>>>> For additional commands, e-mail: dev-h...@commons.apache.org
>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org
