Vadim:
Sorry I have not chimed in earlier.. I tend to
"not look" often at some of my boxes :-)
Glad Michael helped out here .. thanks Michael
Vadim Goncharov wrote:
Hi Michael Tuexen!
On Thu, 6 Mar 2008 09:34:13 +0100; Michael Tuexen wrote about 'Re: SCTP using
questions (API etc.)':
"substreams". SCTP can do it for me, it's wonderful, but in practice
there
are some questions.
How long can be one particular SCTP message? Can I rely on the fact
that it
can be unbounded, e.g. I want to emulate a stream with transfer of
6 Gig-sized file?
Protocol wise there is no limitation of the message size. API wise,
for
this size of a message you need to use the explicit EOR mode to be
able
to pass this large message using multiple sequential send() calls.
And how should I determine from my/remote stack an optimal size for
message
parts when entire message is guaranteed to not fit into buffers/
windows of
both peers?
If the sendbuffer is too small for the message to fit, the send call
will return -1 and errno being set to EMSGSIZE. Or you do it in the
application
by inspecting the sendbuffer size. You do not have to deal with the
recv buffer
of the peer.
So this means I need no subscription to unsent messages and simply can try
to resend message in several steps without EOR, after getting EMSGSIZE ?
So, if you have put your socket int EEOR mode, then
you could send multiple sends down the socket (to the same stream)
until you get back a EWOULDBLOCK. You would only get a
EMSGSIZE if the value you are sending is larger than the entire
size of the send socket buffer..
So lets say the send buffer for the socket is 100k
You could do
while(ok) {
send(1k[index]);
if(ret == -1) && error == EWOULDBLOCK)
hit full buffer (100k is inqueue)
go do wait or other thing
resume send(1k[index]
else
index++;
Either all of the buffer or none of the buffer will be sent.
Can a message be of zero-length data (only headers) ?
Empty user messages, i.e. a DATA chunk without payload is not
allowed.
An empty SCTP message, i.e. only the common header without any chunks
is allowed and processed by FreeBSD when received, but ever send
(well,
I do not know a way to force the FreeBSD implementation to send it).
OK, understood. So I should include at least 1 byte of my own
headers into
data and do receive into *iov with at least to parts to ensure good
align
for non-header part?
What header are you talking about? An application header or any SCTP
header?
You will never receive any SCTP header as part of a user message via
a recv() call. SCTP will give you as much of a message that fits into
the buffer you provide or it has, if the partial delivery API has been
invoked.
My applicaion-protocol header, of course. Does this mean also that I should
always enable partial delivery on receiving? Or what will happen if received
msg is too big and don't fir into my buffers?
Well, you have no control over this per.se. You can get partial delivery
events.. there is only one. the partial delivery was aborted.. you probably
need this if you are going to do EEOR mode.
Basically the kernel will start a partial delivery when 1/2 of the recv
buffer
is in use. Note there is a socket option to control this value, so you can
change it if you like...
What is the relation between SCTP streams in both directions? Can
streams
be opened and closed on-demand, like SSH port forwarding (yet again
multiplexing example) or they are preallocated at connection setup
all
together? What is the minimum number of streams application can rely
upon
(or it just one stream 0 in general case) ?
If you restrict to protocols being in RFC status, there is no way of
modifying the number of streams during the lifetime of an
association.
The number of streams in each direction is negotiated during the
association setup. The streams in bother directions are completely
independent. There is always at least one stream in each direction,
which
is stream 0.
However, there is an extension (currently specified in an Internet
Draft)
which allows the addition of streams during the lifetime of an
association.
The ID is at least partially supported by the FreeBSD implementation.
https://datatracker.ietf.org/drafts/draft-stewart-sctpstrrst/
OK. Are there recommended defaults for various stacks about how many
streams they are creating by default / what maximum of them
application
can ever request?
The maximum number to request is 2^16 - 1. It is controllable by the
applications via socket options. Defaults in OSes are in the order of
10, 16, 32...
Can I be sure that every OS can give me maximum number of streams if I
request it?
The ceiling for the number of streams is actually a defined contnsatn
in the BSD stack (and in most).
For bsd its defined in
sctp_constants.h
and is
#define MAX_SCTP_STREAMS 2048
I probably should make this a configurable item ... hmm..
Each stream outbound costs about 16 bytes.
Each stream inbound costs about 16 bytes...
Thus my desire to limit to some extent resources used.. I think
most kernels do this as well.
You of course can twiddle the define, and I think for 8.x I will see about
making it an option.
How can I put request to kernel for a connect, for example, and then
sleep
until connect will complete or event in some another descriptor will
occur?
If you use the 1-to-1 style API, it should be similar to using TCP.
Put the socket in non-blocking mode, enable notifications,
call connect() and wait until the fd becomes readable. You should
get an
indication that that association has been established or could not
start.
Yes, that's possible, as I see after reading draft-ietf-tsvwg-
sctpsocket.
But several more questions arise. What notifications do I really need
on 1-to-1 non-blocking socket API mode? What use is 'context' in this
1-to-1 context and why after a failed send I must receive entire
failed
sent message (which can be very long) instead of just an error code?
The context is something you provide in the send call and is given
back to you. So you can use it to find some state/buffer/whatever again.
It was unclear from draft whether context is one per SCTP association or per
send call. And what the hell are all that unsent messages, why I must
retrieve entire unsent message - can I fire-and-forget a 2M msg and receive
only context of it instead of all 2 megs? And on which condition such event
can ever occur - with TCP it's simple, I either do write() a number of bytes
successfully or receive an error from write() - be that EAGAIN for just
blocking of peer's recv() or connection termination error. What concept is
under unsent msgs?
The idea is that you can see the message that did not get sent. And
you can know if it was every sent .. i.e. put on the wire but
unack'd or never put on the wire.
We don't currently have a way to not get the entire message up (sorry
no one ever asked for that)...
The context is kept per message if I remember right.. Its copied from
the sinfo_context field and then carried with the queued data
until its acked and freed.
I believe you can set a default context as well..
In usual FSM I can use kqueue()/kevent() with arbitrary void* to my
data, also telling me how many bytes I can read from or write to
the socket (RCVLOWAT etc.), as well as indicating error/EOF conditions
so I don't need to do additional syscalls. Is this working with SCTP?
Haven't tried it... Sounds like it would make sense to make sure that
it works.
Oh, can you please check it?.. Would be good to support all features
described in kqueue(2).
I rather doubt this works, since we don't use socket buffers.. pe.se.
I will have to go take a look at it and will proabably need to add
that to my TODO list.
Michael just finished getting it to work INET only.. (no v6).. good work
Michael :-D
If I can't write to TCP socket (due to window shortage from peer),
I leave data in my own application buffers, but SCTP tells something
about unsent messages delivered later, looks somewhat weird, do I
really
need this? Also, all that msg*/cmsg* staff is too complex, and I see
there are simplier sctp_send()/sctp_sendx() interfaces, will they be
enough and really simplier for me?..
sctp_sendx() purpose is to use the multiple addresses provided during
the implicit setup of the association. So I think you are not looking
for
Ok.
this. sctp_send() can be used to provide the stream id, payload protocol
identifier and to on with using the CMSG stuff. So you might be looking
for this function.
With CMSG? May be you wanted to say 'without' ?..
Yep,
The sctp_xxx send calls are true function calls so they do not
have the intense overhead of the app encoding ancillary data
and the kernel un-encoding it.. much better :-)
How can I put each client to it's fd and then do a kqueue()/kevent()
on a
set of those fd's (and other items) ? It is very handy to have this
architecture as kevent() allows to store an arbitrary void* in it's
structure which I can later use to quickly dispatch events.
And, of course, all this usual C10K-problem-solving-TCP-server
tricks I want
with basic SCTP SEQPACKET benefits: multiple streams and message
record
separation (I don't need other SCTP features currently). Where can I
find
answers to these questions, like it was with W.R.Stevens books for
TCP ?..
Have you looked at the third edition of 'Unix Network Programming'?
Randall Stewart wrote a couple of sections covering SCTP...
Unfortunately, I have only 2nd edition currently available here,
though
heard about 3rd, yes. May be several months later...
It is really worth buying if you are interested in SCTP socket
programming...
I know, but in my province it is currently unavailable for some months...
you know, Siberia, bears walking on the streets :) but it is not critical
for actual SCTP programming (TCP version will be debugged first), but I need
to take architectural decisions now.
Also, are there some examples of real-world SCTP applications with source
code available? May be something is getting to integrate into our base
system?..
I could probably find some of my test code and send it to you..
I have a pretty intesensive test app that we use sctp_test_app that
does about every socket option etc.. its not pretty..(it grew organically)..
but it does cover lots of stuff..
R
--
Randall Stewart
NSSTG - Cisco Systems Inc.
803-345-0369 <or> 803-317-4952 (cell)
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
