Sun, 21 Feb 2016 14:09:26 +0100 Federico Giannici <giann...@neomedia.it>
> On 02/21/16 01:05, li...@wrant.com wrote:
> > Sat, 20 Feb 2016 20:06:57 +0100 Federico Giannici <giann...@neomedia.it>
> >> In a server (OpenBSD amd64 5.7) with many concurrent perl programs that
> >> have to open a lot of SSH connections, I get many errors like this:
> >
> > Have you tried connection sharing with ssh(1) yet? Does the Net:SSH
> > give you this?
> >
> >> connect() on closed socket GEN136 at
> >> /usr/local/libdata/perl5/site_perl/Net/SSH/Perl.pm line 216.
> >>
> >> Maybe at some point no more sockets can be opened because of some limit
> >> is reached?
> >>
> >> I already tried to set these in sysctl.conf:
> >>
> >> kern.maxfiles=20000
> >> kern.somaxconn=1024
> >>
> >> And set this in login.conf:
> >>
> >> openfiles-cur=1024
> >>
> >> But it didn't solved.
> >>
> >> Any suggestion of what a can look at?
> >
> > You could start simple and then advance to the complete programmatic
> > solution. Possibly, but not necessarily limited to these:
> >
> > ssh(1) connection sharing
>
> Connections are to (a lot of) different devices, so cannot do connection
> sharing.
Can you count these or provide a range (up to a decimal order of
magnitude precision) for the sake of narrowing it down?
Can you start putting prints at various locations and see which parts
are not getting lucky with completion?
Can you work at scheduling / queuing issues so you won't have to work
on these when it's too late (after you bump up your temp stall now)?
> > ksh(1) ulimit
> > login.conf(5)
>
> I already increased "openfiles-cur".
Intentionally leave out sysctl(3) out of your way first iteration:
sysctl.conf(5) if you think it's sockets (do you think it's air you're
breathing)
It's not everything there is to limits, so quick revisit on this:
shell limits, login class, sysctl, options (wait a moment)...
interpreter (Perl), module/class (Net::SSH) and before all this your
program algorithm.
When the program is as good as it gets, then work your way into the
environment it runs on each pass revisiting first your program.
(general hobby?is?t logic)
> Are there other limits that can give problems to socket opening
> ("connect() on closed socket" or "Bad file descriptor")?
ksh(1) ulimit
> The program works, the problems arise when there are too many concurrent
> connections.
Definitely, but see if there is some obvious large resource waste on
program inefficiency / sub-optimal organisation which can spread the
contention over time, or organise it to not ever reach limits.
Twisting knobs should be more conservative each escalation level you
pass. Speaking from user level (not authoritative).
> Thanks.
>
> > Maybe just then work on the eventual Perl related issues after you
> > confirm you can achieve the desired results with plain shell commands,
> > one liner (scripts) and direct ssh client connect using connection
> > sharing.
