iocage following stable?
Hi — I am running ezjail for some years now, but I intend to migrate to iocage. Not that I am "disappointed" with ezjail, but I do want to give VNET a try. After having read iocage's documentation and some google research, I am left with the following questions: 1) Currently I am using ezjail's functionality to update/upgrade my basejail from svn following STABLE. I am unsure if I can stick to running STABLE in iocage jails, and if so, how? "iocage chroot" and compile in /usr/src? 2) Is there an equivalent for "ezjail-admin console -e 'pkg upgrade -y -f' jailname", namely running commands inside a jail without entering it? Thanks in advance and with kind regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: iocage following stable?
Sorry, possibly not precise enough: > "iocage chroot" and compile in /usr/src? "iocage chroot" into basejail and compile in /usr/src? ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: iocage following stable?
Allan Jude wrote: > On 2015-12-18 16:18, Michael Grimm wrote: >> I am running ezjail for some years now, but I intend to migrate to iocage. >> Not that I am "disappointed" with ezjail, but I do want to give VNET a try. >> >> After having read iocage's documentation and some google research, I am left >> with the following questions: >> >> 1) Currently I am using ezjail's functionality to update/upgrade my basejail >> from svn following STABLE. I am unsure if I can stick to running STABLE in >> iocage jails, and if so, how? "iocage chroot" and compile in /usr/src? > > I don't know on this part Well, I did try that approach in the meantime, and yes, one might fake iocage in this regard. >> 2) Is there an equivalent for "ezjail-admin console -e 'pkg upgrade -y -f' >> jailname", namely running commands inside a jail without entering it? > > The generic jail command 'jexec' can work for this. > > jexec pkg upgrade -y -f > > Also, pkg itself has native support for jails. > > pkg upgrade -j -y -f Thanks I didn't see the obvious :-( Thanks. After testing iocage today I do come to the conclusion that it does what it is made for: perfect wrapper script for jail functionality. I did get sam test jails running vnet, quite easily. But, I will not use iocage as a substitute for ezjail, not due to lacking functionality, no en contre, rather due to "inflexibility" from *my* point of view. Those UUIDs irritate me, and they make "zfs list" rather "odd looking". And, I do miss "flexibility" regarding where to nail "iocage" into my zpool filesystem. I am using beadm boot environments, and I am used to stick ezjail's basejail to those. If upgrading a basejail went wrong I will have a fallback basejail at hand. Anyway, I am currently testing jail management "the hard way" aka own scripts ;-) Regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: iocage following stable?
[Sorry for my private mail. That has been intended to go to the ML instead.] Ernie Luzar wrote: > Michael Grimm wrote: >> I am running ezjail for some years now, but I intend to migrate to iocage. >> Not that I am "disappointed" with ezjail, but I do want to give VNET a try. > > Qjail is a fork of ezjail and qjail has vnet and same console function your > used to. Please, don't get me wrong, but I did follow the "beer license and qjail forking ezjail" issue in length 2.5 years ago, and I declared my servers "qjail free areas" ever since. Even if it might be a piece of software that suits my demands, perfectly, I will stick to my decision made 2.5 years ago. But anyway, I'd kindly thank you for your suggestion, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
How to define the order of starting jails?
Hi — [Background: I wish to run (some of my) ezjail-made jails and VNET which cannot be done by ezjail, natively.] But I found a way to mix both ezjail and basic jail(8) functionality. It's quite easy to fire up ezjail-made jails defined in jail.conf by jail. And, now I can apply VNET to those jails I do wish to run their own network stacks. That's all working well. But I am bit stuck in finding a way to start my jails in a pre-defined order (e.g. first DNS, then mail, …). Well, I can achieve that during boot time by using jail_list="dns mail …" in rc.conf. But, this is respected during boot time, *only*. Whenever I do run a "jail -rc '*'" that shutdown and starting order becomes arbitrary. It doesn't follow the sequence of my jail definitions in jail.conf, either. I thoroughly checked the jail.conf(5) man page for a functionality that would allow me to define a startup/shutdown sequence, but I couldn't find it. Thus, I might have overlooked it, is there a way to achieve my goal using jail and jail.conf? Or something else? Thanks and regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: How to define the order of starting jails?
Michael B. Eichorn wrote > On Sun, 2015-12-20 at 17:50 +0100, Michael Grimm wrote: >> But I am bit stuck in finding a way to start my jails in a pre- >> defined order (e.g. first DNS, then mail, …). […] >> Thus, I might have overlooked it, is there a way to achieve my goal >> using jail and jail.conf? >> Or something else? > jail(8)'s '*' operates on everything without concern for rc.conf, as > such jail_list is not respected. Perhaps try something tied to the rc.d > system. Does `service jail restart` do what you are looking for? Bingo! That made the trick! Thank you very, very much! Cool! That command is stopping and starting jails in the order as listed in "jail_list". Ok, it isn't stopping jails in the reverse order, but that is no big deal, though. [Arrrgh, I never heard of "service" before (after so many years with FBSD) :-( What a shame ...] Thanks and with kind regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: How to define the order of starting jails?
dweimer wrote:
> On 2015-12-20 1:25 pm, Michael B. Eichorn wrote:
>> On Sun, 2015-12-20 at 17:50 +0100, Michael Grimm wrote:
[starting sequence definition, how to?]
>>> Thus, I might have overlooked it, is there a way to achieve my goal
>>> using jail and jail.conf?
>>> Or something else?
>> jail(8)'s '*' operates on everything without concern for rc.conf, as
>> such jail_list is not respected. Perhaps try something tied to the rc.d
>> system. Does `service jail restart` do what you are looking for?
>> Otherwise I would just go with simple restart script such as:
>> #!/bin/sh
>> set -e
>> jail -r '*'
>> jail -c dns
>> jail -c mail
>
> You can also define a jail dependency to make sure a jail starts before
> another one
>
> dns {
> ...
> }
> mail {
> ...
> depend = "dns"
> }
Yep! That is working as well. And, now it will stop jails in the reverse order.
*BUT*, that doesn't work with "jail -rc '*'", reproducibly …
| testing> jail -rc '*'
| ifconfig: : bad value
| jail: dns: /sbin/ifconfig em0 inet netmask 255.255.255.255 @ alias:
failed
… and leaving me with all stopped but not started a single jail :-(
"service jail restart" works much better, never failing.
Thanks and with kind regards,
Michael
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: How to define the order of starting jails?
Boris Samorodov wrote: > > 20.12.15 19:50, Michael Grimm пишет: > >> But I am bit stuck in finding a way to start my jails in a >> pre-defined order (e.g. first DNS, then mail, …). > > AFAIK ezjails start jails in reverse natural (by name) order and stops > in natural order... So I used to name a jail to be started first as > "zz..." one. ezail does the following: # To specify the start up order of your ezjails, use these lines to # create a Jail dependency tree. See rcorder(8) for more details. # # PROVIDE: mail # REQUIRE: dns # BEFORE: [defined in /usr/local/etc/ezjail/mail] But that isn't available any longer, if one goes for "/etc/jail.conf" and "jail", instead. Regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: How to define the order of starting jails?
Michael B. Eichorn wrote: > > On Sun, 2015-12-20 at 20:57 +0100, Michael Grimm wrote: >> Michael B. Eichorn wrote: >>> Does `service jail restart` do what you are looking for? >> >> Bingo! That made the trick! Thank you very, very much! Cool! That >> command is stopping and starting jails in the order as listed in >> "jail_list". Ok, it isn't stopping jails in the reverse order, but >> that is no big deal, though. > > I think that this came up before and there was a patch sumbitted to > stop in the reverse order, It might be fixed in current. I don't really > remember the specifics and I cannot find it now, but something was done > about this before. Ok. I'm following stable, thus it might arrive soon. But, at least at my servers, that is no big deal. Stopping is so fast … (only 8 service jails). But there might be conditions where reverse order might be of advantage. >> [Arrrgh, I never heard of "service" before (after so many years with >> FBSD) :-( What a shame ...] > > Always more to learn. :) Yep, definitely so! [And I did start programming/computing with https://en.wikipedia.org/wiki/PDP-11 ;-)] With kind regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: How to define the order of starting jails?
Allan Jude wrote: > > On 2015-12-20 14:57, Michael Grimm wrote: >> [Arrrgh, I never heard of "service" before (after so many years with FBSD) >> :-( What a shame …] > 'service' was not introduced until FreeBSD 8, iirc. It is mostly a > redhat thing, but is very handy. No, I am not excused, because I arrived at FBSD-6.1. I should have noticed it in the meantime :-C > Re: jail reverse order, there is a patch to address this, and a PR > somewhere I think: > > https://reviews.freebsd.org/D2088 August 2015, hmm. Well, it might make it into FBSD-11? With kind regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
ipsec tunnel and vnet jails: routing, howto?
Hi, I am currently stuck, somehow, and I do need your input. Thus, let me explain, what I do want to achieve: I do have two servers connected via an ipsec/tunnel ... [A] dead:beef:1234:abcd::1 <—> dead:feed:abcd:1234::1 [B] … which is sending all traffic destined for dead:beef:1234:abcd::/64 and dead:feed:abcd:1234::/64 through the tunnel, and vice versa. That did run perfectly well during the last years until I decided to give VNET jails a try. Previously, some of my old fashioned jails got an IPv6 address attached like dead:beef:1234:abcd:1:2::3, and I could reach that address from the remote server without any routing/re-directing or alike, necessary. Now, after having moved those jails to VNET jails (having those addresses bound to their epairXXb interfaces), I cannot reach those addresses within those jails any longer. From my point of view and understanding this must have to do with lack of proper routing, but I am not sure, if that is correct, thus my questions to the experts: 1) Is my assumption correct, that my tunnel is "ending" after having passed my firewalls at each server, *bevor* decrypting its ESP traffic into its final destination (yes, I do have pf rules to allow for esp traffic to pass my outer internet facing interface)? 2) If that is true, racoon has to decide where to deliver those packets, finally? 3) If that is true, I do have an issue with routing that *cannot* be solved by pf firewall rules, right? 4) If that is true, what do I have to look for? What am I missing? How can I route incoming and finally decrypted traffic to its final destination within a VNET jail? 5) Do I need to look for a completely different approach? Every hint is highly welcome. Thanks in advance and with kind regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: ipsec tunnel and vnet jails: routing, howto?
Julian Elischer wrote:
>
> On 27/12/2015 4:24 AM, Michael Grimm wrote:
>> I am currently stuck, somehow, and I do need your input. Thus, let me
>> explain, what I do want to achieve:
>>
>> I do have two servers connected via an ipsec/tunnel ...
>> [A] dead:beef:1234:abcd::1 <—> dead:feed:abcd:1234::1 [B]
>> … which is sending all traffic destined for dead:beef:1234:abcd::/64 and
>> dead:feed:abcd:1234::/64 through the tunnel, and vice versa.
>>
>> That did run perfectly well during the last years until I decided to give
>> VNET jails a try. Previously, some of my old fashioned jails got an IPv6
>> address attached like dead:beef:1234:abcd:1:2::3, and I could reach that
>> address from the remote server without any routing/re-directing or alike,
>> necessary. Now, after having moved those jails to VNET jails (having those
>> addresses bound to their epairXXb interfaces), I cannot reach those
>> addresses within those jails any longer.
>>
>> >From my point of view and understanding this must have to do with lack of
>> >proper routing, but I am not sure, if that is correct, thus my questions to
>> >the experts:
>>
>> 1) Is my assumption correct, that my tunnel is "ending" after having passed
>> my firewalls at each server, *bevor* decrypting its ESP traffic into its
>> final destination (yes, I do have pf rules to allow for esp traffic to pass
>> my outer internet facing interface)?
>>
>> 2) If that is true, racoon has to decide where to deliver those packets,
>> finally?
>>
>> 3) If that is true, I do have an issue with routing that *cannot* be solved
>> by pf firewall rules, right?
>>
>> 4) If that is true, what do I have to look for? What am I missing? How can I
>> route incoming and finally decrypted traffic to its final destination within
>> a VNET jail?
>>
>> 5) Do I need to look for a completely different approach? Every hint is
>> highly welcome.
>
> basically you have to treat the jails as if they are totally separate
> machines that are reached through the vpn endpoints instead of being the
> endpoints themselves.
> This will require a different setup. for example your tunnel will need to be
> exactly that a tunnel and not just an encapsulation. And you will need full
> routing information for the other end at each end.
Thanks for your input. In the meantime I got it running, somehow. The "somehow"
refers to: I am not sure if that's the way its supposed to be.
What I did (I do only show the part of host [A], the other host is configured
accordingly):
1. ipsec/tunnel between [A] dead:beef:1234:abcd::1 <—> dead:feed:abcd:1234::1
[B]
/path-to-racoon/setkey.conf:
spdadd dead:beef:1234:abcd::/56 dead:feed:abcd:1234:1:2::3 any -P out
ipsec esp/tunnel/dead:beef:1234:abcd::1-dead:feed:abcd:1234::1/require;
spdadd dead:feed:abcd:1234::/56 dead:beef:1234:abcd:1:2::3 any -P in
ipsec esp/tunnel/dead:feed:abcd:1234::1-dead:beef:1234:abcd::1/require;
2. routing at [A]:
/etc/rc.conf:
ipv6_static_routes="jail1"
# that's for the route from host system [A] into jail1 with IPv6
address of fd00::::::1
—> ipv6_route_mail="-host dead:beef:1234:abcd:1:2::3 -host
fd00::::::1"
/etc/jail.conf:
#
# host dependent global settings
#
$ip6prefix = "dead:beef:1234:abcd";
$ip6prefix_remote_host = "dead:feed:abcd:1234";
#
# global jail settings
#
host.hostname= "${name}";
path = "/usr/home/jails/${name}";
mount.fstab = "/etc/fstab.${name}";
exec.consolelog = "/var/log/jail_${name}_console.log";
vnet = "new";
vnet.interface = "epair${jailID}b";
exec.clean;
mount.devfs;
persist;
#
# network settings to apply/destroy during start/stop of every jail
#
exec.prestart= "sleep 2";
exec.prestart += "ifconfig epair${jailID} create up";
exec.prestart += "ifconfig bridge0 addm epair${jailID}a";
exec.start = "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start += "/sbin/ifconfig epair${jailID}b inet
${ip4_addr}";
exec.start += "/sbin/ifconfig epair${jailID}b inet6
${ip6_addr}";
exec.start += "/sbin/route add default -gateway
10.x.x.254";
exec.sta
Re: [Bug 196152] jail_list is not reversed when stopping jails
On 10.02.2016, at 17:14, bugzilla-nore...@freebsd.org wrote: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=196152 > > --- Comment #9 from commit-h...@freebsd.org --- > A commit references this bug: > > Author: feld > Date: Wed Feb 10 16:13:59 UTC 2016 ^^^ > New revision: 295471 > URL: https://svnweb.freebsd.org/changeset/base/295471 > > Log: > Add new rc.conf parameter "jail_reverse_stop" > > When a user defines "jail_list" in rc.conf the jails are started in the > order defined. Currently the jails are not are stopped in reverse order > which may break dependencies between jails/services and prevent a clean > shutdown. The new parameter "jail_reverse_stop" will shutdown jails in > "jail_list" in reverse order when set to "YES". > > Please note that this does not affect manual invocation of the jail rc > script. If a user runs the command > ># service jail stop jail1 jail2 jail3 > > the jails will be stopped in exactly the order specified regardless of > jail_reverse_stop being defined in rc.conf. > > PR: 196152 > Approved by: jamie > MFC after:1 week Hmm. Any chance to get that improvement MFC'd? > Relnotes: yes > Differential Revision:https://reviews.freebsd.org/D5233 > > Changes: > head/etc/defaults/rc.conf > head/etc/rc.d/jail Regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: [Bug 196152] jail_list is not reversed when stopping jails
James Gritton wrote: > On 2016-04-29 13:09, Michael Grimm wrote: >> On 10.02.2016, at 17:14, bugzilla-nore...@freebsd.org wrote: >>> URL: https://svnweb.freebsd.org/changeset/base/295471 >> Hmm. Any chance to get that improvement MFC'd? > > Sure, I'll push one through. Thanks for the prompt reaction. Regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: [Bug 196152] jail_list is not reversed when stopping jails
Michael Grimm wrote: > > James Gritton wrote: >> On 2016-04-29 13:09, Michael Grimm wrote: >>> On 10.02.2016, at 17:14, bugzilla-nore...@freebsd.org wrote: > >>>> URL: https://svnweb.freebsd.org/changeset/base/295471 >>> Hmm. Any chance to get that improvement MFC'd? >> Sure, I'll push one through. > Thanks for the prompt reaction. And, it works like a charm. and, I have been waiting for this for a very long time ;-) Thanks to all involved, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: deploy multiple vnets with VIMAGE/VNET + Production Ready?
Sebastián Maruca via freebsd-jail wrote:
> Now we're talking about 10.3-HEAD wiht Jails+vnet... but then again, has
> anyone tried it? Roger, it seems you are thumbing up my challenge...
> But I guess i'll have to stick with netgraph instead epair/if_bridge because
> the later is not so documented as the first one…
Preamble: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, now
10.3-STABLE, and haven't seen any issues, sofar. Currently I do have 10 jails
running, firewall is pf at the host, only. My servers are not big scaled ISP
like, more small business-like, though. I am considering myself a hobby admin.
Here's my configuration that may show you one way to get that running, but I am
sure your will have to tweak it to your needs:
1) Jails have been created by ezjail in the past, thus they are still at
ezjail's infrastructure. But I do no longer use ezjail for starting or stopping
my jails due to ezjail's lack of dealing with VNET jails (yet). So I do still
have fstab definitions in /etc for all jails, e.g.:
/etc/fstab.www
/path-to-your/jails/basejail /path-to-your/jails/www/basejail
nullfs ro 0 0
2) All external IPv4 or IPv6 addresses are NAT'ed or NAT66'ed to 10.1.1.x or
fd00:dead:dead:beef::x
3) Networking regarding VNET jails defined in /etc/rc.conf:
# set up one bridge interface
cloned_interfaces="bridge0"
# needed for default routes within jails
ifconfig_bridge0="inet 10.1.1.254 netmask 255.255.255.0"
ifconfig_bridge0_ipv6="inet6 fd00:dead:dead:beef::254 prefixlen 64"
4) Thus, jails are controlled by jail(8) (shown for 3 example jails):
/etc/rc.conf
———BEGIN
jail_enable="YES"
jail_reverse_stop="YES"
jail_list="dns www mail"
———-END
/etc/jail.conf:
#
# host dependent global settings
#
$ip6prefixLOCAL = "fd00:dead:dead:beef";
#
# global jail settings
#
host.hostname= "${name}";
path = "/path-to-your/jails/${name}";
mount.fstab = "/etc/fstab.${name}";
exec.consolelog = "/var/log/jail_${name}_console.log";
vnet = "new";
vnet.interface = "epair${jailID}b";
exec.clean;
mount.devfs;
persist;
#
# network settings to apply/destroy during start/stop of every
jail
#
exec.prestart= "sleep 2";
exec.prestart += "ifconfig epair${jailID} create up";
exec.prestart += "ifconfig bridge0 addm
epair${jailID}a";
exec.start = "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start += "/sbin/ifconfig epair${jailID}b inet
${ip4_addr}";
exec.start += "/sbin/ifconfig epair${jailID}b
inet6 ${ip6_addr}";
exec.start += "/sbin/route add default -gateway
10.1.1.254";
exec.start += "/sbin/route add -inet6 default
-gateway ${ip6prefixLOCAL}::254";
#exec.stop = "/sbin/route del default";
#exec.stop += "/sbin/route del -inet6 default";
exec.stop += "/bin/sh /etc/rc.shutdown";
exec.poststop= "ifconfig epair${jailID}a destroy";
#
# individual jail settings
#
mail {
$jailID = 1;
$ip4_addr= 10.1.1.1;
$ip6_addr= ${ip6prefixLOCAL}::1/64;
exec.start += "/bin/sh /etc/rc";
}
www {
$jailID = 2;
$ip4_addr= 10.1.1.2;
$ip6_addr= ${ip6prefixLOCAL}::2/64;
exec.start += "/bin/sh /etc/rc";
}
dns {
$jailID = 3;
$ip4_addr= 10.1.1.3;
$ip4_addr_2 = 10.1.1.4;
$ip6_addr= ${ip6prefixLOCAL}::3/64;
$ip6_addr_2 = ${ip6prefixLOCAL}::4/64;
exec.start += "/sbin/ifconfig epair${jailID}b inet
${ip4_addr_2} alias";
exec.start += "/sbin/ifconfig epair${jailID}b
inet6 ${ip6_addr_2} alias";
exec.start
Re: deploy multiple vnets with VIMAGE/VNET + Production Ready?
On 2016-06-03 0:05, Ernie Luzar wrote: Michael Grimm wrote: I switched to VNET+epair/if_bridge jails starting 10.2-STABLE, now 10.3-STABLE, and haven't seen any issues, sofar. You left out whether you had to compile the kernel with the vimage option or whether vimage was already included in the kernel? Obviously, I had had to compile a custom kernel because I am running 10.x-STABLE. And, because I am using IPSec, I have to do so anyway: includeGENERIC# include GENERIC definitions, first ident CUSTOM # custom kernel name cpuHAMMER # amd64 and intel 64bit cpus device crypto # IPsec depends on this optionsIPSEC # IP security optionsIPSEC_NAT_T# IP security NAT-T in transport mode optionsVIMAGE # network virtualization Regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: multiple interfaces for jail.conf(1) and jail_set(2)
Isaac (.ike) Levy wrote:
> Can I specify multiple IP interfaces and assign IP’s to them using jail.conf?
Not sure if I understand your question correctly, but I do define the following
in my jail.conf for VNET jails:
#
# host dependent global settings
#
$ip6prefixLOCAL = "fd00:dead:beef:1234";
#
# global jail settings
#
host.hostname= "${name}";
path = "/usr/home/jails/${name}";
mount.fstab = "/etc/fstab.${name}";
exec.consolelog = "/var/log/jail_${name}_console.log";
vnet = "new";
vnet.interface = "epair${jailID}b";
exec.clean;
mount.devfs;
persist;
#
# network settings to apply/destroy during start/stop of every jail
#
exec.prestart= "sleep 2";
exec.prestart += "/sbin/ifconfig epair${jailID} create up";
exec.prestart += "/sbin/ifconfig bridge0 addm epair${jailID}a";
exec.start = "/sbin/sysctl net.inet6.ip6.dad_count=0";
exec.start += "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}";
exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}";
exec.start += "/sbin/route add default -gateway 10.1.1.254";
exec.start += "/sbin/route add -inet6 default -gateway
${ip6prefixLOCAL}::254";
exec.stop= "/sbin/route del default";
exec.stop += "/sbin/route del -inet6 default";
exec.stop += "/bin/sh /etc/rc.shutdown";
exec.poststop= "/sbin/ifconfig epair${jailID}a destroy";
#
# individual jail settings
#
dns {
$jailID = 1;
$ip4_addr= 10.1.1.1;
$ip4_addr_2 = 10.1.1.2;
$ip6_addr= ${ip6prefixLOCAL}::1/64;
$ip6_addr_2 = ${ip6prefixLOCAL}::2/64;
exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr_2}
alias";
exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2}
alias";
exec.start += "/bin/sh /etc/rc";
}
etc.
Again, not sure if I do understand your issue correctly, but the shown examples
of exec.start, exec.stop, etc. are quite versatile to use.
I do start/stop my jails by "service jail start/stop".
Hope that helps,
Michael
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: multiple interfaces for jail.conf(1) and jail_set(2)
Isaac (.ike) Levy wrote
> Wow, that’s rad Michael,
>
>> On Dec 14, 2016, at 3:30 PM, Michael Grimm wrote:
>>
>> Isaac (.ike) Levy wrote:
>>
>>> Can I specify multiple IP interfaces and assign IP’s to them using
>>> jail.conf?
>>
>> Not sure if I understand your question correctly, but I do define the
>> following in my jail.conf for VNET jails:
>>
>> #
>> # host dependent global settings
>> #
>> $ip6prefixLOCAL = "fd00:dead:beef:1234";
>>
>> #
>> # global jail settings
>> #
>> host.hostname = "${name}";
>> path = "/usr/home/jails/${name}";
>> mount.fstab = "/etc/fstab.${name}";
>> exec.consolelog = "/var/log/jail_${name}_console.log";
>> vnet = "new";
>> vnet.interface= "epair${jailID}b";
>> exec.clean;
>> mount.devfs;
>> persist;
>>
>> #
>> # network settings to apply/destroy during start/stop of every jail
>> #
>> exec.prestart = "sleep 2";
>> exec.prestart+= "/sbin/ifconfig epair${jailID} create up";
>> exec.prestart+= "/sbin/ifconfig bridge0 addm
>> epair${jailID}a";
>> exec.start= "/sbin/sysctl net.inet6.ip6.dad_count=0";
>> exec.start += "/sbin/ifconfig lo0 127.0.0.1 up";
>> exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}";
>> exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}";
>> exec.start += "/sbin/route add default -gateway 10.1.1.254";
>> exec.start += "/sbin/route add -inet6 default -gateway
>> ${ip6prefixLOCAL}::254";
>> exec.stop = "/sbin/route del default";
>> exec.stop+= "/sbin/route del -inet6 default";
>> exec.stop+= "/bin/sh /etc/rc.shutdown";
>> exec.poststop = "/sbin/ifconfig epair${jailID}a destroy";
>>
>> #
>> # individual jail settings
>> #
>> dns {
>> $jailID = 1;
>> $ip4_addr= 10.1.1.1;
>> $ip4_addr_2 = 10.1.1.2;
>> $ip6_addr= ${ip6prefixLOCAL}::1/64;
>> $ip6_addr_2 = ${ip6prefixLOCAL}::2/64;
>> exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr_2}
>> alias";
>> exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr_2}
>> alias";
>> exec.start += "/bin/sh /etc/rc";
>> }
>>
>> etc.
>
> I’ll need to study/look up some of that syntax, to fully grok this, but that
> comprehensive example appears to hit the nail on the head several times over
> with the exec.start/exec.stop action.
>
> Two questions though:
>
> - I’m confused how you define the shell style $ variables in your individual
> jail settings above, e.g. ‘$ip4_addr_2 = 10.1.1.2;’, why/how does that work?
> Is that a variable to be expanded, or some other behavior?
This is described in jail.conf(5) under the section "variables". I do have 10
jails running, and those $ variables/parameters are very helpful, indeed.
>> Again, not sure if I do understand your issue correctly, but the shown
>> examples of exec.start, exec.stop, etc. are quite versatile to use.
>>
>> I do start/stop my jails by "service jail start/stop”.
>
> - Obviously you state you’re using service to start/stop jails, but shouldn’t
> this work with ‘jail -c ’, or are these subsystems not
> interoperable?
Hmm. I do have to admit that I never tried 'jail -c ', but I just
gave it a try, and yes, it works as well :-)
I do use "service jail start/stop" because that will obey my pre-defined
starting/stopping order of jails (which I do need to have, e.g. dns before mail
and such) in /etc/rc.conf
jail_enable="YES"
jail_reverse_stop="YES"
jail_list="dns mail ..."
Regards,
Michael
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: multiple interfaces for jail.conf(1) and jail_set(2)
Isaac (.ike) Levy wrote: >> I do use "service jail start/stop" because that will obey my pre-defined >> starting/stopping order of jails (which I do need to have, e.g. dns before >> mail and such) in /etc/rc.conf >> >> jail_enable="YES" >> jail_reverse_stop="YES" >> jail_list="dns mail …” > > Awesome! For my use, I’m averse to starting jails at host boot- so I’m > really excited this works. It works very well (for quite some years now). > Thanks so much Michael- this totally answered my question, I’m back on the > right path to using jail.conf with my setup! You are welcome, and I am glad having been able to help (as I have been helped by others before). Regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: multiple interfaces for jail.conf(1) and jail_set(2)
[cc'd to freebsd-jail@FreeBSD.org where that thread originated]
Valeri Galtsev wrote:
> On Wed, December 14, 2016 2:30 pm, Michael Grimm wrote:
>> #
>> # network settings to apply/destroy during start/stop of every jail
>> #
>> exec.prestart = "sleep 2";
>> exec.prestart+= "/sbin/ifconfig epair${jailID} create up";
>> exec.prestart+= "/sbin/ifconfig bridge0 addm
>> epair${jailID}a";
>> exec.start= "/sbin/sysctl net.inet6.ip6.dad_count=0";
>> exec.start += "/sbin/ifconfig lo0 127.0.0.1 up";
>> exec.start += "/sbin/ifconfig epair${jailID}b inet ${ip4_addr}";
>> exec.start += "/sbin/ifconfig epair${jailID}b inet6 ${ip6_addr}";
>> exec.start += "/sbin/route add default -gateway 10.1.1.254";
>> exec.start += "/sbin/route add -inet6 default -gateway
>> ${ip6prefixLOCAL}::254";
>> exec.stop = "/sbin/route del default";
>> exec.stop+= "/sbin/route del -inet6 default";
>> exec.stop+= "/bin/sh /etc/rc.shutdown";
>> exec.poststop = "/sbin/ifconfig epair${jailID}a destroy";
>>
>> #
>> # individual jail settings
>> #
>> dns {
>> $jailID = 1;
>> $ip4_addr= 10.1.1.1;
>> $ip4_addr_2 = 10.1.1.2;
[…]
> Michael, is it possible to have two addresses belonging to two different
> networks (through two different network interfaces)?
>
> Say, on host system:
>
> ifconfig_igb0="inet 172.20.9.22 ...
> ifconfig_igb1="inet 10.1.1.17 ...
>
>
> and in some jail
>
> $ip4_addr= 172.20.9.22;
> $ip4_addr_2 = 10.1.1.17;
>
> - will that work? This is what didn't work for me in the past when
> configured jails old style in /etc/rc.conf
I can't answer that because I have never tried it before.
Those exec. will give you a very fine-grained control over which commands
are run at the host environment (exec.prestart, exec.poststop, exec.poststart)
or within the jail's environment (exec.start, exec.stop, exec.prestop) in which
order. Have a look at jail(8) for details (and presumably more exec.).
Therefore I am quite confident that whatever could be done with jails might be
"coded" into jail.conf.
Regards,
Michael
___
freebsd-jail@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-jail
To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Update jails strategy
Hi -- [FYI: I am running some service jails with an ezjail-type basejail approach.] Until now I did stop all jails before updating and restarting them afterwards. Now I am wondering if that is necessary at all. Wouldn't it be sufficient to update basejail with running jails and restart them after this updating? (In analogy to a "make installworld; make installkernel; reboot") Background of my question: Until now I do have two identical servers running in parallel (simple failover approach), but I am thinking about migrating both servers to a single cloud instance, and keeping downtimes of my service jails whilst updating the OS to a bare minimum. Thanks for your input in advance, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Update jails strategy
Miroslav Lachman <000.f...@quip.cz> wrote: > Michael Grimm wrote on 2017/11/11 21:42: >> [FYI: I am running some service jails with an ezjail-type basejail approach.] >> >> Until now I did stop all jails before updating and restarting them >> afterwards. Now I am wondering if that is necessary at all. >> >> Wouldn't it be sufficient to update basejail with running jails and restart >> them after this updating? (In analogy to a "make installworld; make >> installkernel; reboot") > > Update of basejail and restart of jails is enough. I am doing it this way for > a years. Thanks, that's what I wanted to hear ;-) > We have build server with prepared updates mounted thru NFS and then just > make installkernel && make installworld (+ mergemaster) in host system. After > the host is updated (before reboot) I run my local script to rsync changes > from host to basejail (sync just bin, sbin, libs, but not home, etc, var), > then mergemaster in all jails and finally reboot the server. I never stop > jails before update. Ok. Until now I did something similar: #) make buildworld && make buildkernel at fastest server #) zfs send /usr/obj and /usr/src to remote server #) make installworld && make installkernel at remote server I did use "ezjail-admin update" to update basejail only after stopping all jails. That I will omit from now on, thanks. Regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
performance issue within VNET jail
Hi [ I did recently migrate my servers from bare metal to cloud instances (OpenStack at OVH) ] [ FreeBSD 11.1-STABLE #0 r327055 ] My setup is as follows and didn't change for the last couple of years: extIF0/pf/NAT <—> epairXa (bridge0) epairXb <-> jail Downloading a file (by wget) at the host is around 30 MB/s, and an example tcpdump at extIF0 looks as follows: 19:32:10.711769 IP (tos 0x20, ttl 56, id 37539, offset 0, flags [DF], proto TCP (6), length 8680) remote.http > myhost.14367: Flags [.], cksum 0x64ed (incorrect -> 0x3223), seq 5753:14381, ack 146, win 235, options [nop,nop,TS val 1007145732 ecr 3995852], length 8628: HTTP 19:32:10.713851 IP (tos 0x20, ttl 56, id 37545, offset 0, flags [DF], proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> 0x8d1e), seq 14381:15819, ack 146, win 235, options [nop,nop,TS val 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713899 IP (tos 0x20, ttl 56, id 37546, offset 0, flags [DF], proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> 0x6ade), seq 15819:17257, ack 146, win 235, options [nop,nop,TS val 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713934 IP (tos 0x20, ttl 56, id 37547, offset 0, flags [DF], proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> 0x1173), seq 17257:18695, ack 146, win 235, options [nop,nop,TS val 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713962 IP (tos 0x20, ttl 56, id 37548, offset 0, flags [DF], proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> 0xcf7a), seq 18695:20133, ack 146, win 235, options [nop,nop,TS val 1007145732 ecr 3995852], length 1438: HTTP When downloading the very same file within a VIMAGE jail the performance drops to around 80 KB/s, quite a dramatic loss. An example tcpdump at exitIF0 looks as follows: 19:34:36.284175 IP (tos 0x0, ttl 56, id 28618, offset 0, flags [DF], proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> 0x4478), seq 1449:4345, ack 146, win 235, options [nop,nop,TS val 1007182125 ecr 4141429], length 2896: HTTP 19:34:36.481904 IP (tos 0x0, ttl 56, id 28620, offset 0, flags [DF], proto TCP (6), length 1500) remote.http > myhost.63382: Flags [.], cksum 0xd11d (correct), seq 1449:2897, ack 146, win 235, options [nop,nop,TS val 1007182175 ecr 4141429], length 1448: HTTP 19:34:36.484109 IP (tos 0x0, ttl 56, id 28621, offset 0, flags [DF], proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> 0x2e5b), seq 15929:18825, ack 146, win 235, options [nop,nop,TS val 1007182175 ecr 4141629], length 2896: HTTP 19:34:36.682006 IP (tos 0x0, ttl 56, id 28623, offset 0, flags [DF], proto TCP (6), length 1500) remote.http > myhost.63382: Flags [.], cksum 0x4ab6 (correct), seq 2897:4345, ack 146, win 235, options [nop,nop,TS val 1007182225 ecr 4141629], length 1448: HTTP 19:34:36.684159 IP (tos 0x0, ttl 56, id 28624, offset 0, flags [DF], proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> 0xd7db), seq 18825:21721, ack 146, win 235, options [nop,nop,TS val 1007182225 ecr 4141829], length 2896: HTTP A tcpdump at epairXa looks comparable. I did reduce all MTU settings at the involved interfaces from their initial settings (1490) to an experimental setting of 1400, just to be on the save side, to no avail. (FYI: I did have to reduce from 1500 to 1490 to please IPSec after migration from bare metal to cloud infrastructure.) Then, I did test the following settings found in the Net, to no avail either: sysctl net.inet.tcp.tso=0 sysctl net.link.bridge.pfil_onlyip=0 sysctl net.link.bridge.pfil_bridge=0 sysctl net.link.bridge.pfil_member=0 sysctl net.add_addr_allfibs=0 I do have to admit that I am lost here, and that I cannot think about what is going wrong. The last download I did try at my old severs has been some weeks ago. Ever since I did upgrade FreeBSD 11.1-STABLE, and I did move my infrastructure from bare metal to cloud, thus I cannot test anymore if my old servers would have shown that performance issue in the meantime. Thus any feedback is highly recommended! Thanks in advance and regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: performance issue within VNET jail
Kristof Provost wrote: > > On 21 Dec 2017, at 21:24, Michael Grimm wrote: >> I do have to admit that I am lost here, and that I cannot think about what >> is going wrong. The last download I did try at my old severs has been some >> weeks ago. Ever since I did upgrade FreeBSD 11.1-STABLE, and I did move my >> infrastructure from bare metal to cloud, thus I cannot test anymore if my >> old servers would have shown that performance issue in the meantime. >> >> Thus any feedback is highly recommended! > Can you try turning off TSO? (`ifconfig $ifname -tso`) > > There have been issues with pf and TSO checksums, which looked a lot like > this (i.e. bad TCP performance). Those problems should be fixed, but this is > easy to test. > I did try it, but without success. This only worked for the external interface, though. Both epairX interfaces didn't accept that command: ifconfig: -tso: Invalid argument I did mention that I previously tried "sysctl net.inet.tcp.tso=0". That shoukld do the same, right? Thanks and regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: performance issue within VNET jail
Kristof Provost wrote > On 21 Dec 2017, at 21:50, Michael Grimm wrote: >> Kristof Provost wrote: >>> Can you try turning off TSO? (`ifconfig $ifname -tso`) >>> >>> There have been issues with pf and TSO checksums, which looked a lot like >>> this (i.e. bad TCP performance). Those problems should be fixed, but this >>> is easy to test. >> I did try it, but without success. > Hmm. I’ve got no ideas at the moment. I run a very similar setup (although on > CURRENT), and see no performance issues from my jails. > Can you test a performance test without pf? Perhaps from the local LAN for > example? That should help narrow it down a bit, at least. Well I prepared on of my webservers running at hostB/jailX to serve a sample file for local downloading tests: 1) hostAwget from hostB/jailX sample file: about 30 MB/s 2) hostA/jailY wget from hostB/jailX sample file: about 30 MB/s 3) hostBwget from hostB/jailX sample file: about 190 MB/s 4) hostB/jailY wget from hostB/jailX sample file: about 190 MB/s Hmm. At least tests 3) and 4) omit the pf firewall. Tests 1) qnd 2) include passing two firewalls, one at each host. BUT: Both hosts are connected via an IPSec tunnel, and that's esp not tcp. Can anyone draw conclusions from this test? I cannot ;-) Thanks and regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: performance issue within VNET jail
> On 21. Dec 2017, at 22:48, Eugene Grosbein wrote: > > 22.12.2017 4:42, Michael Grimm wrote: > >> Well I prepared on of my webservers running at hostB/jailX to serve a sample >> file for local downloading tests: >> >> 1) hostA wget from hostB/jailX sample file: about 30 MB/s >> 2) hostA/jailY wget from hostB/jailX sample file: about 30 MB/s >> 3) hostB wget from hostB/jailX sample file: about 190 MB/s >> 4) hostB/jailY wget from hostB/jailX sample file: about 190 MB/s >> >> Hmm. At least tests 3) and 4) omit the pf firewall. Tests 1) qnd 2) include >> passing two firewalls, one at each host. BUT: Both hosts are connected via >> an IPSec tunnel, and that's esp not tcp. >> >> Can anyone draw conclusions from this test? >> I cannot ;-) > > Make sure and double check that your ESP packets do not get fragmented. Hmm, I do not know how to achieve that. May the following tcpdump excerpts answer your question, or do you want me to look somewhere else? At hostA while downloading from hostB/jailX and "tcpdump -i extIF esp -vv" 22:52:42.341023 IP (tos 0x0, ttl 64, id 40481, offset 0, flags [none], proto ESP (50), length 140) hostA > hostB: ESP(spi=0x01d9ec34,seq=0x5fe699), length 120 22:52:42.341079 IP (tos 0x0, ttl 53, id 64310, offset 1480, flags [none], proto ESP (50), length 100) hostB > hostA: ip-proto-50 22:52:42.341151 IP (tos 0x0, ttl 64, id 40483, offset 0, flags [none], proto ESP (50), length 140) hostA > hostB: ESP(spi=0x01d9ec34,seq=0x5fe69a), length 120 22:52:42.341169 IP (tos 0x0, ttl 53, id 64312, offset 1480, flags [none], proto ESP (50), length 100) hostB > hostA: ip-proto-50 22:52:42.341238 IP (tos 0x0, ttl 53, id 64314, offset 1480, flags [none], proto ESP (50), length 100) hostB > hostA: ip-proto-50 At hostB the same dump looks like: 22:52:42.463511 IP (tos 0x0, ttl 53, id 41153, offset 0, flags [none], proto ESP (50), length 124) hostA > hostB: ESP(spi=0x01d9ec34,seq=0x5feaa8), length 104 22:52:42.463518 IP (tos 0x0, ttl 53, id 41155, offset 0, flags [none], proto ESP (50), length 124) hostA > hostB: ESP(spi=0x01d9ec34,seq=0x5feaa9), length 104 22:52:42.463593 IP (tos 0x0, ttl 53, id 41157, offset 0, flags [none], proto ESP (50), length 124) hostA > hostB: ESP(spi=0x01d9ec34,seq=0x5feaaa), length 104 22:52:42.463601 IP (tos 0x0, ttl 53, id 41159, offset 0, flags [none], proto ESP (50), length 124) hostA > hostB: ESP(spi=0x01d9ec34,seq=0x5feaab), length 104 22:52:42.463673 IP (tos 0x0, ttl 53, id 41161, offset 0, flags [none], proto ESP (50), length 124) hostA > hostB: ESP(spi=0x01d9ec34,seq=0x5feaac), length 104 Thanks and regards, Michael > > > ___ > freebsd-...@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: performance issue within VNET jail
Eugene Grosbein wrote: > 22.12.2017 4:59, Michael Grimm wrote: >>> Make sure and double check that your ESP packets do not get fragmented. >> >> >> Hmm, I do not know how to achieve that. May the following tcpdump excerpts >> answer your question, or do you want me to look somewhere else? >> >> At hostA while downloading from hostB/jailX and "tcpdump -i extIF esp -vv" >> >> 22:52:42.341023 IP (tos 0x0, ttl 64, id 40481, offset 0, flags [none], proto >> ESP (50), length 140) >>hostA > hostB: ESP(spi=0x01d9ec34,seq=0x5fe699), length 120 >> 22:52:42.341079 IP (tos 0x0, ttl 53, id 64310, offset 1480, flags [none], >> proto ESP (50), length 100) >>hostB > hostA: ip-proto-50 > > It shows non-zero offsets, so your ESP packets *are* fragmented. > I guess, this is the reason of your problems as fragmented ESP packets are > known to cause problems > due to different reasons. Simpliest way to avoid such issues is to decrease > MTU of IPSEC tunnel > and/or TCP MSS so that incapsulated ESP packets do not get fragmented. Well, you already helped me out with IPSEC very recently, and I already did decrease my MTU from 1500 to 1490. That increased my tunnel performance dramatically, already. Thanks, I will decrease MTU further. BUT: In this thread I did report that I already had decreased MTU for testing purposes on all involved interfaces down to 1400 to no avail, and that my performance issue is regarding downloads within VNET jails using TCP, not ESP. The very same external interfaces do not show a performance drop if connected via ESP tunnel, but when trying to download files from the internet, and only when the download is started within a VNET jail. At the host downloads are only limited by the bandwidth provided by the hosting company. BUT: It might well be that I did completely misunderstood your reply instead ;-) Thanks and regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: performance issue within VNET jail
Kristof Provost wrote: > I run a very similar setup (although on CURRENT), and see no performance > issues from my jails. In utter despair I did upgrade one server to CURRENT (#327076) today, but that hasn't been successful :-( Ok, right now I do know: (#) there is *no* performance loss (TCP) when: (-) fetching files from outside through PF/extIF to host (-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to host (-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to jail via bridge (-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) to host (-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) and then via bridge to jail (#) there is a *dramatic* performance loss (TCP) when: (-) fetching files from outside through PF/extIF via bridge to jail (#) I did try to tweak the following settings *without* success: (-) sysctl net.inet.tcp.tso=0 (-) sysctl net.link.bridge.pfil_onlyip=0 (-) sysctl net.link.bridge.pfil_bridge=0 (-) sysctl net.link.bridge.pfil_member=0 (-) reducing mtu to 1400 (1490 before) on all interfaces extIF, bridge, epairXs (-) deactivating "scrub in all" and "scrub out on $extIF all random-id" in /etc/pf.conf (-) setting "set require-order yes" and "set require-order no" in /etc/pf.conf [1] [1] I do see more a lot of out-of-order packages within a jail "netstat -s -p tcp" after those slow downloads, but not after downloads via IPSEC tunnel from partner host. That leads me to the conclusions: (#) the bridge is not to blame (#) it's either the PF/NATing or something else, right? Thanks for your suggestions so far, but I am lost here. Any ideas? Regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: vnet NAT'd jails extremely slow, connection dies
Farhan Khan wrote: > On Mon, Feb 4, 2019 at 2:29 PM Farhan Khan wrote: >> I have a jail NAT'd to a base system, but the connection is extremely >> slow and frequently disconnects drops, whereas the base is fine has >> perfectly fine connectivity. >> >> My configuration is as follows: >> vtnet0: Has routeable IPv4 address and 172.16.0.1/16 >> Jail uses epair4b, base has epair4a. Jail's IP is 172.16.0.5/16. >> The base and jail can ping each other. >> bridge0: contains vtnet0 and epair4a. >> >> I have gateway_enable="YES" >> My pf.conf is as follows: >> nat pass from 172.16.0.0/16 to any -> (vtnet0) >> >> When I try to run clamav, the connectivity stalls after a few minutes >> and eventually disconnects. I ran tcpdump on the bridge and saw a lot >> of HTTP seq and ack packets but no actual data. I am not using IPv6 >> yet. > > Just to provide more context to my previous email, outside of the jail > I can download the FreeBSD ISO installer image at 3 MBps. Within the > jail it drops to 12KBps. This sounds familiar to me ;-) Please have a look at https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html Solution in https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049484.html I ended up with the following additions to /boot/loader.conf (and a subsequent reboot): # needs to become turned off (LRO) in order to restore tcp performance within VNET jails: hw.vtnet.lro_disable="1" hw.vtnet.tso_disable="1" HTH, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: vnet NAT'd jails extremely slow, connection dies
Kristof Provost wrote: > These are questions for anyone who's running pf on top of a hypervisor > and has vnet or other jails, and has seen slowdowns. > > * What hypervisor are you running? I do not know. It is a cloud hosted on OVH infrastructure in France https://www.ovh.co.uk/public-cloud/instances/technologies/ But I do not know which hypervisor they use, sorry. > * Does the problem affect only the jails, or also the host system? The host didn't show any slowdown. Only within a jail the performance drop dramatically. > * Does it only happen with NAT, or with routed packets as well? In my case it happens with NAT, never tried routed packets. HTH, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Performance issues with VNET/bridge/VLAN
Am 2019-02-22 11:31, schrieb Patrick M. Hausen: [x-posted to freebsd-jail@freebsd.org] The machine is an iocage jail host, all jails with VNET. The problem is: network performance in the jails (not on the host!) is abysmal with the second setup. Not consistently so, everything *seems* to work but e.g. a customer complained that checking out a project from github happend at 15k/s … that’s when we started to investigate. [...] *Any* idea what might be going on here? We use VNET all the same on all the hosts and it is still labelled „experimental", yes. But all the parts that make up the different setups - bridge(4), vlan(4) - have been in FreeBSD for ages. I’m just combining features orthogonally like every good sysadmin ;-) If someone is willing to do some investigation, I think I can provide a test system and remote access … This sounds familiar to me, please have a look at the following two threads: https://lists.freebsd.org/pipermail/freebsd-jail/2019-February/003684.html https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html If your hosts run on cloud infrastructure odds are that the mentioned settings will work in your case. Regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
Re: Performance issues with VNET/bridge/VLAN
Hi On 22. Feb 2019, at 19:48, Patrick M. Hausen wrote: > epair(4) interfaces added to the bridge These are my number one suspects when it comes to performance loss within a VNET jail compared to the host system. > But I’ll fiddle with LRO nonetheless and report if that changes anything. I'm interested to learn if bare metal might behave comparable to cloud infrastructure or not? Regards, Michael ___ freebsd-jail@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-jail To unsubscribe, send any mail to "freebsd-jail-unsubscr...@freebsd.org"
