Re: epair and vnet jail loose connection.

2022-03-17 Thread Michael Gmelin
On Thu, 17 Mar 2022 13:37:28 +0100 Johan Hendriks wrote: > On 16/03/2022 11:36, Michael Gmelin wrote: > > > > On Wed, 16 Mar 2022 11:10:30 +0100 > > Santiago Martinez wrote: > > > >> Guys, do you want me to run a pre/post patch perf test? > >> > >> i can quickly do  Spirent -> bridge - jail

Re: epair and vnet jail loose connection.

2022-03-17 Thread Johan Hendriks
On 16/03/2022 11:36, Michael Gmelin wrote: On Wed, 16 Mar 2022 11:10:30 +0100 Santiago Martinez wrote: Guys, do you want me to run a pre/post patch perf test? i can quickly do  Spirent -> bridge - jail - bridge - jail - bridge -> spirent. Best regards. Santi Hi Santi, Thanks for offer

Re: epair and vnet jail loose connection.

2022-03-16 Thread Michael Gmelin
On Wed, 16 Mar 2022 11:10:30 +0100 Santiago Martinez wrote: > Guys, do you want me to run a pre/post patch perf test? > > i can quickly do  Spirent -> bridge - jail - bridge - jail - bridge > -> spirent. > > Best regards. > > Santi > Hi Santi, Thanks for offering your help. Kristof alre

Re: epair and vnet jail loose connection.

2022-03-16 Thread Santiago Martinez
Guys, do you want me to run a pre/post patch perf test? i can quickly do  Spirent -> bridge - jail - bridge - jail - bridge -> spirent. Best regards. Santi On 3/15/22 19:17, Kristof Provost wrote: On 15 Mar 2022, at 11:48, Michael Gmelin wrote: On Tue, 15 Mar 2022 10:30:41 -0600 Kristof P

Re: epair and vnet jail loose connection.

2022-03-15 Thread Kristof Provost
On 15 Mar 2022, at 11:48, Michael Gmelin wrote: > On Tue, 15 Mar 2022 10:30:41 -0600 > Kristof Provost wrote: >> Can you write up a commit message and post it on phabricator? >> > > Please see https://reviews.freebsd.org/D34569 > Super, thanks! I’ll try to review & commit that today. I’ll also co

Re: epair and vnet jail loose connection.

2022-03-15 Thread Patrick M. Hausen
Just wanted to say: thanks for all your incredible work, guys! Kind regards, Patrick -- punkt.de GmbH Patrick M. Hausen .infrastructure Kaiserallee 13a 76133 Karlsruhe Tel. +49 721 9109500 https://infrastructure.punkt.de i...@punkt.de AG Mannheim 108285 Geschäftsführer: Jürgen Egeling, Daniel

Re: epair and vnet jail loose connection.

2022-03-15 Thread Michael Gmelin
On Tue, 15 Mar 2022 10:30:41 -0600 Kristof Provost wrote: > On 14 Mar 2022, at 18:02, Michael Gmelin wrote: > > On Mon, 14 Mar 2022 09:09:49 -0600 > > Kristof Provost wrote: > > > >> On 14 Mar 2022, at 7:44, Michael Gmelin wrote: > >>> On Sun, 13 Mar 2022 17:53:44 + > >>> "Bjoern A. Z

Re: epair and vnet jail loose connection.

2022-03-15 Thread Kristof Provost
On 14 Mar 2022, at 18:02, Michael Gmelin wrote: > On Mon, 14 Mar 2022 09:09:49 -0600 > Kristof Provost wrote: > >> On 14 Mar 2022, at 7:44, Michael Gmelin wrote: >>> On Sun, 13 Mar 2022 17:53:44 + >>> "Bjoern A. Zeeb" wrote: >>> On 13 Mar 2022, at 17:45, Michael Gmelin wrote: >>

Re: epair and vnet jail loose connection.

2022-03-15 Thread Michael Gmelin
On Tue, 15 Mar 2022 01:02:30 +0100 Michael Gmelin wrote: > snip . > Hi Kristof, > > This sounds plausible. I spent a few hours getting familiar with the > epair code and came up with a patch that seems to fix the issue at > hand (both with and without RSS). I'm not certain that it is

Re: epair and vnet jail loose connection.

2022-03-14 Thread Michael Gmelin
On Mon, 14 Mar 2022 09:09:49 -0600 Kristof Provost wrote: > On 14 Mar 2022, at 7:44, Michael Gmelin wrote: > > On Sun, 13 Mar 2022 17:53:44 + > > "Bjoern A. Zeeb" wrote: > > > >> On 13 Mar 2022, at 17:45, Michael Gmelin wrote: > >> > On 13. Mar 2022, at 18:16, Bjoern A. Zeeb > >>

Re: epair and vnet jail loose connection.

2022-03-14 Thread Johan Hendriks
On 14/03/2022 16:09, Kristof Provost wrote: On 14 Mar 2022, at 7:44, Michael Gmelin wrote: On Sun, 13 Mar 2022 17:53:44 + "Bjoern A. Zeeb" wrote: On 13 Mar 2022, at 17:45, Michael Gmelin wrote: On 13. Mar 2022, at 18:16, Bjoern A. Zeeb wr

Re: epair and vnet jail loose connection.

2022-03-14 Thread Kristof Provost
On 14 Mar 2022, at 7:44, Michael Gmelin wrote: On Sun, 13 Mar 2022 17:53:44 + "Bjoern A. Zeeb" wrote: On 13 Mar 2022, at 17:45, Michael Gmelin wrote: On 13. Mar 2022, at 18:16, Bjoern A. Zeeb wrote: On 13 Mar 2022, at 16:33, Michael Gmelin wrote: It's important to point out that this

Re: epair and vnet jail loose connection.

2022-03-13 Thread Bjoern A. Zeeb
On 13 Mar 2022, at 17:45, Michael Gmelin wrote: On 13. Mar 2022, at 18:16, Bjoern A. Zeeb wrote: On 13 Mar 2022, at 16:33, Michael Gmelin wrote: It's important to point out that this only happens with kern.ncpu>1. With kern.ncpu==1 nothing gets stuck. This perfectly fits into the picture,

Re: epair and vnet jail loose connection.

2022-03-13 Thread Michael Gmelin
> On 13. Mar 2022, at 18:16, Bjoern A. Zeeb > wrote: > > On 13 Mar 2022, at 16:33, Michael Gmelin wrote: >> It's important to point out that this only happens with kern.ncpu>1. >> With kern.ncpu==1 nothing gets stuck. >> >> This perfectly fits into the picture, since, as pointed out by Joha

Re: epair and vnet jail loose connection.

2022-03-13 Thread Bjoern A. Zeeb
On 13 Mar 2022, at 16:33, Michael Gmelin wrote: It's important to point out that this only happens with kern.ncpu>1. With kern.ncpu==1 nothing gets stuck. This perfectly fits into the picture, since, as pointed out by Johan, the first commit that is affected[0] is about multicore support. Igno

Re: epair and vnet jail loose connection.

2022-03-13 Thread Michael Gmelin
On Sun, 13 Mar 2022 14:32:50 +0100 Johan Hendriks wrote: > On 13/03/2022 14:06, Patrick M. Hausen wrote: > > Hi all, > > > > i was a bit puzzled by Michael using bhyve trying to reproduce. > > Up until now I thought bhyve uses tap and not epair? > > > > Anyway ... > > > >> Am 13.03.2022 um 1

Re: epair and vnet jail loose connection.

2022-03-13 Thread Johan Hendriks
On 13/03/2022 14:06, Patrick M. Hausen wrote: Hi all, i was a bit puzzled by Michael using bhyve trying to reproduce. Up until now I thought bhyve uses tap and not epair? Anyway ... Am 13.03.2022 um 14:01 schrieb Johan Hendriks : I have no idea why it does not work on my setup, which is not

Re: epair and vnet jail loose connection.

2022-03-13 Thread Michael Gmelin
> On 13. Mar 2022, at 14:07, Patrick M. Hausen wrote: > > Hi all, > > i was a bit puzzled by Michael using bhyve trying to reproduce. > Up until now I thought bhyve uses tap and not epair? > In my setup, FreeBSD 14 runs on a bhyve vm, hosting the jails, which use vnet. Bare metal -> FreeB

Re: epair and vnet jail loose connection.

2022-03-13 Thread Kristof Provost
> On 13 Mar 2022, at 08:01, Johan Hendriks wrote: > >  >> On 13/03/2022 13:37, Kristof Provost wrote: >>> On 13 Mar 2022, at 5:26, Johan Hendriks wrote: >>> Copyd my haproxy en web01 jail to this machine and have the same problem. >>> >> Do you mean you can or cannot reproduce it on the seco

Re: epair and vnet jail loose connection.

2022-03-13 Thread Patrick M. Hausen
Hi all, i was a bit puzzled by Michael using bhyve trying to reproduce. Up until now I thought bhyve uses tap and not epair? Anyway ... > Am 13.03.2022 um 14:01 schrieb Johan Hendriks : > I have no idea why it does not work on my setup, which is nothing out of the > ordinary i think, basic full

Re: epair and vnet jail loose connection.

2022-03-13 Thread Johan Hendriks
On 13/03/2022 13:37, Kristof Provost wrote: On 13 Mar 2022, at 5:26, Johan Hendriks wrote: Copyd my haproxy en web01 jail to this machine and have the same problem. Do you mean you can or cannot reproduce it on the second machine? I have the same problem. Could it be a sysctl i use? or b

Re: epair and vnet jail loose connection.

2022-03-13 Thread Kristof Provost
On 13 Mar 2022, at 5:26, Johan Hendriks wrote: > Copyd my haproxy en web01 jail to this machine and have the same problem. > Do you mean you can or cannot reproduce it on the second machine? > Could it be a sysctl i use? or boot/loader.conf setting. > None of those settings strike me as likely to

Re: epair and vnet jail loose connection.

2022-03-13 Thread Michael Gmelin
> On 13. Mar 2022, at 11:27, Johan Hendriks wrote: >  > > > Op zo 13 mrt. 2022 01:17 schreef Michael Gmelin : >> I also gave it another go (this time with multiple CPUs assigned to the vm), >> still works just fine - so I think we would need more details about the >> setup. >> >> Would it

Re: epair and vnet jail loose connection.

2022-03-13 Thread Johan Hendriks
Op zo 13 mrt. 2022 01:17 schreef Michael Gmelin : > I also gave it another go (this time with multiple CPUs assigned to the > vm), still works just fine - so I think we would need more details about > the setup. > > Would it make sense to share our test setups, so Johan can try to > reproduce with

Re: epair and vnet jail loose connection.

2022-03-12 Thread Michael Gmelin
I also gave it another go (this time with multiple CPUs assigned to the vm), still works just fine - so I think we would need more details about the setup. Would it make sense to share our test setups, so Johan can try to reproduce with them? -m > On 13. Mar 2022, at 00:48, Kristof Provost wr

Re: epair and vnet jail loose connection.

2022-03-12 Thread Kristof Provost
I’m still failing to reproduce. Is pf absolutely required to trigger the issue? Is haproxy (i.e. can you trigger it with iperf)? Is the bridge strictly required? Kristof On 12 Mar 2022, at 8:18, Johan Hendriks wrote: For me this minimal setup let me see the drop off of the network from the

Re: epair and vnet jail loose connection.

2022-03-12 Thread Michael Gmelin
Hi Johan, I created a setup similar to yours - so it's based on your config files, but I had to tweak a few things to make it work. Basically this what it looks like: Host: Bare metal Runs bhyve, bridge has 10.1.1.1, does NAT to the world Bhyve VM runs 14-CURRENT (latest snapshot and has). vt

Re: epair and vnet jail loose connection.

2022-03-12 Thread Johan Hendriks
For me this minimal setup let me see the drop off of the network from the haproxy server. 2 jails, one with haproxy, one with nginx which is using the following html file to be served. Page Title My First Heading My first paragraph. >From a remote machine i do a hey -h2 -n 10 -c 10 -z

Re: epair and vnet jail loose connection.

2022-03-11 Thread Kristof Provost
On 11 Mar 2022, at 18:55, Michael Gmelin wrote: >> On 12. Mar 2022, at 01:21, Kristof Provost wrote: >> >> On 11 Mar 2022, at 17:44, Johan Hendriks wrote: On 09/03/2022 20:55, Johan Hendriks wrote: The problem: I have a FreeBSD 14 machine and a FreeBSD 13-stable machine, both runni

Re: epair and vnet jail loose connection.

2022-03-11 Thread Michael Gmelin
> On 12. Mar 2022, at 01:21, Kristof Provost wrote: > > On 11 Mar 2022, at 17:44, Johan Hendriks wrote: >>> On 09/03/2022 20:55, Johan Hendriks wrote: >>> The problem: >>> I have a FreeBSD 14 machine and a FreeBSD 13-stable machine, both running >>> the same jails just to test the workings.

Re: epair and vnet jail loose connection.

2022-03-11 Thread Kristof Provost
On 11 Mar 2022, at 17:44, Johan Hendriks wrote: > On 09/03/2022 20:55, Johan Hendriks wrote: >> The problem: >> I have a FreeBSD 14 machine and a FreeBSD 13-stable machine, both running >> the same jails just to test the workings. >> >> The jails that are running are a salt master, a haproxy jail

Re: epair and vnet jail loose connection.

2022-03-11 Thread Johan Hendriks
On 09/03/2022 20:55, Johan Hendriks wrote: The problem: I have a FreeBSD 14 machine and a FreeBSD 13-stable machine, both running the same jails just to test the workings. The jails that are running are a salt master, a haproxy jail, 2 webservers, 2 varnish servers, 2 php jails one for php8

Re: epair and vnet jail loose connection.

2022-03-10 Thread Johan Hendriks
On 10/03/2022 13:37, Wolfgang Zenker wrote: Hi Kristof, Am Thu, Mar 10, 2022 at 12:44:00PM +0100 schrieb Kristof Provost: On 10 Mar 2022, at 10:13, Johan Hendriks wrote: On 10/03/2022 08:54, Patrick M. Hausen wrote: Hi Johan, we experience the same on 13.1-PRERELEASE. Currently trying to c

Re: epair and vnet jail loose connection.

2022-03-10 Thread Bjoern A. Zeeb
On Thu, 10 Mar 2022, Wolfgang Zenker wrote: Hi, I did do a  hey -h2 -n 10 -c 10 -z 60s https://wp.test.nl to that machine and in the 60 seconds the jail became unresponsive. Then i did run the dtrace.sh script above like so /root/bin/dtrace.sh > /root/dtrace_output I hope this helps, if you

Re: epair and vnet jail loose connection.

2022-03-10 Thread Kristof Provost
On 10 Mar 2022, at 10:13, Johan Hendriks wrote: > On 10/03/2022 08:54, Patrick M. Hausen wrote: >> Hi Johan, >> >> we experience the same on 13.1-PRERELEASE. Currently trying to collect some >> evidence >> (dtrace) to send to Kristof Provost who was so kind to assist. We are hit by >> the problem

Re: epair and vnet jail loose connection.

2022-03-10 Thread Johan Hendriks
On 10/03/2022 08:54, Patrick M. Hausen wrote: Hi Johan, we experience the same on 13.1-PRERELEASE. Currently trying to collect some evidence (dtrace) to send to Kristof Provost who was so kind to assist. We are hit by the problem in production in 12-24 hour intervals. Have not done any artif

Re: epair and vnet jail loose connection.

2022-03-09 Thread Patrick M. Hausen
Hi Johan, we experience the same on 13.1-PRERELEASE. Currently trying to collect some evidence (dtrace) to send to Kristof Provost who was so kind to assist. We are hit by the problem in production in 12-24 hour intervals. Have not done any artificial load tests, yet. May I ask you to run this

Re: epair and vnet jail loose connection.

2022-03-09 Thread Johan Hendriks
I remembered that it used to work, so i thought lets go back in time. So i did a git reset --hard 375fdb6e161ea78a957314efeecd5ee0654a2793 which is a commit from january the first of 2022. [root]@[jhost001] - [ ~ ] > uname -a FreeBSD jhost001 13.0-STABLE FreeBSD 13.0-STABLE #0 stable/13-n248793-37

epair and vnet jail loose connection.

2022-03-09 Thread Johan Hendriks
The problem: I have a FreeBSD 14 machine and a FreeBSD 13-stable machine, both running the same jails just to test the workings. The jails that are running are a salt master, a haproxy jail, 2 webservers, 2 varnish servers, 2 php jails one for php8.0 and one with 8.1. All the jails are