Peers using heavily single cpu core

receiving fixes around Q2 2026. Known bugs: http://www.haproxy.org/bugs/bugs-2.4.14.html Running on: Linux 4.15.0-154-generic #161-Ubuntu SMP Fri Jul 30 13:04:17 UTC 2021 x86_64 Kind regards, Maciej Zdeb

Re: Peers using heavily single cpu core

49) an_exp= rex=5s wex= buf=0x7f0aa402e2d8 data=(nil) o=0 p=0 i=0 size=0 res=0x7f0aa402e330 (f=0x80448202 an=0x0 pipe=0 tofwd=-1 total=9322481446) an_exp= rex= wex= buf=0x7f0aa402e338 data=(nil) o=0 p=0 i=0 size=0 pt., 11 mar 2022 o 18:09 Willy Tarreau napisał(a): > Hi

Re: Peers using heavily single cpu core

00 epoch=0x4 age=0s calls=1 rate=1 cpu=0 lat=0 rq[f=c08000h,i=0,an=00h,rx=,wx=,ax=] rp[f=80008002h,i=0,an=00h,rx=,wx=,ax=] s0=[8,240008h,fd=30,ex=] s1=[8,204018h,fd=-1,ex=] exp= Kind regards, sob., 12 mar 2022 o 00:20 Willy Tarreau napisał(a): > On Fri, Mar 11, 2022 at 10:19:09PM +0100, Macie

Re: Peers using heavily single cpu core

Hi Willy, Thanks for the patch! Now I'll be able to rebalance peers connections manually. Do you know why so many of them are landing on the same thread? Kind regards czw., 31 mar 2022 o 15:16 Willy Tarreau napisał(a): > On Thu, Mar 31, 2022 at 11:56:13AM +0200, Willy Tarreau wrote: > > > I've

Re: Peers using heavily single cpu core

pt., 1 kwi 2022 o 09:06 Willy Tarreau napisał(a): > I seem to remember a discussion about this in the past and that the > conclusion basically was that outgoing connections are sent on the > "current" thread when the management task creates them, while incoming > connections are balanced dependin

Re: Peers using heavily single cpu core

and the "flow" between applet and session (and how/when task_queue is called in this flow). I would appreciate any help or tips on how to proceed. Kind regards, pt., 1 kwi 2022 o 14:34 Willy Tarreau napisał(a): > On Fri, Apr 01, 2022 at 11:23:34AM +0200, Maciej Zdeb wrote: > &g

Re: Peers using heavily single cpu core

Hi Willy, I adjusted the 3rd patch to use HA_ATOMIC_INC/HA_ATOMIC_DEC. Thanks! czw., 7 kwi 2022 o 14:02 Willy Tarreau napisał(a): > There are some ongoing changes in that area that are about to be merged > (probably early > next week) and that will help for this task, so we'll have a closer look

Re: Peers using heavily single cpu core

Kind regards czw., 7 kwi 2022 o 22:32 Maciej Zdeb napisał(a): > Hi Willy, > I adjusted the 3rd patch to use HA_ATOMIC_INC/HA_ATOMIC_DEC. Thanks! > > czw., 7 kwi 2022 o 14:02 Willy Tarreau napisał(a): > >> There are some ongoing changes in that area that are about to be

Re: Peers using heavily single cpu core

No worries, thanks for the update. Kind regards, Maciej śr., 20 kwi 2022 o 21:59 Willy Tarreau napisał(a): > Hi Maciej, > > On Wed, Apr 20, 2022 at 02:51:32PM +0200, Maciej Zdeb wrote: > > Hi Willy, > > I saw Christopher changes are now merged. I was wondering how to proc

Session terminated with status IH-- after upgrade to 2.4.16

Hi, after the HAProxy upgrade from 2.4.15 to 2.4.16 we observe in logs that very small percent of sessions are terminated with IH-- status. Docs say that "I" should never happen and should be reported. All the terminated sessions used one of the POST/PATCH/PUT methods and all of them were on HTTP/

Re: Session terminated with status IH-- after upgrade to 2.4.16

Ok, I've found the github issue https://github.com/haproxy/haproxy/issues/1684 sorry for the duplicate. Kind regards, Maciej czw., 12 maj 2022 o 13:34 Maciej Zdeb napisał(a): > Hi, > after the HAProxy upgrade from 2.4.15 to 2.4.16 we observe in logs that > very small percent o

Re: Peers using heavily single cpu core

xprt=RAW Kind regards, wt., 17 maj 2022 o 16:25 Christopher Faulet napisał(a): > Le 4/20/22 à 14:51, Maciej Zdeb a écrit : > > Hi Willy, > > I saw Christopher changes are now merged. I was wondering how to proceed > with my > > issue. Right now in stream_new() I'm

2.4.22 and maxconn setting problem

Hi, I'm observing a strange issue with haproxy 2.4.22 (but it was also on previous versions). I have set maxconn to 20 in global and defaults configuration section and with following configuration frontend front mode http option http-keep-alive bind 10.0.0.10:443 ssl crt /etc/ce

Re: 2.4.22 and maxconn setting problem

wt., 21 mar 2023 o 11:39 Willy Tarreau napisał(a): > Just to be clear on these last few points, when you say you cannot > connect, you mean in fact that the connection establishes to haproxy > but your request cannot reach the server, right ? 503 will indeed > indicate a failure to find a server

Re: 2.4.22 and maxconn setting problem

wt., 21 mar 2023 o 15:03 Willy Tarreau napisał(a): > What you *seem* to be missing in these logs is "%bc" and "%bq" to see > the state of the backend itself. My suspicion is that %bq is not zero > despite %bc being zero. > I will add these to logs/experiment with balance source and will return wi

[2.4.22] Segmentation fault when using spoe + disabled keyword

Hi, I'm experiencing a segmentation fault caused by adding "disabled" ( http://docs.haproxy.org/2.4/configuration.html#4-disabled) to the frontend section of haproxy configuration file. That frontend does not handle any traffic yet. The "disabled" keyword was used on other frontends on the same HA

Re: [2.4.22] Segmentation fault when using spoe + disabled keyword

-frontend-http-request if route_to_abc wt., 9 maj 2023 o 09:14 Christopher Faulet napisał(a): > Le 5/2/23 à 13:58, Maciej Zdeb a écrit : > > Hi, > > I'm experiencing a segmentation fault caused by adding "disabled" > > (http://docs.haproxy.org/2.4/c

Re: [2.4.22] Segmentation fault when using spoe + disabled keyword

Awesome, thanks! czw., 11 maj 2023 o 09:38 Christopher Faulet napisał(a): > Le 5/9/23 à 14:29, Maciej Zdeb a écrit : > > Hi Christopher, > > no problem. :) Yes I'm using the same spoe backend for multiple > frontends. This > > is my spoe configuration: > > >

[2.0.17] crash with coredump

Hi, Our HAProxy (2.0.14) started to crash, so first we upgraded to 2.0.17 but it didn't help. Below you'll find traces from coredump Version: HA-Proxy version 2.0.17 2020/07/31 - https://haproxy.org/ Build options : TARGET = linux-glibc CPU = generic CC = gcc CFLAGS = -O0 -g -f

Re: [2.0.17] crash with coredump

Hi, Our config is quite complex and I'm trying to narrow it down. It is occurring only on one production haproxy cluster (which consists of 6 servers in each of two data centers) with significant load - crashes occurs on random servers so I would exclude memory corruption. I'm suspecting SPOE or/

Re: [2.0.17] crash with coredump

Hi Kirill, Thanks for your hints and time! Unfortunately, I think lrandom is not the cause of crash. We're using lrandom with threads for couple of months on our other servers without any crash. I think lua in HAproxy is executed in a single thread so your analysis is correct but this assumption i

Re: [2.0.17] crash with coredump

> Here I can suggest to implement Yarrow PRGN (that is very simple to > implement) with some lua-pure cryptographic hash function. We're using lrandom because of the algorithm Mersenne Twister and its well known weaknesses and strengths. > In fact I know it's possible to call haproxy's internal s

Re: [2.0.17] crash with coredump

Yes at the same place with same value: (gdb) bt full #0 0x559ce98b964b in h2s_notify_recv (h2s=0x559cef94e7e0) at src/mux_h2.c:783 sw = 0x pt., 25 wrz 2020 o 15:42 Kirill A. Korinsky napisał(a): > > On 25. Sep 2020, at 15:26, Maciej Zdeb wrote: > > >

Re: [2.0.17] crash with coredump

ommit/f96508aae6b49277dcf142caa35042678cf8e2ca > > Maybe it is worth to try 2.2 or 2.3 branch? > > Yes, it is a blind shot and just a guess. > > -- > wbr, Kirill > > On 25. Sep 2020, at 16:01, Maciej Zdeb wrote: > > Yes at the same place with same value: > > (g

Re: [2.0.17] crash with coredump

t moved to somewhere else. > > -- > wbr, Kirill > > On 2. Nov 2020, at 10:58, Maciej Zdeb wrote: > > Hi, > > Update for people on the list that might be interested in the issue, > because part of discussion was private. > > I wan

Re: [2.0.17] crash with coredump

n adding in h2c->send_list or h2c->fctl_lsit */ }; pon., 2 lis 2020 o 12:42 Maciej Zdeb napisał(a): > Great idea Kirill, > > With such modification: > > struct h2s { > [...] > struct tasklet *shut_tl; > struct wait_event *recv_wait; /* recv wait_eve

Re: [2.0.17] crash with coredump

her place. > > Willy do you agree? > > -- > wbr, Kirill > > On 2. Nov 2020, at 15:34, Maciej Zdeb wrote: > > So after Kirill suggestion to modify h2s struct in a way that tasklet > "shut_tl" is before recv_wait I verified if in 2.2.4 the same crash will > occu

Re: [2.0.17] crash with coredump

://git.haproxy.org/?p=haproxy-2.2.git;a=commit;h=5c8be272c732e4f42ccd6b3d65f25aa7425a2aba which alters tasks processing. pon., 2 lis 2020 o 15:46 Maciej Zdeb napisał(a): > I'm wondering, the corrupted address was always at "wait_event" in h2s > struct, after its removal in: >

Re: [2.0.17] crash with coredump

Hi, This time h2s = 0x30 ;) it crashed here: void testcorrupt(void *ptr) { [...] if (h2s->cs != cs) return; [...] Program terminated with signal SIGSEGV, Segmentation fault. #0 0x556b617f0562 in testcorrupt (ptr=0x7f99741d85a0) at src/mux_h2.c:6228 6228 src/mux_h2.c: No such

Re: [2.0.17] crash with coredump

;s->si[1], srv_conn); // FAIL TEST_STRM(s); // if (!srv_cs) { conn_free(srv_conn); return SF_ERR_RESOURCE; } [...] } pon., 9 lis 2020 o 11:51 Maciej Zdeb napisał(a): > Hi, > > This time h2s = 0x30

Re: [2.0.17] crash with coredump

= 0x, list = {n = 0x7f75842445c8, p = 0x7f75842445c8}, shut_tl = 0x7f75842df0d0} pon., 9 lis 2020 o 15:07 Christopher Faulet napisał(a): > Le 09/11/2020 à 13:10, Maciej Zdeb a écrit : > > I've played little bit with the patch and it led me to backend.c file > and >

Re: [2.0.17] crash with coredump

): haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 10608 pon., 9 lis 2020 o 16:01 Maciej Zdeb napisał(a): > It crashed now on first test in process_stream: > > struct task *process_stream(struct task *t, void *context, unsigned short > state) > { >

Re: [2.0.17] crash with coredump

Hi, I'm so happy you're able to replicate it! :) With that patch that disabled pool_flush I still can reproduce on my r&d server and on production, just different places of crash: on r&d: (gdb) bt #0 tasklet_wakeup (tl=0xd720c300a000) at include/haproxy/task.h:328 #1 h2s_notify_recv (h2s=h

Re: [2.0.17] crash with coredump

gt; On Tue, Nov 10, 2020 at 09:17:15PM +0100, Christopher Faulet wrote: > > Le 10/11/2020 à 18:12, Maciej Zdeb a écrit : > > > Hi, > > > > > > I'm so happy you're able to replicate it! :) > > > > > > With that patch that disabled pool_flush I

Re: [2.0.17] crash with coredump

śr., 11 lis 2020 o 12:53 Willy Tarreau napisał(a): > Two months of chasing a non reproducible > memory corruption with zero initial info is quite an achievement, many > thanks for doing that! > Initially it crashed (once every few hours) only on our most critical HAProxy servers and with SPOA fr

HTTP fetch req.cook_cnt() always return 0

Hi, With such sample config: frontend front mode http http-request set-var(txn.abc) req.cook_cnt() http-response set-header abc %[var(txn.abc)] bind x.x.x.x:80 default_backend back backend back mode http server srv y.y.y.y:80 check When sending 3 cookies with curl: cu

Re: HTTP fetch req.cook_cnt() always return 0

I've a patch that is fixing the issue (I hope) ;) It's my first patch so any hints are very welcome :) pt., 13 lis 2020 o 09:45 Christopher Faulet napisał(a): > Le 12/11/2020 à 15:08, Maciej Zdeb a écrit : > > Hi, > > > > With such sample config: > >

Re: HTTP fetch req.cook_cnt() always return 0

Fixed warnings during compilation and fixed req.cook() which should return first and not last value when no cookie name specified. pt., 13 lis 2020 o 10:48 Maciej Zdeb napisał(a): > I've a patch that is fixing the issue (I hope) ;) It's my first patch so > any hints are very w

Re: HTTP fetch req.cook_cnt() always return 0

It is great Christopher! :) Thank you for fixing it! W dniu pt., 13.11.2020 o 14:06 Christopher Faulet napisał(a): > Le 13/11/2020 à 11:27, Maciej Zdeb a écrit : > > Fixed warnings during compilation and fixed req.cook() which should > return first > > and not last value w

Re: [PATCH] REGTESTS: Add sample_fetches/cook.vtc

Great idea Tim! Thank you! pt., 13 lis 2020 o 19:49 Christopher Faulet napisał(a): > Le 13/11/2020 à 19:36, Tim Duesterhus a écrit : > > Add a reg-test verifying the fix in > dea7c209f8a77b471323dd97bdc1ac4d7a17b812. > > > > Thanks Tim! merged now. > > -- > Christopher Faulet >

Re: [2.2.5] High cpu usage after switch to threads

txn.http:req_del_header(name) end end end core.register_action('req_header_filter', { 'http-req' }, req_header_filter, 1) śr., 18 lis 2020 o 12:46 Julien Pivotto napisał(a): > On 18 Nov 12:33, Maciej Zdeb wrote: > > Hi again, >

Re: [2.2.5] High cpu usage after switch to threads

cea -m end I'll try to implement it in the free time. śr., 18 lis 2020 o 13:20 Maciej Zdeb napisał(a): > Sure, the biggest problem is to delete header by matching prefix: > > load_blacklist = function(service) > local prefix = '/etc/haproxy/configs/maps/header_

Re: [2.2.5] High cpu usage after switch to threads

wants to implement it asap, then just let me know. W dniu śr., 18.11.2020 o 16:49 Tim Düsterhus napisał(a): > Maciej, > > Am 18.11.20 um 14:22 schrieb Maciej Zdeb: > > I've found an earlier discussion about replacing reqidel (and others) in > > 2.x: https://www.mail-archiv

Re: [2.2.5] High cpu usage after switch to threads

Hi, Alaksandar I've looked into code and... :) śr., 18 lis 2020 o 15:30 Aleksandar Lazic napisał(a): > Can you think to respectthe '-i'. > > http://git.haproxy.org/?p=haproxy.git&a=search&h=HEAD&st=grep&s=PAT_MF_IGNORE_CASE > I'm not sure if I understand you correctly, but in case of http-reque

[PATCH] minor bug + http-request del-header supporting -m flag

Hi, I've implemented a http-request del-header with optional flag -m [beg, sub, str, end]. I'm not sure if my approach is correct so I'm not adding support for the "reg" method in these patches. I've doubts about (ab)using the action field in the act_rule struct for storing selected matching metho

Re: [PATCH] minor bug + http-request del-header supporting -m flag

pt., 20 lis 2020 o 16:56 Tim Düsterhus napisał(a): > There's a duplicate semicolon there. Fixed! > Merge the test into the previous commit, please. This ensures the test > is always there when the feature is there and absent if it is not. > True! Fixed. Thank you Tim! 0001-BUG-MINOR-http_ht

Re: [PATCH] minor bug + http-request del-header supporting -m flag

I've added to commit msg "This is related to GitHub issue #909" because there is still no support for reg matching in this patch. pt., 20 lis 2020 o 17:54 Tim Düsterhus napisał(a): > > +#REQUIRE_VERSION=2.2 > > This should either be 2.4 or be removed. The removal should be safe, > because it is

Re: [PATCH] minor bug + http-request del-header supporting -m flag

x->conf.args.line; It is on various places in http_act.c and so I've also placed that in my patch, however I'm not sure if it's really necessary. I just don't understand this fragment, why it is there. :( pt., 20 lis 2020 o 19:49 Tim Düsterhus napisał(a): > Maciej, &

Re: [PATCH] minor bug + http-request del-header supporting -m flag

Hi, Willy thanks for clarification about lfs_file, I've removed it from patch. As Christopher mentioned, when developing the patch I've noticed the comment that when "action_ptr" is set, "action" can be set to any meaningful value. However, I also had doubts about it. :) Christopher I fixed the p

Re: [2.2.5] High cpu usage after switch to threads

sob., 21 lis 2020 o 07:13 Willy Tarreau napisał(a): > So I guess we'll use you as a beta tester once we're starting to see > promising solutions ;-) > I'll test it happily :)

[PATCH] Fix lacking init of log-format list

Hi, By accident I removed initialization of log-format list during implementation of -m flag for http-request/response del-header. :( This patch restores the init. 0001-BUG-MINOR-http_act-Restore-init-of-log-format-list.patch Description: Binary data

Re: [PATCH] Fix lacking init of log-format list

8 Tim Düsterhus napisał(a): > Maciej, > > Am 23.11.20 um 17:15 schrieb Maciej Zdeb: > > By accident I removed initialization of log-format list during > > implementation of -m flag for http-request/response del-header. :( > > > > This patch restores the init. >

Re: [PATCH] Fix lacking init of log-format list

wt., 24 lis 2020 o 10:34 Willy Tarreau napisał(a): > I prefer the former. If there's a list head, it must be valid. We must > not start to look at list pointers to guess whether or not they are > properly initialized, there are enough exceptions and special cases > everywhere in the code not to a

Re: [ANNOUNCE] haproxy-2.4-dev1

ttle bit of crt-list breakage that was quickly addressed. A bug in > the http-after-response rules could possibly cause random crashes. An > old bug in the SPOE with a dangling pointer could cause random crashes > (Many thanks to Maciej Zdeb for working hard for two months to isolate > thi

[PATCH] DOC: Clarify %HP description in log-format

Hi, Since 30ee1efe676e8264af16bab833c621d60a72a4d7 "MEDIUM: h2: use the normalized URI encoding for absolute form requests" we're using absolute URI when possible. As stated in the commit message this also changed the way it is reported in logs, especially for H2. Docs describes %HP as "HTTP requ

Re: [PATCH] DOC: Clarify %HP description in log-format

Hi I have a small patch which adds a new log-format variable %HPR for logging relative path. I hope it is clean and useful! :) Kind regards, czw., 26 lis 2020 o 19:08 Willy Tarreau napisał(a): > Hi Maciej, > > On Thu, Nov 26, 2020 at 12:39:19PM +0100, Maciej Zdeb wrot

Re: [PATCH] DOC: Clarify %HP description in log-format

I'm overwriting it to be more verbose. wt., 1 gru 2020 o 04:12 Willy Tarreau napisał(a): > Hi Maciej, > > On Mon, Nov 30, 2020 at 08:17:42PM +0100, Maciej Zdeb wrote: > > Hi > > > > I have a small patch which adds a new log-format variable %HPR for > logging &g

Re: [PATCH] DOC: Clarify %HP description in log-format

eaders' which is not available here. What I'm doing wrong? :) wt., 1 gru 2020 o 06:56 Willy Tarreau napisał(a): > On Tue, Dec 01, 2020 at 06:54:16AM +0100, Maciej Zdeb wrote: > > There's no difference with %[path] in fact this patch only exposes "path" > > fetch

Re: [PATCH] DOC: Clarify %HP description in log-format

bove config). Idea with named defaults is brilliant! :) wt., 1 gru 2020 o 07:26 Willy Tarreau napisał(a): > On Tue, Dec 01, 2020 at 07:13:32AM +0100, Maciej Zdeb wrote: > > For such defaults section: > > defaults > > log stdout local0 > > log-format 'path:%[path]'

Re: [PATCH] DOC: Clarify %HP description in log-format

wt., 1 gru 2020 o 08:14 Willy Tarreau napisał(a): > Ah I'm stupid, sorry! The problem is not caused by the defaults section > but by the fact that %[path] extracts the path from the request while your > proposal takes it from the copy of the URI that was kept for logging. > No need to be sorry. :

ring buffer + spoe = segfault

Hi, By accident I discovered that when enabling logging with line "log ring@spoe-ring local2" in SPOE immediate segmentation fault occurs. Steps to reproduce (configuration files attached): # execute spoa-server from contrib directory: ./spoa -d -p 8080 # execute haproxy: haproxy -f haproxy.cfg

Re: ring buffer + spoe = segfault

Great, thanks for the update. :) W dniu pt., 19.02.2021 o 18:06 Christopher Faulet napisał(a): > Le 19/02/2021 à 09:31, Maciej Zdeb a écrit : > > Hi, > > > > By accident I discovered that when enabling logging with line "log > > ring@spoe-ring local2" in SPO

[2.2.9] 100% CPU usage

Hi, Sometimes after HAProxy reload it starts to loop infinitely, for example 9 of 10 threads using 100% CPU (gdb sessions attached). I've also dumped the core file from gdb. # haproxy -v HA-Proxy version 2.2.9-a947cc2 2021/02/06 - https://haproxy.org/ Status: long-term supported branch - will sto

Re: [2.2.9] 100% CPU usage

ulet napisał(a): > Le 04/03/2021 à 14:01, Maciej Zdeb a écrit : > > Hi, > > > > Sometimes after HAProxy reload it starts to loop infinitely, for example > 9 of 10 > > threads using 100% CPU (gdb sessions attached). I've also dumped the > core file > > from

Re: [2.2.9] 100% CPU usage

Hi, After applying the patch, the issue did not occur, however I'm still not sure it is fixed. Unfortunately I don't have a reliable way to trigger it. pt., 5 mar 2021 o 22:07 Willy Tarreau napisał(a): > Note, before 2.4, a single thread can execute Lua scripts at once, > with the others waitin

Re: [2.2.9] 100% CPU usage

at 09:04:43AM +0100, Maciej Zdeb wrote: > > Hi, > > > > After applying the patch, the issue did not occur, however I'm still not > > sure it is fixed. Unfortunately I don't have a reliable way to trigger > it. > > OK. If it's related, it's very possi

Re: [2.2.9] 100% CPU usage

Sure, patch from Christopher attached. :) wt., 16 mar 2021 o 10:58 Willy Tarreau napisał(a): > Hi Maciej, > > On Tue, Mar 16, 2021 at 10:55:11AM +0100, Maciej Zdeb wrote: > > Hi, > > > > I'm returning with bad news, the patch did not help and the issue > oc

Re: [2.2.9] 100% CPU usage

ess_stream 0.19% [kernel][k] audit_filter_rules.constprop.13 0.17% haproxy [.] si_applet_wake_cb 0.14% libcrypto.so.1.1[.] rsaz_1024_mul_avx2 wt., 16 mar 2021 o 12:00 Maciej Zdeb

Re: [2.2.9] 100% CPU usage

metadata about each request to external service), - peers (we have a cluster of 12 HAProxy servers connected to each other). Kind regards, wt., 16 mar 2021 o 13:05 Maciej Zdeb napisał(a): > Below is the output from perf top - it happens during reload (all threads > on an old process spike an

Re: [2.2.9] 100% CPU usage

napisał(a): > Le 16/03/2021 à 13:46, Maciej Zdeb a écrit : > > Sorry for spam. In the last message I said that the old process (after > reload) > > is consuming cpu for lua processing and that's not true, it is > processing other > > things also. > > > > I&#x

Re: [2.2.9] 100% CPU usage

Hi Christopher, Thanks! I'm building a patched version and will return with feedback! Kind regards, pt., 19 mar 2021 o 16:40 Christopher Faulet napisał(a): > Le 16/03/2021 à 13:46, Maciej Zdeb a écrit : > > Sorry for spam. In the last message I said that the old process (

Re: [2.2.9] 100% CPU usage

Hi Christopher, Bad news, patches didn't help. Attaching stacktraces, now it looks that spoe that executes lua scripts (free_thread_spue_lua.txt) tried to malloc twice. :( Kind regards, pon., 22 mar 2021 o 08:39 Maciej Zdeb napisał(a): > Hi Christopher, > > Thanks! I'm

Re: [2.2.9] 100% CPU usage

Hi, wt., 23 mar 2021 o 18:36 Willy Tarreau napisał(a): > > It is most probably because of compiler optimizations. Some compiler > > barriers are necessary to avoid instructions reordering. It is the > purpose > > of attached patches. Sorry to ask you it again, but could you make some > > tests ?

Re: [2.2.9] 100% CPU usage

Wow, that's it! :) 0x55d94949e965 <+53>: addl $0x1,%fs:0xfffdd688 0x55d94949e96e <+62>: callq 0x55d9494782c0 0x55d94949e973 <+67>: subl $0x1,%fs:0xfffdd688 [...] 0x55d94949e99f <+111>: ja 0x55d94949e9b0 0x55d94949e9a1 <+113>: mov%rb

Re: [2.2.9] 100% CPU usage

śr., 24 mar 2021 o 10:37 Christopher Faulet napisał(a): > However, reading the other trace Maciej sent (bussy_thread_peers.txt), it > seems > possible to stop a memory allocation from other places. Thus, I guess we > must > find a more global way to prevent the lua stack dump. > I'm not sure whi

Re: [2.2.9] 100% CPU usage

d regards, śr., 24 mar 2021 o 10:51 Maciej Zdeb napisał(a): > śr., 24 mar 2021 o 10:37 Christopher Faulet > napisał(a): > >> However, reading the other trace Maciej sent (bussy_thread_peers.txt), it >> seems >> possible to stop a memory allocation from other places. Thu

Re: [2.2.9] 100% CPU usage

dgets[queue]--; (gdb) 440 ti->flags &= ~TI_FL_STUCK; // this thread is still running (gdb) 437 t = (struct task *)LIST_ELEM(tl_queues[queue].n, struct tasklet *, list); (gdb) 438 state = t->state & (TASK_SHARED_WQ|TASK_SELF_WAKING|TASK_KILLED); (gdb) 440 ti->flags &= ~TI_FL_STUCK;

Re: [2.2.9] 100% CPU usage

I've forgot to mention that the backtrace is from 2.2.11 built from http://git.haproxy.org/?p=haproxy-2.2.git;a=commit;h=601704962bc9d82b3b1cc97d90d2763db0ae4479 śr., 31 mar 2021 o 13:28 Maciej Zdeb napisał(a): > Hi, > > Well it's a bit better situation than earlier because

Re: [2.2.9] 100% CPU usage

31/03/2021 à 13:28, Maciej Zdeb a écrit : > > Hi, > > > > Well it's a bit better situation than earlier because only one thread is > looping > > forever and the rest is working properly. I've tried to verify where > exactly the > > thread looped but

Re: Still 100% CPU usage in 2.3.9 & 2.2.13 (Was: Re: [2.2.9] 100% CPU usage)

Hi Robin, W dniu pt., 9.04.2021 o 19:26 Robin H. Johnson napisał(a): > Maciej had said they were going to create a new thread, but I didn't see > one yet. > > I want to start by noting problem was much worse on 2.2.8 & 2.2.9, and > that 2.2.13 & 2.3.9 don't get entirely hung at 100% anymore: a b

[2.2.11] 100% CPU again

Hi, After a couple weeks running HAProxy 2.2.11, one server started to loop on thread 9. If I'm reading this correctly something went wrong on h2c at 0x7fd7b08d0530. ### show threads Thread 1 : id=0x7fd8855a2200 act=0 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0 stuck=0 prof=0 harmless=1 want

Re: [2.2.11] 100% CPU again

3.9, because as for now it looks like a different bug. Kind regards Maciej, wt., 20 kwi 2021 o 18:38 Christopher Faulet napisał(a): > Le 19/04/2021 à 19:54, Maciej Zdeb a écrit : > > Hi, > > > > After a couple weeks running HAProxy 2.2.11, one server started to loop &g

Re: [2.2.11] 100% CPU again

śr., 21 kwi 2021 o 13:53 Christopher Faulet napisał(a): > The fix was merge in upstream : > > * BUG/MAJOR: mux-h2: Properly detect too large frames when decoding headers >(http://git.haproxy.org/?p=haproxy.git;a=commit;h=07f88d75) > Thanks! Building 2.2.13 and 2.3.9 with patches. I'll return

Re: HAPROXY CAN NOT POINT IN TO PORT 5000 OF PATRONI

Hi, try removing those two lines from config: option httpchk http-check expect status 200 it is postgres (tcp backend), you should not expect http response on health check. Kind regards, Maciej czw., 22 kwi 2021 o 02:38 thủy bùi napisał(a): > I have open all port by setting firewall rules, But

[1.8] tune.ssl.cachesize

Hi, Thank you for your hard work on HAProxy 1.8 - great job! I tried to update haproxy from 1.7 on our r&d machine to 1.8 version however stumbled upon problem with tune.ssl.cachesize directive. In 1.7.3 with tune.ssl.cachesize = 1000: ps aux | grep haproxy USER PID %CPU %MEMVSZ

Re: [1.8] tune.ssl.cachesize

, Nov 27, 2017 at 09:15:22AM +0100, Maciej Zdeb wrote: > > Hi, > > > > Thank you for your hard work on HAProxy 1.8 - great job! > > > > I tried to update haproxy from 1.7 on our r&d machine to 1.8 version > > however stumbled upon problem with tune.ssl.cachesi

Re: [1.8] tune.ssl.cachesize

r/run/haproxy.pid -D - 2017-11-28 11:17 GMT+01:00 William Lallemand : > Hi, > > On Tue, Nov 28, 2017 at 09:08:59AM +0100, Maciej Zdeb wrote: > > Hi Willy, > > > > Thanks for your response. In change > > http://git.haproxy.org/?p=haproxy-1.8.git;a=commit;h= >

[1.9.6] One of haproxy processes using 100% CPU

Hi, After haproxy starts everything is fine, but after some time one of haproxy processes uses 100% of CPU. I've managed to dump some "show" commands (I have also "show fd", "show sess all" and "strace" but if needed I would prefer to send it privately). If you could give any advice howto debug it

Re: [1.9.6] One of haproxy processes using 100% CPU

I'm very sorry, it is probably "real" traffic that haproxy needs to handle. Some local DDoS occurred just in time when I upgraded to 1.9 (what terribly bad luck). Please ignore previous email! Sorry for wasting your time. czw., 4 kwi 2019 o 14:36 Maciej Zdeb napisał(a): > Hi,

Re: [1.9.6] One of haproxy processes using 100% CPU

would be very helpful. czw., 4 kwi 2019 o 14:56 Maciej Zdeb napisał(a): > I'm very sorry, it is probably "real" traffic that haproxy needs to > handle. Some local DDoS occurred just in time when I upgraded to 1.9 (what > terribly bad luck). > > Please ignore previous

Re: [1.9.6] One of haproxy processes using 100% CPU

pidfd = (gdb) generate-core-file warning: target file /proc/75107/cmdline contained unexpected null characters warning: Memory read failed for corefile section, 12288 bytes at 0x7ffd0e1c5000. Saved corefile core.75107 (gdb) quit pt., 5 kwi 2019 o 09:19 Maciej Zdeb napisał(a): > Hi again, > &g

Re: [1.9.6] One of haproxy processes using 100% CPU

n_exp= rex=30s wex= buf=0x20eec408 data=0x20e2a530 o=0 p=0 req.next=0 i=0 size=16384 res=0x20eec460 (f=0x80008002 an=0x0 pipe=0 tofwd=-1 total=1465) an_exp= rex= wex= buf=0x20eec468 data=0x20d5bbd0 o=1465 p=1465 rsp.next=0 i=0 size=16384 netstat -untap | grep S.S.S.S pt., 5 kwi 2019 o

Re: Infinite loop after 39cc020af BUG/MEDIUM: streams: Don't remove the SI_FL_ERR flag in si_update_both()

Hi Richard, Those patches from Olivier (in streams) are related to my report from thread "[1.9.6] One of haproxy processes using 100% CPU", but as it turned out it wasn't a single bug and issue is not entirely fixed yet. Currently I'm testing some additional patches from Olivier which hopefully f

[1.9.7] One of haproxy processes using 100% CPU

Hi, I'm returning with problem similar as in thread "[1.9.6] One of haproxy processes using 100% CPU". Thanks to Olivier hard work, some issues where fixed but still not all of them. :( Currently it is much harder to trigger and it occurs on https virtual server and not tcp one. I'm observing thi

Re: [1.9.7] One of haproxy processes using 100% CPU

annot be specified using 'proto' keyword) h2 : mode=HTTP side=FE h2 : mode=HTXside=FE|BE : mode=HTXside=FE|BE : mode=TCP|HTTP side=FE|BE Available filters : [SPOE] spoe [COMP] compression [CACHE] cache [TRACE] trace wt.,

Re: [1.9.7] One of haproxy processes using 100% CPU

.flg=0x1 .nbst=0 .nbcs=1 .fctl_cnt=0 .send_cnt=0 .tree_cnt=1 .orph_cnt=0 .sub=0 .dsi=89 .dbuf=0@(nil)+0/0 .msi=-1 .mbuf=0@(nil)+0/0 last_h2s=0x82f5900 .id=15 .flg=0x4005 .rxbuf=0@(nil)+0/0 .cs=0x7e027a0 .cs.flg=0x00106a00 .cs.data=0x6e5d398 wt., 30 kwi 2019 o 08:31 Maciej Zdeb napisał(a

Re: [1.9.7] One of haproxy processes using 100% CPU

Hi Olivier, Thank you very much. I'll test it and get back with feedback! Regards, wt., 30 kwi 2019 o 13:12 Olivier Houchard napisał(a): > Hi Maciej, > > On Tue, Apr 30, 2019 at 08:43:07AM +0200, Maciej Zdeb wrote: > > Filtered results from show fd for that parti

Re: [1.9.7] One of haproxy processes using 100% CPU

Hi, I confirm Willy patch fixed the problem! Thanks! wt., 30 kwi 2019 o 13:49 Maciej Zdeb napisał(a): > Hi Olivier, > > Thank you very much. I'll test it and get back with feedback! > > Regards, > > wt., 30 kwi 2019 o 13:12 Olivier Houchard > napisał(a): > &g

[1.9 HEAD] HAProxy using 100% CPU

Hi, I've got another bug with 100% CPU on HAProxy process, it is built from HEAD of 1.9 branch. One of processes stuck in infinite loop, admin socket is not responsive so I've got information only from gdb: 0x00484ab8 in h2_process_mux (h2c=0x2e8ff30) at src/mux_h2.c:2589 2589

Re: [1.9 HEAD] HAProxy using 100% CPU

I'll gladly test Olivier patch after backporting. :) śr., 8.05.2019, 15:29 użytkownik Willy Tarreau napisał: > On Wed, May 08, 2019 at 03:03:23PM +0200, Olivier Houchard wrote: > > > > I can't seem to remember :) > > > > > > Given the number of bugs we've dealt with in the last few weeks, you're

  1   2   >