On 14.04.2019 22:40, John Fastabend wrote:
On 4/13/19 6:56 PM, Andre Tomt wrote:
On 13.04.2019 17:34, Steinar H. Gunderson wrote:
Hi,
I've been using kTLS for a while, with my video reflector Cubemap
(https://git.sesse.net/?p=cubemap). After I upgraded my server from
4.18.11 to 5.0.6, seemingly I've started seeing corruption. The data sent
with send() (HTTP headers, HLS playlists) appears to be fine, but sendfile()
(actual video data, from a file on tmpfs) is not; after ~20 kB of data
(19626 in one test here), the data appears to be randomly corrupted. Diffing
non-TLS (good) and TLS (bad) video data:
00004c70: fa 70 c5 71 b5 f5 b7 ac 74 b0 ca 80 02 4c 06 3f .p.q....t....L.?
00004c80: 5c 5b 0c b3 e0 a0 c3 21 93 d3 6e 65 36 70 0a 27 \[.....!..ne6p.'
00004c90: 84 67 16 2c 95 c0 55 e1 04 76 52 10 50 5d 00 26 .g.,..U..vR.P].&
-00004ca0: 0c b8 84 70 7e ed 12 8f 5e 7e 18 c0 06 20 02 54 ...p~...^~... .T
+00004ca0: 0c b8 84 70 7e ed 12 8f 5e 7e 0a 60 9f 1f 97 f2 ...p~...^~.`....
-00004cb0: 1e 4c c1 71 7d 0b 91 28 23 98 09 ae c4 95 ae 7f .L.q}..(#.......
+00004cb0: 6e 17 50 03 67 fa 2f 83 b0 88 eb fc 54 f2 0b 00 n.P.g./.....T...
-00004cc0: a2 92 20 b8 f2 b6 72 2a e8 7e d7 27 99 65 56 70 .. ...r*.~.'.eVp
+00004cc0: 6c 9e a1 02 b4 30 11 25 d7 58 b0 0c c0 6c e1 bd l....0.%.X...l..
It never appears to get back into sync after that. Interestingly, it is
_consistently_ wrong; if I download the same fragment multiple times, it
breaks at the same place and gives the same garbage (but different fragments
give different divergence points). Tested with both wget and Chrome.
Does anyone know what could be wrong?
(It is, unfortunately, not easy for me to reboot this server at will, so a
bisect could be hard.)
Please Cc me on any replies, I'm not subscribed to netdev.
/* Steinar */
Reproduced and bisected, the problem showed up in v4.20-rc1. Unfortunately the
commit seems to have some significant dependencies so I was unable to verify by
reverting it on 4.20.
Hi thanks I'll take a look this evening or first thing tomorrow. I have
a couple other fixes queued up for ktls as well so I'll see if we can get
a fix for this included in that series.
Thanks!
John
Hi John
Have you had any luck tracking this down?
Just gave net.git a spin and it is still serving up corrupted data when
ktls is active and using sendfile. FWIW I only tested without ktls
offload capable hardware (ie in software mode) and no bpf. Same sendfile
usage on a non-ktls socket works fine.
