On Fri, Dec 02, 2016 at 06:18:39PM +0000, Keen, Joe wrote: > > > On 12/2/16, 1:29 AM, "Mehdi Abaakouk" <sil...@sileht.net> wrote: > > >On Fri, Dec 02, 2016 at 03:29:59PM +1100, Tony Breeds wrote: > >>On Thu, Dec 01, 2016 at 04:52:52PM +0000, Keen, Joe wrote: > >> > >>> Unfortunately there¹s nothing wrong on the Monasca side so far as we > >>>know. > >>> We test new versions of the kafka-python library outside of Monasca > >>> before we bother to try integrating a new version. Since 1.0 the > >>> kafka-python library has suffered from crashes and memory leaks severe > >>> enough that we¹ve never attempted using it in Monasca itself. We > >>>reported > >>> the bugs we found to the kafka-python project but they were closed once > >>> they released a new version. > >> > >>So Opening bugs isn't working. What about writing code? > > > >The bug https://github.com/dpkp/kafka-python/issues/55 > > > >Reopening it would be the right solution here. > > > >I can't reproduce the segfault neither and I agree with dpkp, that looks > >like a > >ujson issue. > > > The bug I had was: https://github.com/dpkp/kafka-python/issues/551 > > In the case of that bug ujson was not an issue. The behaviour remained > even using the standard json library. The primary issue I found with it > was a memory leak over successive runs of the test script. Eventually the > leak became so bad that the OOM killer killed the process which caused the > segfault I was seeing. The last version I tested was 1.2.1 and it still > leaked badly. I¹ll need to let the benchmark script run for a while and > make sure it¹s not still leaking.
So you write that on Friday so you shoudl know by now if it's leaking care to give us an update? > >And my bench seems to confirm the perf issue have been solved: > >(but not in the pointed version...) > > > >$ pifpaf run kafka python kafka_test.py > >kafka-python version: 0.9.5 > >... > >fetch size 179200 -> 45681.8728864 messages per second > >fetch size 204800 -> 47724.3810674 messages per second > >fetch size 230400 -> 47209.9841092 messages per second > >fetch size 256000 -> 48340.7719787 messages per second > >fetch size 281600 -> 49192.9896743 messages per second > >fetch size 307200 -> 50915.3291133 messages per second > > > >$ pifpaf run kafka python kafka_test.py > >kafka-python version: 1.0.2 > >.... > >fetch size 179200 -> 8546.77931323 messages per second > >fetch size 204800 -> 9213.30958314 messages per second > >fetch size 230400 -> 10316.668006 messages per second > >fetch size 256000 -> 11476.2285269 messages per second > >fetch size 281600 -> 12353.7254386 messages per second > >fetch size 307200 -> 13131.2367288 messages per second > > > >(1.1.1 and 1.2.5 have also the same issue) > > > >$ pifpaf run kafka python kafka_test.py > >kafka-python version: 1.3.1 > >fetch size 179200 -> 44636.9371873 messages per second > >fetch size 204800 -> 44324.7085365 messages per second > >fetch size 230400 -> 45235.8283208 messages per second > >fetch size 256000 -> 45793.1044121 messages per second > >fetch size 281600 -> 44648.6357019 messages per second > >fetch size 307200 -> 44877.8445987 messages per second > >fetch size 332800 -> 47166.9176281 messages per second > >fetch size 358400 -> 47391.0057622 messages per second > > > >Looks like it works well now :) > > It¹s good that the performance problem has been fixed. The remaining > issues on the Monasca side are verifying that the batch send method we > were using in 0.9.5 still works with the new async behaviour, seeing if > our consumer auto balance still functions or converting to use the Kafka > internal auto balance in Kafka 0.10, and finding a way to do efficient > synchronous writes with the new async methods. Can you +1 https://review.openstack.org/404878 to make it clear we can proceed down this path. Yours Tony.
signature.asc
Description: PGP signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
