2019-04-16 22:22:42 UTC - Anthony Corletti: @Anthony Corletti has joined the channel ---- 2019-04-16 22:29:17 UTC - Anthony Corletti: hi there! I was hoping to see if anyone could help troubleshoot why I'm seeing really slow rate out with my cluster. rate in looks as expected but my rate out is about 10x slower :disappointed: shoot me a mention or dm! ---- 2019-04-16 22:56:06 UTC - Ali Ahmed: @Anthony Corletti what is your cluster setup like ? ---- 2019-04-16 23:03:45 UTC - Anthony Corletti: @Ali Ahmed right now running everything in gke – I can send over some .yml configs in a bit ... any particular params I should double check for slow consumers? ---- 2019-04-16 23:04:30 UTC - Ali Ahmed: hard to say are you using the websocket interface ? ---- 2019-04-17 02:48:27 UTC - Karthik Ramasamy: What is the instance that you are running ---- 2019-04-17 02:48:54 UTC - Karthik Ramasamy: What is your input rate vs output rate that you are seeing? ---- 2019-04-17 02:49:10 UTC - Karthik Ramasamy: We have heavily used gke as well raised_hands : Anthony Corletti ---- 2019-04-17 03:51:16 UTC - Anthony Corletti: @Karthik Ramasamy @Ali Ahmed thanks for reaching out. what do you mean by web-socket interface? putting together some details ---- 2019-04-17 03:52:05 UTC - Ali Ahmed: pulsar has both a binary interface and websocket one not sure what you are using ---- 2019-04-17 03:57:21 UTC - Anthony Corletti: ok – have a hunch im using the websocket interface but am not entirely sure. what would I check to know? ---- 2019-04-17 03:57:32 UTC - Anthony Corletti: here's basically the setup im messing with now ---- 2019-04-17 03:57:39 UTC - Anthony Corletti: ---- 2019-04-17 03:57:59 UTC - Ali Ahmed: what’s the url of the connection ? ---- 2019-04-17 04:00:06 UTC - Anthony Corletti: seeing an input rate of about 1.2 MB/s vs an output rate of 111 kb/s ---- 2019-04-17 04:00:33 UTC - Anthony Corletti: running this on `n1-standard-8`s ---- 2019-04-17 04:00:38 UTC - Anthony Corletti: dont have one right now – I could spin it up ---- 2019-04-17 04:00:50 UTC - Anthony Corletti: is there anything you could tell from those configs? ---- 2019-04-17 04:01:28 UTC - Ali Ahmed: what did you use to get these results ? ``` seeing an input rate of about 1.2 MB/s vs an output rate of 111 kb/s ``` ---- 2019-04-17 04:01:33 UTC - Ali Ahmed: pulsar-perf ? ---- 2019-04-17 04:02:44 UTC - Anthony Corletti: nope - used the python client to build a basic producer and consumer ---- 2019-04-17 04:03:29 UTC - Anthony Corletti: went to stress test the cluster by spinning up tons of requests to the producer – which held up great. but the consumer didn't scale along with the way the producer did ---- 2019-04-17 04:03:53 UTC - Ali Ahmed: I would first use the build in utility ``` pulsar-perf Usage: pulsar-perf <command> where command is one of: ``` ---- 2019-04-17 04:04:03 UTC - Ali Ahmed: to check the health of the cluster ---- 2019-04-17 04:04:22 UTC - Ali Ahmed: if things are fine they can isolate the issues in python client ---- 2019-04-17 04:05:10 UTC - Anthony Corletti: I ran the sample pulsar perf examples and everything behaved as expected ---- 2019-04-17 04:06:12 UTC - Anthony Corletti: but when I started to create more load, it seemed like something was getting slowed down between producer and consumer ---- 2019-04-17 04:06:17 UTC - Ali Ahmed: what’s the python version and pulsar-client version ? ---- 2019-04-17 04:06:58 UTC - Anthony Corletti: python: 3.7 pulsar-client: 2.3.0 ---- 2019-04-17 04:07:10 UTC - Ali Ahmed: can you try with pulsar-client: 2.3.1 ---- 2019-04-17 04:07:36 UTC - Anthony Corletti: sure! why do you think that will help? ---- 2019-04-17 04:07:53 UTC - Ali Ahmed: some changes were made ---- 2019-04-17 04:17:44 UTC - Samuel Sun: @Samuel Sun has joined the channel ---- 2019-04-17 04:37:10 UTC - Karthik Ramasamy: Ok that is good ---- 2019-04-17 04:38:02 UTC - Karthik Ramasamy: If you want you can try the pulsar from Streamlio on gke market place, it is free ---- 2019-04-17 04:38:36 UTC - Karthik Ramasamy: That is really odd ---- 2019-04-17 04:55:27 UTC - Anthony Corletti: @Ali Ahmed unfortunately seeing similar results after upgrading the python client to 2.3.1 – rate out isn't really keeping up with rate in. havent been submitting as much load now but was hoping for alteast a closer rate in/out ratio ---- 2019-04-17 04:56:19 UTC - Ali Ahmed: not sure @Matteo Merli may be able to point something out +1 : Anthony Corletti ---- 2019-04-17 04:58:19 UTC - Matteo Merli: Can you try consuming with `pulsar-perf consume` to get a baseline? +1 : Anthony Corletti ---- 2019-04-17 05:01:38 UTC - Joe Francis: and also enable clinet stats logging? +1 : Anthony Corletti ---- 2019-04-17 05:03:47 UTC - Matteo Merli: These should be enabled by default and they will be printing stats every 1min or so +1 : Anthony Corletti ---- 2019-04-17 05:23:06 UTC - Anthony Corletti: awesome – so as expected running a baseline pulsar-perf consume/ produce worked well ... must be something with how im configuring my consumer. trying with custom producer and a baseline consumer ---- 2019-04-17 06:05:23 UTC - Jacob O'Farrell: @Jacob O'Farrell has joined the channel ----
