Hello all,
I had one other idea on how Zoom is handling this.
I watched a Zoom conference where there were up to 37 participants.
The participants were spread over 2 internal Zoom pages. The first page
was in a 5x5 array (25) and the second was also in a 5x5 array that
including the remaining 12 participants and 13 from the first
page.Perhaps they saw a client limitations at about 25 participants
with live video and audio and the paging limits the stream to 25
participants per page. Also maybe they auto adjust the resolution down
with more participants.
Perhaps OM would be similar. We could only prove that by a test with
the demo server with at least 25 devices and clustering for kms. This
may require a coordinated effort among us.
Dennis
On Tue, 2020-11-17 at 22:39 +0700, Maxim Solodovnik wrote:
> On Tue, 17 Nov 2020 at 20:55, dww <[email protected]> wrote:
> > I apologize for accidently sending this twice.
> > On Tue, 2020-11-17 at 08:40 -0500, dww wrote:
> > > Hello Denis,
> > >
> > > I was looking forward to your feedback soon.
> > >
> > > It seems there are mainly 2 major stress points.
> > >
> > > One, is the stream number limit per kms instance of about 200.
> > > That should allow about 10 or 11 users in a room with both video
> > > and microphone but this can be overcome by clustering starting
> > > with a docker swarm and the containers can be spread over
> > > multiple servers.
> > >
> > > Two, is the question of how many simultanious pods from users
> > > with both video and microphone can be accommodated on the client
> > > browser for the room. When I ran the bash script on a laptop I
> > > created 4 guests so that was 4 instances of Chrome each with 5
> > > pods on them. Things got unstable after that. So I surmise that
> > > 20-25 would be the limit there. I don't see how that limit could
> > > be passed without some design changes.
> > >
> > > Any ideas would be welcome.
> > >
> > > I think Maxim mentioned that Zoom draws the entire screen on
> > > their servers and streams that to the clients.
>
> FTR this was just my opinion, I haven't checked this (and not (yet)
> sure how this can be checked)
> > > Dennis
> > > On Fri, 2020-11-13 at 01:23 -0600, Denis Noctor wrote:
> > > > Hi there Maxim... I did a test with 8 computers and 2 tablets
> > > > last night (spread across 2 WiFis)... please don’t delete the
> > > > logs on the OM demo server (next)... I will come back to you
> > > > all with some feedback and pics later tomorrow (if that’s
> > > > okay)... however, for reference... I started the process in the
> > > > public room #7...start time around 8.22pm (12th Nov) (México...
> > > > 6 hrs behind) and end time 9.50pm... (if you want to check the
> > > > logs) .... the short version is that 8 users experienced
> > > > relatively stable performance. Will give you a more detailed
> > > > feedback once I deal with a personal issue. All the best,
> > > > Denis.
> > > >
> > > > Sent from my iPhone
> > > > On Nov 11, 2020, at 9:09 PM, Maxim Solodovnik <
> > > > [email protected]> wrote:
> > > >
> > > > > Hello All,
> > > > > I'll try to answer in one email :)
> > > > >
> > > > > On Wed, 11 Nov 2020 at 20:32, dww <[email protected]>
> > > > > wrote:
> > > > > > However, Denis, I think your experiment with multiple
> > > > > > devices would be
> > > > > >
> > > > > > valuable as then there is only one browser tab or window
> > > > > > with the OM
> > > > > >
> > > > > > room open as a guest on each device. Perhaps that will make
> > > > > > a
> > > > > >
> > > > > > difference.
> > > > >
> > > > > yes, this would be better test (even if "fake" camera is
> > > > > used)
> > > > > >
> > > > > > Dennis
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, 2020-11-11 at 08:24 -0500, dww wrote:
> > > > > >
> > > > > > > Thanks, Denis,
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > Back on Oct. 17 Maxim provided the following Bash script
> > > > > > to be run on
> > > > > >
> > > > > > > the machine with a client side browser for the psuedo
> > > > > > guest users. (
> > > > > >
> > > > > > > Use another machine to create the room administratively
> > > > > > and send
> > > > > >
> > > > > > > invitations) This is a far simpler way to stress test the
> > > > > > client side
> > > > > >
> > > > > > > browser.
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > Dennis
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > Hello,
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > i just have tried the following script
> > > > > >
> > > > > > > started as `./run10.sh 5`
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > everything seems to work, but my CPU was 800% busy (all
> > > > > > cores were
> > > > > >
> > > > > > > 100%
> > > > > >
> > > > > > > busy)
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > without `--use-fake-device-for-media-stream` parameter I
> > > > > > had lots of
> > > > > >
> > > > > > > permission errors due to camera was "captured" by first
> > > > > > browser
> > > > > >
> > > > > > > other have reported "Camera busy" error
> > > > > >
> > > > > > >
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > _HASH_HERE_ - should be replaced with real hash (I have
> > > > > > created
> > > > > >
> > > > > > > endless
> > > > > >
> > > > > > > invitation hash to the private conference room)
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > the script
> > > > > >
> > > > > > > ===============================================
> > > > > >
> > > > > > > #!/bin/bash
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > i=$1
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > if [ -z "${i}" ]; then
> > > > > >
> > > > > > > i=30
> > > > > >
> > > > > > > fi
> > > > > >
> > > > > > > let "i += 0"
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > rm -rf /tmp/delme*
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > while ((i--)); do
> > > > > >
> > > > > > > #echo "${i}"
> > > > > >
> > > > > > > mkdir /tmp/delme${i}
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > #local conference
> > > > > >
> > > > > > > chromium-browser --user-data-dir=/tmp/delme${i} --
> > > > > > disable-infobars
> > > > > >
> > > > > > > --no-default-browser-check --allow-insecure-localhost
> > > > > >
> > > > > > > --use-fake-device-for-media-stream '
> > > > > >
> > > > > > >
> > > > > > https://localhost:5443/openmeetings/hash?invitation=_HASH_HERE_&language=1'
> > > > > >
> > > > > > > &
> > > > > >
> > > > > > > done
> > > > > >
> > > > > > >
> > > > > >
> > > > > > >
> > > > > >
> > > > > > > On Wed, 2020-11-11 at 01:53 -0600, Denis Noctor wrote:
> > > > > >
> > > > > > > > Hi there everyone, this seems to be the “elephant in
> > > > > > the room”
> > > > > >
> > > > > > > > discussion, while there has been a HUGE amount of
> > > > > > development and
> > > > > >
> > > > > > > > progress in OM since March (thank you so much @Maxim)
> > > > > > ... there is
> > > > > >
> > > > > > > > the whole issue of, for example, the number of users
> > > > > > per room...
> > > > > >
> > > > > > > > which seems to be about 5-6 (and maybe even to 7) when
> > > > > > pushed to
> > > > > >
> > > > > > > > the
> > > > > >
> > > > > > > > limit... with both audio and video being broadcasted
> > > > > > from all
> > > > > >
> > > > > > > > users... and, something else.. if there are
> > > > > > simultaneous
> > > > > >
> > > > > > > > classes/sessions being held on the same server... will
> > > > > > this
> > > > > >
> > > > > > > > restrict
> > > > > >
> > > > > > > > things even further? Is this an overall limitation
> > > > >
> > > > > Sebastian did some AWS based testingAnd, if i'm not mistaken,
> > > > > the server with 4GB RAM was able to handle at least 3 rooms
> > > > > of 5 people(5.1.0-SNAPSHOT should behave better than 5.0.1)
> > > > > to increase the number of rooms you can use cluster
> > > > > > to using a
> > > > > >
> > > > > > > > browser
> > > > > >
> > > > > > > > based approach... or should we be taking approach?
> > > > >
> > > > > well,there is "The Limit"KMS can handle only certain amount
> > > > > of multimedia connectionsadditionally there are other
> > > > > limits:- bandwidth- CPU- RAM- open files (network socket is a
> > > > > file)
> > > > > "The Limit" is something I'm not sure how to deal with (yet)
> > > > > > > >
> > > > > >
> > > > > > > > It was my intention to test out the OM “demo servers”
> > > > > > over the last
> > > > > >
> > > > > > > > 2
> > > > > >
> > > > > > > > weeks but will take today off and try to test 10 real
> > > > > > device
> > > > > >
> > > > > > > > connections... with a combination of desktops, laptops,
> > > > > > android
> > > > > >
> > > > > > > > tablets and maybe even the odd iPhone or two.
> > > > >
> > > > > Apple devices has issues with sound (outgoing)I'm still
> > > > > investigating this one
> > > > > > > >
> > > > > >
> > > > > > > > My million dollar question is... prior to WebRTC and
> > > > > > Kurento... was
> > > > > >
> > > > > > > > it possible to have 5-10 users in a room with audio and
> > > > > > video
> > > > > >
> > > > > > > > working
> > > > > >
> > > > > > > > seamlessly in previous versions (for example, the old
> > > > > > “flash” setup
> > > > > >
> > > > > > > > (which will be redundant after Christmas... Chrome etc
> > > > > >
> > > > > > > > notifications)
> > > > > >
> > > > > > > > and if so, what has changed?
> > > > >
> > > > > Yes this was possibleOM_before_5 was based on Red5 media
> > > > > serverUnfortunately it's open source version has no WebRTC
> > > > > support
> > > > > > > >
> > > > > >
> > > > > > > > If there is anyone out there that has no problem with
> > > > > > user numbers
> > > > > >
> > > > > > > > (using audio and vid)... exceeding a body of 7-10+,
> > > > > > please let us
> > > > > >
> > > > > > > > know.
> > > > > >
> > > > > > > >
> > > > > >
> > > > > > > > In the meantime, I’ll give you my feedback on my tests.
> > > > > >
> > > > > > > >
> > > > > >
> > > > > > > > I really appreciate everything that has been done to
> > > > > > date.
> > > > > >
> > > > > > > >
> > > > > >
> > > > > > > > Thanks.
> > > > > >
> > > > > > > >
> > > > > >
> > > > > > > > Sent from my iPhone
> > > > > >
> > > > > > > >
> > > > > >
> > > > > > > > > On Nov 9, 2020, at 4:50 PM, dww <[email protected]
> > > > > > > wrote:
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > Hello Maxim,
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > A couple of weeks ago there was an email thread about
> > > > > > the 5 total
> > > > > >
> > > > > > > > > users
> > > > > >
> > > > > > > > > for one room, each user with video/microphone under
> > > > > > the
> > > > > >
> > > > > > > > > Subject: "docker container clustering experiments
> > > > > > #1".
> > > > >
> > > > > For whatever reason you love to start new mail threads :))))
> > > > > > In this
> > > > > >
> > > > > > > > > case
> > > > > >
> > > > > > > > > it
> > > > > >
> > > > > > > > > appears the bottleneck is the CPU usage on the client
> > > > > > machine
> > > > > >
> > > > > > > > > with
> > > > > >
> > > > > > > > > the
> > > > > >
> > > > > > > > > browser.
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > In a response to Denis Noctor on a similar thread you
> > > > > > mentioned
> > > > > >
> > > > > > > > > to
> > > > > >
> > > > > > > > > try
> > > > > >
> > > > > > > > > the following:
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > "please check allowed amount of opened files for the
> > > > > > user who
> > > > > >
> > > > > > > > > starts
> > > > > >
> > > > > > > > > OM/KMS/TURN
> > > > > >
> > > > > > > > > increasing it might help"
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > Might this help with the issue we discussed? Where
> > > > > > approximately
> > > > > >
> > > > > > > > > do
> > > > > >
> > > > > > > > > I
> > > > > >
> > > > > > > > > set the allowed amount of opened files?
> > > > >
> > > > > KMS seems to drop connections when there is not enough
> > > > > files(network socket is a file)you can check the limit for
> > > > > current user using `ulimit -n` (`ulimit -a` to see all
> > > > > limits)
> > > > > to check limit for `nobody` user `su nobody --shell /bin/bash
> > > > > --command "ulimit -n"`
> > > > > to increase the limit i'm changing
> > > > > `/etc/security/limits.conf` file
> > > > > https://github.com/openmeetings/openmeetings-docker/blob/48b72f4d0f38a0fab2021a0a2e4d6693c61c00be/scripts/om_euser.sh#L35
> > > > >
> > > > > (seems to work at Ubuntu)
> > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > > Also are there any other things that can be tried to
> > > > > > improve this
> > > > > >
> > > > > > > > > scalability? Are there areas in the code that can be
> > > > > > examined to
> > > > > >
> > > > > > > > > investigate how to improve this?
> > > > >
> > > > > KMS cluster would be ultimate solution, I guess
> > > > > > > > >
> > > > > >
> > > > > > > > > Thanks,
> > > > > >
> > > > > > > > > Dennis
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
>
>
