Denis, we have OM and BigBlue Button in use. BBB is hosted on a provider machine.
OM we have on internal VL, with LDAP connection. Also we have a physical server with 12 Core 2Ghz and 32G since some days. If I will find some time, we will do an stress test. We have some employees in home offices… I was also surprised, that BBB shows much more users, but using the same technology. The key probably is, that BBB is for presenting by default, my understanding. With OM you have a conference with all, what end’s in a lots of streams. We are a production company, so conference is okay for us in 99% of cases. But our engineers do some online trainings sometimes, and the number of listeners can be more then 100. Maybe I did not got all the features and settings for OM. But I would think in a class room mode with many people there should be only the teacher presenting his video and audio. There ,( that’s my opinion) is no need that every listener can talk to all by default and show his video by default. Yes, it´s not like sitting in the same room… However, still my favourite is OM. I learned a lot about this technology in the last 8 months and OM has moved forward really. For internal use with LDAP it’s simple. This forum is really active and the installation documents are nearly perfect. I will let you know, when a stress test is done. Maybe we have in Germany only virtual Christmas parties…. So a good time for stress tests. Gerald. Von: Denis Noctor [mailto:denisnoc...@gmail.com] Gesendet: Montag, 23. November 2020 07:58 An: user@openmeetings.apache.org Betreff: Re: Users per room client browser scalability Hi there everyone, My sincerest apologies for only getting back to you now. As I had mentioned in my previous email that I was going through a personal issue and it took longer than I had anticipated to get back in touch. However, as mentioned before, I have been keeping up to date by reading all the emails in the forum over the last few weeks... and some direct emails also. I also apologize for the length of the email... so maybe you should grab a coffee. As mentioned in my previous email, I set up 10 devices to connect to Room 7 on the OM demo server - all of which where running the latest version of Chrome. 3 machines running Windows 10, 1 running Windows 8, 4 running Windows 7, 2 Amazon Fires (set up to run Chrome) with varying degrees of ram (2, 4, 8 and 12 gigs) The results are as follows: After logging in around 8.22pm (Mexico time), 12th November (OpenMeetings - Next, 5.1.0-SNAPSHOT, Revision: db7be4b, Build date: 2020-11-09T14:57:23Z , I gradually added other devices to the room. I got to 8. There was a little but of a time lapse... in the sense that I would move from one computer to another... and could still see myself in one feed after I had move to another. It is important to note that 2 of the computers (older HP's) have a slight webcam issue... (I think there is a fauly cable... sometimes works sometimes doesn't - but audio/mic was working fine). When I added 2 the last two devices, things started to break down. The audio quality was clearly reduced... there was a lot of crackling sounds... and some of the users video pods disappeared from some of the devices´screens.... or "empty" video pods filled some screens... on some devices.... but were viewable on others. Some users appeared to be disconnected, though they could continue to view the whiteboards... but had their audio and video disconnected (icons in orange)... when they tried to reconnect... they couldn't... they clicked on the audio / vid icons but with no effect... refreshing the screen sometimes seemed to correct this. While 8 users seemed to be able to connect okay... there was a little bit of a time delay. As you can understand, I don't have headphones and microphones for each and every computer... so I spaced them around my house... when I talked... I could hear my voice being repeated... (I am not referring to echo feedback).... there was a slight time delay by a couple of seconds on some of the devices... moving from device to device. However, with just 5 users in a room, this was not really an issue. From time to time users experienced other users being disconnected or whereby they could see the "empty video pod" with the green border flashing on and off as someone spoke.... but again no audio or video being received.... but it was possible to see the same users on other devices. Users would try to "refresh" the page... again only having access to 4 - 5 users on the page.... and not necessarily seeing the moderator. I finished testing around 9.50pm. Some additional observations: Based on some of the emails over the last few weeks. It appears to be that one OM instance can only deal with 3 simultaneous rooms with 5 users approx in each room (using audio and video)... and based on the above maybe a little more, but at a stretch. This appears to boil down to limitations due to number Kurento / WebRTC connections.... some of you have mention somewhere in the range of 200 - 300 connections. As a result I took a look at a few sites regarding BigBlueButton (BBB), as it also uses Kurento and WebRTC to get a general idea as to how many users can be in a room (with camera and audio). However, a lot of digging had to be done as many of the numbers that are used are about how many participants can be in a room (without cam and mic) with a moderator (using cam and mic). Now I apologize for bringing up BBB in conversation, as I am not endorsing the platform....reminding me of Harry Potter, (Voldemort) "He who shall not be named" :) However, it might be worth investigating for ideas on how to increase the number of cams / mics in an OM room. You can view this information here: https://support.blindsidenetworks.com/hc/en-us/articles/360042272991-How-many-users-can-I-have-in-a-BigBlueButton-session-at-one-time- Similarly, if you look at the following link it suggests that the more number of concurrent users... the number of rooms will be less (but again not taking into consideration cams and mics) : https://docs.bigbluebutton.org/support/faq.html#how-many-simultaneous-users-can-bigbluebutton-support However, regarding the limited number of users (with cam and mic), BBB seems to have got arround this by having a window of 5 cams that can be scrolled left or right. It appears moderators can still view up to 25 cams etc. Take a look at this: https://support.blindsidenetworks.com/hc/en-us/articles/360049799851-September-16-2020-Webcam-viewing-and-recording-updates _________________________ OM Demo Next Server Specs: @Maxim, is it possible to share the specs of the demo server, ram, cpu, etc, so that we can get a general idea as to what to benchmark against. At the moment I am using AWS, t3a.large (8 gigs Ram , 2 vCPUS). However, I am under the impression that even if I upgrade my AWS server... I am not really going to see any major improvements given the fact that I am still limited to the number of users (with cam and mic) per room. Before this whole covid situation, my school had 10 classrooms, class size average 10. I would like to be able to have a similar virtual set up... but based on all the info above I would need 3 - 4 instances (using clustering, which I have never done before) - but will still have a problem having 11 users (10 students / 1 moderator) in a room. And while I know the following question (as an alternative) might be considered ridiculous - can a multiple number of OM installations (with multiple KMS etc) be conducted in one server - which more ram, cpu power etc? It would be great if anyone out there has a successful clustering model that they could share - even to test across 2 instances. At present, I am using Ubuntu 18.04 on AWS as described above. Either way it seems the main obstacle at the moment seems to be how Kurento and WebRTC can be set up to overcome these limitations Apologies once again for the length of this email and for taking so long to get back in touch. (I've added a few screenshots regarding my test below (one computer's time is 2 hours behind for some strange reason) :)) All the best, Denis. [https://ipmcdn.avast.com/images/icons/icon-envelope-tick-green-avg-v1.png]<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free. www.avg.com<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> On Sat, Nov 14, 2020 at 12:26 AM Maxim Solodovnik <solomax...@gmail.com<mailto:solomax...@gmail.com>> wrote: On Fri, 13 Nov 2020 at 14:24, Denis Noctor <denisnoc...@gmail.com<mailto:denisnoc...@gmail.com>> wrote: Hi there Maxim... I did a test with 8 computers and 2 tablets last night (spread across 2 WiFis)... please don’t delete the logs on the OM demo server (next)... I just backup the logs I will come back to you all with some feedback and pics later tomorrow (if that’s okay)... however, for reference... I started the process in the public room #7...start time around 8.22pm (12th Nov) (México... 6 hrs behind) and end time 9.50pm... (if you want to check the logs) .... the short version is that 8 users experienced relatively stable performance. Looking forward to hear the full version :)) Will give you a more detailed feedback once I deal with a personal issue. All the best, Denis. Sent from my iPhone On Nov 11, 2020, at 9:09 PM, Maxim Solodovnik <solomax...@gmail.com<mailto:solomax...@gmail.com>> wrote: Hello All, I'll try to answer in one email :) On Wed, 11 Nov 2020 at 20:32, dww <dwort...@mykolab.com<mailto:dwort...@mykolab.com>> wrote: However, Denis, I think your experiment with multiple devices would be valuable as then there is only one browser tab or window with the OM room open as a guest on each device. Perhaps that will make a difference. yes, this would be better test (even if "fake" camera is used) Dennis On Wed, 2020-11-11 at 08:24 -0500, dww wrote: > Thanks, Denis, > > Back on Oct. 17 Maxim provided the following Bash script to be run on > the machine with a client side browser for the psuedo guest users. ( > Use another machine to create the room administratively and send > invitations) This is a far simpler way to stress test the client side > browser. > > Dennis > > Hello, > > i just have tried the following script > started as `./run10.sh 5` > > everything seems to work, but my CPU was 800% busy (all cores were > 100% > busy) > > without `--use-fake-device-for-media-stream` parameter I had lots of > permission errors due to camera was "captured" by first browser > other have reported "Camera busy" error > > > _HASH_HERE_ - should be replaced with real hash (I have created > endless > invitation hash to the private conference room) > > the script > =============================================== > #!/bin/bash > > i=$1 > > if [ -z "${i}" ]; then > i=30 > fi > let "i += 0" > > rm -rf /tmp/delme* > > while ((i--)); do > #echo "${i}" > mkdir /tmp/delme${i} > > #local conference > chromium-browser --user-data-dir=/tmp/delme${i} --disable-infobars > --no-default-browser-check --allow-insecure-localhost > --use-fake-device-for-media-stream ' > https://localhost:5443/openmeetings/hash?invitation=_HASH_HERE_&language=1' > & > done > > > On Wed, 2020-11-11 at 01:53 -0600, Denis Noctor wrote: > > Hi there everyone, this seems to be the “elephant in the room” > > discussion, while there has been a HUGE amount of development and > > progress in OM since March (thank you so much @Maxim) ... there is > > the whole issue of, for example, the number of users per room... > > which seems to be about 5-6 (and maybe even to 7) when pushed to > > the > > limit... with both audio and video being broadcasted from all > > users... and, something else.. if there are simultaneous > > classes/sessions being held on the same server... will this > > restrict > > things even further? Is this an overall limitation Sebastian did some AWS based testing And, if i'm not mistaken, the server with 4GB RAM was able to handle at least 3 rooms of 5 people (5.1.0-SNAPSHOT should behave better than 5.0.1) to increase the number of rooms you can use cluster to using a > > browser > > based approach... or should we be taking approach? well, there is "The Limit" KMS can handle only certain amount of multimedia connections additionally there are other limits: - bandwidth - CPU - RAM - open files (network socket is a file) "The Limit" is something I'm not sure how to deal with (yet) > > > > It was my intention to test out the OM “demo servers” over the last > > 2 > > weeks but will take today off and try to test 10 real device > > connections... with a combination of desktops, laptops, android > > tablets and maybe even the odd iPhone or two. Apple devices has issues with sound (outgoing) I'm still investigating this one > > > > My million dollar question is... prior to WebRTC and Kurento... was > > it possible to have 5-10 users in a room with audio and video > > working > > seamlessly in previous versions (for example, the old “flash” setup > > (which will be redundant after Christmas... Chrome etc > > notifications) > > and if so, what has changed? Yes this was possible OM_before_5 was based on Red5 media server Unfortunately it's open source version has no WebRTC support > > > > If there is anyone out there that has no problem with user numbers > > (using audio and vid)... exceeding a body of 7-10+, please let us > > know. > > > > In the meantime, I’ll give you my feedback on my tests. > > > > I really appreciate everything that has been done to date. > > > > Thanks. > > > > Sent from my iPhone > > > > > On Nov 9, 2020, at 4:50 PM, dww > > > <dwort...@mykolab.com<mailto:dwort...@mykolab.com>> wrote: > > > > > > Hello Maxim, > > > > > > A couple of weeks ago there was an email thread about the 5 total > > > users > > > for one room, each user with video/microphone under the > > > Subject: "docker container clustering experiments #1". For whatever reason you love to start new mail threads :)))) In this > > > case > > > it > > > appears the bottleneck is the CPU usage on the client machine > > > with > > > the > > > browser. > > > > > > In a response to Denis Noctor on a similar thread you mentioned > > > to > > > try > > > the following: > > > > > > "please check allowed amount of opened files for the user who > > > starts > > > OM/KMS/TURN > > > increasing it might help" > > > > > > Might this help with the issue we discussed? Where approximately > > > do > > > I > > > set the allowed amount of opened files? KMS seems to drop connections when there is not enough files (network socket is a file) you can check the limit for current user using `ulimit -n` (`ulimit -a` to see all limits) to check limit for `nobody` user `su nobody --shell /bin/bash --command "ulimit -n"` to increase the limit i'm changing `/etc/security/limits.conf` file https://github.com/openmeetings/openmeetings-docker/blob/48b72f4d0f38a0fab2021a0a2e4d6693c61c00be/scripts/om_euser.sh#L35 (seems to work at Ubuntu) > > > > > > Also are there any other things that can be tried to improve this > > > scalability? Are there areas in the code that can be examined to > > > investigate how to improve this? KMS cluster would be ultimate solution, I guess > > > > > > Thanks, > > > Dennis > > > > > > -- Best regards, Maxim -- Best regards, Maxim [https://ipmcdn.avast.com/images/icons/icon-envelope-tick-green-avg-v1.png]<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free. www.avg.com<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
