2016-01-13 18:30 GMT+03:00 Grant Saicom <grant.sai...@gmail.com>: > I might have the wrong end of the stick here, but I was under the > impression that kannel/bearerbox used openSMPP libraries to implement its > smpp functionality. I have done a bit of googling and cannot see anything > confirming this. So I guess this is not the case :) >
No. Kannel is basically is a set of bearerbox, smsbox, wapbox. sqlbox is a box too as well. opensmppbox is kinda -box too. each *box is connected to bearerbox or sometimes sqlbox connected to smsbox for further processing. each *box got its own functionality, but main thing is bearerbox which routes all messages / manages your smsc uplinks and processing own internal queues. so if you want to just send messages through kannel you need only: 1) bearerbox 2) sqlbox you just insert into send_sms your MT message with certain values in fields and it's submitted to bearerbox, which sends it to upstream SMSC you specified (smsc_id field). this is basically how it works. > With respect to mtr (both gateways are same device, juniper mx-140 with > dual 10Gbit direct connection to blade hypervisor data fabric): > Host Loss% Snt Last Avg > Best Wrst StDev > 1. smsc-gateway 0.0% 95 0.4 0.5 0.3 > 0.9 0.0 > 2. kannel 0.0% 94 0.8 0.6 > 0.4 1.0 0.0 > > Host Loss% Snt Last Avg > Best Wrst StDev > 1. kannel-gateway 0.0% 73 0.4 0.3 0.3 > 0.5 0.0 > 2. smsc 0.0% 72 0.5 0.6 > 0.4 2.7 0.4 > Better run mtr for at least 1000 packets, but I guess since both hosts are on the same network there shouldn't be any problem. > > Average on both is sub 1ms. MTU are is 1500 on both. > What's the software on SMSC server? Or It's some commercial solution? It could be this is the problem. > > I am loathe to think it is the hypervisor or hardware as it is all service > provider grade. I have a spare HP DL380 in the rack at the DC which was > scheduled for collection. I may just wipe that and put kannel there and see > how it behaves there. > Waiting for your results. > > Thank you for the advice. > > On 13 Jan 2016, at 15:09, spameden <spame...@gmail.com> wrote: > > > > 2016-01-13 15:14 GMT+03:00 Grant Saicom <grant.sai...@gmail.com>: > >> Thanks for the advice. >> >> Which smpp implementation does kannel code ship with from the repo? Just >> want to clarify as it sounds it isn’t opensmpp in the answer. >> > > Why do you even ask about opensmppbox? I thought original question was > about sqlbox. OpenSMPPBox is basically a simple proxy to kannel for outside > clients. There is not much in it, there is no flow control or accounting. > What bearerbox/kannel implements for smsc is basically what opensmppbox > offers in terms of connections to upstream links. > > > >> With regards to network, all the machines are on the same subnet, >> including the smsc. We have an instance within our network. On top of that, >> they are all on the same hypervisor and data fabric. >> > > Another thought I just got: could your network problems be due your > virtualization used? Try using kannel on a bare machine and see if there is > any TCP retransmissions going? > > Did you check mtr output as well to your upstream smsc? And also check > reverse network path if it's same or not. > > > > >> >> On 13 Jan 2016, at 14:02, spameden <spame...@gmail.com> wrote: >> >> opensmppbox is generally not recommended for production, it's very basic >> and there is no accounting at all, for SMPP-server I'd recommended >> contacting Stipe Tolk (you can find his e-mail on the lists archive in >> google), he has commercial solution carrier-grade. >> >> about TCP retransmissions you might need tuning either Linux network >> settings, e.g. MTU if you're behind some sorta NAT or better contact your >> provider with mtr output and tcpdump captures there might be unoptimal >> direct or/and reverse network path between you and your SMSC provider. >> >> 2016-01-13 14:07 GMT+03:00 Grant Saicom <grant.sai...@gmail.com>: >> >>> I suspect the issue I am still experiencing is because of TCP >>> retransmission between Kannel and SMSC. The time out for ack is 210ms on >>> the SMSC we are sending to and the delay can sometimes be as long as 370ms. >>> >>> I have not found a solution, but am exploring further optimisations and >>> avenues. >>> >>> Another Kannel user advised me saying that openSMPP can be problematic >>> some times. Is this a generally known and confirmed issue? >>> >>> Kind regards >>> Grant >>> >>> On 15 Dec 2015, at 12:53, spameden <spame...@gmail.com> wrote: >>> >>> store-type = spool >>> store-location = "/tmp/kannel-spool/" >>> >>> put these 2 lines below group = core in /etc/kannel/kannel.conf >>> >>> do you use for dlr MySQL as well? >>> >>> worth adding index on dlr table as well: KEY `smsc_ts` (`smsc`,`ts`) >>> >>> ALTER table dlr add index `smsc_ts` (`smsc`,`ts`); >>> >>> >>> >>> >>> 2015-12-15 13:43 GMT+03:00 spameden <spame...@gmail.com>: >>> >>>> Yes. >>>> >>>> The only thing that comes to my mind is to use kannel-store = spool and >>>> move your spool store to /tmp dir, this way queue will be in multiple files >>>> instead of the single file and in RAM, I found this way it's very fast. >>>> >>>> Let me know if it helps. >>>> >>>> If you want to preserve queue (in case of hard reset or something) you >>>> can rsync it's contents every 2 minutes or so in some permanent directory. >>>> >>>> 2015-12-15 12:56 GMT+03:00 Grant Saicom <grant.sai...@gmail.com>: >>>> >>>>> Haven’t really found my answer yet, but I have another question along >>>>> this line of thought. >>>>> >>>>> I see the queue in sqlbox rises quite high, but the queue in the smpp >>>>> connector to the smsc barely goes above 0. >>>>> >>>>> Is there a way to control.modify the flow of messages from sqlbox to >>>>> the smpp queue? >>>>> >>>>> Architecturally, is this correct in terms of the flow of messages: >>>>> sqlbox queue -> bearerbox sms store-> smpp queue >>>>> >>>>> Regards >>>>> Grant >>>>> >>>>> On 09 Dec 2015, at 11:56, spameden <spame...@gmail.com> wrote: >>>>> >>>>> >>>>> >>>>> 2015-12-09 12:43 GMT+03:00 Grant Saicom <grant.sai...@gmail.com>: >>>>> >>>>>> We process the sent_sms into another table on they fly. Maximum size >>>>>> the sent_sms table gets is maybe 40k tops but mostly it averages around >>>>>> 10K. We see this once 1 week maybe. >>>>>> >>>>>> I have really made every attempt to remove any bottlenecks in terms >>>>>> unwieldy database sizes to allow kannel to work in a favourable >>>>>> environment. >>>>>> >>>>>> Is there reason to add multiple sqlboxes to feed bearer box? >>>>>> >>>>>> Is there maybe a concurrency setting I can do for bearer box to >>>>>> receive the messages? I did not come across documentation aside from >>>>>> email >>>>>> posts with respect to the limit-per-cycle setting. >>>>>> >>>>>> I have another question, would we be able to get faster performance >>>>>> if we went flat file for the kannel operations? >>>>>> >>>>> >>>>> Well you can exclude bottlenecks by simply testing same setup with >>>>> fakesmsc daemon and see if speed will be better. >>>>> >>>>> It might be that delay is caused by your SMSC uplinks overall speed >>>>> and not database. >>>>> >>>>> You can also try classic smsbox implementation for sending instead of >>>>> sqlbox. But I think sqlbox is fastest and more convinient way because of >>>>> DB >>>>> storage. >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>> Regards >>>>>> >>>>>> >>>>>> On 08 Dec 2015, at 15:12, spameden <spame...@gmail.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> 2015-12-08 12:51 GMT+03:00 Grant Vorenberg <gr...@saicomvoice.co.za>: >>>>>> >>>>>>> <saicom-voice-services.gif> >>>>>>> <https://branding.synaq.com/api/r/id/22474050/map/0> >>>>>>> >>>>>>> Hi >>>>>>> >>>>>>> We manage how big send_sms gets. The queue builder puts in 500 >>>>>>> messages at a time to a total maximum of 3000 from a larger main queue >>>>>>> which can go as big as 2M. >>>>>>> >>>>>> >>>>>> 2M is kinda big table, how big is sent_sms? 10-30M ? >>>>>> >>>>>> I think your issue happens when kannel tries to move from send_sms to >>>>>> sent_sms table already submitted message this is where it hangs. You can >>>>>> try testing it yourself with simple query: >>>>>> >>>>>> INSERT INTO sent_sms SELECT * from send_sms where sql_id=XXXX and >>>>>> measure time per query. >>>>>> >>>>>> if it's instant there should be no problem. >>>>>> >>>>>> Generally it's better to leave sent_sms table at around 1M records >>>>>> not more, old records you can move to other table daily. >>>>>> >>>>>> >>>>>>> >>>>>>> Actual hardware is a VCenter on blades with plenty ram, cpu and hp >>>>>>> 3PAR(144GB raid card ram for caching in total) fibre attached storage >>>>>>> with >>>>>>> dedicated SSD specifically for DB. Calculated IOPS are stupidly good. >>>>>>> >>>>>>> The VMs are as follows: >>>>>>> Queuebuilder: 4 vcpu, 16Gb on SAS >>>>>>> Kannel: 4 vcpu, 8GB on SAS >>>>>>> MysqlDB-Master: 8 vcpu, 32GB on SSD >>>>>>> MysqlDB-Slave: 8 vcpu, 32GB on SSD >>>>>>> >>>>>> >>>>>> MySQL on SSDs should work just fine and you should have big number of >>>>>> iops. Btw, I recommend to use MariaDB instead of regular MySQL ( >>>>>> mariadb.org) it's very fast and reliable, for InnoDB it uses >>>>>> modified XtraDB engine which has some tweaks. >>>>>> >>>>>> did you check mysqladmin status to indicate number of queries >>>>>> processed per second? >>>>>> >>>>>> >>>>>>> The Load on the MysqlDB-Master averages around 0.4 with max of 0.6 >>>>>>> (single spike). Memory usage hangs around 24GB. I will need to check the >>>>>>> process list to double check, but we generally don’t see much strain >>>>>>> here >>>>>>> either, but I stand to be corrected. >>>>>>> >>>>>> DB optimasations as follows: >>>>>>> >>>>>>> key_buffer = 16M >>>>>>> max_allowed_packet = 16M >>>>>>> >>>>>> >>>>>> maybe increase a bit max_allowed_packet to 64mb >>>>>> >>>>>> innodb_buffer_pool_size = 12G >>>>>>> >>>>>> >>>>>> this is fine >>>>>> >>>>>> query_cache_limit = 20M >>>>>>> query_cache_size = 128M >>>>>>> >>>>>> >>>>>> query_cache in kannel's case doesn't affect much, so it's ok to have >>>>>> it like that. >>>>>> >>>>>> >>>>>>> >>>>>>> No extra indexes on the send_sms as we limit its size to 3000. >>>>>>> >>>>>> >>>>>> make sure both send_sms and sent_sms tables are InnoDB type. >>>>>> >>>>>> >>>>>>> >>>>>>> All reporting is done on the slaveDB, so no extra strain on >>>>>>> monitoring and reporting. >>>>>>> >>>>>> >>>>>> under 'reporting' do you mean your dlr_url is called to external >>>>>> script which is connected to slaveDB? >>>>>> >>>>>> >>>>>>> >>>>>>> Historically, we have queued at the SMPP connector and not at the >>>>>>> smsbox. We generally reached a top (AVG) speed of 73 msg/s but when I >>>>>>> see >>>>>>> the smsbox queued figure rise and the internal smpp queue drop to 0, we >>>>>>> only hit half of this speed. >>>>>>> >>>>>>> I did not see the limit-per-cycle setting in the sqlbox >>>>>>> documentation (2011). I also checked the code and saw that the select >>>>>>> limit >>>>>>> was a variable instead of hard-limited. >>>>>>> >>>>>> >>>>>> Yeah, it was introduced some time ago alongside with other >>>>>> optimisations (year ago or so, can't remember now), I think it should be >>>>>> in >>>>>> new documentation. However, you need to build documentation to get more >>>>>> recent version of it. >>>>>> >>>>>> >>>>>>> Regards >>>>>>> G >>>>>>> >>>>>>> On 08 Dec 2015, at 10:52, spameden <spame...@gmail.com> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Grant Vorenberg >>>>>>> >>>>>>> Technical Manager >>>>>>> Office 0861 SAICOM (724 266) Direct 010 140 5012 Support 010 140 >>>>>>> 5050 Cell - Fax 010 140 5001 Email gr...@saicomvoice.co.za Visit >>>>>>> our website www.saicomvoice.co.za >>>>>>> >>>>>>> <logo-image.png> <http://www.saicomvoice.co.za/> >>>>>>> >>>>>>> 2015-12-08 11:23 GMT+03:00 Grant Vorenberg <gr...@saicomvoice.co.za> >>>>>>> : >>>>>>> >>>>>>>> <saicom-now-offers-cloud-pbx.gif> >>>>>>>> <https://branding.synaq.com/api/r/id/22469522/map/0> >>>>>>>> >>>>>>>> Here is my config: >>>>>>>> >>>>>>>> group = sqlbox >>>>>>>> id = sqlbox-db >>>>>>>> smsbox-id = sqlbox >>>>>>>> #global-sender = "" >>>>>>>> bearerbox-host = localhost >>>>>>>> bearerbox-port = 13001 >>>>>>>> smsbox-port = 13005 >>>>>>>> smsbox-port-ssl = false >>>>>>>> sql-log-table = sent_sms >>>>>>>> sql-insert-table = send_sms >>>>>>>> log-file = "/var/log/kannel/kannel-sqlbox.log" >>>>>>>> log-level = 0 >>>>>>>> #sqlbox optimisation GAV 20151207 >>>>>>>> limit-per-cycle = 100 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> Try monitoring your database workload. >>>>>>> >>>>>>> Is send_sms table is big? >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On 07 Dec 2015, at 15:06, spameden <spame...@gmail.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Grant Vorenberg >>>>>>>> >>>>>>>> Technical Manager >>>>>>>> Office 0861 SAICOM (724 266) Direct 010 140 5012 Support 010 >>>>>>>> 140 5050 Cell - Fax 010 140 5001 Email >>>>>>>> gr...@saicomvoice.co.za Visit our website www.saicomvoice.co.za >>>>>>>> >>>>>>>> <logo-image.png> <http://www.saicomvoice.co.za/> >>>>>>>> >>>>>>>> 2015-12-07 14:03 GMT+03:00 Grant Vorenberg <gr...@saicomvoice.co.za >>>>>>>> >: >>>>>>>> >>>>>>>>> <saicom-now-offers-cloud-pbx.gif> >>>>>>>>> <https://branding.synaq.com/api/r/id/22451334/map/0> >>>>>>>>> >>>>>>>>> Hi Guys >>>>>>>>> >>>>>>>>> I am a new to the list subscription and am looking for a little >>>>>>>>> clarity on the SQLBOX. >>>>>>>>> >>>>>>>>> I have a Debian Wheezy box running: >>>>>>>>> Kannel sqlbox version 1.4.4 >>>>>>>>> Libxml version 2.7.2 >>>>>>>>> MySQL 5.5.43 >>>>>>>>> >>>>>>>>> The front end is custom and drops message into the send_sms table. >>>>>>>>> These messages are terminated via smpp to another system of ours. We >>>>>>>>> process and clean out the sent_sms table. >>>>>>>>> >>>>>>>>> I gather stats on the performance of the system using the status >>>>>>>>> page: http://localhost:13000/status >>>>>>>>> >>>>>>>>> I am trying to understand the following output from the screen: >>>>>>>>> Box connections: >>>>>>>>> smsbox:sqlbox, IP 127.0.0.1 (2532 queued), (on-line 0d 2h >>>>>>>>> 48m 19s) >>>>>>>>> >>>>>>>> >>>>>>>> This means there is a queue on sqlbox. >>>>>>>> >>>>>>>> Queue works like this: >>>>>>>> >>>>>>>> First sqlbox gets messages from DB backend, then it adds messages >>>>>>>> to own queue, then it sends message into bearerbox queue and bearerbox >>>>>>>> splits this queue over your connected SMSC/upstream operators. >>>>>>>> >>>>>>>> So if there is a huge queue on sqlbox it means there is a big >>>>>>>> amount of MTs in your send_sms table and sqlbox is still submitting >>>>>>>> those >>>>>>>> to bearerbox. >>>>>>>> >>>>>>>> To optimize sqlbox I'd recommend adding this parameter into sqlbox >>>>>>>> section (after group = sqlbox): >>>>>>>> limit-per-cycle = 50 this means sqlbox will get 50 bulk messages >>>>>>>> from DB per query at once instead of 1. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> What I noticed is when our send speeds start dipping on the smpp >>>>>>>>> connection (internal/default route), this smsbox:sql queue starts >>>>>>>>> building >>>>>>>>> up. >>>>>>>>> >>>>>>>>> When the smsbox:sqlbox queue starts building up like this: >>>>>>>>> 1)What causes this? >>>>>>>>> 2) What does this signify? >>>>>>>>> >>>>>>>>> We generally don’t see this behaviour very often, but its effect >>>>>>>>> is detrimental to the performance of the system so I would like to >>>>>>>>> know >>>>>>>>> what causes the growth and how to combat it so our send speed is >>>>>>>>> safe-guarded. >>>>>>>>> >>>>>>>>> >>>>>>>>> Here is an excerpt from my config files: >>>>>>>>> >>>>>>>>> <smsbox conf>: >>>>>>>>> group = sqlbox >>>>>>>>> id = sqlbox-db >>>>>>>>> smsbox-id = sqlbox >>>>>>>>> #global-sender = "" >>>>>>>>> bearerbox-host = localhost >>>>>>>>> bearerbox-port = 13001 >>>>>>>>> smsbox-port = 13005 >>>>>>>>> smsbox-port-ssl = false >>>>>>>>> sql-log-table = sent_sms >>>>>>>>> sql-insert-table = send_sms >>>>>>>>> >>>>>>>>> >>>>>>>>> <kannel.conf>: >>>>>>>>> group = smsbox >>>>>>>>> bearerbox-host = 127.0.0.1 >>>>>>>>> sendsms-port = 13013 >>>>>>>>> global-sender = 13013 >>>>>>>>> smsbox-id=my_smsbox >>>>>>>>> >>>>>>>>> group=smsbox-route >>>>>>>>> smsbox-id=sqlbox >>>>>>>>> smsc-id=internal >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Grant Vorenberg >>>>>>>>> >>>>>>>>> Technical Manager >>>>>>>>> Office 0861 SAICOM (724 266) Direct 010 140 5012 Support 010 >>>>>>>>> 140 5050 Cell - Fax 010 140 5001 Email >>>>>>>>> gr...@saicomvoice.co.za Visit our website www.saicomvoice.co.za >>>>>>>>> >>>>>>>>> <logo-image.png> <http://www.saicomvoice.co.za/> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>> >>> >> >> > >
