Hello Everyone I have the answer to the speed issues I have been encountering for the last 2 months.
The issue was access speeds on the dlr table which is used as temporary storage. This was corrected by adding an index on the smsc which dramatically increased speeds. Whats good to know for a kannel with sqlbox is that it is not necessary to use a SQL table for dlr storage (setting dlr-storage in kannel.conf). It is better to use a spool file based inside an ext2 partition for good speed. To identify this issue (and possibly a lot of others), you need to watch the actual network traffic using something like tshark. A healthy smpp communication (excluding other DLRs) will have the following order (assuming max-pending-submit=10) kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm SMSC -> kannel: submit_sm : Resp OK kannel -> SMSC: submit_sm SMSC -> kannel: submit_sm : Resp OK kannel -> SMSC: submit_sm SMSC -> kannel: submit_sm : Resp OK kannel -> SMSC: submit_sm SMSC -> kannel: submit_sm : Resp OK kannel -> SMSC: submit_sm SMSC -> kannel: submit_sm : Resp OK You will see that kannel sends its first 10, then waits for the acknowledgements before sending more. kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm SMSC -> kannel: submit_sm : Resp OK kannel -> SMSC: submit_sm SMSC -> kannel: submit_sm : Resp OK SMSC -> kannel: submit_sm : Resp OK SMSC -> kannel: submit_sm : Resp OK SMSC -> kannel: submit_sm : Resp OK SMSC -> kannel: submit_sm : Resp OK SMSC -> kannel: submit_sm : Resp OK kannel -> SMSC: submit_sm kannel -> SMSC: submit_sm SMSC -> kannel: submit_sm : Resp OK SMSC -> kannel: submit_sm : Resp OK SMSC -> kannel: submit_sm : Resp OK SMSC -> kannel: submit_sm : Resp OK This out of sequence behaviour indicated that there was a delay occurring somewhere and in our case, it was the updates to the dlr storage. I really hope this helps other people out there. Regards Grant > On 13 Jan 2016, at 17:30, Grant Saicom <grant.sai...@gmail.com> wrote: > > I might have the wrong end of the stick here, but I was under the impression > that kannel/bearerbox used openSMPP libraries to implement its smpp > functionality. I have done a bit of googling and cannot see anything > confirming this. So I guess this is not the case :) > > With respect to mtr (both gateways are same device, juniper mx-140 with dual > 10Gbit direct connection to blade hypervisor data fabric): > Host Loss% Snt Last Avg > Best Wrst StDev > 1. smsc-gateway 0.0% 95 0.4 0.5 0.3 0.9 > 0.0 > 2. kannel 0.0% 94 0.8 0.6 0.4 > 1.0 0.0 > > Host Loss% Snt Last Avg > Best Wrst StDev > 1. kannel-gateway 0.0% 73 0.4 0.3 0.3 0.5 > 0.0 > 2. smsc 0.0% 72 0.5 0.6 0.4 > 2.7 0.4 > > Average on both is sub 1ms. MTU are is 1500 on both. > > I am loathe to think it is the hypervisor or hardware as it is all service > provider grade. I have a spare HP DL380 in the rack at the DC which was > scheduled for collection. I may just wipe that and put kannel there and see > how it behaves there. > > Thank you for the advice. >> On 13 Jan 2016, at 15:09, spameden <spame...@gmail.com >> <mailto:spame...@gmail.com>> wrote: >> >> >> >> 2016-01-13 15:14 GMT+03:00 Grant Saicom <grant.sai...@gmail.com >> <mailto:grant.sai...@gmail.com>>: >> Thanks for the advice. >> >> Which smpp implementation does kannel code ship with from the repo? Just >> want to clarify as it sounds it isn’t opensmpp in the answer. >> >> Why do you even ask about opensmppbox? I thought original question was about >> sqlbox. OpenSMPPBox is basically a simple proxy to kannel for outside >> clients. There is not much in it, there is no flow control or accounting. >> What bearerbox/kannel implements for smsc is basically what opensmppbox >> offers in terms of connections to upstream links. >> >> >> >> With regards to network, all the machines are on the same subnet, including >> the smsc. We have an instance within our network. On top of that, they are >> all on the same hypervisor and data fabric. >> >> Another thought I just got: could your network problems be due your >> virtualization used? Try using kannel on a bare machine and see if there is >> any TCP retransmissions going? >> >> Did you check mtr output as well to your upstream smsc? And also check >> reverse network path if it's same or not. >> >> >> >> >>> On 13 Jan 2016, at 14:02, spameden <spame...@gmail.com >>> <mailto:spame...@gmail.com>> wrote: >>> >>> opensmppbox is generally not recommended for production, it's very basic >>> and there is no accounting at all, for SMPP-server I'd recommended >>> contacting Stipe Tolk (you can find his e-mail on the lists archive in >>> google), he has commercial solution carrier-grade. >>> >>> about TCP retransmissions you might need tuning either Linux network >>> settings, e.g. MTU if you're behind some sorta NAT or better contact your >>> provider with mtr output and tcpdump captures there might be unoptimal >>> direct or/and reverse network path between you and your SMSC provider. >>> >>> 2016-01-13 14:07 GMT+03:00 Grant Saicom <grant.sai...@gmail.com >>> <mailto:grant.sai...@gmail.com>>: >>> I suspect the issue I am still experiencing is because of TCP >>> retransmission between Kannel and SMSC. The time out for ack is 210ms on >>> the SMSC we are sending to and the delay can sometimes be as long as 370ms. >>> >>> I have not found a solution, but am exploring further optimisations and >>> avenues. >>> >>> Another Kannel user advised me saying that openSMPP can be problematic some >>> times. Is this a generally known and confirmed issue? >>> >>> Kind regards >>> Grant >>> >>>> On 15 Dec 2015, at 12:53, spameden <spame...@gmail.com >>>> <mailto:spame...@gmail.com>> wrote: >>>> >>>> store-type = spool >>>> store-location = "/tmp/kannel-spool/" >>>> >>>> put these 2 lines below group = core in /etc/kannel/kannel.conf >>>> >>>> do you use for dlr MySQL as well? >>>> >>>> worth adding index on dlr table as well: KEY `smsc_ts` (`smsc`,`ts`) >>>> >>>> ALTER table dlr add index `smsc_ts` (`smsc`,`ts`); >>>> >>>> >>>> >>>> >>>> 2015-12-15 13:43 GMT+03:00 spameden <spame...@gmail.com >>>> <mailto:spame...@gmail.com>>: >>>> Yes. >>>> >>>> The only thing that comes to my mind is to use kannel-store = spool and >>>> move your spool store to /tmp dir, this way queue will be in multiple >>>> files instead of the single file and in RAM, I found this way it's very >>>> fast. >>>> >>>> Let me know if it helps. >>>> >>>> If you want to preserve queue (in case of hard reset or something) you can >>>> rsync it's contents every 2 minutes or so in some permanent directory. >>>> >>>> 2015-12-15 12:56 GMT+03:00 Grant Saicom <grant.sai...@gmail.com >>>> <mailto:grant.sai...@gmail.com>>: >>>> Haven’t really found my answer yet, but I have another question along this >>>> line of thought. >>>> >>>> I see the queue in sqlbox rises quite high, but the queue in the smpp >>>> connector to the smsc barely goes above 0. >>>> >>>> Is there a way to control.modify the flow of messages from sqlbox to the >>>> smpp queue? >>>> >>>> Architecturally, is this correct in terms of the flow of messages: >>>> sqlbox queue -> bearerbox sms store-> smpp queue >>>> >>>> Regards >>>> Grant >>>> >>>>> On 09 Dec 2015, at 11:56, spameden <spame...@gmail.com >>>>> <mailto:spame...@gmail.com>> wrote: >>>>> >>>>> >>>>> >>>>> 2015-12-09 12:43 GMT+03:00 Grant Saicom <grant.sai...@gmail.com >>>>> <mailto:grant.sai...@gmail.com>>: >>>>> We process the sent_sms into another table on they fly. Maximum size the >>>>> sent_sms table gets is maybe 40k tops but mostly it averages around 10K. >>>>> We see this once 1 week maybe. >>>>> >>>>> I have really made every attempt to remove any bottlenecks in terms >>>>> unwieldy database sizes to allow kannel to work in a favourable >>>>> environment. >>>>> >>>>> Is there reason to add multiple sqlboxes to feed bearer box? >>>>> >>>>> Is there maybe a concurrency setting I can do for bearer box to receive >>>>> the messages? I did not come across documentation aside from email posts >>>>> with respect to the limit-per-cycle setting. >>>>> >>>>> I have another question, would we be able to get faster performance if we >>>>> went flat file for the kannel operations? >>>>> >>>>> Well you can exclude bottlenecks by simply testing same setup with >>>>> fakesmsc daemon and see if speed will be better. >>>>> >>>>> It might be that delay is caused by your SMSC uplinks overall speed and >>>>> not database. >>>>> >>>>> You can also try classic smsbox implementation for sending instead of >>>>> sqlbox. But I think sqlbox is fastest and more convinient way because of >>>>> DB storage. >>>>> >>>>> >>>>> >>>>> >>>>> Regards >>>>> >>>>> >>>>>> On 08 Dec 2015, at 15:12, spameden <spame...@gmail.com >>>>>> <mailto:spame...@gmail.com>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> 2015-12-08 12:51 GMT+03:00 Grant Vorenberg <gr...@saicomvoice.co.za >>>>>> <mailto:gr...@saicomvoice.co.za>>: >>>>>> <saicom-voice-services.gif> >>>>>> <https://branding.synaq.com/api/r/id/22474050/map/0> >>>>>> >>>>>> Hi >>>>>> >>>>>> We manage how big send_sms gets. The queue builder puts in 500 messages >>>>>> at a time to a total maximum of 3000 from a larger main queue which can >>>>>> go as big as 2M. >>>>>> >>>>>> 2M is kinda big table, how big is sent_sms? 10-30M ? >>>>>> >>>>>> I think your issue happens when kannel tries to move from send_sms to >>>>>> sent_sms table already submitted message this is where it hangs. You can >>>>>> try testing it yourself with simple query: >>>>>> >>>>>> INSERT INTO sent_sms SELECT * from send_sms where sql_id=XXXX and >>>>>> measure time per query. >>>>>> >>>>>> if it's instant there should be no problem. >>>>>> >>>>>> Generally it's better to leave sent_sms table at around 1M records not >>>>>> more, old records you can move to other table daily. >>>>>> >>>>>> >>>>>> Actual hardware is a VCenter on blades with plenty ram, cpu and hp >>>>>> 3PAR(144GB raid card ram for caching in total) fibre attached storage >>>>>> with dedicated SSD specifically for DB. Calculated IOPS are stupidly >>>>>> good. >>>>>> >>>>>> The VMs are as follows: >>>>>> Queuebuilder: 4 vcpu, 16Gb on SAS >>>>>> Kannel: 4 vcpu, 8GB on SAS >>>>>> MysqlDB-Master: 8 vcpu, 32GB on SSD >>>>>> MysqlDB-Slave: 8 vcpu, 32GB on SSD >>>>>> >>>>>> MySQL on SSDs should work just fine and you should have big number of >>>>>> iops. Btw, I recommend to use MariaDB instead of regular MySQL >>>>>> (mariadb.org <http://mariadb.org/>) it's very fast and reliable, for >>>>>> InnoDB it uses modified XtraDB engine which has some tweaks. >>>>>> >>>>>> did you check mysqladmin status to indicate number of queries processed >>>>>> per second? >>>>>> >>>>>> >>>>>> The Load on the MysqlDB-Master averages around 0.4 with max of 0.6 >>>>>> (single spike). Memory usage hangs around 24GB. I will need to check the >>>>>> process list to double check, but we generally don’t see much strain >>>>>> here either, but I stand to be corrected. >>>>>> DB optimasations as follows: >>>>>> >>>>>> key_buffer >>>>>> = 16M >>>>>> max_allowed_packet >>>>>> = 16M >>>>>> >>>>>> maybe increase a bit max_allowed_packet to 64mb >>>>>> >>>>>> innodb_buffer_pool_size >>>>>> = 12G >>>>>> >>>>>> this is fine >>>>>> >>>>>> query_cache_limit >>>>>> = 20M >>>>>> query_cache_size >>>>>> = 128M >>>>>> >>>>>> query_cache in kannel's case doesn't affect much, so it's ok to have it >>>>>> like that. >>>>>> >>>>>> >>>>>> No extra indexes on the send_sms as we limit its size to 3000. >>>>>> >>>>>> make sure both send_sms and sent_sms tables are InnoDB type. >>>>>> >>>>>> >>>>>> All reporting is done on the slaveDB, so no extra strain on monitoring >>>>>> and reporting. >>>>>> >>>>>> under 'reporting' do you mean your dlr_url is called to external script >>>>>> which is connected to slaveDB? >>>>>> >>>>>> >>>>>> Historically, we have queued at the SMPP connector and not at the >>>>>> smsbox. We generally reached a top (AVG) speed of 73 msg/s but when I >>>>>> see the smsbox queued figure rise and the internal smpp queue drop to 0, >>>>>> we only hit half of this speed. >>>>>> >>>>>> I did not see the limit-per-cycle setting in the sqlbox documentation >>>>>> (2011). I also checked the code and saw that the select limit was a >>>>>> variable instead of hard-limited. >>>>>> >>>>>> Yeah, it was introduced some time ago alongside with other optimisations >>>>>> (year ago or so, can't remember now), I think it should be in new >>>>>> documentation. However, you need to build documentation to get more >>>>>> recent version of it. >>>>>> >>>>>> >>>>>> Regards >>>>>> G >>>>>> >>>>>>> On 08 Dec 2015, at 10:52, spameden <spame...@gmail.com >>>>>>> <mailto:spame...@gmail.com>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Grant Vorenberg >>>>>>> Technical Manager >>>>>>> Office 0861 SAICOM (724 266) >>>>>>> Direct 010 140 5012 >>>>>>> Support 010 140 5050 >>>>>>> Cell - >>>>>>> Fax 010 140 5001 >>>>>>> Email gr...@saicomvoice.co.za <mailto:gr...@saicomvoice.co.za> >>>>>>> Visit our website www.saicomvoice.co.za >>>>>>> <http://www.saicomvoice.co.za/> >>>>>>> >>>>>>> <logo-image.png> <http://www.saicomvoice.co.za/> >>>>>>> >>>>>>> 2015-12-08 11:23 GMT+03:00 Grant Vorenberg <gr...@saicomvoice.co.za >>>>>>> <mailto:gr...@saicomvoice.co.za>>: >>>>>>> <saicom-now-offers-cloud-pbx.gif> >>>>>>> <https://branding.synaq.com/api/r/id/22469522/map/0> >>>>>>> >>>>>>> Here is my config: >>>>>>> >>>>>>> group = sqlbox >>>>>>> id = sqlbox-db >>>>>>> smsbox-id = sqlbox >>>>>>> #global-sender = "" >>>>>>> bearerbox-host = localhost >>>>>>> bearerbox-port = 13001 >>>>>>> smsbox-port = 13005 >>>>>>> smsbox-port-ssl = false >>>>>>> sql-log-table = sent_sms >>>>>>> sql-insert-table = send_sms >>>>>>> log-file = "/var/log/kannel/kannel-sqlbox.log" >>>>>>> log-level = 0 >>>>>>> #sqlbox optimisation GAV 20151207 >>>>>>> limit-per-cycle = 100 >>>>>>> >>>>>>> >>>>>>> Try monitoring your database workload. >>>>>>> >>>>>>> Is send_sms table is big? >>>>>>> >>>>>>> >>>>>>>> On 07 Dec 2015, at 15:06, spameden <spame...@gmail.com >>>>>>>> <mailto:spame...@gmail.com>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Grant Vorenberg >>>>>>>> Technical Manager >>>>>>>> Office 0861 SAICOM (724 266) >>>>>>>> Direct 010 140 5012 >>>>>>>> Support 010 140 5050 >>>>>>>> Cell - >>>>>>>> Fax 010 140 5001 >>>>>>>> Email gr...@saicomvoice.co.za <mailto:gr...@saicomvoice.co.za> >>>>>>>> Visit our website www.saicomvoice.co.za >>>>>>>> <http://www.saicomvoice.co.za/> >>>>>>>> >>>>>>>> <logo-image.png> <http://www.saicomvoice.co.za/> >>>>>>>> 2015-12-07 14:03 GMT+03:00 Grant Vorenberg <gr...@saicomvoice.co.za >>>>>>>> <mailto:gr...@saicomvoice.co.za>>: >>>>>>>> <saicom-now-offers-cloud-pbx.gif> >>>>>>>> <https://branding.synaq.com/api/r/id/22451334/map/0> >>>>>>>> >>>>>>>> Hi Guys >>>>>>>> >>>>>>>> I am a new to the list subscription and am looking for a little >>>>>>>> clarity on the SQLBOX. >>>>>>>> >>>>>>>> I have a Debian Wheezy box running: >>>>>>>> Kannel sqlbox version 1.4.4 >>>>>>>> Libxml version 2.7.2 >>>>>>>> MySQL 5.5.43 >>>>>>>> >>>>>>>> The front end is custom and drops message into the send_sms table. >>>>>>>> These messages are terminated via smpp to another system of ours. We >>>>>>>> process and clean out the sent_sms table. >>>>>>>> >>>>>>>> I gather stats on the performance of the system using the status page: >>>>>>>> http://localhost:13000/status <http://localhost:13000/status> >>>>>>>> >>>>>>>> I am trying to understand the following output from the screen: >>>>>>>> Box connections: >>>>>>>> smsbox:sqlbox, IP 127.0.0.1 (2532 queued), (on-line 0d 2h 48m >>>>>>>> 19s) >>>>>>>> >>>>>>>> This means there is a queue on sqlbox. >>>>>>>> >>>>>>>> Queue works like this: >>>>>>>> >>>>>>>> First sqlbox gets messages from DB backend, then it adds messages to >>>>>>>> own queue, then it sends message into bearerbox queue and bearerbox >>>>>>>> splits this queue over your connected SMSC/upstream operators. >>>>>>>> >>>>>>>> So if there is a huge queue on sqlbox it means there is a big amount >>>>>>>> of MTs in your send_sms table and sqlbox is still submitting those to >>>>>>>> bearerbox. >>>>>>>> >>>>>>>> To optimize sqlbox I'd recommend adding this parameter into sqlbox >>>>>>>> section (after group = sqlbox): >>>>>>>> limit-per-cycle = 50 this means sqlbox will get 50 bulk messages from >>>>>>>> DB per query at once instead of 1. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> What I noticed is when our send speeds start dipping on the smpp >>>>>>>> connection (internal/default route), this smsbox:sql queue starts >>>>>>>> building up. >>>>>>>> >>>>>>>> When the smsbox:sqlbox queue starts building up like this: >>>>>>>> 1)What causes this? >>>>>>>> 2) What does this signify? >>>>>>>> >>>>>>>> We generally don’t see this behaviour very often, but its effect is >>>>>>>> detrimental to the performance of the system so I would like to know >>>>>>>> what causes the growth and how to combat it so our send speed is >>>>>>>> safe-guarded. >>>>>>>> >>>>>>>> >>>>>>>> Here is an excerpt from my config files: >>>>>>>> >>>>>>>> <smsbox conf>: >>>>>>>> group = sqlbox >>>>>>>> id = sqlbox-db >>>>>>>> smsbox-id = sqlbox >>>>>>>> #global-sender = "" >>>>>>>> bearerbox-host = localhost >>>>>>>> bearerbox-port = 13001 >>>>>>>> smsbox-port = 13005 >>>>>>>> smsbox-port-ssl = false >>>>>>>> sql-log-table = sent_sms >>>>>>>> sql-insert-table = send_sms >>>>>>>> >>>>>>>> >>>>>>>> <kannel.conf>: >>>>>>>> group = smsbox >>>>>>>> bearerbox-host = 127.0.0.1 >>>>>>>> sendsms-port = 13013 >>>>>>>> global-sender = 13013 >>>>>>>> smsbox-id=my_smsbox >>>>>>>> >>>>>>>> group=smsbox-route >>>>>>>> smsbox-id=sqlbox >>>>>>>> smsc-id=internal >>>>>>>> >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Grant Vorenberg >>>>>>>> Technical Manager >>>>>>>> Office 0861 SAICOM (724 266) >>>>>>>> Direct 010 140 5012 >>>>>>>> Support 010 140 5050 >>>>>>>> Cell - >>>>>>>> Fax 010 140 5001 >>>>>>>> Email gr...@saicomvoice.co.za <mailto:gr...@saicomvoice.co.za> >>>>>>>> Visit our website www.saicomvoice.co.za >>>>>>>> <http://www.saicomvoice.co.za/> >>>>>>>> >>>>>>>> <logo-image.png> <http://www.saicomvoice.co.za/> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >> >> >
