Agreed, java can be faster than scripts like php, if designed carefully. PHP,
just like java, is interpreted, needs ~7 passes before it is rendered by a
webserver, and doesn't even have the benefit of intermediate byteCode that Java
has. Java of course has the very annoying garbadge collection, for people who
can't handle their own memory, that can bring response to a halt (again subject
to design).
But an application server is a completely different animal. It is designed for
the general case, and the code that it generates is seriously underoptimized
for any application in particular. I doubt that DLR handling is such an
involved job that it needs an application server.
Indeed backend needs a good, careful design, but once such is implemented, it
should be able to handle that kind of traffic. On the same server, w/ 1 dual
core CPU @2.16 GHz, 4 GB RAM, and lighttpd, I can GET 3kb pages at 3000/s (pure
webserver, no php or DB). A light php (or java) implementation that processes
only 1 line of text should be even faster than that. From then on it is up to
the database design. Mysql is much lighter and faster than Oracle, but less
robust.
I know this is quite subjective, but I hope people will resist the temptation
to flame.
BR,
Nikos
----- Original Message -----
From: Alejandro Guerrieri
To: Gabor Maros ; users@kannel.org
Sent: Friday, April 02, 2010 12:11 AM
Subject: Re: Too many dlr at once
Hmm, I wouldn't throw an opinion about the technologies involved without
really knowing what's really going on infrastructure-wise.
IMHO there's nothing wrong with your setup from a high level perspective. Of
course I can't tell a thing about how it is configured, dimensioned and/or
being used, but seems to be an enterprise-grade platform and it should be able
to handle tons of traffic if used properly.
To give you a real example, we use Tomcat to handle MO's and DLR's.
I can tell you we handle many, many hundreds of requests per second without
even noticing the load on the servers. WAY more than what we were doing with
our former php solution.
NOTE: However if the application layer is poorly designed, nothing like java
to bring a system to its knees in no time ;) but it wouldn't be a Java problem
but a design problem. With enough (or too little) skills, you can trash a
server using any language of choice :D
Now seriously, kannel doesn't spawn infinite threads to hit the application
server when a DLR arrives, I'm not sure about how many requests per second
you're getting, but using the proper approach you should be able to deal with
it with a cluster setup for sure.
I don't know what you're doing when you receive the dlrs, but whatever it is,
if you're expecting high traffic you need to take some strategies in order to
be able to handle it. My rather uninformed recommendations for your case are:
1. Try to do as little as possible when receiving the DLR. You can
post-process them afterwards in a cron, batch, separate daemon, pull from a
queue, whatever.
2. Try to finish the process as fast as possible. This is more or less the
same as 1: the less time you take processing each dlr, the better.
3. Profile the applications involved. Sometimes a silly bug somewhere holds
things longer than needed and brings everything to a crawl. e.g. a
misconfigured DB pool, objects being held in memory longer than needed or not
enough memory assigned to an application.
4. Last but not least: Load balancers, message queues (JMS) and memory caches
(ehcache, memcached, etc) are your best friends when designing high-performance
platforms ;)
Hope it helps,
Alex
2010/4/1 Nikos Balkanas <nbalka...@gmail.com>
Big, big, big mistake. You are using a java application server, to do the
work a c web server. That is your problem. I am surprised that your system is
not frozen from the many connections. You could at least invest in a JBoss that
is open source.
Still my question stands: DLRs are not synchronous. That means you don't
get them all at once when you stop sending SMS. SMS can be delivered at any
time, and therefore DLRs are received at the time of delivery, i.e. any time.
With 25 SMS/s DLRs should be coming in at about the same throughput.
Anyway to answer your question. Kannel cannot limit its DLR output, since
it doesn't have a queue specific for this, and it sends them as it receives
them (OK there is a queue at the SMSc driver, but that's not it). You have a
problem in your architecture of your backend. Maybe you can talk to your SMSc
and ask them to observe the same inbound throuput restrictions that they have
for outbound traffic. 25 SMS/s in, 25 SMS/s out.
Nikos
----- Original Message ----- From: "Gabor Maros" <gabor.ma...@erstebank.hu>
To: <users@kannel.org>
Sent: Thursday, April 01, 2010 9:22 PM
Subject: Re: Too many dlr at once
Hi Seikath,
Dlrs are very important for us they are in an Oracle database, and the web
application is running in a WebLogic cluster (it makes DB connectio
pooling). I can not use other infrastructure. I think opening 40k network
connection at once is not the best and effective thing (allowing so many
open files is risky) so I just like a solution that doesn't allow kannel to
open such amount of connections at once. I like a solution where I can tell
kannel not to send more dlrs back then 100/sec. This is a normal pooling
system a lot of applications are using such tuning parameters to save others
and I think kannel use the same thing on smsc side, but I want it on the
reverse side (how flow-control and window work).
The problem is that cimd2 has the less possibility to manadge according to
the documentation.
Bye,
Gabor
seikath wrote:
In general DLR is not so important info to be injected right away into the
database.
if you have high load of MO/DLR, consider db pooling and even better, drop
the http requests.
The Apache or Lighty or even ISS can handle the traffic you have mentioned
with no issues.
What I do for high load of MO/DLR, is either use sqlbox to handle it,
either simply write directly to simple xml files.
OR, you may parse the kannel logs, which will require some regexp skills.
I used to implement all of the above, according to the specific projects.
The XML files easily can be loaded later in a queue in the database.
On 04/01/2010 06:33 PM, Gabor Maros wrote:
Thanks Nikos,
it may help but there is another problem i haven't mentioned before. We
have
a webapplication that receives dlrs from kannel. If kannel gets 10k dlr
in
one sec then kannel tries to send all of them in the same sec to the
app.
This behaviour kills the app (and the database behind it) because it
gets
10000 http connections in one sec which is quite huge amount according
to
our peaktime when there is 25 SMs/sec.
Unfortunately we are not the NASA with unimaginable computing capacity,
so
the ideal solution for us would be a parameter that tells kannel how
many
connections are allowed in one sec.
Bye,
Gabor
Nikos Balkanas wrote:
Hi,
Check if you havd /etc/hosts, and if you do you should have specified
your
gateway.
Also check if named is running (Linux)
BR,
Nikos
----- Original Message ----- From: "Gabor Maros"
<gabor.ma...@erstebank.hu>
To: <users@kannel.org>
Sent: Thursday, April 01, 2010 12:58 PM
Subject: Too many dlr at once
Hi,
I've got a kannel install with emi smsc connection.
When we send lots of sms to the smsc at once the delivery
notifications
only
come at the end when kannel's queue is empty. Smsc only accepts 10-15
SM/sec
but can send back horrible amount at once. This is a problem for us
because
kannel gets thousands of dlrs in one second and ERROR messages appear
in
smsbox.log:
2010-04-01 08:21:17 [4834] [4] INFO: Starting delivery report <sms>
from
<0036303444481>
2010-04-01 08:21:17 [4834] [4] INFO: Starting delivery report <sms>
from
<0036303444481>
2010-04-01 08:21:17 [4834] [4] INFO: Starting delivery report <sms>
from
<0036303444481>
2010-04-01 08:21:17 [4834] [4] INFO: Starting delivery report <sms>
from
<0036303444481>
2010-04-01 08:21:17 [4834] [4] INFO: Starting delivery report <sms>
from
<0036303444481>
…after thousands of such normal logrecords we can see thousands
of the
following:
2010-04-01 08:21:18 [4834] [9] ERROR: Error while gw_gethostbyname
occurs.
2010-04-01 08:21:18 [4834] [9] ERROR: System error 2: No such file or
directory
2010-04-01 08:21:18 [4834] [9] ERROR: gethostbyname failed
2010-04-01 08:21:18 [4834] [9] ERROR: error connecting to server
`xxxx'
at
port `yyy'
2010-04-01 08:21:18 [4834] [9] ERROR: Couldn't send request to
<https://xyz>
2010-04-01 08:21:18 [4834] [9] ERROR: Error while gw_gethostbyname
occurs.
2010-04-01 08:21:18 [4834] [9] ERROR: System error 2: No such file or
directory
2010-04-01 08:21:18 [4834] [9] ERROR: gethostbyname failed
2010-04-01 08:21:18 [4834] [9] ERROR: error connecting to server
`xxxx'
at
port `yyy'
2010-04-01 08:21:18 [4834] [9] ERROR: Couldn't send request to
<https://xyz>
2010-04-01 08:21:18 [4834] [9] ERROR: Error while gw_gethostbyname
occurs.
2010-04-01 08:21:18 [4834] [9] ERROR: System error 2: No such file or
directory
2010-04-01 08:21:18 [4834] [9] ERROR: gethostbyname failed
2010-04-01 08:21:18 [4834] [9] ERROR: error connecting to server
`xxxx'
at
port `yyy'
2010-04-01 08:21:18 [4834] [9] ERROR: Couldn't send request to
<https://xyz>
2010-04-01 08:21:18 [4834] [9] ERROR: Error while gw_gethostbyname
occurs.
2010-04-01 08:21:18 [4834] [9] ERROR: System error 2: No such file or
directory
2010-04-01 08:21:18 [4834] [9] ERROR: gethostbyname failed
2010-04-01 08:21:18 [4834] [9] ERROR: error connecting to server
`xxxx'
at
port `yyy'
2010-04-01 08:21:18 [4834] [9] ERROR: Couldn't send request to
<https://xyz>
2010-04-01 08:21:18 [4834] [9] ERROR: Error while gw_gethostbyname
occurs.
2010-04-01 08:21:18 [4834] [9] ERROR: System error 2: No such file or
directory
Is there a configuration parameter that change this behavior and we
can
slow it down?
I donβ€ℶt know why it is happen but there must be some kind of
limit (I
think
it is not an open file issue but something similar).
Maybe there is another side effect (but Iβ€ℶm not sure yet) in
connection
with
DLR database because the number of SMs that are not in the end phase
(delivered or canβ€ℶt be delivered) are growing.
Thanks,
Gabor
--
View this message in context:
http://old.nabble.com/Too-many-dlr-at-once-tp28106589p28106589.html
Sent from the Kannel - User mailing list archive at Nabble.com.
--
View this message in context:
http://old.nabble.com/Too-many-dlr-at-once-tp28106589p28112070.html
Sent from the Kannel - User mailing list archive at Nabble.com.
