We deal with a similar rate of data acquisition, and have taken a different approach. (We get, on average, about 10,000,000 UDP messages / day)
We have a very small UDP listener (about 50 lines of Python 3 code), and _all_ it does is receive the message and dump it into a Celery queue. We then run multiple Celery worker tasks to process that data and write it to the database. There are some times during peak period in the day where the queue might get backed up 1,000,000 messages or so, but we never have any problems getting them all processed that day. (UDP packets effectively stops around 9 PM.) We may be losing some packets - with UDP, how can you tell? But the numbers of packets processed are well within 95% of projections, and so for our purposes it's quite satisfactory. (There is absolutely no way for us to determine if some upstream device is being saturated and dropping packets that we should otherwise be receiving. Or more accurately, how many packets are being dropped upstream from our collection point. UDP is definitely not the protocol to be using if you absolutely need to receive 100% of your data.) On Monday, January 14, 2019 at 9:31:21 AM UTC-5, Sahil Mangotra wrote: > > I am facing a problem in my django web server. > > We are using python3, django2, django-rest-framework3.8 and channels2.x > > Scenario is we are receiving DATA from a UDP connection at very fast rate > (~100 messages per second). The data revived is in proto format (you can > say we are receiving byte data). some data gets starved in this process as > *Rate > of production >>> rate of consumption* we are implementing throttling but > still at 100 concurrent users data starves again. Can anyone help us in > this scenario. > > If anyone has any new architecture idea please share. > > *This is surely an interesting problem. This is about stock market feed* > > PS :- I cannot post any code as it is my companies. but i can help any > time you need clarification on any point. > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/55ca74e0-1164-40cc-90a0-7e1e1b0baad3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
