[BangPypers] real use case of gevent context switch gevent.sleep(0)

[anu sree]

Hi,

I have seen an example of gevent context switch
http://sdiehl.github.io/gevent-tutorial/#synchronous-asynchronous-execution

Could you please give some real use case with example.
It would be fine If you can share link of github project which uses gevent
context gevent.sleep(0).

Thanks,
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] real use case of gevent context switch gevent.sleep(0)

[anu sree]

Thanks naufal and Krace.

I have tried following code. I have given gevent.sleep(0.1), that means
each greenlet let others to execute 0.1 secoond. But here each greenlet
waiting for others to complete, why?  Is it because of greenlet.joinall ?

Here I have created 3 greenlet threads (A, B, C). I think, A is the parent
greenlet, right ? Once C get finished control should goes to B, right ?
But here it going to A.
Please check the ouput.

import gevent

def task(num):
thread_id = id(gevent.getcurrent())
print "Start: ", num, thread_id

for i in range(10001):
if i == 1:
print num + str(i)

gevent.sleep(0.1)

for i in range(10001):
if i == 1:
print num + str(i)

print "End: ", num, thread_id

threads = [gevent.spawn(task, x) for x in ['A', 'B', 'C']]
gevent.joinall(threads)


Output
===

Start:  A 139984038735696
A1
Start:  B 139984008421616
B1
Start:  C 139984008421776
C1
A1
End:  A 139984038735696
B1
End:  B 139984008421616
C1
End:  C 139984008421776
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] real use case of gevent context switch gevent.sleep(0)

[anu sree]

Hi Krace,

I am still not understanding from code (practical example) where we are
getting the benefit of PAUSE and let other greenlet to execute.

I have created a simple worker program from your example
https://github.com/dcramer/taskmaster/blob/79a312c5cb3c34d00829fe9cf4204aeb478a0166/src/taskmaster/client.py#L26
.

This code has two pause (gevent.sleep(0)), in Consumer.start and Worker.run.
Here control goes to Worker.run when Consumer.start pauses and
Consumer.start gets control back when Worker.run pauses. There may be
benefit from this switching, but I am still not understanding it.

Please check the attached code and output. Is my example code valid ?


CODE
==

import gevent

class Worker(object):

def __init__(self, consumer, target):
self.consumer = consumer
self.target = target

def run(self):
print "Worker.run, START"
self.started = True

while self.started:
print "Greenlet thread (from Worker.run) Pauses, START"
gevent.sleep(0)
print "Greenlet thread (from Worker.run) Pauses, END"
job = self.consumer.get_job()
self.target(job)

print "Worker.run, END"

class Consumer(object):

def __init__(self, target):
self.target = target

def start(self):
print "Consumer.start, START"
self.started = True

worker = Worker(self, self.target)
gevent.spawn(worker.run)

while self.started:
print "Main Interpreter greenlet (from Consumer.start) Pauses,
START"
gevent.sleep(0)
print "Main Interpreter greenlet (from Consumer.start) Pauses,
END \n"

print "Consumer.start, END"

def get_job(self):
return 1

#
def my_target_1(n):
print "my_target_1, START"
nums = []
while n > 0:
nums.append(n)
n -= 1
print "my_target_1, END"

c = Consumer(my_target_1)
c.start()


OUTPUT
===

Consumer.start, START
Main Interpreter greenlet (from Consumer.start) Pauses, START
Worker.run, START
Greenlet thread (from Worker.run) Pauses, START
Main Interpreter greenlet (from Consumer.start) Pauses, END

Main Interpreter greenlet (from Consumer.start) Pauses, START
Greenlet thread (from Worker.run) Pauses, END
my_target_1, START
my_target_1, END
Greenlet thread (from Worker.run) Pauses, START
Main Interpreter greenlet (from Consumer.start) Pauses, END

Main Interpreter greenlet (from Consumer.start) Pauses, START
Greenlet thread (from Worker.run) Pauses, END
my_target_1, START
my_target_1, END
Greenlet thread (from Worker.run) Pauses, START
Main Interpreter greenlet (from Consumer.start) Pauses, END
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] real use case of gevent context switch gevent.sleep(0)

[anu sree]

Thanks noufal,

I got it.

After gevent.spawn, we have to do join or gevent.joinall or gevent.sleep or
patched library call to start the greenlets, right ?

I have see the code which not using any of these after gevent.spawn, How it
is working there ?
https://github.com/Juniper/contrail-controller/blob/master/src/opserver/uveserver.py#L656


Thanks,



On Sun, Jun 21, 2015 at 1:01 PM, Noufal Ibrahim KV 
wrote:

> On Sun, Jun 21 2015, anu sree wrote:
>
>
> [...]
>
> > This code has two pause (gevent.sleep(0)), in Consumer.start and
> > Worker.run.  Here control goes to Worker.run when Consumer.start
> > pauses and Consumer.start gets control back when Worker.run
> > pauses. There may be benefit from this switching, but I am still not
> > understanding it.
>
> [...]
>
> The idea is that any non blocking call is an opportunity to switch
> between greenlets. The monkey patching system will enable this.
>
> If you don't have any place in your greenlets that are monkey patched
> (and therefore allow switching), you can be a good citizen and drop in a
> patched sleep call so that other greenlets will get time to
> run. Otherwise, you'll simply hog the interpreter and no one else will
> be able to run.
>
> This the principle. I still don't understand your question.
>
>
> --
> Cordially,
> Noufal
> http://nibrahim.net.in
> ___
> BangPypers mailing list
> BangPypers@python.org
> https://mail.python.org/mailman/listinfo/bangpypers
>
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

[BangPypers] UI less Rest API server with flask

[anu sree]

Hi,

I want to develop an API server (don't need GUI).
I am planning to use flask to develop this UI less API server.

Question-1:
I have seen an example here https://github.com/miguelgrinberg/api-pycon2015
But above example does not use flask_restful extension.
Here I need your help to take decision, Should I use with flask_restful
extension ?
what are the benefits of using flask_restful extension?

Question-1:
When I start or restart api server, it should fetch huge data from
somewhere (Eg:database) and process it in separate thread (it also make
some rest api call different services during the processing).
I am planning to create a thread inside the api server using *gevent* to do
this while starting the api server, is it right decision?
This fetching and processing of data should not affect the other operation
of the API server, that is I should able to make rest api call
to api server while it fetching and processing the data.

Thanks,
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] UI less Rest API server with flask

[anu sree]

Thanks anuvrat,

My first question was, Should I use only flask to develop REST API like in
this example https://github.com/miguelgrinberg/api-pycon2015

or fask + flask_restful ?


On Mon, Jun 22, 2015 at 3:00 PM, Anuvrat Parashar 
wrote:

> On Mon, Jun 22, 2015 at 2:06 PM, anu sree  wrote:
>
> > Hi,
> >
> > I want to develop an API server (don't need GUI).
> > I am planning to use flask to develop this UI less API server.
> >
> > Question-1:
> > I have seen an example here
> > https://github.com/miguelgrinberg/api-pycon2015
> > But above example does not use flask_restful extension.
> > Here I need your help to take decision, Should I use with flask_restful
> > extension ?
> >
>
> There are quite a few extensions for making restful apis - flask-restful,
> flask-restless etc ...
> While researching also consider python-eve
> https://github.com/nicolaiarocci/eve
>
>
> > what are the benefits of using flask_restful extension?
> >
> > Question-1:
> > When I start or restart api server, it should fetch huge data from
> > somewhere (Eg:database) and process it in separate thread (it also make
> > some rest api call different services during the processing).
> > I am planning to create a thread inside the api server using *gevent* to
> do
> > this while starting the api server, is it right decision?
> > This fetching and processing of data should not affect the other
> operation
> > of the API server, that is I should able to make rest api call
> > to api server while it fetching and processing the data.
> >
>
> For this deferred processing too you have various options namely: gevent,
> celery, twisted etc ...
>
> Sharing more information on what you are trying to accomplish would enable
> people here to suggest better solutions.
> Peace.
>
> >
> > Thanks,
> > ___
> > BangPypers mailing list
> > BangPypers@python.org
> > https://mail.python.org/mailman/listinfo/bangpypers
> >
>
>
>
> --
> Anuvrat Parashar <http://anuvrat.in>
> http://anuvrat.in
> ___
> BangPypers mailing list
> BangPypers@python.org
> https://mail.python.org/mailman/listinfo/bangpypers
>
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] UI less Rest API server with flask

[anu sree]

Thanks anuvrat and krace,

How do we do validation of data in the POST and PUT API request, if we use
only native fask ?.

I have seen a way in flask-restful, but I am planing to use only native
fask.
http://flask-restful.readthedocs.org/en/latest/api.html#reqparse.RequestParser

Thanks,



On Mon, Jun 22, 2015 at 6:28 PM, kracekumar ramaraju <
kracethekingma...@gmail.com> wrote:

> The biggest problem with all rest apis except Django REST framework is no
> distinction between serializer and model. Soon that will be problem for the
> kind of tasks you want to do.
>
> I don't use any framework to create apis in Flask.
>
> I follow different approach, use schematics [1] for serialization, Flask
> method view and use model methods/service layer to interact with dbs,
> queues etc ...
>
> You can find the similar example for Django in schematics example
> directory, it should be easy to implement in Flask too.
>
> [1]: http://github.com/schematics/schematics
> ___
> BangPypers mailing list
> BangPypers@python.org
> https://mail.python.org/mailman/listinfo/bangpypers
>
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] UI less Rest API server with flask

[anu sree]

Thanks dhruv,

That is for validating the form. I don't have forms in my app, i just want
to validate the data in the REST API request.

I think, good choice is
http://flask-restful.readthedocs.org/en/latest/api.html#reqparse.RequestParser


On Mon, Jun 22, 2015 at 7:59 PM, Dhruv Baldawa 
wrote:

> You can either add those yourself, or use something like Flask-WTF[1] or
> validictory[2] based on your requirements.
>
>
> [1]: https://flask-wtf.readthedocs.org/en/latest/
> [2]: https://readthedocs.org/projects/validictory/
>
> On Mon, Jun 22, 2015 at 6:57 PM, anu sree  wrote:
>
> > Thanks anuvrat and krace,
> >
> > How do we do validation of data in the POST and PUT API request, if we
> use
> > only native fask ?.
> >
> > I have seen a way in flask-restful, but I am planing to use only native
> > fask.
> >
> >
> http://flask-restful.readthedocs.org/en/latest/api.html#reqparse.RequestParser
> >
> > Thanks,
> >
> >
> >
> > On Mon, Jun 22, 2015 at 6:28 PM, kracekumar ramaraju <
> > kracethekingma...@gmail.com> wrote:
> >
> > > The biggest problem with all rest apis except Django REST framework is
> no
> > > distinction between serializer and model. Soon that will be problem for
> > the
> > > kind of tasks you want to do.
> > >
> > > I don't use any framework to create apis in Flask.
> > >
> > > I follow different approach, use schematics [1] for serialization,
> Flask
> > > method view and use model methods/service layer to interact with dbs,
> > > queues etc ...
> > >
> > > You can find the similar example for Django in schematics example
> > > directory, it should be easy to implement in Flask too.
> > >
> > > [1]: http://github.com/schematics/schematics
> > > ___
> > > BangPypers mailing list
> > > BangPypers@python.org
> > > https://mail.python.org/mailman/listinfo/bangpypers
> > >
> > ___
> > BangPypers mailing list
> > BangPypers@python.org
> > https://mail.python.org/mailman/listinfo/bangpypers
> >
>
>
>
> --
> Dhruv Baldawa
> (http://www.dhruvb.com)
> ___
> BangPypers mailing list
> BangPypers@python.org
> https://mail.python.org/mailman/listinfo/bangpypers
>
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] UI less Rest API server with flask

[anu sree]

Thanks krace,

Good suggestion, I am going to use schematics with flask. I red
about schematics and think that will help me keep validation code clean and
reduce number of statements in the code.

Thanks

On Mon, Jun 22, 2015 at 10:48 PM, kracekumar ramaraju <
kracethekingma...@gmail.com> wrote:

> On Jun 22, 2015 19:03, "anu sree"  wrote:
> >
> > Thanks anuvrat and krace,
> >
> > How do we do validation of data in the POST and PUT API request, if we
> use
> > only native fask ?.
>
> That is where serialization library like schematics comes to play. Pass the
> request data dict to serialization and call  validate method.
>
> > I have seen a way in flask-restful, but I am planing to use only native
> > fask.
> >
>
> http://flask-restful.readthedocs.org/en/latest/api.html#reqparse.RequestParser
> >
> > Thanks,
> >
> >
> >
> > On Mon, Jun 22, 2015 at 6:28 PM, kracekumar ramaraju <
> > kracethekingma...@gmail.com> wrote:
> >
> > > The biggest problem with all rest apis except Django REST framework is
> no
> > > distinction between serializer and model. Soon that will be problem for
> the
> > > kind of tasks you want to do.
> > >
> > > I don't use any framework to create apis in Flask.
> > >
> > > I follow different approach, use schematics [1] for serialization,
> Flask
> > > method view and use model methods/service layer to interact with dbs,
> > > queues etc ...
> > >
> > > You can find the similar example for Django in schematics example
> > > directory, it should be easy to implement in Flask too.
> > >
> > > [1]: http://github.com/schematics/schematics
> > > ___
> > > BangPypers mailing list
> > > BangPypers@python.org
> > > https://mail.python.org/mailman/listinfo/bangpypers
> > >
> > ___
> > BangPypers mailing list
> > BangPypers@python.org
> > https://mail.python.org/mailman/listinfo/bangpypers
> ___
> BangPypers mailing list
> BangPypers@python.org
> https://mail.python.org/mailman/listinfo/bangpypers
>
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] UI less Rest API server with flask

[anu sree]

Thanks gora,

I am happy to use django-DRF if I can use it without doing syncdb. I don't
want to use models and orm.

The REST API app which I am going to develop doesn't have models now.

I also don't want the apps which are included in settings.INSTALLED_APPS by
default.

I also want to run some asynchronous task (to read a large xml file and
process it) from this REST API App, I think I can use celery + gevent here.

I just started reading http://www.django-rest-framework.org/

Thanks,

On Tue, Jun 23, 2015 at 12:41 AM, Gora Mohanty  wrote:

> Hi,
>
> Since you seem to be determined to reinvent the wheel, may I ask what is
> your use case to choose flask + largely-roll-one's-own-rest-api vs. Django
> + DRF?
>
> The same goes for all the recent hoopla in this list over Flask/Bottle over
> Django, if you ask me. Django is not all that difficult to get into, has a
> well-established track record, and is eminently scalable. For serious
> development, one also needs to worry about the community size, and support.
> I do not even know that Flask/Bottle/what have an edge over Django for any
> use case, but IMHO if so one would be talking about the scale of one the
> largest sites on the Internet. Remember, premature oprtimisation is the
> root of all evil. Sorry, but if you ask me I smell buzzword-driven
> development.
>
> Regards,
> Gora
>
>
> On 23 June 2015 at 00:20, anu sree  wrote:
>
> > Thanks krace,
> >
> > Good suggestion, I am going to use schematics with flask. I red
> > about schematics and think that will help me keep validation code clean
> and
> > reduce number of statements in the code.
> >
> > Thanks
> >
> > On Mon, Jun 22, 2015 at 10:48 PM, kracekumar ramaraju <
> > kracethekingma...@gmail.com> wrote:
> >
> > > On Jun 22, 2015 19:03, "anu sree"  wrote:
> > > >
> > > > Thanks anuvrat and krace,
> > > >
> > > > How do we do validation of data in the POST and PUT API request, if
> we
> > > use
> > > > only native fask ?.
> > >
> > > That is where serialization library like schematics comes to play. Pass
> > the
> > > request data dict to serialization and call  validate method.
> > >
> > > > I have seen a way in flask-restful, but I am planing to use only
> native
> > > > fask.
> > > >
> > >
> > >
> >
> http://flask-restful.readthedocs.org/en/latest/api.html#reqparse.RequestParser
> > > >
> > > > Thanks,
> > > >
> > > >
> > > >
> > > > On Mon, Jun 22, 2015 at 6:28 PM, kracekumar ramaraju <
> > > > kracethekingma...@gmail.com> wrote:
> > > >
> > > > > The biggest problem with all rest apis except Django REST framework
> > is
> > > no
> > > > > distinction between serializer and model. Soon that will be problem
> > for
> > > the
> > > > > kind of tasks you want to do.
> > > > >
> > > > > I don't use any framework to create apis in Flask.
> > > > >
> > > > > I follow different approach, use schematics [1] for serialization,
> > > Flask
> > > > > method view and use model methods/service layer to interact with
> dbs,
> > > > > queues etc ...
> > > > >
> > > > > You can find the similar example for Django in schematics example
> > > > > directory, it should be easy to implement in Flask too.
> > > > >
> > > > > [1]: http://github.com/schematics/schematics
> > > > > ___
> > > > > BangPypers mailing list
> > > > > BangPypers@python.org
> > > > > https://mail.python.org/mailman/listinfo/bangpypers
> > > > >
> > > > ___
> > > > BangPypers mailing list
> > > > BangPypers@python.org
> > > > https://mail.python.org/mailman/listinfo/bangpypers
> > > ___
> > > BangPypers mailing list
> > > BangPypers@python.org
> > > https://mail.python.org/mailman/listinfo/bangpypers
> > >
> > ___
> > BangPypers mailing list
> > BangPypers@python.org
> > https://mail.python.org/mailman/listinfo/bangpypers
> >
> ___
> BangPypers mailing list
> BangPypers@python.org
> https://mail.python.org/mailman/listinfo/bangpypers
>
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] UI less Rest API server with flask

[anu sree]

Thanks gora and guruprasad,

I am not using database or other store. My API App just make different API
requests to other rest api servers.

Does celery need db ?


Thanks,

On Tue, Jun 23, 2015 at 1:28 PM, L. Guruprasad  wrote:

> Hi Anu,
>
> On Tue, Jun 23, 2015 at 1:21 PM, anu sree  wrote:
> > I just started reading http://www.django-rest-framework.org/
>
> Once you have learnt enough of DRF to start writing code, this -
> http://www.cdrf.co/ will be a handy reference.
>
> Hope this helps :)
>
> Thanks & Regards,
> Guruprasad
> ___
> BangPypers mailing list
> BangPypers@python.org
> https://mail.python.org/mailman/listinfo/bangpypers
>
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] Connecting developers and companies

[anu sree]

+1. It will be helpful for beginner like me to get job.
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] Online communication channel

[anu sree]

Slack is good, but I think we have to pay to keep chat history.
On Aug 15, 2015 8:59 PM, "kracekumar ramaraju" 
wrote:

> Hi
>
>  BangPypers doesn't have any online channel to discuss. Most of the Open
> source/Free software projects use IRC as tool for instant communication.
>
> Lately, lot of people have asked in meetup and at various places does one
> exist one for BangPypers. Many organizations like Wordpress [1], gopher
> [2], d3js [3] are using slack and indeed most people suggested the same.
>
> IRC is default tool for most the projects/organization, but slack has whole
> lot of advantages like mobile app, multi account login, email notification
> etc ... Also slack has bridge for IRC [4].
>
> Thoughts ?
>
>
> [1]: https://make.wordpress.org/chat/
> [2]: http://blog.gopheracademy.com/gophers-slack-community/
> [3]: https://d3js.slack.com/
> [4]:
>
> https://slack.zendesk.com/hc/en-us/articles/201727913-Connecting-to-Slack-over-IRC-and-XMPP
>
> --
>
> *Thanks & Regardskracekumar"Talk is cheap, show me the code" -- Linus
> Torvaldshttp://kracekumar.com *
> ___
> BangPypers mailing list
> BangPypers@python.org
> https://mail.python.org/mailman/listinfo/bangpypers
>
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

[BangPypers] framework to write functional test cases

[anu sree]

Hi,

I am looking for a framework which helps to write functional test cases.
Right now i am using unittest framework to do that, but that is ugly. How
you guys are writing functional test cases.I also want to do stress test by
sending thousands of concurrent requests (run functional test cases in
different threads).
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

Re: [BangPypers] Python Software Foundation

[anu sree]

congrats
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers

[BangPypers] How to configure python celery in active standby mode

[anu sree]

Hi,

How to configure python celery scheduler in active standby mode.

I have 2 nodes setup:
Node-1: Running celery worker and rabbitmq server
Node-2: Running celery worker pointing to rabbitmq server in node-1

Requirement:
I want to run a periodic task in this setup and that task should run when
node-1 or node-2 goes down.
How to do it ?

I have seen that, we can run celery scheduler in one node. But periodic
task will not run if that node goes down.

Thanks,
___
BangPypers mailing list
BangPypers@python.org
https://mail.python.org/mailman/listinfo/bangpypers