Thanks for pulling this list together, Oleksii. More comments inline. - Doug
On Wed, Mar 4, 2015, at 12:10 PM, ozamiatin wrote: > Hi, > > By this e-mail I'd like to start a discussion about current zmq driver > internal design problems I've found out. > I wish to collect here all proposals and known issues. I hope this > discussion will be continued on Liberty design summit. > And hope it will drive our further zmq driver development efforts. > > ZMQ Driver issues list (I address all issues with # and references are > in []): > > 1. ZMQContext per socket (blocker is neutron improper usage of messaging > via fork) [3] It looks like I had a question about managing backwards-compatibility on that, and Mehdi responded that he thinks things are broken enough ZMQ can't actually be used in production now. If that's true, then I agree we don't need to be concerned about upgrades. Can you add a comment to the review with your impression of the current version's suitability for production use? > > 2. Too many different contexts. > We have InternalContext used for ZmqProxy, RPCContext used in > ZmqReactor, and ZmqListener. > There is also zmq.Context which is zmq API entity. We need to > consider a possibility to unify their usage over inheritance (maybe > stick to RPCContext) > or to hide them as internal entities in their modules (see > refactoring #6) > > > 3. Topic related code everywhere. We have no topic entity. It is all > string operations. > We need some topics management entity and topic itself as an entity > (not a string). > It causes issues like [4], [5]. (I'm already working on it). > There was a spec related [7]. > > > 4. Manual implementation of messaging patterns. > Now we can observe poor usage of zmq features in zmq driver. Almost > everything is implemented over PUSH/PULL. > > 4.1 Manual polling - use zmq.Poller (listening and replying for > multiple sockets) > 4.2 Manual request/reply implementation for call [1]. > Using of REQ/REP (ROUTER/DEALER) socket solves many issues. A > lot of code may be reduced. > 4.3 Timeouts waiting > > > 5. Add possibility to work without eventlet [2]. #4.1 is also related > here, we can reuse many of the implemented solutions > like zmq.Poller over asynchronous sockets in one separate thread > (instead of spawning on each new socket). > I will update the spec [2] on that. > > > 6. Put all zmq driver related stuff (matchmakers, most classes from > zmq_impl) into a separate package. > Don't keep all classes (ZmqClient, ZmqProxy, Topics management, > ZmqListener, ZmqSocket, ZmqReactor) > in one impl_zmq.py module. The AMQP 1.0 driver work did something similar under a "protocols" directory. It would be nice to be consistent with the existing work on organizing driver-related files. > > _drivers (package) > +-- impl_rabbit.py > +-- impl_zmq.py <- leave only ZmqDriver class here > +-- zmq_driver (package) > | +--- matchmaker.py > | +--- matchmaker_ring.py > | +--- matchmaker_redis.py > | +--- matchmaker_.py > ... > | +--- client.py > | +--- reactor.py > | +--- proxy.py > | +--- topic.py > ... > > 7. Need more technical documentation on the driver like [6]. > I'm willing to prepare a current driver architecture overview with > some graphics UML charts, and to continue discuss the driver > architecture. > > Please feel free to add or to argue about any issue, I'd like to have > your feedback on these issues. This looks like a good list, and I'm encouraged to see activity around the ZMQ driver. I would like to see more participation in reviews for the ZMQ-related specs before the summit, so we can use our time together in person to resolve remaining issues rather than starting from scratch. Doug > Thanks. > > Oleksii Zamiatin > > > References: > > [1] https://review.openstack.org/#/c/154094/ > [2] https://review.openstack.org/#/c/151185/ > [3] https://review.openstack.org/#/c/150735/ > [4] https://bugs.launchpad.net/oslo.messaging/+bug/1282297 > [5] https://bugs.launchpad.net/oslo.messaging/+bug/1381972 > [6] https://review.openstack.org/#/c/130943/8 > [7] https://review.openstack.org/#/c/144149/1 > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
