+1 "NFTablesDriver"! Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example: https://home.regit.org/2014/02/suricata-and-nftables/
Then, I'm wondering here... What benefits might come for OpenStack Nova / Neutron, if it comes with a NFTables driver, instead of the current IPTables?! * Efficient Security Group design? * Better FWaaS, maybe with NAT(44/66) support? * Native support for IPv6, with the defamed NAT66 built-in, simpler "Floating IP" implementation, for both v4 and v6 networks under a single implementation (*I don't like NAT66, I prefer a `routed Floating IPv6` version*)? * Metadata over IPv6 still using NAT(66) (*I don't like NAT66*), single implementation? * Suricata-as-a-Service?! It sounds pretty cool! :-) On 20 August 2014 23:16, Baohua Yang <yangbao...@gmail.com> wrote: > Great！ > We met similar problems. > The current mechanisms produce too many iptables rules, and it's hard to > debug. > Really look forward to seeing a more efficient security group design. > > > On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery <mest...@noironetworks.com> > wrote: > >> On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang <ayshihanzh...@126.com> >> wrote: >> > >> > With the deployment 'nova + neutron + openvswitch', when we bulk create >> > about 500 VM with a default security group, the CPU usage of >> neutron-server >> > and openvswitch agent is very high, especially the CPU usage of >> openvswitch >> > agent will be 100%, this will cause creating VMs failed. >> > >> > With the method discussed in mailist: >> > >> > 1) ipset optimization (https://review.openstack.org/#/c/100761/) >> > >> > 3) sg rpc optimization (with fanout) >> > (https://review.openstack.org/#/c/104522/) >> > >> > I have implement these two scheme in my deployment, when we again bulk >> > create about 500 VM with a default security group, the CPU usage of >> > openvswitch agent will reduce to 10%, even lower than 10%, so I think >> the >> > iprovement of these two options are very efficient. >> > >> > Who can help us to review our spec? >> > >> This is great work! These are on my list of things to review in detail >> soon, but given the Neutron sprint this week, I haven't had time yet. >> I'll try to remedy that by the weekend. >> >> Thanks! >> Kyle >> >> > Best regards, >> > shihanzhang >> > >> > >> > >> > >> > >> > At 2014-07-03 10:08:21, "Ihar Hrachyshka" <ihrac...@redhat.com> wrote: >> >>-----BEGIN PGP SIGNED MESSAGE----- >> >>Hash: SHA512 >> >> >> >>Oh, so you have the enhancement implemented? Great! Any numbers that >> >>shows how much we gain from that? >> >> >> >>/Ihar >> >> >> >>On 03/07/14 02:49, shihanzhang wrote: >> >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today >> >>> I will modify my spec, when the spec is approved, I will commit the >> >>> codes as soon as possilbe! >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majop...@redhat.com> >> >>> wrote: >> >>>> >> >>>> Nice Shihanzhang, >> >>>> >> >>>> Do you mean the ipset implementation is ready, or just the >> >>>> spec?. >> >>>> >> >>>> >> >>>> For the SG group refactor, I don't worry about who does it, or >> >>>> who takes the credit, but I believe it's important we address >> >>>> this bottleneck during Juno trying to match nova's scalability. >> >>>> >> >>>> Best regards, Miguel Ángel. >> >>>> >> >>>> >> >>>> On 07/02/2014 02:50 PM, shihanzhang wrote: >> >>>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that >> >>>>> split the work in several specs, I have finished the work ( >> >>>>> ipset optimization), you can do 'sg rpc optimization (without >> >>>>> fanout)'. as the third part(sg rpc optimization (with fanout)), >> >>>>> I think we need talk about it, because just using ipset to >> >>>>> optimize security group agent codes does not bring the best >> >>>>> results! >> >>>>> >> >>>>> Best regards, shihanzhang. >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrac...@redhat.com> >> >>>>> wrote: >> >>> On 02/07/14 10:12, Miguel Angel Ajo wrote: >> >>> >> >>>> Shihazhang, >> >>> >> >>>> I really believe we need the RPC refactor done for this cycle, >> >>>> and given the close deadlines we have (July 10 for spec >> >>>> submission and July 20 for spec approval). >> >>> >> >>>> Don't you think it's going to be better to split the work in >> >>>> several specs? >> >>> >> >>>> 1) ipset optimization (you) 2) sg rpc optimization (without >> >>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you >> >>>> , me) >> >>> >> >>> >> >>>> This way we increase the chances of having part of this for the >> >>>> Juno cycle. If we go for something too complicated is going to >> >>>> take more time for approval. >> >>> >> >>> >> >>> I agree. And it not only increases chances to get at least some of >> >>> those highly demanded performance enhancements to get into Juno, >> >>> it's also "the right thing to do" (c). It's counterproductive to >> >>> put multiple vaguely related enhancements in single spec. This >> >>> would dim review focus and put us into position of getting >> >>> 'all-or-nothing'. We can't afford that. >> >>> >> >>> Let's leave one spec per enhancement. @Shihazhang, what do you >> >>> think? >> >>> >> >>> >> >>>> Also, I proposed the details of "2", trying to bring awareness >> >>>> on the topic, as I have been working with the scale lab in Red >> >>>> Hat to find and understand those issues, I have a very good >> >>>> knowledge of the problem and I believe I could make a very fast >> >>>> advance on the issue at the RPC level. >> >>> >> >>>> Given that, I'd like to work on this specific part, whether or >> >>>> not we split the specs, as it's something we believe critical >> >>>> for neutron scalability and thus, *nova parity*. >> >>> >> >>>> I will start a separate spec for "2", later on, if you find it >> >>>> ok, we keep them as separate ones, if you believe having just 1 >> >>>> spec (for 1 & 2) is going be safer for juno-* approval, then we >> >>>> can incorporate my spec in yours, but then >> >>>> "add-ipset-to-security" is not a good spec title to put all this >> >>>> together. >> >>> >> >>> >> >>>> Best regards, Miguel Ángel. >> >>> >> >>> >> >>>> On 07/02/2014 03:37 AM, shihanzhang wrote: >> >>>>> >> >>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my >> >>>>> spes, but I will also optimization the RPC from security group >> >>>>> agent to neutron server. Now the modle is >> >>>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1, >> >>>>> sg2..]', this can reduce the size of RPC respose message from >> >>>>> neutron server to security group agent. >> >>>>> >> >>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo" >> >>>>> <mangel...@redhat.com> wrote: >> >>>>>> >> >>>>>> >> >>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to >> >>>>>> work into this problem, I could file an specific spec for >> >>>>>> the security group RPC optimization, a masterplan in two >> >>>>>> steps: >> >>>>>> >> >>>>>> 1) Refactor the current RPC communication for >> >>>>>> security_groups_for_devices, which could be used for full >> >>>>>> syncs, etc.. >> >>>>>> >> >>>>>> 2) Benchmark && make use of a fanout queue per security >> >>>>>> group to make sure only the hosts with instances on a >> >>>>>> certain security group get the updates as they happen. >> >>>>>> >> >>>>>> @shihanzhang do you find it reasonable? >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> ----- Original Message ----- >> >>>>>>> ----- Original Message ----- >> >>>>>>>> @Nachi: Yes that could a good improvement to factorize >> >>>>>>>> the RPC >> >>>>>>> mechanism. >> >>>>>>>> >> >>>>>>>> Another idea: What about creating a RPC topic per >> >>>>>>>> security group (quid of the >> >>>>>>> RPC topic >> >>>>>>>> scalability) on which an agent subscribes if one of its >> >>>>>>>> ports is >> >>>>>>> associated >> >>>>>>>> to the security group? >> >>>>>>>> >> >>>>>>>> Regards, Édouard. >> >>>>>>>> >> >>>>>>>> >> >>> >> >>> >> >>>>>>> Hmm, Interesting, >> >>> >> >>>>>>> @Nachi, I'm not sure I fully understood: >> >>> >> >>> >> >>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] .. >> >>>>>>> port[SG_ID1, SG_ID2], port2 , port3 >> >>> >> >>> >> >>>>>>> Probably we may need to include also the SG_IP_LIST = >> >>>>>>> [SG_IP1, SG_IP2] ... >> >>> >> >>> >> >>>>>>> and let the agent do all the combination work. >> >>> >> >>>>>>> Something like this could make sense? >> >>> >> >>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....], >> >>>>>>> SG2:{IPs:[....],RULES:[....]} } >> >>> >> >>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... } >> >>> >> >>> >> >>>>>>> @Edouard, actually I like the idea of having the agent >> >>>>>>> subscribed to security groups they have ports on... That >> >>>>>>> would remove the need to include all the security groups >> >>>>>>> information on every call... >> >>> >> >>>>>>> But would need another call to get the full information of >> >>>>>>> a set of security groups at start/resync if we don't >> >>>>>>> already have any. >> >>> >> >>> >> >>>>>>>> >> >>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang < >> >>>>>>> ayshihanzh...@126.com > >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> hi Miguel Ángel, I am very agree with you about the >> >>>>>>>> following point: >> >>>>>>>>> * physical implementation on the hosts (ipsets, >> >>>>>>>>> nftables, ... ) >> >>>>>>>> --this can reduce the load of compute node. >> >>>>>>>>> * rpc communication mechanisms. >> >>>>>>>> -- this can reduce the load of neutron server can you >> >>>>>>>> help me to review my BP specs? >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" < >> >>>>>>> mangel...@redhat.com > >> >>>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>> Hi it's a very interesting topic, I was getting ready >> >>>>>>>>> to raise the same concerns about our security groups >> >>>>>>>>> implementation, >> >>>>>>> shihanzhang >> >>>>>>>>> thank you for starting this topic. >> >>>>>>>>> >> >>>>>>>>> Not only at low level where (with our default security >> >>>>>>>>> group rules -allow all incoming from 'default' sg- the >> >>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the >> >>>>>>>>> "security_group_rules_for_devices" rpc call from >> >>>>>>>>> ovs-agent to neutron-server grows to message sizes of >> >>>>>>>>>> 100MB, >> >>>>>>>>> generating serious scalability issues or >> >>>>>>>>> timeouts/retries that totally break neutron service. >> >>>>>>>>> >> >>>>>>>>> (example trace of that RPC call with a few instances >> >>>>>>>>> http://www.fpaste.org/104401/14008522/ ) >> >>>>>>>>> >> >>>>>>>>> I believe that we also need to review the RPC calling >> >>>>>>>>> mechanism for the OVS agent here, there are several >> >>>>>>>>> possible approaches to >> >>>>>>> breaking >> >>>>>>>>> down (or/and CIDR compressing) the information we >> >>>>>>>>> return via this >> >>>>>>> api >> >>>>>>>>> call. >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> So we have to look at two things here: >> >>>>>>>>> >> >>>>>>>>> * physical implementation on the hosts (ipsets, >> >>>>>>>>> nftables, ... ) * rpc communication mechanisms. >> >>>>>>>>> >> >>>>>>>>> Best regards, Miguel Ángel. >> >>>>>>>>> >> >>>>>>>>> ----- Mensaje original ----- >> >>>>>>>>> >> >>>>>>>>>> Do you though about nftables that will replace >> >>>>>>> {ip,ip6,arp,eb}tables? >> >>>>>>>>>> It also based on the rule set mechanism. The issue >> >>>>>>>>>> in that proposition, it's only stable since the >> >>>>>>>>>> begin >> >>>>>>> of the >> >>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros >> >>>>>>>>>> I don't list here (leverage iptables >> >>>>>>> limitation, >> >>>>>>>>>> efficient update rule, rule set, standardization of >> >>>>>>>>>> netfilter commands...). >> >>>>>>>>> >> >>>>>>>>>> Édouard. >> >>>>>>>>> >> >>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < >> >>>>>>>>>> henry4...@gmail.com > wrote: >> >>>>>>>>> >> >>>>>>>>>>> we have done some tests, but have different >> >>>>>>>>>>> result: the >> >>>>>>> performance is >> >>>>>>>>>>> nearly the same for empty and 5k rules in iptable, >> >>>>>>>>>>> but huge gap between enable/disable iptable hook >> >>>>>>>>>>> on linux bridge >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang < >> >>>>>>> ayshihanzh...@126.com >> >>>>>>>>>>>> >> >>>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>>>> Now I have not get accurate test data, but I can >> >>>>>>>>>>>> confirm the following points? >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM >> >>>>>>>>>>>> is liner, >> >>>>>>> iptable >> >>>>>>>>>>>> filter it one by one, if a VM in default >> >>>>>>>>>>>> security group and this default security group >> >>>>>>>>>>>> have many members, but ipset chain is set, the >> >>>>>>>>>>>> time >> >>>>>>> ipset >> >>>>>>>>>>>> filter one and many member is not much >> >>>>>>>>>>>> difference. >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>>> 2. when the iptable rule is very large, the >> >>>>>>>>>>>> probability of >> >>>>>>> failure >> >>>>>>>>>>>> that iptable-save save the iptable rule is very >> >>>>>>>>>>>> large. >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" < >> >>>>>>>>>>>> blak...@gmail.com >> >>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>>>>> This sounds like a good idea to handle some of >> >>>>>>>>>>>>> the >> >>>>>>> performance >> >>>>>>>>>>>>> issues until the ovs firewall can be >> >>>>>>>>>>>>> implemented down the the line. >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>>>> Do you have any performance comparisons? >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" < >> >>>>>>> ayshihanzh...@126.com > >> >>>>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>>>>>> Hello all, >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>>>>>> Now in neutron, it use iptable implementing >> >>>>>>>>>>>>>> security >> >>>>>>> group, but >> >>>>>>>>>>>>>> the performance of this implementation is >> >>>>>>>>>>>>>> very poor, there >> >>>>>>> is a bug: >> >>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272 >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>to >> >>>>>>> reflect this >> >>>>>>>>>>>>>> problem. In his test, w ith default security >> >>>>>>>>>>>>>> groups(which has remote security group), >> >>>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k >> >>>>>>>>>>>>>> Iptable rules >> >>>>>>> on evry >> >>>>>>>>>>>>>> compute node, although his patch can reduce >> >>>>>>>>>>>>>> the processing time, but >> >>>>>>> it don't >> >>>>>>>>>>>>>> solve this problem fundamentally. I have >> >>>>>>>>>>>>>> commit a BP to solve this >> >>>>>>> problem: >> >>>>>>>>>>>>>> >> >>>>>>> >> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security >> >>> >> >>>>>>> >> >>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>>>>> There are other people interested in this >> >>>>>>>>>>>>>> it? >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>>>>>> _______________________________________________ >> >>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>>>>> OpenStack-dev mailing list >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>>>>> OpenStack-dev@lists.openstack.org >> > > > >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>> >> >>>>>>> >> >>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>>>> _______________________________________________ >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>>> OpenStack-dev mailing list >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>>>> OpenStack-dev@lists.openstack.org >> > >> >>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>>>>>> >> >>>>>>> >> >>>> >> >>>>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>>> _______________________________________________ >> >>>>>>>>>> >> >>>>>>>>>>> OpenStack-dev mailing list >> >>>>>>>>>> >> >>>>>>>>>>> OpenStack-dev@lists.openstack.org >> >> >>>>>>>>>>> >> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>>>>>> >> >>>>>>> >> >>>> >> >>>>>>>>> >> >>>>>>>>>> _______________________________________________ >> >>>>>>>>>> OpenStack-dev mailing list >> >>>>>>>>>> OpenStack-dev@lists.openstack.org >> >> >>>>>>>>>> >> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>>>>>> >> >>>>>>> >> >>> >> >>>>>>>>> _______________________________________________ >> >>>>>>>>> OpenStack-dev mailing list >> >>>>>>>>> OpenStack-dev@lists.openstack.org > >> >>>>>>>>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>> >> >>>>>>>>> >> >>>>>>>>> >> >>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> _______________________________________________ >> >>>>>>>> OpenStack-dev mailing list >> >>>>>>>> OpenStack-dev@lists.openstack.org >> >>>>>>>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>> >> >>>>>>>> >> >>>>>>>> >> >>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> _______________________________________________ >> >>>>>>>> OpenStack-dev mailing list >> >>>>>>>> OpenStack-dev@lists.openstack.org >> >>>>>>>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>> >> >>>>>>>> >> >>>>>>>> >> >>> >> >>> >> >>>>>>> _______________________________________________ >> >>>>>>> OpenStack-dev mailing list >> >>>>>>> OpenStack-dev@lists.openstack.org >> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>> >> >>> _______________________________________________ >> >>>>>> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>>> >> >>> >> >>>>>> >> >>_______________________________________________ >> >>>>> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>>>> >> >>> >> >>>>> >> >>>>> >> >>>> _______________________________________________ OpenStack-dev >> >>>> mailing list OpenStack-dev@lists.openstack.org >> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>>>>> >> >>>>>>_______________________________________________ >> >>>>>>OpenStack-dev >> >>>> >> >>mailing list >> >>>>>> OpenStack-dev@lists.openstack.org >> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>>> >> >>_______________________________________________ >> >>>>> OpenStack-dev mailing list OpenStack-dev@lists.openstack.org >> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>>>> >> >>>> >> >>>>_______________________________________________ >> >>>>OpenStack-dev >> >>>>> >> >>mailing list >> >>>> OpenStack-dev@lists.openstack.org >> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>> >> >>>> >> >>> >> >>> >> >>> >> >>> >> >>> _______________________________________________ OpenStack-dev >> >>> mailing list OpenStack-dev@lists.openstack.org >> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>> >> >>-----BEGIN PGP SIGNATURE----- >> >>Version: GnuPG/MacGPG2 v2.0.22 (Darwin) >> >>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >> >> >> >>iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR >> >>IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE >> >>WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros >> >>7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23 >> >>1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di >> >>/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc= >> >>=TxAT >> >>-----END PGP SIGNATURE----- >> >> >> >>_______________________________________________ >> >>OpenStack-dev mailing list >> >>OpenStack-dev@lists.openstack.org >> >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > >> > >> > >> > _______________________________________________ >> > OpenStack-dev mailing list >> > OpenStack-dev@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > > -- > Best wishes! > Baohua > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
