- Original Message -
> From: "Robert Collins"
> To: "OpenStack Development Mailing List (not for usage questions)"
>
> Sent: Tuesday, May 12, 2015 3:06:21 AM
> Subject: Re: [openstack-dev] [all] Replace mysql-python with mysqlclient
>
>
On 5/11/15 9:17 PM, Robert Collins wrote:
On 12 May 2015 at 10:44, Mike Bayer wrote:
What we have today in our standard architecture for OpenStack is
optimised for IO bound workloads: waiting on the
network/subprocesses/disk/libvirt etc. Running high numbers of
eventlet handlers in a single
>Arguably if we're actually seeing performance issues then it's not a
distraction but rather a real problem that needs fixing.
The important take-away from the thread is that we aren't anywhere near
hitting python limits though. Our main bottleneck is due to the fact that
we are serializing all DB
On 05/11/2015 08:22 PM, Jay Pipes wrote:
c) Many OpenStack services, including Nova, Cinder, and Neutron, when looked at
from a thousand-foot level, are little more than glue code that pipes out to a
shell to execute system commands (sorry, but it's true).
No apologies necessary. :)
So, botto
On 05/11/2015 09:02 PM, Dieterly, Deklan wrote:
Given Python’s inherent inability to scale (GIL) relative to other
languages/platforms, have there been any serious discussions on
allowing other more scalable languages into the OpenStack ecosystem
when concurrency/scalability is paramount?
Rober
On 12 May 2015 at 13:02, Dieterly, Deklan wrote:
> Given Python’s inherent inability to scale (GIL) relative to other
> languages/platforms, have there been any serious discussions on allowing
> other more scalable languages into the OpenStack ecosystem when
> concurrency/scalability is paramou
On 12 May 2015 at 11:35, Clint Byrum wrote:
> Excerpts from Mike Bayer's message of 2015-05-11 15:44:30 -0700:
> Anyway, there is additional thought that might change the decision
> a bit. There is one "pro" to changing to use pymsql vs. changing to
> use threads, and that is that it isolates the
On 12 May 2015 at 10:44, Mike Bayer wrote:
>> What we have today in our standard architecture for OpenStack is
>> optimised for IO bound workloads: waiting on the
>> network/subprocesses/disk/libvirt etc. Running high numbers of
>> eventlet handlers in a single process only works when the majorit
On Tue, 12 May 2015 at 05:08 Mike Bayer wrote:
>
> On 5/11/15 2:02 PM, Attila Fazekas wrote:
> > The scary part of a blocking I/O call is when you have two
> > python thread (or green thread) and one of them is holding a DB lock the
> other
> > is waiting for the same lock in a native blocking I/O
On 12 May 2015 at 10:12, Attila Fazekas wrote:
>
>
>
>
>> If you can illustrate a test script that demonstrates the actual failing
>> of OS threads that does not occur greenlets here, that would make it
>> immediately apparent what it is you're getting at here.
>>
>
> http://www.fpaste.org/220824
pment Mailing List (not for usage questions)"
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [all] Replace mysql-python with mysqlclient
> All production Openstack applications today are fully serialized to only be
> able to emit a single query to
> All production Openstack applications today are fully serialized to only
be able to emit a single query to the database at a time;
True. That's why any deployment configures tons (tens) of workers of any
significant service.
> When I talk about moving to threads, this is not a "won't help or hu
Mike,
Thank you for saying all that you said above.
Best regards,
Boris Pavlovic
On Tue, May 12, 2015 at 2:35 AM, Clint Byrum wrote:
> Excerpts from Mike Bayer's message of 2015-05-11 15:44:30 -0700:
> >
> > On 5/11/15 5:25 PM, Robert Collins wrote:
> > >
> > > Details: Skip over this bit if y
Excerpts from Mike Bayer's message of 2015-05-11 15:44:30 -0700:
>
> On 5/11/15 5:25 PM, Robert Collins wrote:
> >
> > Details: Skip over this bit if you know it all already.
> >
> > The GIL plays a big factor here: if you want to scale the amount of
> > CPU available to a Python service, you have
On 5/11/15 5:25 PM, Robert Collins wrote:
Details: Skip over this bit if you know it all already.
The GIL plays a big factor here: if you want to scale the amount of
CPU available to a Python service, you have two routes:
A) move work to a different process through some RPC - be that DB's
usi
- Original Message -
> From: "Mike Bayer"
> To: openstack-dev@lists.openstack.org
> Sent: Monday, May 11, 2015 9:07:13 PM
> Subject: Re: [openstack-dev] [all] Replace mysql-python with mysqlclient
>
>
>
> On 5/11/15 2:02 PM, Attila Fazekas
On 10 May 2015 at 03:26, John Garbutt wrote:
> On 9 May 2015 at 15:02, Mike Bayer wrote:
>> On 5/9/15 6:45 AM, John Garbutt wrote:
>>>
>>> I am leaning towards us moving to making DB calls with a thread pool and
>>> some fast C based library, so we get the 'best' performance. Is that a crazy
>>>
On 5/11/15 2:02 PM, Attila Fazekas wrote:
Not just with local database connections,
the 10G network itself also fast. Is is possible you spend more time even on
the kernel side tcp/ip stack (and the context switch..) (Not in physical I/O
wait)
than in the actual work on the DB side. (Check ne
- Original Message -
> From: "Mike Bayer"
> To: openstack-dev@lists.openstack.org
> Sent: Monday, May 11, 2015 4:44:58 PM
> Subject: Re: [openstack-dev] [all] Replace mysql-python with mysqlclient
>
>
>
> On 5/11/15 9:58 AM, Attila Fazekas wro
On 5/11/15 9:58 AM, Attila Fazekas wrote:
- Original Message -
From: "John Garbutt"
To: "OpenStack Development Mailing List (not for usage questions)"
Cc: "Dan Smith"
Sent: Saturday, May 9, 2015 12:45:26 PM
Subject: Re: [openstack-dev] [
- Original Message -
> From: "John Garbutt"
> To: "OpenStack Development Mailing List (not for usage questions)"
>
> Cc: "Dan Smith"
> Sent: Saturday, May 9, 2015 12:45:26 PM
> Subject: Re: [openstack-dev] [all] Replace mysql-python
On 9 May 2015 at 15:02, Mike Bayer wrote:
> On 5/9/15 6:45 AM, John Garbutt wrote:
>>
>> I am leaning towards us moving to making DB calls with a thread pool and
>> some fast C based library, so we get the 'best' performance. Is that a crazy
>> thing to be thinking? What am I missing here? Thanks,
On 5/9/15 6:45 AM, John Garbutt wrote:
I am leaning towards us moving to making DB calls with a thread pool
and some fast C based library, so we get the 'best' performance. Is
that a crazy thing to be thinking? What am I missing here? Thanks, John
I'd like to do that but I want the whole Op
On 30 April 2015 at 18:54, Mike Bayer wrote:
> On 4/30/15 11:16 AM, Dan Smith wrote:
>>> There is an open discussion to replace mysql-python with PyMySQL, but
>>> PyMySQL has worse performance:
>>>
>>> https://wiki.openstack.org/wiki/PyMySQL_evaluation
>>
>> My major concern with not moving to som
On 5/8/15 10:41 AM, Ronald Bradford wrote:
I guess I may have spoken too soon.
https://wiki.openstack.org/wiki/PyMySQL_evaluation states " Oracle
refuses to publish MySQL-connector-Python on Pypi, which is critical
to the Openstack infrastructure".
I am unclear when this statement was made
If that could get published, please do make it happen!
As for as who tried to contact oracle, and never got a response, I am
not sure about that question (or answer). But if we can get that to
happen it would be great for the whole python community (IMHO).
-Josh
Ronald Bradford wrote:
I gue
Excerpts from Ronald Bradford's message of 2015-05-08 10:41:30 -0400:
> I guess I may have spoken too soon.
> https://wiki.openstack.org/wiki/PyMySQL_evaluation states " Oracle refuses
> to publish MySQL-connector-Python on Pypi, which is critical to the
> Openstack infrastructure".
>
> I am uncl
I guess I may have spoken too soon.
https://wiki.openstack.org/wiki/PyMySQL_evaluation states " Oracle refuses
to publish MySQL-connector-Python on Pypi, which is critical to the
Openstack infrastructure".
I am unclear when this statement was made and who is involved in this
discussion. As I hav
Has anybody considered the native python connector for MySQL that supports
Python 3.
Here are the Ubuntu Packages.
$ apt-get show python-mysql.connector
E: Invalid operation show
rbradfor@rubble:~$ apt-cache show python-mysql.connector
Package: python-mysql.connector
Priority: optional
Section:
On 5/7/15 5:32 PM, Thomas Goirand wrote:
If there are really fixes and features we
need in Py2K then of course we have to either convince MySQLdb to merge
them or switch to mysqlclient.
Given the "no reply in 6 months" I think that's enough to say it:
mysql-python is a dangerous package wit
On 05/05/2015 08:41 PM, Mike Bayer wrote:
On 5/4/15 6:48 PM, Thomas Goirand wrote:
I don't see what it would break. If I do:
Package: python-mysqlclient
Breaks: python-mysqldb
Replaces: python-mysqldb
Provides: python-mysqldb
everything is fine, and python-mysqlclient becomes another
imple
On 05/05/2015 09:56 PM, Mike Bayer wrote:
Having two packages that both install into the same name is the least
ideal arrangement
From your point of view, and for testing against both, certainly. But
for a distribution, avoiding dot have 2 packages clashing each other and
deciding on only a
On 5/5/15 3:07 PM, Mike Bayer wrote:
On 5/5/15 1:11 PM, Thomas Goirand wrote:
On 04/30/2015 05:00 PM, Victor Stinner wrote:
Hi,
I propose to replace mysql-python with mysqlclient in OpenStack
applications to get Python 3 support, bug fixes and some new
features (support MariaDB's libmy
On 5/5/15 1:11 PM, Thomas Goirand wrote:
On 04/30/2015 05:00 PM, Victor Stinner wrote:
Hi,
I propose to replace mysql-python with mysqlclient in OpenStack
applications to get Python 3 support, bug fixes and some new features
(support MariaDB's libmysqlclient.so, support microsecond in TIM
On 5/4/15 6:48 PM, Thomas Goirand wrote:
I don't see what it would break. If I do:
Package: python-mysqlclient
Breaks: python-mysqldb
Replaces: python-mysqldb
Provides: python-mysqldb
everything is fine, and python-mysqlclient becomes another
implementation of the same thing. Then I believe
On 04/30/2015 05:00 PM, Victor Stinner wrote:
Hi,
I propose to replace mysql-python with mysqlclient in OpenStack applications to
get Python 3 support, bug fixes and some new features (support MariaDB's
libmysqlclient.so, support microsecond in TIME column).
In fact, when looking at the py
On 04/30/2015 07:48 PM, Mike Bayer wrote:
On 4/30/15 11:00 AM, Victor Stinner wrote:
Hi,
I propose to replace mysql-python with mysqlclient in OpenStack
applications to get Python 3 support, bug fixes and some new features
(support MariaDB's libmysqlclient.so, support microsecond in TIME
co
> I propose to replace mysql-python with mysqlclient in OpenStack applications
> to get Python 3 support, bug fixes and some new features (support MariaDB's
> libmysqlclient.so, support microsecond in TIME column).
I just proposed a change to add mysqlclient dependency to global requirements:
Hi,
Mike Bayer wrote:
> It is not feasible to use MySQLclient in Python 2 because it uses the
> same module name as Python-MySQL, and would wreak havoc with distro
> packaging and many other things.
IMO mysqlclient is just the new upstream for MySQL-Python, since MySQL-Python
is no more maintain
On 4/30/15 11:16 AM, Dan Smith wrote:
There is an open discussion to replace mysql-python with PyMySQL, but
PyMySQL has worse performance:
https://wiki.openstack.org/wiki/PyMySQL_evaluation
My major concern with not moving to something different (i.e. not based
on the C library) is the thread
On 4/30/15 11:00 AM, Victor Stinner wrote:
Hi,
I propose to replace mysql-python with mysqlclient in OpenStack applications to
get Python 3 support, bug fixes and some new features (support MariaDB's
libmysqlclient.so, support microsecond in TIME column).
It is not feasible to use MySQLcli
> If we're going to move, shouldn't we be looking at something that
> supports our threading model?
I would prefer to make baby steps, and first fix the Python 3 compatibility.
Enhance concurrency/parallelism is a much more complex project than just
replacing a single line in dependencies ;-)
S
> There is an open discussion to replace mysql-python with PyMySQL, but
> PyMySQL has worse performance:
>
> https://wiki.openstack.org/wiki/PyMySQL_evaluation
My major concern with not moving to something different (i.e. not based
on the C library) is the threading problem. Especially as we move
Hi,
I propose to replace mysql-python with mysqlclient in OpenStack applications to
get Python 3 support, bug fixes and some new features (support MariaDB's
libmysqlclient.so, support microsecond in TIME column).
The MySQL database is popular, but the Python driver mysql-python doesn't look
to
44 matches
Mail list logo
