Question on virtualenv (python) and FreeBSD
Hey folks, First off, thank you all very much for all the work being put into the Python ports infrastructure. We are using it extensively in our project and it's been a huge boon to us. I do have a question about how Python is setup on FreeBSD by default and it has to do with "virtualenv". http://docs.python-guide.org/en/latest/dev/virtualenvs/ Virtualenv allows you to make almost a python "chroot" to keep all your dependencies together so that tracking your deps is very easy. It's very much like Ruby's rvm. From the site: > A Virtual Environment, put simply, is an isolated working copy of Python which allows you to work on a specific project without worry of affecting other projects. > > For example, you can work on a project which requires Django 1.3 while also maintaining a project which requires Django 1.0. It's very cool and useful for allowing multiple apps to exist on a box with different deps. The problem I came across is that it seems "broken" if you are coming from OS X or Linux. Specifically it seems like FreeBSD is shipping a relatively stripped down version of Python that makes using virtualenv not work. ~/gitbridge/app % python db_repo/manage.py version_control sqlite:///db.db db_repo Traceback (most recent call last): File "db_repo/manage.py", line 5, in main(debug='False') File "/home/alfred/gitbridge/lib/python2.7/site-packages/migrate/versioning/shell.py", line 207, in main ret = command_func(**kwargs) File "", line 2, in version_control File "/home/alfred/gitbridge/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 155, in with_engine engine = construct_engine(url, **kw) File "/home/alfred/gitbridge/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 140, in construct_engine return create_engine(engine, **kwargs) File "/home/alfred/gitbridge/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 344, in create_engine return strategy.create(*args, **kwargs) File "/home/alfred/gitbridge/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 73, in create dbapi =dialect_cls.dbapi(**dbapi_args) File "/home/alfred/gitbridge/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py", line 297, in dbapi raise e ImportError: No module named pysqlite2 I have details and a workaround on my blog here (less messy than pasting the entire 200+ lines of info into this message): http://splbio.wordpress.com/2014/04/19/using-djangosqlalchemy-and-virtualenv-on-freebsd/ Basically it seems that because we don't by default ship sqlite with Python we wind up with this odd situation where the virtualenv'd Python can't be used for the apps that I tend to use. I'm wondering can something be done to make virtualenv work better out of the box for people? -Alfred ___ freebsd-python@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-python To unsubscribe, send any mail to "freebsd-python-unsubscr...@freebsd.org"
Re: Question on virtualenv (python) and FreeBSD
On, Sat May 31, 2014, Alfred Perlstein wrote: [...] > Specifically it seems like FreeBSD is shipping a relatively stripped > down version of Python that makes using virtualenv not work. Please be precise here. It does not work for your specific use case. > > ~/gitbridge/app % python db_repo/manage.py version_control sqlite:///db.db > > db_repo > > Traceback (most recent call last): > > File "db_repo/manage.py", line 5, in > > main(debug='False') > > File > > "/home/alfred/gitbridge/lib/python2.7/site-packages/migrate/versioning/shell.py", > > line 207, in main > > ret = command_func(**kwargs) > > File "", line 2, in version_control > > File > > "/home/alfred/gitbridge/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", > > line 155, in with_engine > > engine = construct_engine(url, **kw) > > File > > "/home/alfred/gitbridge/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", > > line 140, in construct_engine > > return create_engine(engine, **kwargs) > > File > > "/home/alfred/gitbridge/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", > > line 344, in create_engine > > return strategy.create(*args, **kwargs) > > File > > "/home/alfred/gitbridge/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", > > line 73, in create > > dbapi =dialect_cls.dbapi(**dbapi_args) > > File > > "/home/alfred/gitbridge/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py", > > line 297, in dbapi > > raise e > > ImportError: No module named pysqlite2 > > > > I have details and a workaround on my blog here (less messy than pasting > the entire 200+ lines of info into this message): > http://splbio.wordpress.com/2014/04/19/using-djangosqlalchemy-and-virtualenv-on-freebsd/ > > Basically it seems that because we don't by default ship sqlite with > Python we wind up with this odd situation where the virtualenv'd Python > can't be used for the apps that I tend to use. > > I'm wondering can something be done to make virtualenv work better out > of the box for people? Shouldn't that be rephrased to "make virtualenv work better out of the box with my applications requiring package X?" Sorry for nit-picking here. We strip certain parts of a Python default installation, which others might or might not do. One could create a FAQ, what to watch or which additional packages are recommended for installation. Right now, I do not see a huge issue here, though. If this is a problem that someone sees as more critical for users, we should rethink the customized Python installation process, which however also means that users would have to install more dependencies, when they just want a simple Python interpreter. Cheers Marcus pgpkeYPI1R7dt.pgp Description: PGP signature
Re: Question on virtualenv (python) and FreeBSD
On 5/31/14 12:50 PM, Marcus von Appen wrote: On, Sat May 31, 2014, Alfred Perlstein wrote: I have details and a workaround on my blog here (less messy than pasting the entire 200+ lines of info into this message): http://splbio.wordpress.com/2014/04/19/using-djangosqlalchemy-and-virtualenv-on-freebsd/ Basically it seems that because we don't by default ship sqlite with Python we wind up with this odd situation where the virtualenv'd Python can't be used for the apps that I tend to use. I'm wondering can something be done to make virtualenv work better out of the box for people? Shouldn't that be rephrased to "make virtualenv work better out of the box with my applications requiring package X?" Sorry for nit-picking here. We strip certain parts of a Python default installation, which others might or might not do. One could create a FAQ, what to watch or which additional packages are recommended for installation. Right now, I do not see a huge issue here, though. If this is a problem that someone sees as more critical for users, we should rethink the customized Python installation process, which however also means that users would have to install more dependencies, when they just want a simple Python interpreter. No problem with nit-picking here. In the opposite direction I wouldn't say "my", I would say "the majority of people using python in web stacks" which is huge these days. Basically the suggested way of setting up django and sqlalchemy does not work via the HOWTOs for Linux and OS X that are out there without what appears to be somewhat hard to figure out commands to "fixup" the virtualenv install. As far as a huge issue, I actually see one, the recommended way to do apps these days is using these virtual containers, that being virtualenv under python and rvm under ruby. In fact it's the best way to make your apps easily distributable across your org. At the end of the day taking a look at what other OS's are doing and how the HOWTOs are structured would go a long way to getting FreeBSD more traction in the webstack community, this is not only my opnion, but the opinion of the younger people on my team who suggested that I bring this up. They, the younger devs on my team would not have the time nor inclination to debug this and would simply say "lol, I guess it's time to download ubuntu where this howto will work". For them the OS is a tool that should just work, not really what they are all that interested in for day to day. The fact that I get to hack on FreeBSD is a product of FreeBSD being a great platform for these python apps. Sorry for the harsh words, it's not intended, I'm just not sure how to otherwise express the need here for us to support these stacks out of the box. A few questions, but feel free to comments on all: 1) Does the concept of a "python-lite" port make sense where the default is a larger subset of the default python packages you get on other machines? -Or- 2) perhaps we should simply just add sqlite as that seems to be the only blocker I've seen thus far to get this working for the two important web stacks? 3) do we care about making it out-of-the-box easy for people running web services/stacks like django and sqlalchemy+flask on FreeBSD in the recommended fashion? 3) other? thank you Marcus. -Alfred ___ freebsd-python@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-python To unsubscribe, send any mail to "freebsd-python-unsubscr...@freebsd.org"
Re: Question on virtualenv (python) and FreeBSD
On, Sat May 31, 2014, Alfred Perlstein wrote: > On 5/31/14 12:50 PM, Marcus von Appen wrote: > > On, Sat May 31, 2014, Alfred Perlstein wrote: > > > >> I have details and a workaround on my blog here (less messy than pasting > >> the entire 200+ lines of info into this message): > >> http://splbio.wordpress.com/2014/04/19/using-djangosqlalchemy-and-virtualenv-on-freebsd/ > >> > >> Basically it seems that because we don't by default ship sqlite with > >> Python we wind up with this odd situation where the virtualenv'd Python > >> can't be used for the apps that I tend to use. > >> > >> I'm wondering can something be done to make virtualenv work better out > >> of the box for people? > > Shouldn't that be rephrased to "make virtualenv work better out of the box > > with my applications requiring package X?" > > > > Sorry for nit-picking here. We strip certain parts of a Python default > > installation, which others might or might not do. One could create a FAQ, > > what > > to watch or which additional packages are recommended for installation. > > Right > > now, I do not see a huge issue here, though. > > > > If this is a problem that someone sees as more critical for users, we should > > rethink the customized Python installation process, which however also means > > that users would have to install more dependencies, when they just want a > > simple Python interpreter. > > No problem with nit-picking here. In the opposite direction I wouldn't > say "my", I would say "the majority of people using python in web > stacks" which is huge these days. Basically the suggested way of > setting up django and sqlalchemy does not work via the HOWTOs for Linux > and OS X that are out there without what appears to be somewhat hard to > figure out commands to "fixup" the virtualenv install. They target Linux and OS X, not BSD and not Windows, either ;-). > As far as a huge issue, I actually see one, the recommended way to do > apps these days is using these virtual containers, that being virtualenv > under python and rvm under ruby. > > In fact it's the best way to make your apps easily distributable across > your org. This depends on the org and distribution method. > At the end of the day taking a look at what other OS's are doing and how > the HOWTOs are structured would go a long way to getting FreeBSD more > traction in the webstack community, this is not only my opnion, but the > opinion of the younger people on my team who suggested that I bring this up. > > They, the younger devs on my team would not have the time nor > inclination to debug this and would simply say "lol, I guess it's time > to download ubuntu where this howto will work". For them the OS is a I could write a lot in response to that... > tool that should just work, not really what they are all that interested > in for day to day. The fact that I get to hack on FreeBSD is a product > of FreeBSD being a great platform for these python apps. The problem is not about the OS, but about a *tool* they want to use. That's a huge difference, when it comes to maintenance. Our py-sqlite3 installation depends on databases/sqlite3, which is highly configurable and, depending on the purpose of the host and other applications, may need special care. Given your venv approach, the admin might update sqlite3 in a slightly different way, effectively breaking your app - since you lack control over it, even with venv (I won't mention installing django from ports, since you know that way well enough and probably evaluated the pros and cons against venv and your production and dev environments already). > Sorry for the harsh words, it's not intended, I'm just not sure how to > otherwise express the need here for us to support these stacks out of > the box. Create a toolchain for them, which deals with all necessary bits, e.g. a shell script, doing something like the following: 1) check for pythonXX and py-sqlite3 being installed on the host 2) create venv 3) copy/link the py-sqlite3 bindings into venv (or adjust venv's path), if necessary > A few questions, but feel free to comments on all: > 1) Does the concept of a "python-lite" port make sense where the default > is a larger subset of the default python packages you get on other > machines? We have the python-lite port already as default, if one may say so. sqlite3, gdbm, tkinter and bsdbb are stripped from it. How about adjusting your server or dev host installations and to add sqlite3 and friends to the installation? Would that be sufficient or otherwise problematic? > -Or- > 2) perhaps we should simply just add sqlite as that seems to be the only > blocker I've seen thus far to get this working for the two important web > stacks? See above for the dependency chain. I see a risk here for pythonXX to be updated on every sqlite update. Is that really necessary, if we only speak about a single port to be installed on the host? > 3) do we care about making it out-of-the-box easy for people ru
Re: Question on virtualenv (python) and FreeBSD
On 5/31/14 3:51 PM, Marcus von Appen wrote: On, Sat May 31, 2014, Alfred Perlstein wrote: 3) other? thank you Marcus. I won't block the process in either direction here, since I am generally pretty indifferent about pulling in sqlite3 as dependency for lang/python. It is wide-spread enough that it would end up on a majority of web application systems anyway (although most large-scale web apps probably will use a scalable database back-end, such as mysql, pg or oracle). Which brings me to the target group for your request: it'd be (mostly) developers, not? Which would bring me to the toolchain script and FAQ and responsibility over the dev machines again :-). [1] https://docs.djangoproject.com/en/dev/intro/install/ Sure, that's how to install it onto the host when using a single version of django AND it requires root for that. Virtualenv (once installed) lets people install almost all packages as non-root and leaves the worry off of the sysadmin. If you google "django virtualenv" you'll get this list of links: http://www.jeffknupp.com/blog/2012/02/09/starting-a-django-project-the-right-way/ http://thecodeship.com/deployment/deploy-django-apache-virtualenv-and-mod_wsgi/ https://docs.djangoproject.com/en/1.6/topics/install/ : "(optional) Take a look at virtualenv and virtualenvwrapper. These tools provide isolated Python environments, which are more practical than installing packages systemwide. They also allow installing packages without administrator privileges. It’s up to you to decide if you want to learn and use them." http://www.tonido.com/blog/index.php/2013/11/25/working-with-virtualenv-on-django-projects/#.U4phVZRdX2g https://community.webfaction.com/questions/4253/simple-virtualenv-usage-with-django I really believe the only blocker is sqlite right now. What I really want is people to be able to use virtualenv without having to switch OS, because the method I recommend is way complex. Both python and ruby teams are heavily recommending these "virtual environments" we should support it out of the box, particularly since the change seems very light. -Alfred ___ freebsd-python@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-python To unsubscribe, send any mail to "freebsd-python-unsubscr...@freebsd.org"
