If you are new to virtualenv, Fabric or pip is, Alex Clemesha’s excellent “Tools of the Modern Python Hacker” is a must-read.
In short: virtualenv lets you switch seamlessly between isolated Python environments, Fabric automates remote deployment, while pip takes care of installing required packages and dependencies. If you have ever had to wrestle with more than one development project at the same time, then virtualenv is one of those tools that, once mastered, you can’t see yourself living without. Fabric and pip are somewhat immature, but still highly useful in their present shapes. It is likely that you will end up learning them anyway. Best of all, these three tools play very nicely together.
Except on Cygwin.
Here at Atbrox, we spend quite a lot of our time on Windows platforms. While Cygwin adds a fair amount of unix functionality to Windows, configuring certain applications can be difficult. This article describes the steps we go through to get an operational virtualenv, Fabric and pip setup on Windows Vista. It also gives you a brief taster of how virtualenv and Fabric works.
Step 1 – Install Cygwin: If you haven’t already, Cygwin can be installed from this page. Click the “View” button once to get a full list of available packages. Make sure to include at least the following packages (the numbers in the parentheses indicate the versions used at the time of writing):
- python (2.5.2-1)
- python-paramiko (1.7.4-1)
- python-crypto (2.0.1-1)
- gcc (3.4.4-999)
- wget (1.11.4-3)
- openssh (5.1p1-10)
Now would also be a good time to install other common packages such as vim, git, etc.—but you can always go back and install them at a later time.
Note that we are using Cygwin Python rather than the standard Windows Python. I had nothing but trouble trying to get Windows Python to play nicely along with virtualenv and Fabric, so this is a compromise. The downside is that you are stuck with a rather dated and somewhat buggy version of Python. If someone manages to get this setup working with Windows Python, then let me know!
Step 2 – Get paramiko working: The python-paramiko and python-crypto packages are required to get Fabric deployment over SSH working properly. If you are lucky, paramiko should work out of the box. If you don’t get the following error message when importing paramiko then skip the rest of this step:
$ python Python 2.5.2 (r252:60911, Dec 2 2008, 09:26:14) [GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin Type "help", "copyright", "credits" or "license" for more information. >>> import paramiko Traceback (most recent call last): File "<stdin>", line 1, in <module> File "__init__.py", line 69, in <module> File "transport.py", line 32, in <module> File "util.py", line 31, in <module> File "common.py", line 101, in <module> File "rng.py", line 69, in __init__ File "randpool.py", line 87, in __init__ File "randpool.py", line 120, in _randomize IOError: [Errno 0] Error
According to the discussion here, this appears to be a lingering Cygwin bug. The workaround is to change line 120 in /usr/lib/python2.5/site-packages/Crypto/Util/randpool.py
from
if num!=2 : raise IOError, (num, msg)
to
if num!=2 and num!=0 : raise IOError, (num, msg)
Paramiko should now import without any complaints.
Step 3 – Install setuptools: Setuptools are required for installing the rest of the required Python packages. Instructions for Cygwin are found on the setuptools pages—but just enter the following and you’ll be all set:
$ wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg $ sh setuptools-0.6c9-py2.5.egg
Step 4 – Install pip, virtualenv and virtualenvwrapper: We haven’t said anything about virtualenvwrapper so far. This extension to virtualenv streamlines working with multiple environments and is well recommended:
$ easy_install pip $ easy_install virtualenv $ easy_install virtualenvwrapper $ mkdir ~/.virtualenvs
That last line creates a working directory for your virtual Python environments. When e.g. working with an environment named myenv
, all packages will be installed in ~/.virtualenvs/myenv
.
I find it useful to create and activate a default environment called sandbox
. This helps prevent package installations to the default Python site-packages
. It’s a good strategy in general to avoid polluting the main package directory so that almost all package installations are per project and virtual environment. Run the following commands to create the sandbox
environment:
$ export WORKON_HOME=$HOME/.virtualenvs $ export PIP_VIRTUALENV_BASE=$WORKON_HOME $ source /usr/bin/virtualenvwrapper_bashrc $ mkvirtualenv sandbox
mkvirtualenv
is a virtualenvwrapper command that creates the given environment. If you get an IOError: [Errno 2] No such file or directory: '/usr/local/bin/python2.5'
you will have to add a symbolic link to the Python executable:
$ ln -s /usr/bin/python2.5.exe /usr/bin/python2.5
Note that whenever you execute a shell command, the bash prompt will remind you of the active environment:
$ echo "foo" foo (sandbox)
To make the sandbox activation permanent, append the following lines to your ~/.bashrc
:
export WORKON_HOME=$HOME/.virtualenvs export PIP_VIRTUALENV_BASE=$WORKON_HOME source /usr/bin/virtualenvwrapper_bashrc workon sandbox
The workon
is another virtualenvwrapper extension that switches you to the given environment. To get a full list of available environments, type workon
without an argument. Other useful commands are deactivate
to step out of the currently active environment, and rmvirtualenv
to delete an environment. Refer to the virtualenvwrapper documentation for the whole story.
As a sanity check, try exiting and restarting the Cygwin shell. If you have paid attention so far, you should now automatically end up in the sandbox
environment.
Step 5 – Install Fabric: From this point and on, all installed packages, including Fabric, will end up in a virtual environment. Fabric is undergoing a major rewrite right now, so given that its interface is quite unstable it is preferable to have a per-project installation anyway.
First we create a test environment named myproject
:
$ mkvirtualenv myproject
We have to make some modifications to the Fabric source code, so we can’t use pip for installing it. Make sure to use version 0.9 or higher, as version 0.1 is already quite outdated:
$ mkdir ~/tmp $ cd ~/tmp $ wget http://git.fabfile.org/cgit.cgi/fabric/snapshot/fabric-0.9b1.tar.gz $ tar xzf fabric-0.9b1.tar.gz $ cd fabric-0.9b1
Fabric is run using the fab
command, but if we try to install it as is, the following error might show up:
$ fab Traceback (most recent call last): File "/home/brox/.virtualenvs/myproject/bin/fab", line 8, in <module> load_entry_point('Fabric==0.1.1', 'console_scripts', 'fab')() File "/home/brox/.virtualenvs/myproject/lib/python2.5/site-packages/setuptools -0.6c9-py2.5.egg/pkg_resources.py", line 277, in load_entry_point File "/home/brox/.virtualenvs/myproject/lib/python2.5/site-packages/setuptools -0.6c9-py2.5.egg/pkg_resources.py", line 2180, in load_entry_point File "/home/brox/.virtualenvs/myproject/lib/python2.5/site-packages/setuptools -0.6c9-py2.5.egg/pkg_resources.py", line 1913, in load File "/home/brox/.virtualenvs/myproject/lib/python2.5/site-packages/fabric.py" , line 53, in <module> import win32api ImportError: No module named win32api
At the time of writing there is a small bug in Fabric that is likely to be fixed in the near future. For now you have to manually modify a file in fabric/state.py
before you install. Change the line that says
win32 = sys.platform in ['win32', 'cygwin']
to
win32 = sys.platform in ['win32']
This is just to tell Fabric that Cygwin isn’t really Windows and that the win32api module therefore isn’t available. Having made the necessary change, do a regular installation from source:
$ python setup.py install
The following error message about paramiko not being found might pop up; just ignore it:
local packages or download links found for paramiko==1.7.4 error: Could not find suitable distribution for Requirement.parse('paramiko==1.7.4')
And that’s it! You should now have a fully functional virtualenv/Fabric/pip setup. To verify that Fabric works, create a file called fabfile.py
:
from fabric.api import local, run def local_test(): local('echo "foo"') def remote_test(): run('uname -s')
This file, of course, only scratches the surface of what you can do with Fabric—refer to the latest documentation for more information.
To test the fabfile, type the following:
$ fab local_test [localhost] run: echo "foo" Done.
The biggest issue is that of getting Fabric to play along with your SSH installation so that you can deploy on remote servers. (You did install the openssh package, right?). Try the following command, substituting test@atbrox.com
with one of your own accounts:
$ fab remote_test No hosts found. Please specify (single) host string for connection: test@atbrox.com [test@atbrox.com] run: uname -s Password: [test@atbrox.com] out: Linux Done. Disconnecting from test@atbrox.com... done.
The next step would be to set up password-less logins, but that is a different story.
Afterthoughts: While Cygwin is a lifesaver, it has some quirks and annoyances that may or may not be an issue depending on your system configuration. For instance, on my setup the following error tends to show up randomly when using Fabric for remote deployment:
sem_init: Resource temporarily unavailable Traceback (most recent call last): File "build/bdist.cygwin-1.5.25-i686/egg/fabric/main.py", line 454, in main File "/cygdrive/c/Users/brox/workspace/quote_finder/fabfile.py", line 187, in deploy _prepare_host_global() File "/cygdrive/c/Users/brox/workspace/quote_finder/fabfile.py", line 137, in _prepare_host_global if not exists(u'/usr/bin/virtualenvwrapper_bashrc'): File "build/bdist.cygwin-1.5.25-i686/egg/fabric/contrib/files.py", line 32, in exists File "/usr/lib/python2.5/contextlib.py", line 33, in __exit__ self.gen.throw(type, value, traceback) File "/usr/lib/python2.5/contextlib.py", line 118, in nested yield vars File "build/bdist.cygwin-1.5.25-i686/egg/fabric/contrib/files.py", line 32, in exists File "build/bdist.cygwin-1.5.25-i686/egg/fabric/network.py", line 371, in host _prompting_wrapper File "build/bdist.cygwin-1.5.25-i686/egg/fabric/operations.py", line 422, in r un File "channel.py", line 297, in recv_exit_status File "/usr/lib/python2.5/threading.py", line 368, in wait self.__cond.wait(timeout) File "/usr/lib/python2.5/threading.py", line 210, in wait waiter = _allocate_lock() thread.error: can't allocate lock
This is a known problem that is not likely to go away anytime soon, due to an inherent race condition in Cygwin’s implementation of sem_init. Still, having a functional virtualenv/Fabric/pip environment on Windows is all in all pretty convenient.
There is a slew of useful articles out there if you need more information on the tools described in this article. These are my current favorites:
- Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip (note that most of the Fabric articles out there use an outdated version of the Fabric API, so have a look at the latest documentation as well.)
- A Primer on virtualenv
- Django Deployment with virtualenv and pip
- Using pip Requirements
FWIW: virtualenv 1.4.5.post1 seems to work (not very tested) with cygwin & win32 Python if you patch it: http://bitbucket.org/ianb/virtualenv/issue/22/fails-to-install-activate-when-running-win32-python-from-cygwin#comment-119005
i think the anoying problem you been having can be solve by using.
cd \cygwin\bin
ash
PATH=. rebaseall -v
it is my experience, that generally you should avoid using windows and cygwin for serious development work with python – you will bounce your head against more and more problems, lxml is a problem, python-mysql is a problem, and if you have strange errors while debugging your own stuff, you can never be sure, if it is an error you have in your code or if it is a cygwin issue – overall it adds another level of unneeded complexity and lots of boring problems.
To avoid all this just use a virtual machine. You can still use your favorite windows tools if you install samba into the virtual machine and export your home to a samba share, which can be mounted on the host machine.
I spent many ours and always came to this conclusion – development with python on windows is still a pita, use a linux vm to save many hours!
The Fabric fix for cygwin went in, so you can use the master (http://github.com/bitprophet/fabric/tarball/master) with pip until the next release rather than manually patching and manually installing.
Pingback: Building a Facebook App in Django | naraekim
This a post from 3 years ago but it’s still working! Although had to update the version numbers, but still… working! Thank you!
Pingback: 求教在windows下怎么安装python的软件包,如virtualenv,nose,distribute和pip等 - python - 开发者问答
I just followed these instructions. very helpful. one issue: virtualenvwrapper_bashrc appears to have changed to virtualenvwrapper.sh since this was written
Pingback: Nagios OpenStack Installer – Automated monitoring of your OpenStack VMs | InIT Cloud Computing Lab
Pingback: Cygwin使用 | Joez
Pingback: Running ansible from a windows host | Webscalability
Pingback: Python:How do I get virtualenvwrapper and cygwin to co-operate? – IT Sprite