

Fabric Python with Cleaner API and Parallel Deployment - tav
http://tav.espians.com/fabric-python-with-cleaner-api-and-parallel-deployment-support.html

======
bitprophet_
Fabric maintainer here -- Tav's been quite active on our IRC channel while
developing much of this, and it's great to have more people excited about
improving the tool!

As tswicegood says in another comment, some of these changes (@task,
parallelism, etc) aren't unique to Tav's work, so when it comes time to merge
we'll draw on the various implementations to see what feels best.

The features unique to his fork will be evaluated like any other contribution
and merged in if they fit well and are generally useful (which many seem to
be).

The plan right now is post-1.0 (1.0 is scheduled for Pycon or earlier) to
round up a lot of major contributions and get them into core, so people like
tswicegood don't have to work off of their own forks quite as much :)

------
polvi
Another cool trick: Integrating Cloudkick tags with Fabric. Using the
Cloudkick API - you can express things like "run certain_func on everything
tagged 'loadbalancer'".

[https://support.cloudkick.com/Integrating_Fabric_with_Cloudk...](https://support.cloudkick.com/Integrating_Fabric_with_Cloudkick)

We dog food this for all of our production deployments. For example, when we
ship a new version of cloudkick.com, we just run "fab ship", which looks for
all servers tagged "cloudkick.com" and runs the appropriate deployment script
on each of them.

Tagging can be done through the CK UI, automatically with the agent, or with
things like puppet or chef. This allows you to fully automate deployment and
other tasks around groups of servers.

~~~
askedrelic
I figured out something like that using the Python Linode API last week:
[http://asktherelic.com/2011/02/17/dynamic-fabric-commands-
fo...](http://asktherelic.com/2011/02/17/dynamic-fabric-commands-for-managing-
cloud-servers/)

------
sophacles
While I like a lot of the stuff you have incorporated into your fabric, it
seems to me that you are trying to take credit for things that have existed as
branches to the main codebase of fabric for quite a while. Just off the top of
my head: task decorators, parallel execution, better host management and
command-line task listing all exist in other people's patches. If you have
redone the code, good for you, but at least mention that the ideas and
prototype implementations aren't yours. (I don't think a claim to ignorance of
such is really that valid either, since the Redmine for the project has
tickets for most of the things you've done, and has links to the branches
containing them).

As to getting stuff into fabric: a very valid complaint (and one I have
myself, since like a lot of people I run my own hacked version of fabric) is
that fabric development moves very very slowly. There are several different
feature branches in several people's code-bases that could have been released
6 or more months ago. Perhaps if you start pulling other patches that have a
lot of functionality complementary to yours, you could have a product good
enough to push development on the mainline too (ala gcc/egcs of old...)

~~~
tav
Let me get this right. Due to convergent needs, dozens of Fabric users post
various ideas to the issue tracker and mailing lists. And over the years, some
people hack together personal branches solving some of these issues and,
despite the incredible amount of time that's passed, they fail to get their
work into the mainline branch.

Eventually, one hacker comes along — who, having been a Fabric user for a long
time, has independently experienced similar needs and finding no decent
solution, decides to take on the task of implementing the various features
himself — making sure that it's clean, backwards compatible and fully
documented. And your response is to accuse him of trying to take credit for
other people's work?!

I can understand your frustrations with the pace of Fabric development, but
accusing another hacker of taking credit for other people's work simply
beggars belief. Even more incredible is that so many people decided to upvote
you! For the record, I'm not taking credit for anyone else's work. I came up
with the various ideas myself and you can look at the code to see that it's
not based on anyone else's work. Thank you.

~~~
StavrosK
As a side note, your blog is beautiful. Did you design it yourself?

~~~
tav
Thank you for the compliment — I was going to write an article a few weeks
ago, but ended up spending the evening redesigning the blog instead:
<http://tav.espians.com/new-site-design-for-2011.html>

Procrastination has its benefits I guess, heh. If you fancy them, the
css/templates are in <https://github.com/tav/blog> and the site is run using
yatiblog —
[https://github.com/tav/ampify/blob/master/src/pyutil/yatiblo...](https://github.com/tav/ampify/blob/master/src/pyutil/yatiblog.py)
— the source is all public domain, so do with it as you please.

~~~
StavrosK
I'm also migrating my blog right now, so don't mind if I do! Thank you!

------
tswicegood
Just a quick note: these features are definitely experimental. The @task
decorator has been in stable forks of Fabric for over a year at this point
(this is the 3rd, if not 4th independent implementation of it). Likewise,
parallel tasks have also been relatively stable for quite a long time.

~~~
jsdalton
Good to hear, hopefully the task decorator will make it in soon because I
think it's a much superior API.

The little hack on Fabric I use currently takes the opposite approach -- I use
an @internal decorator that hides certain functions from the task list but
keeps the default Fabric behavior around for anything else. It's just a few
lines of codes that mess with a private list Fabric uses to track internal
functions:

    
    
        from fabric.main import _internals
    
        def internal(fn):
            _internals.append(fn)
            return fn
        _internals.append(internal)

~~~
tswicegood
That's an interesting idea. I like the idea of turning things off too. In some
cases, that's more useful than explicitly defining a @task. Care to open a
ticket on code.fabfile.org with it? Maybe we can work that in as an option in
1.1 (when @task is set to hit).

~~~
jsdalton
Sure, I'll try to file one later so it can at least be considered.

What I was trying to say in my earlier comment was that I would actually
prefer the explicit @task decorator be part of the official Fabric API because
it's more clear and, well, explicit. I prefer my @internal decorator as a hack
in the meantime because I imagine it's less surprising to someone who might
stumble upon my fabfile without any explanation from me.

------
traviscline
Did some filter-branching and rebasing to get tav's work on top of current
master: <https://github.com/traviscline/fabric/tree/tav>

------
ghc
This looks nice, but I have to be honest...I'm not going to switch from the
version of fabric I use to this. Why? Even though I do have to deploy to more
than 30 hosts at once, the thought of the time that would need to be spent
verifying that there are no bugs in this would make it impractical to switch.
I have to imagine there are others like me who feel the same way. It's a shame
that the people this would most benefit are probably the least likely to
switch.

~~~
bitprophet_
I don't believe Tav posted this as an actual fork, but simply highlighting his
branches/patches, in the same way that many other Fabric users have code
waiting to be merged into core.

When the time comes to merge/incorporate them, we will definitely be wanting
peoples' opinions on whether they feel the changes would fit into core Fabric
:)

------
cagenut
I admit to substantial ignorance here, but can anyone who's using fabric, or
for whom these patches to fabric are an important improvement, speak to any
kind of pro/con comparison they did with func
(<https://fedorahosted.org/func/>)?

I've been using func for parallel deployment to 15 - 20 servers for about a
year now, but I have to admit I didn't really do any research into other
options at the time. Its written by the guys who wrote yum and cobbler, which
are already core to my setup, so it seemed natural to just stay in their
ecosystem. Thing is I never hear about anyone else using it, so I'm curious if
anyone did more diligence than I and had specific reasons of choosing one or
the other.

~~~
mccutchen
I had not heard of func until this thread. I chose fabric because it was the
only Python-based option I was aware of.

------
encbladexp
The @task decorator is a real good idea. Many good changes in 1.0.

The only thing i miss is a virtualenv context manager (with
virtualenv(blupp):...).

------
rch
Plus one for the task decorator.

Very nice work all around.

------
jedsmith
I've just started playing with Fabric, and I like your take on its
functionality better. If they're resistant to merge you, why not fork into
something new?

You definitely have something very useful there, from a casual read.

~~~
tav
Whilst forking is always a possibility, I'd rather contribute back to the
project and help improve Fabric collectively. The community is made up of
really nice guys and it'd be great if we could figure out a way of getting the
various ideas to work together.

Thanks for checking out the docs and your encouraging words!

~~~
_Fil_
Jeff mentioned on twitter that he's keen on merging some of the stuff post
1.0.

~~~
tswicegood
Yes. Those of us who have been in the Fabric community working with one
another are aware of this. From my conversations on IRC with @tav, I thought
he was as well.

It's always encouraging to see new people get involved. It's quite another to
see someone relatively new to the community try to go around that community by
announcing "new features" for a project they've barely started work on.

Sorry if that comes off a little bitter, but seeing this as the top post on HN
today rubbed me the wrong way.

------
_Fil_
There is definitely some cool stuff in there. I particularly like the @task
decorator, the directory support, the context and of course the parallel
tasks.

Nice work tav.

------
askedrelic
Cool ideas. We're just now moving from a 1 machine deployment to a 4 (and
probably more) machine deployment at work and I can already see the value in
this.

------
simonz05
I cant see how adding such a wast amount of features makes the API cleaner.
You made the library more complex than what it already was. Fixing Fabric
should in my opinion be done, not in a backwards compatible way, but by
forking it and taking away all the crap you don't want, then build a cleaner
API. Adding more features to something which already is bad doesn't make it
better.

------
carlhu
Beautiful work. I've always felt that Fabric strikes a effective balance
between simplicity and flexibility and love how you're extending it without
breaking its spirit.

