
Ask HN: How to automate Python apps deployment? - aalhour
Hello,<p>I am researching how to automate the development environment setup for a team of 15+ developers that is currently manual and requires a lot of repetitive work. In addition to that I am researching how to automate testing, integration and deployment of these web apps to different environments (staging, and production).<p>How would you implement this huge workflow and how would you tackle it? Any success stories of automating the deployment of similar apps (Python, JS, SASS, Config files, git cloning of projects) in other companies&#x2F;projects?<p>The current workflow is that each developer clones all the apps (30+) and sets them up locally via editing config files, then sets up Apache for some internal URLs, and then starts working on them. Finishes work, commits to remote development branch and from their everything is just manual work: review of pull-requests and merging branches...etc. Production releases are manual as well.<p>I am also interested in knowing how to setup&#x2F;refactor&#x2F;change Apps in such away that would enable further automation to be as easy as possible.<p>Cheers!
======
twunde
1) Look into configuration management. Ansible and Salt Stack are both python-
based so everyone should be able to contribute if needed.

2)It's popular to run your automation through CI tools like Jenkins because it
has auditing and logging built in and it's easy to set up.

3) Don't run bower/npm install/update on production because if npm or github
go down then your production site will break because a partial code change has
taken place. You should build (and test) on a separate server and then copy
the code changes over to production.

Really you're looking for a Capistrano for python apps. If you're going to
build a deployment tool on your own it's worth looking at their approach. For
python, fabric is a fairly standard way to automate code deployments. The good
news is that you're on the right path. Having automated deployment procedures
will give devs a lot more time back AND remove a whole class of errors

I've moved my current company's deployment from manual line by line ftp
deployment to an automated build and deploy using Jenkins to trigger ansible
playbooks. Given the choice, I'd probably just use capistrano or fabric
instead of building from scratch. I'd estimate that we've saved several months
of work per a year in both deployment time and in debugging production issues.

~~~
aalhour
That's one comprehensive list of items, thanks for this.

I will take a look at Capistrano, Fabric, Ansible and SaltStack, in addition
to the configuration management topic.

I think your advise of copying source packages and dependencies makes total
sense, I have seen it happen at other places.

My question remains, how would I automate a developer's environment setup
(machine)? Assume I have the OS installed and connected to the internet
already. Can I use the tools you referred to for deploying these apps to
developers machines? What would you say regarding Python's virtualenvs
creation? Can that be incorporated into the configuration management tools?

~~~
aprdm
You can use Vagrant and Ansible, I think it's a killer combo, ansible
provisions your vagrant/virtualbox machine for the dev.

------
NicoJuicy
I could integrate my entire workflow with Jenkins and custom parameters and
custom development of console applications ( creating automated versions of
packages ( npm / nuget / .. ), mostly with the use of the "build number")

It handles 3 situations:

\- local ( not included in Jenkins, but is automated within the development
tools -- eg. Visual Studio. It uses local databases and local paths)

\- development

\- beta

\- live / production

So my entire workflow is automated now, the merging of branches and etc. is
done by the developer. Testing is automated ( unit tests) ... Deploying is
just a matter of minutes instead of 1,5 hours. And best of all, all related
packages are updated automaticly with the correct tag ( development, beta, ..)

Another way would be using TeamCity ( instead of Jenkins)

~~~
aalhour
Thanks for writing this.

We use Python virtual environments to make every app's dependencies in one
place and not installed globally. Also, the frontend dependencies and code
compilation is handled with npm, Gulp and Bower.

The thing is the developer has to issue all these commands in order to "setup"
the app locally and be able to run and troubleshoot it before doing actual
work, and this is universal for all apps. Same thing goes for setting up apps
on staging and production other than the pull-requests and branches-merging
tasks which are all manual.

And if a new developer comes in, they will have to install and configure their
workstation before doing any actual manual setup which is also an overhead.
This scenario would also happen if a developer comes back from a long holiday.

Any ideas on this?

------
dozzie
For deployment: use your OS's packaging system, for both dependencies and
actual applications. `git pull' rarely does good job, but `apt-get install'
often is just enough.

For development: use your OS's packaging system for installing dependencies
and a makefile for compiling things.

Really, I don't understand why programmers tend to avoid using sysadmins'
tools in their work, especially now with this DevOps thing, when programmers
are supposed to see some administration themselves.

~~~
aalhour
I agree with you, I kicked off this workflow implementation phase with a
simple bash script to see what parts need special knowledge and are ugly and
not standard and then pushed back to the team to fix them. Using a bash script
to automate the development environment setup sounds good but I thought I
might have missed something, hence the question.

Do you think this is a good way to automate the boring workflow:

* git clone project

* make virtualenv

* edit config file

* npm install

* bower install

* exectue gulp build task

* enable apache internal url for project

* restart apache

?

~~~
dozzie
> Using a bash script to automate the development environment setup sounds
> good [...]

No, it doesn't sound good even remotely. This way you have method for setting
up things that is very brittle, probably can't be run with confidence on a
machine that was already set up, and this method is not suitable for
production or test deployment, so you need to maintain _two_ setup methods.

You should look at how developers of various distributions (Debian, CentOS/Red
Hat, Fedora, Ubuntu) package their software. All these _foo-dev_ packages
containing static libraries and headers, and _foo_ packages with actual
libraries and tools, either having their respective dependencies (e.g. _foo-
dev_ depending on gcc, so you'll get compiler by just running `apt-get install
foo-dev').

Apache is not a good choice for development tool. It's not pretty to set up a
second instance that runs without root privileges. I use uWSGI for those, I
just use different communication protocols for production (FastCGI or uWSGI's
own thing) and development (HTTP, to avoid setting up fully blown web server).

Running `npm install' or `bower install' or `pip install' to just set up your
environment is a method for a big stall in case of network outage (see this
thread:
[https://news.ycombinator.com/item?id=10984775](https://news.ycombinator.com/item?id=10984775)),
unless you prepare local caches for _all these repositories_. You should have
all the things in your local repository, either source control repository
(your stuff) or packages repository (external dependencies).

~~~
aalhour
Can you elaborate more on the last paragraph? Also what do you exactly refer
to by package repository and how would installing pip, bower or npm modules
become different?

~~~
dozzie
> Can you elaborate more on the last paragraph?

There's not much to say. You depend on external services (and network
connection) to write, build, and deploy locally developed things. All this
fails badly when one of the services experiences problems.

> Also what do you exactly refer to by package repository and how would
> installing pip, bower or npm modules become different?

Dude, really? `apt-get' didn't give a hint?

And all those modules are different from one, single package repository in
that you won't install Python modules using npm, or JavaScript with pip, but
you can do all those with `apt-get'. And you'll get _repeatable_ process,
since you control your package repository.

~~~
aalhour
I am either lacking some experience with apt-get or you're not clarifying your
workflows enough.

How would you clone a git repository, compile the SASS into CSS, uglify the
javascript, and run a verticalization gulp task, after of course installing
the frontend libraries through npm and bower all in apt-get? Oh no hold on,
you won't use neither npm, nor bower to download the frontend deps... oh no,
not even pip. Mind ``explaining like I'm 5``? :)

~~~
dozzie
Apparently you lack training in using your server system if you thought that
binary packages system is to be used for running compilation. No, it is to be
used to distribute, download, and install pre-compiled (e.g. minified) files.
Also, to distribute build tools.

For more explaination how package systems work, ask sysadmins in your company,
as HN comments is not an appropriate medium for tutorial about them. If you
don't have a sysadmin, well, you need to employ one anyway.

And this is exactly what I meant when I said that programmers tend to avoid
using sysadmins' tools.

~~~
aalhour
Apparently you made me think of something else, I managed to do something
similar to what you explained in a different workplace, but it seems that it
needed to take forever to make you clarify your point. Anyway, thanks for
coming back and writing, I really appreciate it.

------
andrewjshults
Longer blog post here: [http://dna.hioscar.com/2015/10/14/scaling-deployments-
at-osc...](http://dna.hioscar.com/2015/10/14/scaling-deployments-at-oscar/)

We work out of a mono-repo and use pants
([https://pantsbuild.github.io/](https://pantsbuild.github.io/)) for
python/jvm/js packaging to deploy on mesos
([https://mesos.apache.org/](https://mesos.apache.org/)). Pants gives us
(effectively) the same development workflow as in production and let's us
treat any application (we have ~500 at this point) as just a command line
executable (for things with weird lib dependencies, we do statically compiled
wheels for each architecture - CentOS and OS X in our case - on our own pypi
mirror). We've got a few kinks to still workout for local development of web
projects (mostly related to JS/CSS) but for things that don't touch the
frontend it's pretty beautiful.

------
afarrell
I am currently finishing up writing a step-by-step tutorial on this very
topic. Please check it out and, if you have a question about any piece of it,
write a comment and I'd be happy to help.

[https://amfarrell.com/saltstack-from-
scratch/](https://amfarrell.com/saltstack-from-scratch/)

Also let me know if any of the integration tests for any of the tests fail on
your setup.

~~~
aalhour
Thanks for sharing this.

I will definitely check it out soon, but since we're doing releases and are
short on time, mind explaining the general headlines of this tutorial? Say if
I do everything manual from development machines setups and installation of
deps and packages, to staging and production releases, what will I be able to
automate after applying your solution? And what steps won't be covered? Can
you give me a detailed list of these items?

~~~
afarrell
You mentioned above that you were planning to check out SaltStack, this
tutorial will give you working understanding of it.

Development machines setups: The tutorial walks you through running your app
on virtualbox machines the run locally, so that is an option. However, it
doesn't cover how to set up a development environment outside of a virtual
machine.

Installation of deps and packages: This it covers thoroughly.

Staging and production releases: It does not walk you through this explicitly,
but once you go through the tutorial, using the tools it teaches to do this is
fairly straightforward.

