
Ask HN: How to deploy into production large software project? - newprint
I work as software developer for a very large corporation...
And yet, in my department we have absolutely outdated process of deploying large software projects. We literary have 2 people, they take a massive spreadsheet and drag and drop (from repository) software onto a large number of servers, after that they manually change .config files (info comes from another massive spreadsheet) and manually start&#x2F;stop services in correct sequence. After that someone else comes in and verifies that software is up &amp; running correctly, by checking logs. All software are Windows services, most don&#x27;t have .MSI installation packages. 
Problem is, last Friday deployment took 21h and we have several per year! I was at work from 7 in the morning till 2 in the morning. I was completely drained after being on my feet for 21 and following the morning we had issues, because of misconfigured applications (connection strings...)
HN help me sleep better. How do you deploy large software projects ?  
Thank you !
======
awinter-py
Set a conservative schedule for improvement (months, not weeks).

Automate the low-hanging fruit first (copying files, maybe, or centralized
health-checking so you don't have to open logs).

For a lot of improvements you can make a business case _other_ than your
quality of life. Health-checking improves reliability, for example. But accept
that these will be hard conversations that take a long time. Don't let others
convince you this isn't a problem.

Research the tools that solve your problems -- it's easy to argue for adopting
an industry-standard tool that solves an industry-standard problem.

Accept that certain pieces of the process will stay manual for a while.

Don't be afraid to go after small improvements -- if you can halve the
difficulty, the remaining steps will fall away easily.

------
fmansoor
[https://octopus.com/](https://octopus.com/) is a decent option. But, I wrote
a windows service that knows how to deploy different types of applications
windows service, iis app, scheduled tasks etc.

The service listens for manifest change message on a topic, when a team wants
to release a new version of an app, they update the url of the jenkins
artifacts in a manifest file in git and publish manifest changed message.

Deployment service on all the servers downloads the latest version of the
manifest file and updates or installs apps as needed.

I also wrote a custom prometheus exporter to export health and version of
services installed on a box. That allows us to track the deployment status.

This setup has worked super reliability for us, very little maintenance and
training required. Since deployment occur in parallel, deployment on hundreds
of servers takes only a few seconds.

------
inertiatic
Automate how you set up your development environment, then automate the
process of running this automation on your production servers.

You can get incredibly fancy and automate everything and deploy tens of times
a day etc. but you don't have to. You just have to pick practices that would
improve your current situation, and in most cases more automation helps.

In my current job, despite us not having everything down perfectly, we've got
to a point where I can tag my branch and run a script that deploys the tagged
code as a separate container locally if I'm doing development.

Then once we want to deploy to production, another set of configuration files
(also versioned, with the smallest diff possible from the dev config) is fed
to the same script, and the service is deployed on the production server.

Then you automate running this process across your servers (are your servers
your own? Are they on the cloud? Use the tools relevant to your situation).

Then, there is no need to perform manual monitoring of the logs. Set up a
monitoring service to notify you of any worrying logs (warn level and up?
Specific patterns? Up to you and depending on what you're logging) and feed
that service your logs. Hell, automate pinging these services so you can check
whether they're in the state you expect, and make a colour coded minimal
dashboard that anyone can check in a sec and find out if the everything is in
the state it should be.

If the business is paranoid, then do smoke testing at the end, but you
shouldn't need to.

------
meetneps
Hey there, I can help you with our product. We provide deployments of software
as service. We also provide customised solution for large enterprises. Take a
look at [http://grandappstudio.com/roll-out](http://grandappstudio.com/roll-
out) Let us know if you have any questions at napoleon@grandappstudio.com

------
mabynogy
Write a daemon yourself to control the deployment on the machines. A kind of
trojan but make it nice and safe. Do it on your spare time at the office and
wait for a good opportunity to show it.

You probably have many levels of hierarchy working against you. If you have
the opportunity to lead or manage anything, take it.

------
ptdel
I'm not familiar with your setup, so I'm not sure how much of this is
applicable, but I've seen a few different approaches, the all more or less go
like this:

* choose a configuration state manager like Salt, Ansible, Chef, Puppet, etc. Of all these I'd probably say Salt is the simplest for somebody approaching this methodology.

* choose a source control management system -- I'd just go with git, and host something on github or gitlab or bitbucket or whatever.

* choose a secure storage medium for the variables that are going to be loaded into your configuration state manager.

* choose some form of automation management tool, Jenkins, Thoughtworks Go, TravisCI, CircleCI, w/e.

You're going to use these in conjunction to gain the following: when a
developer pushes code to a specific branch, this push triggers a build on the
automation server, which is able to package the code, and distribute it to the
nodes managed by your configuration state manager. Aka, a push to your repo
makes it all happen (with testing, etc as gatekeepers to deployment of
course.)

I'll describe a basic setup I've encountered in the wild using Salt in a
master-agent setup (since it's the most basic imo, aside from headless):

-> developer writes some code
    
    
       -> developer pushes that code to github
    
       -> a POST webhook containing branch info is sent to my Jenkins box
    
       -> Jenkins initiates a build job that pulls down that branch of code
    
       -> Jenkins uses make (we use make) to build the code branch.
    
       -> Jenkins deposits the built code artifacts on a fileshare somewhere.
    
       -> Jenkins calls to the salt-master to update all nodes.
    
       -> the salt master (which could just be hosted on the Jenkins i guess) tells all minions to update
    
       -> all minions pull the new code and HUP themselves.
    

Salt uses yaml files to define the desired state you want, this includes
installing deps, installing config files, loading files, blah blah.

I use Jenkins Groovy pipelines (which I'm not that crazy about) to ensure that
the Jenkins jobs themselves are captured as code and are repeated the same way
each time across my make build steps. I work in AWS, so the machines I use,
the bootstraps they boot with, the desired state config files, and everything
are defined as code in some regard.

You should attempt to push for 'desired state' and move towards 'immutable
state'.

I would also strongly consider looking at the other options aside from salt,
it's only so-so.

