
Ask HN: How do you rake production? - andriosr
Ruby on Rails has rake tasks baked in. It let you run a piece of code meant to run only once. How do you do it in other environments&#x2F;languages?<p>How do you fix some database items that got corrupted due to a bug? How do you ensure code review and time logs with the person triggering the execution? Considering that manual database updates in production are not a good thing.<p>I faced this problem with a couple of companies. There was never a good solution.<p>Does anything exist that I am not aware of?
======
sethammons
We use a change management process. Pretty much a jira ticket that outlines
the change that needs to happen. Affected systems need to have someone review
the plan. It has a rollout and rollback process and instructions should be
written in a way that anyone on the team could run it.

Ideally, all steps are scripts that have been code reviewed and tested in a
staging or dev environment.

When it comes to DB migrations, the code is updated to work with the old and
the new. The db migration happens and is ran by a script that is
start/stoppable and has logs. The code is then updated to not need the old db
stuffs.

~~~
andriosr
Nice! This is the best process I can think of. I’ve worked with this model.

But then everybody in the team has access to run the scripts, right?

In my case a few people had, and with scale they became bottlenecks

------
gt565k
Rake is just a task runner, it sounds like maybe you had some rake tasks that
were constantly ran to fix bad data or take care of recurring problems. This
is the wrong approach. I've seen this done by DBAs who had stored procs to fix
issues with data, without ever bothering to check with the dev team and have
them fix the integration that was contributing to the faulty data entering the
db.

Bad data integrations and poor UI validation bleed garbage data into the data
store. That's just the reality of life. If you're constantly fixing records
and cleaning data, you need to fix it at the source. Assuming there's a
pattern, you can fix the data integration, automated process, or poor UI
that's producing those records.

Generally speaking, you want good constraints at the data model layer, in
order to prevent bad data from being persisted to the data store by the
application layer. It takes a good understanding of database design and data
modeling to engineer a good model with constraints. A well structured and
properly normalized schema will outlast multiple application layer backends
and front-ends, for decades perhaps. Making sure you add proper unique
constraints, do not accept NULL values, add required validation to fields,
check constraints for various logic, and proper relationship management and
normalization of the model.

Once you have the proper constraints on the database table, the app layer will
no longer allow poor data to be written that violates those constraints, and
the errors will propagate back out to the end user, without ever being
persisted in the data store.

Of course constraints can't catch all bad data, the rest really relies on
having a good support system and reproducing problems and fixing the processes
and integrations that produce poor/unwanted data records.

~~~
andriosr
Completely agree. These problems should not happen twice, and if they are
frequent there is a problem somewhere else.

But as you mention, even with the best validations, it can happen sometimes.
What processes/tools have you seen work to fix them without Manual work or
untraceable scripts?

