In my opinion, the full explanation is too long to be put as code comment, but is just right as the commit message. The best way is to keep the commit message, but add a short comment like:
// Hack to trigger layout change in latest Mozilla
Then, whoever interested in the hack can read the full explanation from the commit message.
Basically, for simple code with unclear purpose, I'd go with short comment in the code plus full explanation in the commit message. For hard-to-read code with complicated logic, I'd go with long comment in the code. Or refactoring.
Ticket description, commit message, code comment, and the code itself are all necessary to keep the codebase "documented".
Comments are frequently out of date with the code that they're describing. This tendency is somewhat linear with the length if a comment and therefore longer comments will be more incorrect, more quickly. I'm not being facetious, this is a readily observed phenomena.
Long comments get out of date quickly because it isn't clear how the comments relate to the code on a quick scan. In this case, you want a quick note as to why the code is there. You can add external references if needed.
You don't want to make someone who is editing code stop to think about how it affects the comments.
> What's the point of comments then if your general rule is that it should be fine to edit code without changing the comments?
Not without changing the comments. Without stopping to ponder how two paragraphs of comments fit into the code changes. The comments (aside from API documentation) should annotate the code, not the other way around.
If the codebase is associated with a reasonably stable bug-tracker, I'd also put a bug # or a URL in the source comment if it's something nonobvious that is important for future maintainers to be aware of, but considered too long to document inline (if it's purely of historical interest, then yeah, leave it to the VCS history). Or a link to an archived mailing list discussion, or a CVE id, or some kind of hint.
Good article, but I think it touches too little about persistence. The trade-off of EBS vs ephemeral storage, for example, is not mentioned at all.
Getting your application server up and running is the easiest part in operation, whether you do it by hand via SSH, or automate and autoscale everything with ansible/chef/puppet/salt/whatever. Persistence is the hard part.
Good point. We're struggling to see the benefits of EBS for Cassandra that has its own replication strategy (ie data is not lost if an instance is lost), voiding the "only store temporary data on ephemeral stores" argument.
How do you handle entire datacenter outages with ephemeral only setup? You can replicate to another datacenter, but if power is lost to both do you just accept that you'll have to restore from a snapshotted backup?
Our use-case is different (not cassandra or db hosted on ephemeral drives), but what we've found using AWS for about 2 years now is that when an availability zone goes out, it's either linked to or affects EBS. Our setup now is to have base-load data and PG WAL files stored/written to S3, all servers use ephemeral drives, difference in data is loaded at machine creation time, AMI that servers are loaded from is recreated every night. We always deploy to 3 AZs (2 if that's all a region has) with Route 53 latency based DNS lookups that points to an ELB that sits in front of our servers for the region (previously had 1 ELB per AZ as they used DNS lookups to determine where to route someone amongst AZs and some of our sites are the origin for a CDN, so it didn't balance appropriately...this has since been changed) that is in the public+private section of a VPC with all the rest of our infrastructure in the private section of a VPC (VPC across all 3 AZs). We use ELBs internal to the AZ for services that communicate with each other. The entire system is designed to where you can shoot a single server, a single AZ or a single region in the face and the worst you have is degraded performance (say, going to the west coast from the east coast, etc.).
Using this type of setup, we had 100% availability for our customers over a period of 2 years (up until a couple of weeks ago where the flash 12 upgrade caused a small amount of our customers to be impacted). This includes the large outage in US East 1 from the electrical storm, as well as several other EBS related outages. Overall costs are cheaper than setting up our own geo-diverse set of datacenters (or racks in datacenters) thanks to heavy use of reserved instances. We keep looking at the costs and as soon as it makes sense, we'll switch over, but will still use several of the features of AWS (peak load growth, RDS, Route 53).
The short answer is to design your entire system so that any component can randomly be shot in the face, from a single server to the eastern seaboard falling into the ocean to the United States immediately going the way of Mad Max. Design failure into the system and you get to sleep at night a lot more.
Persistence for AWS can be relatively simple if you use a distributed filesystem, such as GlusterFS (http://gluster.org) or our ObjectiveFS (https://objectivefs.com). You get a shared namespace for all your instances and persisting your data becomes as simple as writing files.
Outside of US and EU, it is quite difficult to find a reputable and affordable dedicated hosting provider. We use AWS extensively in ap-southeast-1, and their new C3 Compute Optimized instances are actually very competitive in price-performance ratio. Previously, we mostly used C1 instances, which were just okay. I never like the M1 / M3 instances.
Besides, in this region with messy peering agreement, AWS network also has the lowest ping time and the fewest hops.
So inside of EU/US you do use dedicated server providers ?
AWS is effectively (almost) a transit-only network. That has pros and cons. On the plus side, it should have decent connectivity anywhere. On the minus side, if (almost) any link at all on a network is congested, that will almost certainly congest AWS traffic too. They're relatively unlikely to have long-running issues. Also : they're REALLY expensive.
Also, the packaging is done differently by ubuntu and nginx upstream, so I don't think you can just replace it. They are also bundled with different modules. For example, I have to use the package from ubuntu because the one from nginx upstream lacks the geoip module.
It looks like the Multi-AZ setup is using block level replication such as DRBD instead of the built-in replication:
> Database updates are made concurrently on the primary and standby resources to prevent replication lag.
Makes me feels better for setting up my own pg cluster on EC2 a week ago, which does allow reads from the replication slave. Plus, I can provision <1000 IOPS (provisioned IOPS is damn expensive with AWS), and get to use ZFS.
The time spent researching and building that cluster was time not spent building product. If you are low on capital then it was potentially a good tradeoff, but I'd much rather just scale up my read capacity by adding horsepower until they get around to adding read replicas. My goal is to spend as little time as possible administering my infrastructure, and running my own database cluster is the last thing on my mind.