
Peerd: an AWS VPC peering connection management tool - bmcalary_atl
https://github.com/atlassian-labs/peerd
======
bmcalary_atl
Be kind, this is one of my first python projects. :)

peerd is an AWS VPC Peering Connection management tool. It manages the full
lifecycle of creation, deletion and route table updates needed to make VPC
peerings useful.

Capabilities

    
    
        Capable of creating and accepting cross-account VPC peerings.
        Capable of creating and accepting cross-region VPC peerings.
        Capable of creating full-meshes of VPC peerings.
        Injects, repairs and removes routes as needed from VPC routing tables.
        Capable of managing overlapping meshes with the use of an environment name which is used to tag peerings.
        Overlapping meshes supported through the use of different environment names in configuration file.
    

Yes, Transit Gateway is a valid replacement for this tool. :)

~~~
kesor
Can you explain the "Patent Pending" bit? Why would you need to create a
patent for a short Apache 2.0 licensed API calling Python script?

~~~
KenanSulayman
In addition, the apache license grants patent rights (§3 Grant of Patent
License). Since it seems to be a research project, I assume they're just very
happy to be able to put their name on it with Atlassian funding the patent :-)

~~~
c0restraint
Ya this isn’t something that should be patentable. Tons of prior art in
conference talks on this. Cool tool though, glad it’s released!

~~~
Terretta
Yep. My team wrote this in 2014 or 2015, the features and conf file almost
identical. We’d had to extract out of a custom Ansible module to better do the
full dance.

Shared the approach with AWS ProServe, Stelligent (now Mphasis), and others as
well, also discussed briefly with Atlassian architects probably in 2015-2016
timeframe.

Surprised if this isn’t one of those lots of folks just knocked out and didn’t
think much of, and really should be in AWS toolkit.

------
code4tee
Cool tool and the problem it’s trying to solve is a real one. However on the
complex mesh of connections across accounts isn’t the new way to do this now
not the Transit Gateway?

A tool that could manage complex TGW setups across many accounts would also be
cool. Terraform can do that but it’s not the most elegant solution.

~~~
jcrites
In addition to Transit Gateway, there's also AWS PrivateLink [1], which is my
preferred approach for scenarios where it works. As an alternative to peering
entire networks together, PrivateLink allows you to establish minimal
connectivity between specific clients and services that need to communicate.
When a service supports PrivateLink, any authorized client can create a
private endpoint for that service in their VPC, and the endpoint receives a
local IP address (in that VPC).

This has a couple of benefits. The first is that you don't need to coordinate
IP address ranges in different VPCs, which is necessary if you're peering them
together (unless you're using NAT). IPv4 address management in large networks
can be challenging.

The second benefit is that although clients can communicate with the service,
there isn't an actual route between the two networks. This better achieves the
principle of least privilege at the network level; it avoids the security
risks of connecting networks together such that any machine can connect to any
other. When you design an application network to use PrivateLink from the
ground up, you can start with the expectation that the VPC doesn't need any
connections to any other networks, not even the Internet; it can potentially
be an isolated network bubble, with only PrivateLinks allowing traffic in and
out (using e.g. Session Manager for administration).

There are also practical limits on how large a federated network of peered
VPCs can grow (probably helped by Transit Gateway), but there isn't any limit
I'm aware of on the number of client/server connections that can be
established with PrivateLink.

For these reasons I prefer to use PrivateLink when building systems that
communicate across VPCs, falling back to network peering or Internet-facing
endpoints in scenarios where it doesn't work.

However, one downside compared to network peering is that you need to set up a
PrivateLink connection for each service that each client needs to call, though
this process can be automated during infrastructure provisioning. Services
need to have specifically onboarded with PrivateLink; you can't just deploy a
service on the network and have it work.

[1] [https://aws.amazon.com/privatelink/](https://aws.amazon.com/privatelink/)

~~~
gregwebs
How do you deal with the connections being to only one availability zone?

~~~
jcrites
The connection can be present in multiple availability zones, though the
service does need to decide which ones it will support at setup time. One
simple approach is for the service to accept connections in many or all AZs,
allowing the client to decide which ones are relevant to it. From the endpoint
service documentation [1]:

> For low latency and fault tolerance, we recommend using a Network Load
> Balancer with targets in every Availability Zone of the AWS Region. To help
> achieve high availability (...) you can enable cross-zone load balancing.
> Cross-zone load balancing enables the load balancer to distribute traffic
> across the registered targets in all enabled Availability Zones. For more
> information, see Cross-Zone Load Balancing in the User Guide for Network
> Load Balancers.

When establishing a connection, the client can specify multiple subnets in
which it would like to create an endpoint [2], and those subnets can be in any
AZs supported by the service, if I recall correctly. The client can thus
establish a connection that spans multiple AZ by requesting endpoints in
subnets in those AZs. In most scenarios it's fine if client/service traffic
can cross between AZs; I've only found it necessary to partition traffic by AZ
in specialized applications (usually applications that are themselves trying
to provide a zonal availability model, or that follow cell-based architecture
at the zone level -- most business services don't fall into these categories,
though low-level infrastructure services sometimes do).

[1] [https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-
se...](https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-service.html)

[2]
[https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_C...](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateVpcEndpoint.html)

------
apengwin
Why doesn't Atlassian-labs use Atlassian's own remote git hosting service,
bitbucket?

~~~
slovenlyrobot
BitBucket still has users?

I assume at least one, the person who thought the last redesign was a good
idea.

~~~
meritt
I still use bitbucket because I also use mercurial, but they've deprecated
that and will only offer git beginning later this year.

So, no, I don't know why anyone would use bitbucket at this point.

------
highprofittrade
Vpc peering will be replaced by transit gateway

------
tjalpin
I had this exact use case in my job, trying to connect an app hosted on an ec2
in one account to a redshift instance on another, vpc peering to the rescue.
will checkout your project when i find time.

