I am not Python developer. I develop in Powershell. So don't judge my code. :)
I put this in ghcloud.py.
def __init__(self, org):
self.org = org
url = 'https://api.github.com/orgs/%s/repos' % self.org
data = json.loads(urllib.request.urlopen(url).read().decode('utf-8'))
for d in data: yield d
PYTHONPATH=. cloudmarker --now
That said unless you have a highly specific reason to be writing low level automation code you're almost certainly better off using an abstraction like Terraform or going one level higher with Kubernetes.
Python is the go to language and at least for AWS, there is one module - Boto3. For C#, every AWS resource has its own Nuget package.
And no, K8s is not the magic bullet. There is a lot more to managing cloud resources than just K8s. It doesn’t even begin to cover the different managed services.
> When we began working on this project in 2017, we were aware of similar tools that supported AWS and GCP but none that supported Azure at that time. As a result, we wrote our own tool to support Azure. We later added support for GCP as well
Well, at least that explains the lack of AWS support
For folks interested in this domain, its worth checking out cloud custodian, https://cloudcustodian.io
Its open source and supports AWS, GCP, Azure and provides more functionality (imo) and supports more resources. Its designed for compliance as code gitops style workflows with policies in yaml that query/filter/take action on resources, and provides for reporting, notification, and remediation use cases. It integrates with all the clouds serverless runtimes to provide real time response/inspection of api calls, provisioned directly from its yaml policies, as well as integrating with the cloud provider's specific security/compliance offerings (ConfigRules, GuardDuty, SecurityHub in AWS, CloudSecurityCommandCenter in GCP, etc). Its got developers from AWS and Azure working directly on it, and a community of hundreds of contributors.
I am trying out Cloudmarker. It claims to be extensible.
> As a result of this plugin-based architecture, Cloudmarker can also be used as a framework to develop your own plugins that extend its capabilities by adding support for new types of clouds or data sources, storage or indexing engines, event generation, and alerting destinations.
We have many VMs in Digital ocean. It can be nice if I can make Cloud Custodian or Cloudmarker connect to DigitalOcean. The plug-in framework of Cloudmarker looks neat. I am writing digitalocean.py similar to https://github.com/cloudmarker/cloudmarker/blob/master/cloud.... I can write read() function that returns the JSON from DO. The rest of the framework takes care of running it and putting it in the selected DB (Elasticsearch for me).
If the Digital ocean plugin runs fine I will write a GitHub plugin after it.
We've considered digital ocean support, and I use DO as well, but the primary use cases custodian serves around governance/compliance, security, cost optimization haven't seemed as needed with DO across our user population (ie hasn't been requested), so its not been as high a priority on the roadmap atm, that said its a community project so contributions welcome.
All that said, if your just looking for a periodic dump to a datastore, then perhaps cloud marker is going to be a better fit.
Can you give me Custodian doc or tutorial that will help me do this in Custodian? I saw the Kubernetes link. It has many files to read. Can you give me simpler starting place?