Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Cloudmarker – Cloud monitoring tool and framework (github.com)
73 points by cloudmarker 9 days ago | hide | past | web | favorite | 16 comments

I am trying this tool. First impression: The plugin framework works like charm. Creating a small plugin to get GitHub org repo info. Working with only two files. Plugin is one file and configuration yaml is other file.

I am not Python developer. I develop in Powershell. So don't judge my code. :)

I put this in ghcloud.py.

  import urllib.request
  import json

  class GhCloud:
      def __init__(self, org):
          self.org = org

      def read(self):
          url = 'https://api.github.com/orgs/%s/repos' % self.org
          data = json.loads(urllib.request.urlopen(url).read().decode('utf-8'))
          for d in data: yield d

      def done(self):
I put this in cloudmarker.yaml.

      plugin: ghcloud.GhCloud
        org: python

      - ghplug
      - esstore
      - filestore

  - ghmon
I run tool.

  PYTHONPATH=. cloudmarker --now
Cloudmarker runs my ghcloud.py and puts data into localhost:9200 and /tmp automatically. I can do it without cloning Cloudmarker code. I only hack my code and Cloudmarker runs it.

What is the generally the best language while working on more than one cloud? If I want to deploy VMs into the major three clouds (AWS/GCP/Azure), is Python a good language for automation or am I better off with Java?

It doesn't really matter. All the major clouds have mature libraries for Java/Go/Python.

That said unless you have a highly specific reason to be writing low level automation code you're almost certainly better off using an abstraction like Terraform or going one level higher with Kubernetes.

If you’re just doing automation, while all of the major languages have decent SDK’s, Java or C# is overkill.

Python is the go to language and at least for AWS, there is one module - Boto3. For C#, every AWS resource has its own Nuget package.

And no, K8s is not the magic bullet. There is a lot more to managing cloud resources than just K8s. It doesn’t even begin to cover the different managed services.

Python is the go to language for multicloud automation. For Azure only Powershell is another go to language for it.

You can use Powershell for AWS, but it’s verbose, the community around it isn’t as large and you won’t find as many examples.


You can use any language, all the providers have sdks in available for common languages. Otoh, I think its telling that all three of the major public clouds (AWS, GCP, Azure) have their primary cli in python.

I'm doing some checks like this using Azure powershell. Microsoft has well documented powershell commands for Azure. My powershell scripts have less features and they can't run in unattended mode like this Cloudmarker tool. But I work on windows so powershell is always there. Has anyone run powershell as daemon on Linux? If that is present I will setup a Linux system with my powershell scripts for devops.

It looks like a useful tool for audits and to generate automated alerts to catch mistakes.

> When we began working on this project in 2017, we were aware of similar tools that supported AWS and GCP but none that supported Azure at that time. As a result, we wrote our own tool to support Azure. We later added support for GCP as well

Well, at least that explains the lack of AWS support

If you're looking for a comparable tool that's AWS-focused, CloudSploit [1] is a also open source and has a similar "collect and analyze" architecture (full disclosure: I'm a co-founder of the SaaS product). We've also started adding Azure [2] recently as well, but it's not at the same level as our AWS scans yet.

[1] https://github.com/cloudsploit/scans/tree/master/plugins/aws

[2] https://github.com/cloudsploit/scans/tree/master/plugins/azu...

This seems like periodic api poll and db dump tool across a limited set of resources.

For folks interested in this domain, its worth checking out cloud custodian, https://cloudcustodian.io

Its open source and supports AWS, GCP, Azure and provides more functionality (imo) and supports more resources. Its designed for compliance as code gitops style workflows with policies in yaml that query/filter/take action on resources, and provides for reporting, notification, and remediation use cases. It integrates with all the clouds serverless runtimes to provide real time response/inspection of api calls, provisioned directly from its yaml policies, as well as integrating with the cloud provider's specific security/compliance offerings (ConfigRules, GuardDuty, SecurityHub in AWS, CloudSecurityCommandCenter in GCP, etc). Its got developers from AWS and Azure working directly on it, and a community of hundreds of contributors.

Is Cloud Custodian extensible? Can we make it connect to other clouds?

I am trying out Cloudmarker. It claims to be extensible.

> As a result of this plugin-based architecture, Cloudmarker can also be used as a framework to develop your own plugins that extend its capabilities by adding support for new types of clouds or data sources, storage or indexing engines, event generation, and alerting destinations.

We have many VMs in Digital ocean. It can be nice if I can make Cloud Custodian or Cloudmarker connect to DigitalOcean. The plug-in framework of Cloudmarker looks neat. I am writing digitalocean.py similar to https://github.com/cloudmarker/cloudmarker/blob/master/cloud.... I can write read() function that returns the JSON from DO. The rest of the framework takes care of running it and putting it in the selected DB (Elasticsearch for me).

If the Digital ocean plugin runs fine I will write a GitHub plugin after it.

Sure cloud custodian extensible, its supports multiple cloud providers today (aws, gcp, azure) with work in progress on Kubernetes. The core plugin mechanism (registries) are used across every part, cloud providers, resources in those providers, and filters and actions for resources. As an example have a look at the Kubernetes provider https://github.com/cloud-custodian/cloud-custodian/tree/mast...

We've considered digital ocean support, and I use DO as well, but the primary use cases custodian serves around governance/compliance, security, cost optimization haven't seemed as needed with DO across our user population (ie hasn't been requested), so its not been as high a priority on the roadmap atm, that said its a community project so contributions welcome.

All that said, if your just looking for a periodic dump to a datastore, then perhaps cloud marker is going to be a better fit.

Thanks. I posted a comment update with simple Cloudmarker plugin I tried to get GitHub dump - https://news.ycombinator.com/item?id=19900677

Can you give me Custodian doc or tutorial that will help me do this in Custodian? I saw the Kubernetes link. It has many files to read. Can you give me simpler starting place?

NOTE: Azure and GCP only (no AWS)

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact