
Show HN: Cloudmarker – Cloud monitoring tool and framework - cloudmarker
https://github.com/cloudmarker/cloudmarker
======
msnut
I am trying this tool. First impression: The plugin framework works like
charm. Creating a small plugin to get GitHub org repo info. Working with only
two files. Plugin is one file and configuration yaml is other file.

I am not Python developer. I develop in Powershell. So don't judge my code. :)

I put this in ghcloud.py.

    
    
      import urllib.request
      import json
    
      class GhCloud:
          def __init__(self, org):
              self.org = org
    
          def read(self):
              url = 'https://api.github.com/orgs/%s/repos' % self.org
              data = json.loads(urllib.request.urlopen(url).read().decode('utf-8'))
              for d in data: yield d
    
          def done(self):
              pass
    

I put this in cloudmarker.yaml.

    
    
      plugins:
        ghplug:
          plugin: ghcloud.GhCloud
          params:
            org: python
    
      audits:
        ghmon:
          clouds:
          - ghplug
          stores:
          - esstore
          - filestore
    
      run:
      - ghmon
    

I run tool.

    
    
      PYTHONPATH=. cloudmarker --now
    

Cloudmarker runs my ghcloud.py and puts data into localhost:9200 and /tmp
automatically. I can do it without cloning Cloudmarker code. I only hack my
code and Cloudmarker runs it.

------
200px
What is the generally the best language while working on more than one cloud?
If I want to deploy VMs into the major three clouds (AWS/GCP/Azure), is Python
a good language for automation or am I better off with Java?

~~~
meddlepal
It doesn't really matter. All the major clouds have mature libraries for
Java/Go/Python.

That said unless you have a highly specific reason to be writing low level
automation code you're almost certainly better off using an abstraction like
Terraform or going one level higher with Kubernetes.

~~~
scarface74
If you’re just doing automation, while all of the major languages have decent
SDK’s, Java or C# is overkill.

Python is the go to language and at least for AWS, there is one module -
Boto3. For C#, every AWS resource has its own Nuget package.

And no, K8s is not the magic bullet. There is a lot more to managing cloud
resources than just K8s. It doesn’t even begin to cover the different managed
services.

~~~
msnut
Python is the go to language for multicloud automation. For Azure only
Powershell is another go to language for it.

~~~
scarface74
You _can_ use Powershell for AWS, but it’s verbose, the community around it
isn’t as large and you won’t find as many examples.

[https://aws.amazon.com/powershell/](https://aws.amazon.com/powershell/)

------
msnut
I'm doing some checks like this using Azure powershell. Microsoft has well
documented powershell commands for Azure. My powershell scripts have less
features and they can't run in unattended mode like this Cloudmarker tool. But
I work on windows so powershell is always there. Has anyone run powershell as
daemon on Linux? If that is present I will setup a Linux system with my
powershell scripts for devops.

------
wongarsu
It looks like a useful tool for audits and to generate automated alerts to
catch mistakes.

> When we began working on this project in 2017, we were aware of similar
> tools that supported AWS and GCP but none that supported Azure at that time.
> As a result, we wrote our own tool to support Azure. We later added support
> for GCP as well

Well, at least that explains the lack of AWS support

~~~
cddotdotslash
If you're looking for a comparable tool that's AWS-focused, CloudSploit [1] is
a also open source and has a similar "collect and analyze" architecture (full
disclosure: I'm a co-founder of the SaaS product). We've also started adding
Azure [2] recently as well, but it's not at the same level as our AWS scans
yet.

[1]
[https://github.com/cloudsploit/scans/tree/master/plugins/aws](https://github.com/cloudsploit/scans/tree/master/plugins/aws)

[2]
[https://github.com/cloudsploit/scans/tree/master/plugins/azu...](https://github.com/cloudsploit/scans/tree/master/plugins/azure)

------
kapilvt
This seems like periodic api poll and db dump tool across a limited set of
resources.

For folks interested in this domain, its worth checking out cloud custodian,
[https://cloudcustodian.io](https://cloudcustodian.io)

Its open source and supports AWS, GCP, Azure and provides more functionality
(imo) and supports more resources. Its designed for compliance as code gitops
style workflows with policies in yaml that query/filter/take action on
resources, and provides for reporting, notification, and remediation use
cases. It integrates with all the clouds serverless runtimes to provide real
time response/inspection of api calls, provisioned directly from its yaml
policies, as well as integrating with the cloud provider's specific
security/compliance offerings (ConfigRules, GuardDuty, SecurityHub in AWS,
CloudSecurityCommandCenter in GCP, etc). Its got developers from AWS and Azure
working directly on it, and a community of hundreds of contributors.

~~~
msnut
Is Cloud Custodian extensible? Can we make it connect to other clouds?

I am trying out Cloudmarker. It claims to be extensible.

> As a result of this plugin-based architecture, Cloudmarker can also be used
> as a framework to develop your own plugins that extend its capabilities by
> adding support for new types of clouds or data sources, storage or indexing
> engines, event generation, and alerting destinations.

We have many VMs in Digital ocean. It can be nice if I can make Cloud
Custodian or Cloudmarker connect to DigitalOcean. The plug-in framework of
Cloudmarker looks neat. I am writing digitalocean.py similar to
[https://github.com/cloudmarker/cloudmarker/blob/master/cloud...](https://github.com/cloudmarker/cloudmarker/blob/master/cloudmarker/clouds/azcloud.py).
I can write read() function that returns the JSON from DO. The rest of the
framework takes care of running it and putting it in the selected DB
(Elasticsearch for me).

If the Digital ocean plugin runs fine I will write a GitHub plugin after it.

~~~
kapilvt
Sure cloud custodian extensible, its supports multiple cloud providers today
(aws, gcp, azure) with work in progress on Kubernetes. The core plugin
mechanism (registries) are used across every part, cloud providers, resources
in those providers, and filters and actions for resources. As an example have
a look at the Kubernetes provider [https://github.com/cloud-custodian/cloud-
custodian/tree/mast...](https://github.com/cloud-custodian/cloud-
custodian/tree/master/tools/c7n_kube)

We've considered digital ocean support, and I use DO as well, but the primary
use cases custodian serves around governance/compliance, security, cost
optimization haven't seemed as needed with DO across our user population (ie
hasn't been requested), so its not been as high a priority on the roadmap atm,
that said its a community project so contributions welcome.

All that said, if your just looking for a periodic dump to a datastore, then
perhaps cloud marker is going to be a better fit.

~~~
msnut
Thanks. I posted a comment update with simple Cloudmarker plugin I tried to
get GitHub dump -
[https://news.ycombinator.com/item?id=19900677](https://news.ycombinator.com/item?id=19900677)

Can you give me Custodian doc or tutorial that will help me do this in
Custodian? I saw the Kubernetes link. It has many files to read. Can you give
me simpler starting place?

------
mlrtime
NOTE: Azure and GCP only (no AWS)

~~~
pownr
They may work on it:
[https://github.com/cloudmarker/cloudmarker/issues?q=is%3Aiss...](https://github.com/cloudmarker/cloudmarker/issues?q=is%3Aissue+is%3Aopen+aws)

