This post will get you started adding some metrics to your golang service. It will also link you to some useful resources.

Getting shit monitored:

Use the prometheus golang client to instrument your code:

The Godoc is much better than the github doc and contains lots of examples you can paste:

Read the best practices so that you know wtf you're doing when intrumenting your code:

Read this guys philosophy on monitoring and alerting, and be even more enlightened

Use kubectl port-forward to expose your golang service for testing, and then use a simple prometheus.yml to configure your basic prometheus installation to scrape it

$ kubectl port-forward <podname> <your-api's-metrics-port>
# make sure it's forwarded properly
xdg-open http://localhost:8000/metrics  

The basic prometheus.yml with an additional scrape target for your kuberenetes service

  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # this assumes your metrics are listening on port 8000 and that port is forwarded out of k8s
      - targets: ['localhost:9090','localhost:8000']
      #                   |             |_  your-metrics-service
      #                   |_ _ _ _ _ _ _ _  prometheus-scraping-itself

Run prometheus

$ prometheus

Now open the dashboard at http://localhost:9090


Next Steps:

Deploy prometheus in k8s with helm.


Install the operator from CoreOS


Do it yourself (probably the best way to do it, in my opinion):

hope this helped ;)