This post will get you started adding some metrics to your golang service. It will also link you to some useful resources.

Getting shit monitored:

Use the prometheus golang client to instrument your code:
https://github.com/prometheus/client_golang

The Godoc is much better than the github doc and contains lots of examples you can paste:
https://godoc.org/github.com/prometheus/client_golang/prometheus

Read the best practices so that you know wtf you're doing when intrumenting your code:
https://prometheus.io/docs/practices/naming/

Read this guys philosophy on monitoring and alerting, and be even more enlightened
https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit#

Use kubectl port-forward to expose your golang service for testing, and then use a simple prometheus.yml to configure your basic prometheus installation to scrape it

$ kubectl port-forward <podname> <your-api's-metrics-port>
# make sure it's forwarded properly
xdg-open http://localhost:8000/metrics  

The basic prometheus.yml with an additional scrape target for your kuberenetes service

global:  
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:  
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # this assumes your metrics are listening on port 8000 and that port is forwarded out of k8s
    static_configs:
      - targets: ['localhost:9090','localhost:8000']
      #                   |             |_  your-metrics-service
      #                   |_ _ _ _ _ _ _ _  prometheus-scraping-itself

Run prometheus

$ prometheus

Now open the dashboard at http://localhost:9090

Hooray!

Next Steps:

Deploy prometheus in k8s with helm.
https://github.com/kubernetes/charts/tree/master/stable/prometheus
http://helm.sh

OR

Install the operator from CoreOS
https://coreos.com/blog/the-prometheus-operator.html

OR

Do it yourself (probably the best way to do it, in my opinion):
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml

hope this helped ;)