This post will get you started adding some metrics to your golang service. It will also link you to some useful resources.
Getting shit monitored:
Use the prometheus golang client to instrument your code:
The Godoc is much better than the github doc and contains lots of examples you can paste:
Read the best practices so that you know wtf you're doing when intrumenting your code:
Read this guys philosophy on monitoring and alerting, and be even more enlightened
Use kubectl port-forward to expose your golang service for testing, and then use a simple prometheus.yml to configure your basic prometheus installation to scrape it
$ kubectl port-forward <podname> <your-api's-metrics-port>
# make sure it's forwarded properly xdg-open http://localhost:8000/metrics
The basic prometheus.yml with an additional scrape target for your kuberenetes service
global: scrape_interval: 15s # By default, scrape targets every 15 seconds. # Attach these labels to any time series or alerts when communicating with # external systems (federation, remote storage, Alertmanager). external_labels: monitor: 'codelab-monitor' # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' # Override the global default and scrape targets from this job every 5 seconds. scrape_interval: 5s # this assumes your metrics are listening on port 8000 and that port is forwarded out of k8s static_configs: - targets: ['localhost:9090','localhost:8000'] # | |_ your-metrics-service # |_ _ _ _ _ _ _ _ prometheus-scraping-itself
Now open the dashboard at http://localhost:9090
Deploy prometheus in k8s with helm.
Install the operator from CoreOS
Do it yourself (probably the best way to do it, in my opinion):
hope this helped ;)