Setting up observability for our CI with honeycomb and CircleCI

TL:DR — Allow third-party orbs in admin settings, sign up for honeycomb free tier, follow docs and fiddle with yml till it works!

I’ve been hearing about observability for years now and a friend recommended I add it to our CircleCI config as it was pretty straightforward. He had not anticipated how bad I am at editing yml!

Anyway here’s how I did it.

1. Make sure you’re using CircleCI versions 2.1 in your .circleci/config.yml

# .circleci/config.yml
version: 2.1

2. Sign up for honeycomb.io and get your API Key.

3. Get your CircleCI admin to turn on (Yes) to third-party orbs for your project in Organisation Settings > Security (if its off).

4. Add your Honeycomb API key to your environmental variables of your CircleCI project in “Projects > {Project Name} >Project Settings > Environmental Variables.

5. Include the buildevents: honeycombio/buildevents@0.2.7 orb (see links for documentation and github). This goes at the top of your config.yml just below the version. Here’s how it looks in our file

# .circleci/config.yml
version: 2.1
orbs:
  buildevents: honeycombio/buildevents@0.2.7
  node: circleci/node@4.0.0
  ruby: circleci/ruby@1.1.1
  browser-tools: circleci/browser-tools@1.1.1

6. Add two new jobs setup and watch . in our example *default_docker_ruby_executor is what we use for our rails project. Use whatever you currently use in your project for this. In this example from honeycomb you can see they are using go for a go project.

We also add these new jobs to our workflows and set setup as a requirement for watch and build.

# .circleci/config.yml
....
....
....
jobs:
  setup: 
    docker: 
      - *default_docker_ruby_executor
    steps: 
      - buildevents/start_trace
  watch:
    docker: 
      - *default_docker_ruby_executor
    steps:
      - buildevents/watch_build_and_finish
....
...
workflows:
  version: 2
  build_and_test:
    jobs:
      - setup
      - watch:
          requires:
            - setup
      - build:
          requires:
            - setup
      - test:
          requires:
            - build

7. Now we have our setup and watch jobs defined and in our workflow we need to wrap our other jobs with the buildevents/jobspan

# .circleci/config.yml
....
....
....
jobs:
  setup: 
    docker: 
      - *default_docker_ruby_executor
    steps: 
      - buildevents/start_trace
  watch:
    docker: 
      - *default_docker_ruby_executor
    steps:
      - buildevents/watch_build_and_finish
  build:
    docker:
      - *default_docker_ruby_executor
    steps:
      - buildevents/with_job_span:
          steps:
            - build_vinterior_app
  test:
    <<: *rspec-base-job
    steps:
      - buildevents/with_job_span:
          steps:
            - build_vinterior_app

So any job that we want a separate event for add buildevents/with_job_span before.

8. Go to honeycomb.io and make sure you’re in the buildevents dataset (unless you’ve named it something else this is the default)

Click on any of the lines that appear in requests

Click on Traces from the options just above the graph

Click on the trace id

At the end what we get is something like this (I simplified the code example above, we actually have 3 jobs for our tests).

With query-ability on all these fields

This is my very first effort so I’m keen to learn lots more about honeycomb and what kind of questions we can use it to answer! Expect a lot more posts on this.