10.0: Track email opens with mail templates, draft tokens, and webhook automations

[jstanden]

Introduction

A common metric in email marketing campaigns is tracking which messages actually get opened by the recipient rather than deleted. This data can be used during A/B testing to develop subject lines that consistently perform well.

While Cerb doesn't provide email tracking as a pre-built feature (nor do we endorse the practice), it's simple to implement this functionality yourself using Cerb's tools.

When implementing tracking functionality, it's very important to consider the privacy implications. Email open tracking works by sending a request to a remote server when a message is viewed. This is usually accomplished by using an image (e.g. logo, transparent pixel) and only works with "HTML" messages (e.g. images, colors, fonts, layouts) rather than plain text.

Every time a reader opens an email message that is logged. You will know how often, when, and approximately where (if not obfuscated by a VPN) your messages are read.

In general, this kind of information is already recorded by most web servers: timestamp, URL, IP, and OS/browser versions. But it's not difficult to imagine a case where a stalker could abuse even imprecise location details, or information that a target read their last message multiple times, including in the middle of the night.

There are many ways email applications prevent tracking:

• Only display plaintext email.
• External images can be disabled.
• Known tracking URLs/patterns can be blocked.
• Small fully transparent images can be blocked (e.g. 1x1 transparent pixel).
• IP information can be routed through a proxy/VPN to anonymize it.
• Images can be cached to prevent subsequent requests (either additional reads by the same worker, or by other workers on the same network).

We highly recommend that you respect these preferences when a recipient blocks your tracking.

This guide won't attempt to circumvent such measures, and our provided example is obvious about what it's doing with /open/ in the tracking URL (a pattern that is frequently blocked).

If email tracking is still an important requirement for your use case, this guide demonstrates how you would accomplish it.

NOTE: This guide requires Cerb 10.0.1. Make sure you are fully updated before continuing.

Import the package

Log into Cerb as an administrator.

Navigate to Setup -> Packages -> Import.

Paste the following package:

{
  "package": {
    "name": "Example of email open tracking",
    "requires": {
      "cerb_version": "10.0.1"
    }
  },
  "records": [
    {
      "uid": "automation_tracking",
      "_context": "automation",
      "name": "wgm.example.webhook.email_tracking",
      "extension_id": "cerb.trigger.webhook.respond",
      "description": "Webhook handler for email opened and clicked events",
      "script": "start:\n  decision/path:\n    # GET /open/token\n    outcome/getPixel:\n      if@bool:\n        {{\n          request_method == 'GET' \n          and 2 == request_path|split('/')|length\n          and request_path is prefixed ('open/')\n        }}\n      then:\n        # Load the message by its token\n        record.search:\n          output: message_record\n          inputs:\n            record_type: message\n            record_query: token:${token} limit:1\n            record_query_params:\n              token: {{request_path|split('/')|last}}\n          on_success:\n            decision:\n              outcome/hit:\n                if@bool: {{message_record.id is not empty}}\n                then:\n                  # Mark the message as read\n                  record.update:\n                    output: results\n                    inputs:\n                      record_type: message\n                      record_id: {{message_record.id}}\n                      fields:\n                        trackingOpenedAt@date: now\n                        trackingOpenedIp: {{request_client_ip}}\n                \n            # Return the logo image in any case\n            return:\n              headers:\n                Content-Type: image/png\n              body@base64:\n                R0lGODlhAQABAPAAAAAAAAAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==\n    \n  # Any other request is a 404\n  return:\n    status_code: 404\n    body@text: Not found",
      "policy_kata": "commands:\n  record.search:\n    deny/type@bool: {{inputs.record_type is not record type ('message')}}\n    allow@bool: yes\n  record.update:\n    deny/type@bool: {{inputs.record_type is not record type ('message')}}\n    deny/fields@bool:\n      {{array_diff(inputs.fields|keys, ['trackingOpenedAt','trackingOpenedIp']) is not same as []}}\n    allow@bool: yes"
    },
    {
      "uid": "portal_tracking",
      "_context": "community_portal",
      "name": "Email Tracking",
      "extension_id": "webhooks.portal",
      "uri": "email",
      "params": {
        "automations_kata": "automation/emailTracking:\n  uri: cerb:automation:wgm.example.webhook.email_tracking\n  #disabled@bool: yes\n"
      }
    },
    {
      "uid": "customfieldset_tracking",
      "_context": "custom_fieldset",
      "name": "Tracking",
      "context": "message",
      "owner__context": "app",
      "owner_id": 0
    },
    {
      "uid": "customfield_opened_at",
      "_context": "custom_field",
      "context": "message",
      "name": "Opened At",
      "uri": "trackingOpenedAt",
      "type": "E",
      "custom_fieldset_id": "{{{uid.customfieldset_tracking}}}"
    },
    {
      "uid": "customfield_opened_ip",
      "_context": "custom_field",
      "context": "message",
      "name": "Opened IP",
      "uri": "trackingOpenedIp",
      "type": "S",
      "custom_fieldset_id": "{{{uid.customfieldset_tracking}}}"
    },
    {
      "uid": "mailtemplate_tracking",
      "_context": "html_template",
      "name": "Email Tracking",
      "owner__context": "app",
      "owner_id": 0,
      "content": "<div id=\"body\">\n{{message_body}}\n</div>\n\n<img src=\"{{{default.base_url}}}portal/email/open/{{message_token}}\">\n\n<style type=\"text/css\">\n#body {\n\tfont-family: Arial, Verdana, sans-serif;\n\tfont-size: 10pt;\n}\n\na {\n\tcolor: black;\n}\n\nblockquote {\n\tcolor: #646464;\n\tborder-left: 5px solid;\n\tmargin: 0 0 0 10px;\n\tpadding: 0 0 0 10px;\n}\n\nblockquote a {\n\tcolor: rgb(0, 128, 255);\n}\n\npre > code {\n\tdisplay: block;\n\toverflow-x: scroll;\n\tborder: 1px solid #e8e8e8;\n\tbackground-color: #f6f2f0;\n\tpadding: 10px;\n}\n\np > code {\n\tbackground-color: #f6f2f0;\n\tborder: 1px solid #e8e8e8;\n\tfont-weight: bold;\n\tpadding: 0.1em 0.2em;\n\tline-height: 1.75em;\n}\n\nh1, h2, h3, h4, h5, h6 {\n\tcolor: black;\n\tmargin: 0 0 10px 0;\n\tfont-weight: bold;\n}\n\nh1 { font-size: 2em; }\nh2 { font-size: 1.85em; }\nh3 { font-size: 1.75em; }\nh4 { font-size: 1.5em; }\nh5 { font-size: 1.25em; }\nh6 { font-size: 1.1em; }\n\nimg {\n\tmax-width: 100%;\n}\n\nul, ol {\n\tpadding-left: 2em;\n}\n</style>"
    }
  ]
}

Click the Import button.

How it works

The provided package creates a new mail HTML template named Email Tracking. You'll find it in Search -> Email Templates. You should customize this for your needs (layout/style).

This email template can be selected manually for outgoing messages, or it can be the default for an entire group and/or bucket. You would configure the latter from the HTML template: setting in Search -> Groups or Search -> Buckets.

The email template includes a 1x1 pixel transparent image like:

<img src="https://cerb.example/portal/email/open/{{message_token}}">

The {{message_token}} placeholder is unique and randomized for every draft. When a draft is successfully sent and delivered, the same token is set on the eventual message record.

You can use the token to refer to a message before it has been created. You can also safely disclose a token to customers in public URLs. They don't betray private information in the same way sequential numeric IDs might; and they can't reasonably be guessed to target a specific message.

Email templates are useful for this workflow because they are only applied to the outgoing message. The results of a mail template aren't saved on the copy of the message within Cerb, so workers viewing the history later won't trigger open tracking.

When the recipient reads the message -- and external images are enabled and not blocked -- their computer requests the image from Cerb.

The target in Cerb is a webhook portal named Email Tracking that you'll find in Search -> Portals.

If you're familiar with portals, you may recall that you can access them by name from /portal/ URLs without being logged in. In real-world use, however, you should put a web proxy in front of a portal so you can use your own domain name and SSL certificate, without disclosing the location of your Cerb instance.

Webhook portals receive an arbitrary web request and use an automation to respond based on its properties: client IP, host, path, method, query arguments, payload, etc.

The webhook portal is configured to simply run an automation by name for every request:

automation/emailTracking:
  uri: cerb:automation:wgm.example.webhook.email_tracking

The wgm.example.webhook.email_tracking automation has this logic:

start:
  decision/path:
    # GET /open/token
    outcome/getPixel:
      if@bool:
        {{
          request_method == 'GET' 
          and 2 == request_path|split('/')|length
          and request_path is prefixed ('open/')
        }}
      then:
        # Load the message by its token
        record.search:
          output: message_record
          inputs:
            record_type: message
            record_query: token:${token} limit:1
            record_query_params:
              token: {{request_path|split('/')|last}}
          on_success:
            decision:
              outcome/hit:
                if@bool: {{message_record.id is not empty}}
                then:
                  # Mark the message as read
                  record.update:
                    output: results
                    inputs:
                      record_type: message
                      record_id: {{message_record.id}}
                      fields:
                        trackingOpenedAt@date: now
                        trackingOpenedIp: {{request_client_ip}}
                
            # Return the logo image in any case
            return:
              headers:
                Content-Type: image/png
              body@base64:
                R0lGODlhAQABAPAAAAAAAAAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==
    
  # Any other request is a 404
  return:
    status_code: 404
    body@text: Not found

If it's a GET request for the /open/<token> path, the automation uses the record.search: command to locate the message by token: filter in the record_query: option. It also uses record_query_params: to sanitize untrusted user input so it can't modify the query.

The record_query: also specifies the limit:1 filter, to return a single message rather than a list. With this, the message_record output is a dictionary.

When a matching message record is found, the record.update: command sets two custom fields on that message: trackingOpenedAt (the current date/time) and trackingOpenedIp (the IP of the reader). This could also look up the location by IP, decode the User-Agent: HTTP header for OS/browser details, etc.

Afterwards, a 1x1 pixel transparent image is returned with the Content-Type: image/png and the image bytes encoded in Base-64 as the body@base64: option. You could replace this image with your logo instead and use it at the top of the mail template.

For any other request, the automation returns a 404 (Not found) error.

After that has taken place, you can filter messages by when and/or by whom they were opened.

In Search -> Messages you can use a query like the following to find messages that were opened in the past day:

tracking.openedAt:"-1 day"

This field can also be used for reporting.

Testing email tracking

You can test email tracking by navigating to any ticket profile from Search -> Tickets. Ideally this would be a test ticket.

Click the Reply button.

Set your personal email address to the only To: or Cc: recipient.

Make sure Formatting is enabled.

Select the Email Tracking option in HTML Template. This doesn't require a group or bucket to be configured with the template as the default.

Click the Send message button to deliver the message.

Check your personal email. If external images aren't disabled or blocked, it will ping your Cerb instance and record the fact the message was read.

Reload the ticket profile.

When viewing the message in the timeline, you'll see the tracking custom fields:

Conclusion

The first thing you should do before deploying this workflow in the real-world is to set up a site for the webhook portal on your own hostname using SSL. Then edit the mail template and replace the <img> URL with your new hostname.

You can add additional custom fields for tracking more metrics, or add more custom fieldsets for tracking different campaigns.