Email delivery and bounces

[Note: further changes in this area in January 2023 have made most of this article obsolete].

Cameo now automatically responds to bounced email (when the recipient’s mail server rejects an email). With this, Cameo no longer needs Mailgun to deliver mail-shots. (Mailgun has stopped offering a free tier and spam blockers sometimes block it).

Cameo now directly provides:

• daily summaries of failed emails
• the problem email addresses section
• cancelling quarantined email addresses

Background

Not all email makes it through:

• People change email addresses without telling you.
• Automated systems mark some email as spam.
• Messages sent by others from the same shared server and detected as spam by the recipient’s mail server sometimes cause rejection of emails.

Domestic email systems (like GMail) cannot send bulk email. They will only accept a limited number of emails.

Numerous third-party products are available to manage and monitor bulk email delivery. However, most are expensive. They are out of reach of smaller organisations and out of proportion to the benefits. Until recently, Mailgun was the only product I was aware of that offered a free tier for modest numbers of emails.

However, Mailgun has recently changed its pricing. It has withdrawn the free tier. The equivalent cost is now nearly $100 per year. (For the time being, however, Mailgun apparently continues to honour the free tier for existing subscribers). Experience also shows that spam-blocker services frequently block their shared servers. This leads to rejection of significant number of emails every so often.

There is therefore a need for an alternative.

Bounce handling

When the recipient’s mail server rejects an email, it sends a bounce message. You’ll have seen these occasionally in your domestic mail: typically rather cryptic messages telling you the server did not deliver your email. These also contain some machine-readable information that you don’t see.

Cameo can now handle such bounces for itself. Previously, Mailgun’s API provided bounce reports, but for non-Mailgun clients, there was no equivalent.

When Cameo receives a bounce message, it determines whether it is:

• a hard bounce. Further emails will also fail – for example, if the email account no longer exists), or
• a soft bounce, indicating there is a transient problem. For example, the mail server is not available or the recipient’s mailbox is over its size limit.

Cameo (rather than Mailgun) now suspends delivery to addresses which produce hard bounces. The Problem email addresses panel in the Sent Emails section on the Communications menu provides for removing addresses from this quarantine locally, rather than (or as well as) at Mailgun.

Similarly, Cameo now produces daily aggregated reports of delivery failures for itself, rather than from information provided by Mailgun.

Whether using Mailgun or not, Cameo now also automatically removes soft-bouncing addresses from its problem list once another mail reaches the recipient without bouncing. (Previously, they would have stayed on the list until explicitly removed).

Cameo needs to be able to identify individual emails as a prerequisite to all of this.

The Problem Email Addresses panel

On the Communications menu, Sent Emails contains a Problem Email Addresses panel (Fig 1 – formerly in Library, figure not updated) is now visible in all Cameo installations. For Mailgun clients, who already had this, the section has more to offer.

Problem Email Addresses shows any addresses currently bouncing (Fig 1: 1). This includes:

• when (Fig 1: 2), and
• either quarantined (caused by a hard bounce so that sending email to that address is suspended) or temporary (a soft bounce). You can clear this for either (Fig 1: 4), so that emails can start flowing to that recipient again.

Letting people know there is a problem

New in this section:

• prepare a letter that tells them there is a problem with their email (Fig 1: 5). This uses a template servicing the bouncing email task. You can get a proforma template for this from the bottom of the section. This also:
  • Moves the offending address into a new handle, Bouncing email, so they no longer receive email on it. A new substitution {show: bouncing email} lets you include this address in your letter template.
  • Removes the email from the list of bounces. Mail would no longer be sent to it because the main email field is empty.
• Lists anyone who has such a Bouncing email address. With this, you can see which bounces you have dealt with (Fig 1: 6).

Using a handle to record the problem supersedes an informal way of doing this which used the payment anomalies field.

Later changes

Since problem email addresses was introduced there have been some further enhancements to re-send in bulk, see the reason for the quarantine and refresh the list.

Technical requirements

Cameo uses the same incoming email for bounces as for any other incoming email, so you need to set this up first to manage bounces automatically.

Incoming email uses specific email address(es) for its incoming email (for example tocameo@example.com (possibly with variants for different kinds of content). Unfortunately, though, the way third-party email servers work means that bounces need to use the whole of the email address before the @ sign. Therefore, bounces need a separate sub-domain to receive them. For example, if you send out emails from example.com you might set up bounces.example.com to receive any bounces.

To set this up:

1. Tell Cameo about this sub-domain along with other incoming email credentials in a new box in the incoming email section of organisation settings (Fig 1). This allows outgoing emails to include an appropriate return address for incoming bounces.
   
2. Set up the sub-domain so that it forwards all mail to Cameo’s ordinary incoming email address. You will need to set up at least one so-called MX DNS record for the sub-domain so mail sent to it can be delivered. Your domain name provider will have instructions on how to do this.
   
3. Then, on the email server identified by that record, set up forwarding (often described as wildcard forwarding) to the incoming email address.
   
   For example if your incoming email is received on tocameo@example.com and the sub-domain for bounces is bounces.example.com, then you would need to forward *@bounces.example.com to tocameo@example.com.
   
   If your provider does not offer the facility to do this, Cameo may be able to provide server facilities for this.
   
4. Set up DKIM, SPF and possibly DMARC records in your DNS in your domain registrar’s control panel. Once you have added a bounce domain, the incoming email section shows you what to add.
   
   SPF says what servers are allowed to send mail on your domain’s behalf. It is set for the bounces domain, not the domain in the From address. For example:
   bounces TXT 3600 "v=spf1 include:_spf.mythic-beasts.com ~all"
   
   DKIM provides the means to digitally sign your emails so the recipient can confirm it hasn’t been tampered with. The value for this includes a long string, which is one half of a key-pair used to cryptographically sign the email.
   
   DMARC tells the recipient what to do if they receive a mail purporting to come from your domain, but fails the DKIM or SPF checks, for example:
   _dmarc.bounces TXT 3600 "v=DMARC1; p=none; rua=mailto:dmarc@example.com;"

Manually handling bounces

Even if you don’t use a new domain for bounces, so they get sent to your ordinary in-box, it is still possible to take advantage of bounce processing in a semi-automatic way.

Forward such emails to Cameo’s usual address, either with +bounce in the subject line or appended to the email address (for example, tocameo+bounce@example.com) to process them as if they had been received automatically.