Email

SMTP

By default NotifyBC acts as the SMTP server itself and connects directly to recipient's SMTP server. To setup SMTP relay to a host, say smtp.foo.com, add following smtp config object to /src/config.local.js

module.exports = {
  email: {
    smtp: {
      host: 'smtp.foo.com',
      port: 25,
      pool: true,
      tls: {
        rejectUnauthorized: false,
      },
    },
  },
};

Check out Nodemaileropen in new window for other config options that you can define in smtp object. Using SMTP relay and fine-tuning some options are critical for performance. See benchmark advices.

Throttle

NotifyBC can throttle email requests if SMTP server imposes rate limit. To enable throttle and set rate limit, create following config in file /src/config.local.js

module.exports = {
  email: {
    throttle: {
      enabled: true,
      // minimum request interval in ms
      minTime: 250,
    },
  },
};

where

  • enabled - whether to enable throttle or not. Default to false.
  • minTime - minimum request interval in ms. Example value 250 throttles request rate to 4/sec.

When NotifyBC is deployed from source code, by default the rate limit applies to one Node.js process only. If there are multiple processes, i.e. a cluster, the aggregated rate limit is multiplied by the number of processes. To enforce the rate limit across entire cluster, install Redis and add Redis config to email.throttle

module.exports = {
  email: {
    throttle: {
      enabled: true,
      // minimum request interval in ms
      minTime: 250,
      /* Redis clustering options */
      datastore: 'ioredis',
      clientOptions: {
        host: '127.0.0.1',
        port: 6379,
      },
    },
  },
};

If you installed Redis Sentinel,

module.exports = {
  email: {
    throttle: {
      enabled: true,
      // minimum request interval in ms
      minTime: 250,
      /* Redis clustering options */
      datastore: 'ioredis',
      clientOptions: {
        name: 'mymaster',
        sentinels: [{ host: '127.0.0.1', port: 26379 }],
      },
    },
  },
};

Throttle is implemented using Bottleneckopen in new window and ioredisopen in new window. See their documentations for more configurations. The only deviation made by NotifyBC is using jobExpiration to denote Bottleneck expiration job option with a default value of 2min as defined in config.tsopen in new window.

When NotifyBC is deployed to Kubernetes using Helm, by default throttle, if enabled, uses Redis Sentinel therefore rate limit applies to whole cluster.

Inbound SMTP Server

NotifyBC implemented an inbound SMTP server to handle

In order for the emails from internet to reach the SMTP server, a host where one of the following servers should be listening on port 25 open to internet

  1. NotifyBC, if it can be installed on such internet-facing host directly; otherwise,
  2. a tcp proxy server, such as nginx with stream proxy module that can proxy tcp port 25 traffic to backend NotifyBC instances.

Regardless which above option is chosen, you need to config NotifyBC inbound SMTP server by adding following static config email.inboundSmtpServer to file /src/config.local.js

module.exports = {
  email: {
    inboundSmtpServer: {
      enabled: true,
      domain: 'host.foo.com',
      listeningSmtpPort: 25,
      options: {
        // ...
      },
    },
  },
};

where

  • enabled enables/disables the inbound SMTP server with default to true.
  • domain is the internet-facing host domain. It has no default so must be set.
  • listeningSmtpPort should be set to 25 if option 1 above is chosen. For options 2, listeningSmtpPort can be set to any opening port. On Unix, NotifyBC has to be run under root account to bind to port 25. If missing, NotifyBC will randomly select an available port upon launch which is usually undesirable so it should be set.
  • optional options object defines the behavior of Nodemailer SMTP Serveropen in new window.

Inbound SMTP Server on OpenShift

OpenShift deployment template deploys an inbound SMTP server. Due to the limitation that OpenShift can only expose port 80 and 443 to external, to use the SMTP server, you have to setup a TCP proxy server (i.e. option 2). The inbound SMTP server is exposed as ${INBOUND_SMTP_DOMAIN}:443 , where ${INBOUND_SMTP_DOMAIN} is a template parameter which in absence, a default domain will be created. Configure your TCP proxy server to route traffic to ${INBOUND_SMTP_DOMAIN}:443 over TLS.

TCP Proxy Server

If NotifyBC is not able to bind to port 25 that opens to internet, perhaps due to firewall restriction, you can setup a TCP Proxy Server such as Nginx with ngx_stream_proxy_moduleopen in new window. For example, the following nginx config will proxy SMTP traffic from port 25 to a NotifyBC inbound SMTP server running on OpenShift

stream {
    server {
        listen 25;
        proxy_pass ${INBOUND_SMTP_DOMAIN}:443;
        proxy_ssl on;
        proxy_ssl_verify off;
        proxy_ssl_server_name on;
        proxy_connect_timeout 10s;
    }
}

Replace ${INBOUND_SMTP_DOMAIN} with the inbound SMTP server route domain.

Bounce

Bounces, or Non-Delivery Reports (NDRs), are system-generated emails informing sender of failed delivery. NotifyBC can be configured to receive bounces, record bounces, and automatically unsubscribe all subscriptions of a recipient if the number of recorded hard bounces against the recipient exceeds threshold. A deemed successful notification delivery deletes the bounce record.

Although NotifyBC records all bounce emails, not all of them should count towards unsubscription threshold, but rather only the hard bounces - those which indicate permanent unrecoverable errors such as destination address no longer exists. In principle this can be distinguished using smtp response code. In practice, however, there are some challenges to make the distinction

  • the smtp response code is not fully standardized and may vary by recipient's smtp server so it's unreliable
  • there is no standard smtp header in bounce email to contain smtp response code. Often the response code is embedded in bounce email body.
  • the bounce email template varies by sender's smtp server

To mitigate, NotifyBC defines several customizable string pattern filters in terms of regular expression. Only bounce emails matched the filters count towards unsubscription threshold. It's a matter of trial-and-error to get the correct filter suitable to your smtp server.

to improve hard bounce recognition

Send non-existing emails to several external email systems. Inspect the bounce messages for common string patterns. After gone live, review bounce records in web console from time to time to identify new bounce types and decide whether the bounce types qualify as hard bounce. To avoid false positives resulting in premature unsubscription, it is advisable to start with a high unsubscription threshold.

Bounce handling involves four actions

  • during notification dispatching, envelop address is set to a VERPopen in new window in the form bn-{subscriptionId}-{unsubscriptionCode}@{inboundSmtpServerDomain} routed to NotifyBC inbound smtp server.
  • when a notification finished dispatching, the dispatching start and end time is recorded to all bounce records matching affects recipient addresses
  • when inbound smtp server receives a bounce message, it updates the bounce record by saving the message and incrementing the hard bounce count when the message matches the filter criteria. The filter criteria are regular expressions matched against bounce email subject and body, as well as regular expression to extract recipient's email address from bounce email body. It also unsubscribes the user from all subscriptions when the hard bounce count exceeds a predefined threshold.
  • A cron job runs periodically to delete bounce records if the latest notification is deemed delivered successfully.

To setup bounce handling

  • set up inbound smtp server

  • verify config email.bounce.enabled is set to true or absent in /src/config.local.js

  • verify and adjust unsubscription threshold and bounce filter criteria if needed. Following is the default config in file /src/config.ts compatible with rfc 3464open in new window

    module.exports = {
  email: {
    bounce: {
      enabled: true,
      unsubThreshold: 5,
      subjectRegex: '',
      smtpStatusCodeRegex: '5\\.\\d{1,3}\\.\\d{1,3}',
      failedRecipientRegex:
        '(?:[a-z0-9!#$%&\'*+/=?^_`{|}~-]+(?:\\.[a-z0-9!#$%&\'*+/=?^_`{|}~-]+)*|"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])',
    },
  },
};

    where

    • unsubThreshold is the threshold of hard bounce count above which the user is unsubscribed from all subscriptions

    • subjectRegex is the regular expression that bounce message subject must match in order to count towards the threshold. If subjectRegex is set to empty string or undefined, then this filter is disabled.

    • smtpStatusCodeRegex is the regular expression that smtp status code embedded in the message body must match in order to count towards the threshold. The default value matches all rfc3463open in new window class 5 status codes. For a multi-part bounce message, the body limits to the one of the following parts by content type in descending order

      • message/delivery-status
      • html
      • plain text

    • failedRecipientRegex is the regular expression used to extract recipient's email address from bounce message body. This extracted recipient's email address is compared against the subscription record as a means of validation. If failedRecipientRegex is set to empty string or undefined, then this validation method is skipped. The default RegEx is taken from a stackoverflow answeropen in new window. For a multi-part bounce message, the body limits to the one of the following parts by content type in descending order

      • message/delivery-status
      • html
      • plain text

  • Change config of cron job Delete Notification Bounces if needed

List-unsubscribe by Email

Some email clients provide a consistent UI to unsubscribe if an unsubscription email address is supplied. For example, newer iOS built-in email app will display following banner

list unsubscription

To support this unsubscription method, NotifyBC implements a custom inbound SMTP server to transform received emails sent to address un-{subscriptionId}-{unsubscriptionCode}@{inboundSmtpServerDomain} to NotifyBC unsubscribing API calls. This unsubscription email address is generated by NotifyBC and set in header List-Unsubscribe of all notification emails.

To enable list-unsubscribe by email

  • set up inbound smtp server
  • verify config email.listUnsubscribeByEmail.enabled is set to true or absent in /src/config.local.js

To disable list-unsubscribe by email, set email.listUnsubscribeByEmail.enabled to false in /src/config.local.js

module.exports = {
  email: {
    listUnsubscribeByEmail: { enabled: false },
  },
};