Notification

Configs in this section customize the handling of notification request or generating notifications from RSS feeds. They are all sub-properties of config object notification. Service-agnostic configs are static and service-dependent configs are dynamic.

RSS Feeds

NotifyBC can generate broadcast push notifications automatically by polling RSS feeds periodically and detect changes by comparing with an internally maintained history list. The polling frequency, RSS url, RSS item change detection criteria, and message template can be defined in dynamic configs.

Only first page is retrieved for paginated RSS feeds

If a RSS feed is paginated, NotifyBC only retrieves the first page rather than auto-fetch subsequent pages. Hence paginated RSS feeds should be sorted descendingly by last modified timestamp. Refresh interval should be adjusted small enough such that all new or updated items are contained in first page.

For example, to notify subscribers of myService on updates to feed http://my-serivce/rss, create following config item using POST configuration API

{
  "name": "notification",
  "serviceName": "myService",
  "value": {
    "rss": {
      "url": "http://my-serivce/rss",
      "timeSpec": "* * * * *",
      "itemKeyField": "guid",
      "outdatedItemRetentionGenerations": 1,
      "includeUpdatedItems": true,
      "fieldsToCheckForUpdate": ["title", "pubDate", "description"]
    },
    "httpHost": "http://localhost:3000",
    "messageTemplates": {
      "email": {
        "from": "no_reply@invlid.local",
        "subject": "{title}",
        "textBody": "{description}",
        "htmlBody": "{description}"
      },
      "sms": {
        "textBody": "{description}"
      }
    }
  }
}

The config items in the value field are

• rss
  • url: RSS url
  • timeSpec: RSS poll frequency, a space separated fields conformed to unix crontab format with an optional left-most seconds field. See allowed ranges of each field
  • itemKeyField: rss item's unique key field to identify new items. By default guid
  • outdatedItemRetentionGenerations: number of last consecutive polls from which results an item has to be absent before the item can be removed from the history list. This config is designed to prevent multiple notifications triggered by the same item because RSS poll returns inconsistent results, usually due to a combination of pagination and lack of sorting. By default 1, meaning the history list only keeps the last poll result
  • includeUpdatedItems: whether to notify also updated items or just new items. By default false
  • fieldsToCheckForUpdate: list of fields to check for updates if includeUpdatedItems is true. By default ["pubDate"]
• httpHost: the http protocol, host and port used by mail merge. If missing, the value is auto-populated based on the REST request that creates this config item.
• messageTemplates: channel-specific message templates with channel name as the key. NotifyBC generates a notification for each channel specified in the message templates. Message template fields are the same as those in notification api. Message template fields support dynamic token.

Broadcast Push Notification Task Concurrency

To achieve horizontal scaling, when a broadcast push notification request is received, NotifyBC divides subscribers into chunks and submits a BullMQ job for each chunk. The chunk size is defined by config broadcastSubscriberChunkSize. All subscribers in a chunk are processed concurrently.

The default value for broadcastSubscriberChunkSize is defined in src/config.ts

module.exports = {
  notification: {
    broadcastSubscriberChunkSize: 1000,
  },
};

To customize, create the config with updated value in file src/config.local.js.

When to adjust chunk size?

Redis memory footprint is inversely proportional to chunk size. Increase chunk size if Redis memory usage is approaching physical limit.

Broadcast Push Notification Custom Filter Functions

Advanced Topic

Defining custom function requires knowledge of JavaScript and understanding how external libraries are added and referenced in Node.js. Setting a development environment to test the custom function is also recommended.

To support rule-based notification event filtering, NotifyBC uses a modified version of jmespath to implement json query. The modified version allows defining custom functions that can be used in broadcastPushNotificationFilter field of subscription API and broadcastPushNotificationSubscriptionFilter field of subscription API. The functions must be implemented using JavaScript in config notification.broadcastCustomFilterFunctions. The functions can even be async. For example, the case-insensitive string matching function contains_ci shown in the example of that field can be created in file src/config.local.js

const _ = require('lodash')
module.exports = {
  ...
  notification: {
    broadcastCustomFilterFunctions: {
      contains_ci: {
        _func: async function(resolvedArgs) {
          if (!resolvedArgs[0] || !resolvedArgs[1]) {
            return false
          }
          return _.toLower(resolvedArgs[0]).indexOf(_.toLower(resolvedArgs[1])) >= 0
        },
        _signature: [
          {
            types: [2]
          },
          {
            types: [2]
          }
        ]
      }
    }
  }
}

Consult jmespath.js source code on the functionTable syntax and type constants used by above code. Note the function can use any Node.js modules (lodash in this case).

install additional Node.js modules

The recommended way to install additional Node.js modules is by running command npm install <your_module> from the directory one level above NotifyBC root. For example, if NotifyBC is installed on /data/notifyBC, then run the command from directory /data. The command will then install the module to /data/node_modules/<your_module>.

Guaranteed Broadcast Push Dispatch Processing

As a major enhancement in v3, by default NotifyBC guarantees all subscribers of a broadcast push notification will be processed in spite of node failures during dispatching. Node failure is a concern because the time takes to dispatch broadcast push notification is proportional to number of subscribers, which is potentially large.

NotifyBC is not only resilient to failures of NotifyBC application nodes, but also entire Redis cluster.

The guarantee is achieved by

1. logging the dispatch result to database individually right after each dispatch
2. when subscribers are divided into chunks and a chunk job fails, the job is re-processed by BullMQ
3. a chunk job periodically updates the notification updated timestamp field as heartbeat
4. if redis cluster fails, a cron job detects the failure from the stale timestamp, and re-submits the notification request

Guaranteed processing doesn't mean notification will be dispatched to every intended subscriber, however. Dispatch can still be rejected by smtp/sms server. Furthermore, even if dispatch is successful, it only means the sending is successful. It doesn't guarantee the recipient receives the notification. Bounce may occur for a successful dispatch, for instance; or the recipient may not read the message.

The guarantee comes at a performance penalty because result of each dispatch is written to database one by one, taking a toll on the database. It should be noted that the benchmarks were conducted without the guarantee.

If performance is a higher priority to you, disable both the guarantee and bounce handling by setting config notification.guaranteedBroadcastPushDispatchProcessing and email.bounce.enabled to false in file src/config.local.js

module.exports = {
  notification: {
    guaranteedBroadcastPushDispatchProcessing: false,
  },
  email: {
    bounce: { enabled: false },
  },
};

In such case only failed dispatches are written to dispatch.failed field of the notification.

Also log skipped dispatches for broadcast push notifications

When guaranteedBroadcastPushDispatchProcessing is true, by default only successful and failed dispatches are logged, along with dispatch candidates. Dispatches that are skipped by filters defined at subscription (broadcastPushNotificationFilter) or notification (broadcastPushNotificationSubscriptionFilter) are not logged for performance reason. If you also want skipped dispatches to be logged to dispatch.skipped field of the notification, set logSkippedBroadcastPushDispatches to true in file src/config.local.js

module.exports = {
  ...
  notification: {
    ...
    logSkippedBroadcastPushDispatches: true,
  }
}

Setting logSkippedBroadcastPushDispatches to true only has effect when guaranteedBroadcastPushDispatchProcessing is true.