Skip to main content

How to A/B/n Test your Dunning Strategy

A/B/n testing is a robust method to determine the most effective communication strategies for your dunning (collections) processes. Receeve provides two specific AI-powered strategy steps that can assist you in testing and optimizing message sending times, as well as channel/content variant tone (check Receeve's AI Optimization Steps). However, there are instances where more complex experiments are desired, such as optimizing communication frequency. Here is how to approach such experiments:

Tools Involved & Pre-Requisites​

To conduct a successful A/B Test, you will require access to the following tools:

  1. Write access to the strategy builder
  2. Access to analytics reporting or a direct connection to our data lake

In broad terms, the design and execution of such a test will encompass the following key components:

  1. A strategy that segments/assigns labels using a specific set of reserved metaproperties
  2. The various strategies or collections of steps you wish to compare
  3. A dashboard that enables you to explore the results and analyze your desired metrics

If this is your first time setting up an A/B test, reach out to your Customer Success contact to receive assistance in configuring all the necessary components.

Step 1: Define Your Objectives​

  1. Determine Goals: Identify what you aim to achieve, such as increasing recovery rates, or improving customer satisfaction.
  2. Select Metrics: Choose key performance indicators (KPIs) like payment recovery rate, response rate, customer satisfaction score, or retention rate.

Step 2: Segment Your Audience​

  1. Create Groups: Divide your audience into n segments randomly to ensure representativeness.
  2. Ensure Adequate Sample Size: Ensure each segment is large enough to produce statistically significant results.

In order to create groups you can use strategy builder to implement whatever segmentation strategy you prefer to use. You can even implement random assignment using Javascript in the advanced mode of an IF or a SEGMENTATION step.

Example: Assign 50% of your claims to variant A and 50% to variant B

In order to flag the claims and mark them as part of each testing group, we recommend using one of the following 5 reserved metaproperties:

meta.pathtag1, meta.pathtag2, meta.pathtag3, meta.pathtag4, meta.pathtag5

You can, for example, use meta.pathtag1 in order to distinguish between multiple A/B/b tests and meta.pathtag2 in order to mark each testing path. You can use these variables at your own convinience.

//Example

meta.pathtag1 = "AB202407" // <-- A/B test name

meta.pathtag2 = "A" // <-- Assigned Variant

These metaproperties are there for your convinience but remember that you can use any segmentation technique (incl. Account or Product based). The segmentation technique you use will then have an impact on how the Data Analysis Dashboards will be configure.

Step 3: Develop Variations​

  1. Craft Different Message Strategies: Develop multiple dunning communication variations (A, B, C, etc.) with different tones, content, subject lines, timings, or channels (email, SMS, phone calls). Play with a different message frequency, etc.
  2. Control Group: Include a control group receiving the current standard communication for baseline comparison.

Distribute traffic to the different variants using an IF or a SEGMENTATION Step

Step 4: Analyze Results​

  1. Compare Metrics: Evaluate the performance of each variation against the control group and each other.
  2. Statistical Significance: Use statistical analysis to determine if observed differences are significant and not due to chance.

We will give you access to a dedicated dashboard that will allow you to analyze the results.

More advanced, ad-hoc analysis can also be prepared by our team in case the standard tools are not enough.

Step 5: Iterate and Optimize​

  1. Identify Best Performer: Select the most effective communication strategy based on your analysis.
  2. Refine and Retest: Make necessary adjustments and conduct further A/B/n tests to continuously improve your dunning communications.

By systematically testing and refining your dunning communications, you can significantly improve recovery rates and customer satisfaction. Regular A/B/n testing allows for continuous improvement and ensures your strategies remain effective and responsive to customer needs.