AB testing, also referred to as “split” or “ABn” testing, is an example of statistical hypothesis testing, a process whereby a hypothesis is made about the relationship between two data sets and those data sets are then compared against each other to determine if there is a statistically significant relationship or not.
In the field of Human Resources AB testing can be used in a number of programmes which include (but are not limited to), the following examples:
- Recruitment – Develop two different campaigns whereby you run them both and see what works best.
- Talent development – You give two top potentials similar assignments, and review after six months or a year who performed best.
- Innovation – Split the team in two, A and B. Give both teams the same assignment and see which team comes up with the best idea.
- On-boarding – New joiners split are split into two groups, A & B. Group A goes through an induction programme and group B goes straight into their job. Which team performs better in the long run and stays longer in the organisation?
- CEO-succession – Candidate A gets Division X and candidate B gets division Y. The candidate whose division performs best gets the job.
Why is it important to perform AB testing and how it can assist you in your job as an HR Professional?
- It enables correct business and or HR decisions. Decisions are based on facts as opposed to intuition. This puts you in a better position when you are presenting your case to your executives or decision-makers.
- It is a low-cost way of validating an insight prior to a rollout. For instance, as a training manager, before you embark on an extensive change in how training is delivered e.g. from face to face to online you might want to first perform an AB test exercise to see if the initiative is fit for your organisation. This could save you a lot of costs, time and trust as you would have tested the initiative before rolling out. AB testing can also assist you in determining which group of employees and or what training programs are online training programs applicable.
- To validate a hypothesis that cannot be proved by analysis e.g. new product or feature. At times you might not have historical data to analyse in order to prove whether your initiative will be effective or not. AB testing can help you as an HR Professional to perform a proof of concept before a full-on rollout.
What are the key critical steps to a successful AB test?
- Success Metric – It is important to identify the test success metrics aligned to business goals. In the training example used above (online vs face to face) you could be comparing satisfaction levels between the two groups. Most importantly you also need to determine if training is aligned to your business goals, because if it is not, it could just be another analysis that no one pays attention to.
- Test Design – have a clean, random, representative sample in the test as well as control. Based on the characteristics of your employees e.g. skilled vs unskilled or semi-skilled, age groups, job roles etc choosing a representative sample is a critical step in order to give you valid results.
- Measurement – significant difference in success metric between test and control. Is there a significant difference in satisfaction levels between employees who attended the same training but through different methods (online vs face to face)? What is the difference, e.g. ( 20% / 30% / 40%) this has to be determined upfront and agreed upon by all the relevant stakeholders?
A Practical Approach to How Split Testing can be Applied in HR
Select Initiative You Want To Improve
The process begins by identifying the initiative that you want to improve in this e.g. (online training vs face-to-face). As a training manager, you want to see which of the training methods (online vs face-to-face) yields better results for the organisation (costs) and employees (satisfaction levels). Select your success metric – employees satisfaction, organisation costs.
Hypothesize A Better Variation
Once you have selected your initiative, create a new variation that can be compared against the original. In our training example employees used to attend face to face training and now you have put a test sample of employees through the same training which is offered online.
Collect Results Data
Collect data on satisfaction levels and cost on both groups of employees who attended the face to face and those who attended online training. Tests must be run long enough to achieve statistically significant results.
Select The Winning Training Delivery Method
Once a method has proven to have a statistically higher satisfaction rate and is cost-effective, it is implemented as a permanent solution.
To learn more about A/B test technique and how it can be applied you can contact info@hranalyticsinstitute.com
Buyile Mbanga
Soc Science (Natal), PostGradDip Mktg & Supply Chain Mgmt (Natal), MBA (Henley)
Founder of HR Analytics Institute & Part-Time Lecturer – Wits Business School
