The Data Analytics Blog

Our news and views relating to Data Analytics, Big Data, Machine Learning, and the world of Credit.

All Posts

Model validation and adjustment

September 7, 2020 at 3:32 PM

The time is NOW for model validation and adjustment.

One of the major premises used in credit scoring is that “the future is like the past”. It’s usually a rational assumption and gives us a reasonable platform on which to build scorecards whether they be application scorecards, behavioural scores, collection scores or financial models. That is reasonable until something unprecedented comes along. You can read about this black swan event in our previous two blogs here and here.

In those blogs we covered how and where we believe models will be affected by the COVID-19 crisis. We also went into more detail in a follow-up VLOG series, COVID 19-in-Credit on our YouTube channel. Be sure to check them out if you have not already.


Principa has been conducting our Analytics ICU programme with a variety of lenders to help them understand how their models have changed and to assist them with determining appropriate remedial action. Get in touch with us to find out how we can help you.

COVID-19 and credit consumers’ behaviours

So, what do we do when the future is not like the past? As the COVID-19 crisis continues to grip local and global economies, so the credit consumer’s behaviours are changing. Many have lost or may still lose their jobs, many more have had their income reduced. Lenders offered payment holidays to most borrowers (you can find out more about this here), but for most (in South Africa) that time of financial relief has come to an end. Some businesses collapsed during the shock period (first three months); others collapsed during the immediate aftermath; those businesses still standing will need to adapt to the new-normal – whatever that might be in their industry…COVID19 and credit consumers behaviourWhat is clear is customer behaviour is likely to change and we expect that most models will be heavily affected over the next few months. So, what is to be done?

How can you adapt your models for COVID-19 changes?

At Principa we advocate the following three step process:

  1. Validate

  2. Re-calibrate

  3. Accommodate



All models and strategies should be validated during this time. This involves understanding the current environment with, for example, payment holidays and stressed economic conditions; looking at the scores and model variables; and looking at how the models are being used.

Recommendations can be drawn from the score distribution stability, character stability, risk-based-pricing strategies, IFRS9 flow-rates comparisons, etc. A plan also needs to be drawn up to understand how the models should be adjusted and how they should be used in the short, medium and longer term. Principa’s Analytics ICU programme can assist your business with this process.


Covid model validation - recalibrate

One of the major anticipated trends is the score-to-odds shift of models. All of this is going to mean the anticipated bad-rates are going to be higher at any given score. As strategies have been set on expected bad-rates, so the scores will need to be re-aligned or re-calibrated to represent the expected score-to-odds relationship. These could be through linear-alignments, step-wise alignments or non-linear alignments. Most scorecards expect a significant performance period (12 months being the most common). But we obviously cannot wait 12 months before acting, so instead we need to work off proxies. Proxies can be drawn in many ways dependent on the scorecard. For behavioural scorecards – instead of 3+ ever at 12 months you may consider looking at a 2+ ever at 3 months as an example. The definition you settle on will be dependent on many factors. Use your vintages as a guide.



It may be that certain scores will weaken during this time with significant volatility expected. That’s not to say the models will not work at all. Strategies should therefore be amended to accommodate the weakened models. There will be more manual overrides expected in, for example, originations or with IFRS9 models. When setting your strategies or running provisions, generate various scenarios to help better understand the implication of the changes. Be careful not to double-count during this time.

Should we rebuild our models?

As the new-normal is quite different to where we were, say, 12 months ago, so the models may not be appropriate now and may also not be appropriate going forward (for example 12 months from time). The question is:

When should I rebuild?

Whilst traditional models need a good 24 months of data for a rebuild, this is not possible right now, but if your models are not working what can you do? The plan should be to look into a full rebuild sometime in the future, but in the mean time you should consider scorecard alternatives like Short Outcome Strict Definition – whilst not perfect they will best represent the current environment. As these models have a shorter lifespan, you may want to consider deploying them as Quick Step Machine Learning Models.

How can Principa help you?

The time is NOW to address your models and how to best navigate this crisis. Get in touch with us. We have over 20 years of experience in helping companies charter through market uncertainty in scorecards, strategies, and financial models. We could help you too. It’s time to act. The future may not be currently like the past, but as the inspirational quote goes “The past is in your head, the future is in your hands.”

To better understand how pay-holidays are affecting your book, please contact us on


Contact Us to Discuss Your data analytics Business Requirements

Thomas Maydon
Thomas Maydon
Thomas Maydon is the Head of Credit Solutions at Principa. With over 17 years of experience in the Southern African, West African and Middle Eastern retail credit markets, Tom has primarily been involved in consulting, analytics, credit bureau and predictive modelling services. He has experience in all aspects of the credit life cycle (in multiple industries) including intelligent prospecting, originations, strategy simulation, affordability analysis, behavioural modelling, pricing analysis, collections processes, and provisions (including Basel II) and profitability calculations.

Latest Posts

The Pros and Cons of a Multi-Bureau Strategy in Credit

Although not a new concept, very few lending organisations have deployed a true multi-bureau strategy (MBS). It is however talked about fairly regularly, but often dismissed as “too hard” or “not important enough”. So why should you consider a multi-bureau strategy? What are the key considerations? How do you go about deploying a MBS? This blog hopes to address all these questions.

PART 1: How to Cure the Post Pandemic “Collections” Symptoms

It has been a year and a half since the first case of the coronavirus (COVID-19) was reported from Wuhan, China. As we move into the third wave of the virus, there is an apparent dilution in both collection and recovery yields in the financial services sector, primarily because relief schemes and packages come to an end.

Predicting Customer Behaviour

Propensity modelling attempts to predict the likelihood that visitors, leads and customers will perform certain actions. It’s a statistical approach that accounts for all the independent and confounding variables that affects said behaviour. The propensity score, then, is the actual probability that the visitor, lead, or customer will perform a certain action.