The Data Analytics Blog

Our news and views relating to Data Analytics, Big Data, Machine Learning, and the world of Credit.

All Posts

How To Avoid The Texas Sharpshooter Fallacy In Data Analysis

February 23, 2017 at 11:51 AM

The Texas Sharpshooter Fallacy written on a red barn wall next to a target with bullet holes

The rise of Big Data, data science and predictive analytics to help solve real world problems is just an extension of science marching on. Science is humanity’s tool for better understanding the world. The tools that we use to build models, test hypotheses, look for trends to build value with our brand all derive directly from scientific principles.

With these principles comes a myriad of obstacles. The obstacles are known to philosophers as “logical fallacies”, which I outlined in my previous post "The 7 Logical Fallacies to avoid in Data Analysis."  In this blog post, we focus on the Texas Sharpshooter Fallacy and how to avoid it in your data analysis.

What it is the Texas Sharpshooter Fallacy?

This is a common mistake made by human beings. In essence, it is looking at a large amount of data, identifying small patterns and deriving a conclusion based on the patterns.

The name derives from a story of a Texan marksman who shoots a large amount of bullets at a barn door. He then finds the closest cluster of bullets and draws a target around them and thereby claims that he is a sharpshooter.

How to avoid the Texas Sharpshooter Fallacy

Post-hoc hunting of anomalies and patterns is commonplace in data analytics. There is no real problem with identifying patterns in data through an observational study, but this should result in a hypothesis and not a conclusion. Hypotheses should then be tested against another set of data. To extend our metaphor, the marksman should, after drawing his target, go back and take aim to see whether he can hit the target again.

This is partly why we use hold-out samples when we build models (e.g. we build a model on a random 80% of the population, but will then test the model against the 20% hold-out).

The reality is that all data will have anomalies and we can hunt for these, but we should not rest our conclusions based on these anomalies, we should rather test our hypotheses about the anomalies on hold-out samples, out-of-time tests or new tests.

This logical flaw is well known in applied physics and epidemiology. Certain studies known as “observational studies” may be conducted to look for anomalies in data. These anomalies may be presented, but a conclusion is not drawn as the independent variable is not controlled for. A follow-up study would be a randomised controlled trial to determine whether the results of the observational study could be replicated.

The Texas Sharpshooter fallacy is just one of many statistical pitfalls to avoid in data analysis. Read my post on the 7 Logical Fallacies to avoid in Data Analysis. I'll be covering each logical fallacy covered in my initial blog post on this topic - The 7 Logical Fallacies to avoid in Data Analysis - so make sure to subscribe to our blog to read the new posts in this series.
contact us

Thomas Maydon
Thomas Maydon
Thomas Maydon is the Head of Credit Solutions at Principa. With over 13 years of experience in the Southern African, West African and Middle Eastern retail credit markets, Tom has primarily been involved in consulting, analytics, credit bureau and predictive modelling services. He has experience in all aspects of the credit life cycle (in multiple industries) including intelligent prospecting, originations, strategy simulation, affordability analysis, behavioural modelling, pricing analysis, collections processes, and provisions (including Basel II) and profitability calculations.

Latest Posts

Top 5 LinkedIn Groups To Join On Data Analytics

With LinkedIn usage growing by two new members every second, you simply can’t afford to not be on the platform. Founded in 2003, LinkedIn has 590 million users with 260 million of those active every month.

The EQ Behind The IQ: The Growing Disconnect Between The Data-Driven Business And It's Customers

The EQ Behind the IQ - Jaco Rossouw from PrincipaDecisions

Principa And Atura Announce Partnership To Provide World-Class Chatbot Solutions To The Credit And Collections Industry

Cape Town, South Africa, 20 February 2019 – Principa Decisions, a South African data analytics and software company, and Atura, a software company specialising in instant messaging technology, have announced a partnership that offers the South African credit and collections industry a solution to support originations, customer management and collections via virtual agent chatbots on mobile and desktop devices.