How to simplify complex data

On complexity, Albert Einstein is often quoted as saying, “Everything should be made as simple as possible, but not simpler.”

However…Einstein did not actually say this.

No, that quote happens to be a paraphrase of a longer, more complex (real) quote: “It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.”

Did you finish reading that second quote? It’s quite hard to understand, right? The paraphrased quote makes Einstein’s point but in simpler terms. Ironically, if Einstein had heeded his own advice, maybe he would have come up with the shorter quote.

The role of the data analyst is exactly that: to turn complex data into data that’s “as simple as possible, but not simpler”. Or, data that is reduced down to a useful, actionable insight, but not further.

But what do we mean when we discuss complex data?

Understanding complex data

Data can be complex in two main ways:

1. Data that is extremely heterogeneous, i.e. data that consists of multiple types and formats that don’t combine easily
2. Data which size overloads processing capacity and makes normal data analysis difficult

Data can be heterogeneous in many different ways. It can be due data being differently structured, even if it contains the same information. For example, the same date being written the UK way, the US way, numerically or in words. It could mean messy data, such as data with duplications, missing records and other flaws. Or it could mean unstructured data, like text, images and audio that can’t be represented numerically.

Complex data could also consist of an extraordinary amount of data that poses computational problems based on its sheer size. Like how harvesting a huge field requires a completely different approach to harvesting a small one, even if the basic element – cutting and collecting a crop – is the same. Processing a million rows of data is much different to processing a billion.

Both types of complex data present problems to a data analyst and require simplification before they are usable.

A common cause of data complexity is when an organisation collects everything it can without a plan in place to understand it.

What makes enterprise data complex and challenging to manage?

Enterprise data is challenging to manage partly because it is complex and partly because of inherent complexities of modern organisations.

Enterprise data is all the digital information collected and processed by an organisation. Spanning financial records, sales data, customer data, multimedia content, marketing data and much more, enterprise data presents a challenge to any organisation. Indeed, an enterprise’s data could consist of multiple sets of complex data.

Besides the data itself, organisational complexities also play a big role here. A company needs to abide by various data governance standards. These include internal policies, standards and compliance as well as regulatory requirements. It needs to manage access to data. It has to anticipate future needs, including capacity and processing capability. It needs a strategy to handle data held in legacy systems, which might have material value or be required for compliance. Maintaining data literacy across an organisation as employees come and go requires planning.

Why is simplifying data essential for effective decision-making?

When we talk about simplifying complex data, we can mean three related processes.

1. Making a complex data set simpler by cleansing, transforming, filtering or otherwise manipulating the data
2. Deciding what data is useful to prove a hypothesis
3. Using data visualisation techniques to simplify complex data to communicate its key points

When handling complex data sets, simplifying data is a required step to conduct thorough data analysis. An overwhelming or overly complex data set can cause information overload to even the most skilled data scientist. A data analyst or scientist might find themselves tasked with making sense of a huge volume of messy data.

The first step would be to consolidate and refine it so they can begin to analyse it.

There can be a temptation when presenting data to include as much data as possible. For instance, if presenting on why sales went up, there can be a whole host of reasons, all of which you can show with data. But – most will be peripheral to the main point and their inclusion would confuse rather than clarify, leading to information overload in your audience.

Simplifying data is therefore also the practice of excluding good or accurate but irrelevant data.

The third stage of this data simplification process is using data visualisations. Data visualisations turn a sea of numbers into something an audience can understand. By representing data graphically, it becomes easier to spot, communicate and understand patterns, trends and outliers in data.

With this third simplification complete, you arrive at a stage where your insight is “as simple as possible, but not simpler”. It is in a state where it can be used to make decisions.

What are the fundamental techniques for simplifying complex data?

1. Data aggregation

Data aggregation finds an overview by summarising data, for instance by averaging or totaling data. Aggregation can also mean grouping information by date, location, or type. This helps reduce complexity and facilitates comparison.

2. Data filtering

Data filtering removes irrelevant, less important or corrupted data. Condition-based filtering uses e.g. date ranges and geographic or demographic information to narrow down the data set.

3. Data cleaning

Data cleaning standardises and de-duplicates data, removes incomplete records and fixes broken data.

4. Data normalisation and standardisation

Data normalisation is the process of scaling data to a common range to make it easier to compare. Standardisation is transforming data so that it is useful for algorithms that assume a normal distribution.

5. Data visualisation

Data visualisations make patterns and trends in data sets clear. Usually plotted on an X and Y axis, graphs and charts such as bar charts and line graphs are common visualisations. Dashboards can allow for interactivity between multiple visualisations. Heat maps and geographical maps can be used to show intensity of data points relating to places.

Heat map visualising the most common PIN Numbers used globally. Source: Information Is Beautiful

How can visualisation tools aid in clearer data interpretation?

Tools like Tableau and PowerBI put powerful data visualisation capabilities in the hands of users. They make it easy to create clear and beautiful visualisations that can be read and understood at a glance. Their user interfaces can highlight key insights and emphasise trends. Side-by-side comparisons highlight differences in data. High-quality data visualisations enhance communication, making data easier to interpret.

Tableau, PowerBI and similar programs integrate with other business intelligence tools and applications. They allow businesses to make use of their existing data infrastructure and investments, while benefiting from advanced visualisation capabilities.

Data Literacy Academy offers courses on data visualisation tools, including Tableau, PowerBI and Excel. Confidence with data visualisations is an important pillar of data literacy.

In what ways can automation and AI streamline data simplification?

Artificial intelligence and machine learning (ML) algorithms are transforming data analytics. What AI in particular is useful for is analysing unstructured data, such as text, images, audio and video. Natural language processing (NLP) is a technology that can understand text or audio. Similarly, computer vision can parse and analyse images or videos. This opens up a huge number of potential applications. Any organisation with a large amount of data held in text or multimedia now has a way of analysing it. Data held in paper records can now be photographed and processed without the need for manual data entry.

AI models can also anticipate demand, optimise inventory management and predict maintenance requirements. Altogether, AI and ML provide an ability to quickly and accurately process vast amounts of data in a way that was not possible before.

What are the best practices for simplifying data in Enterprises?

To close out this blog, we’re going to run through some best practices that any company can use to simplify their data.

1. Manage data needs and data quality. Decide which metrics and data points are most important for your business objectives and who/which roles will use the data. Regularly clean, de-dupe and standardise your data.
2. Use data handling tools. Deploy extract, transform, load (ETL) tools automate data collection, transformation and loading processes. Data warehousing centralises data in a data warehouse to provide a single source of truth.
3. Leverage data visualisation. Visualisations are the cornerstone of communicating data for purposeful decision-making.
4. Automate where possible. AI and ML are incredible (if sometimes expensive) tools to simplify data at scale.
5. Promote data literacy. Upskilling a workforce in data literacy is a game-changer when it comes to making data-driven decisions. Offer training so all staff are comfortable communicating data and conducting analysis of their own. This is the best way to identify new use cases that will drive optimisation across teams.

Taking the next steps

Simplifying complex data is a requirement for enterprise teams that want to make effective decisions with data. By employing best practices such as data cleaning, visualisations, leveraging AI and data literacy, businesses can transform overwhelming data into clear, actionable insights.

Data Literacy Academy’s courses are designed to provide workplaces with a foundation for a healthy data culture. When employees can leverage the tools and insights to generate data analysis and communicate it, it accelerates achieving business goals like optimisation, innovation and growth.

‍