The Art of Making Quality Data Analyses | by Jordan Gomes | Dec, 2023

A couple of weeks ago, I wrote about building systems to generate more quality insights. I presented how you could increase the output of your team by working on areas such as processes, tools, culture, etc., but I never defined what I meant by “quality” — so this week, we’ll do a deep-dive into this concept.

Usually when someone talks about quality in regards to a data study — we immediately jump to “making sure the data analysis is sound and the results are reliable”. I believe this is just one part of the definition. From my 8+ years of experience in analytics, for a data analysis to be “good work” — it has to be a blend of three fundamental elements:

• It answers an actual need with timely precision.
• It is underpinned by a robust, tried-and-tested methodology.
• It is digestible by the organization.

Let’s dive in!

For a data analysis to be truly impactful, it’s crucial that it targets a genuine, well-defined need. This means understanding exactly what problem is being addressed, identifying who it affects, recognizing why it’s relevant at this specific moment, and being clear on how the results of the analysis will be concretely used. The precision of this understanding directly correlates with the value your analysis brings to your end users.

And it is vital to pinpoint an actual need — as opposed to a perceived one. This will ensure that the analysis is not just theoretically useful but practically applicable. It will ensure that on the last day of the project, when you present it to your stakeholder — you don’t get questions such as “so… what now?”. It makes the difference between providing insightful, actionable data and offering information that, while interesting, may not be immediately beneficial.

For instance, a retail company might perceive a need to analyze customer demographics broadly, but the actual need could be understanding the purchasing patterns of a specific age group. The latter directly influences marketing strategies and inventory decisions, thereby having a more profound impact on business operations.

Equally important is the timeliness of the analysis. This aspect comprises two key elements: the relevance of the need at the current time, and the speed of providing the analysis.

• Relevance of the Need: The needs of businesses are often time-sensitive and can evolve rapidly — especially if you are in a fast-paced organization. An analysis that addresses a current pressing issue is far more valuable than one that arrives too late, or one that has been done too early. For example, an analysis of consumer trends in the lead-up to a major holiday season can be invaluable for a business in terms of stocking and marketing, but if it comes after the season has started, the opportunity is lost.
• Promptness of Analysis: The speed at which the analysis is delivered is equally critical — as that feeds into the relevance of the need. And this is an important factor to take into account, as sometimes you might have to make trade-offs between thoroughness of the study vs speed (e.g. if there is a new trend on social media and your company wants an analysis to capitalize on a viral topic — you can’t take 2 months to come back with results).

In summary — the odds of success for your data analysis are significantly greater when it precisely identifies and addresses a real, current need and when it is delivered in a timely fashion, ensuring maximum relevance and impact.

Way too often — I see data analyses that are not using any standard methodology. And while this doesn’t necessarily mean the study won’t be good, you highly reduce your chance of making high quality work by not following a proven methodology.

A structured / standardized approach ensures thoroughness and also enhances the credibility and replicability of the analysis.

One methodology that I find easy to follow is the CRoss Industry Standard Process for Data Mining (CRISP-DM) framework. After almost a decade in the field, that’s still my go-to framework when starting an analysis from scratch. This framework — which is said to be the standard “data science” / “data analysis” process — has 6 main phases:

• Business Understanding: During this phase, the data analyst should be thorough in understanding the “business context” of the ask: what is the pain point we’re trying to solve, what did we do in the past, who are the “actors”, what are the risks, resources, etc. — and also very importantly, what would be the success criteria for the project.
• Data Understanding: This phase involves getting acquainted with the data — it is about descriptive & exploratory analysis of the data, and the identification of data quality issues. It’s your own “preliminary survey,” where you start to grasp the nuances and potential of the data.
• Data Preparation: This phase is about selecting the data you want to work with — with the rationale for inclusion/exclusion — then cleaning and transforming the data into a format suitable for analysis. It’s like preparing the ingredients before cooking a meal — essential for a good outcome.
• Modeling: The idea of “modeling” can be daunting for some people — but modeling can be as easy as “creating a certain threshold” for a true/false metric (for instance, if your project is understanding/defining churn). During this phase, various modeling techniques are applied to the prepared data, so that you can benchmark them against each other and understand which ones are the most successful ones.
• Evaluation: The models are now critically assessed to ensure they meet the business objectives, and the success criteria that was set in phase #1. This often leads to insights that you can use to circle back to and revise your business understanding.
• Deployment: The final phase involves applying the model to real-world data and situations, effectively putting the analysis into action, and starting to use the insights to improve the operations of the team.

This framework increases the odds that your analysis is more robust by forcing you to go through those different steps — while leaving room for creativity.

Digestibility is not just about simplifying complex information and making your slide deck easier to understand. It involves two integral aspects: (1) fostering a deep level of comprehension from the audience, and (2) enabling them to apply these insights in a practical, impactful manner. This process is akin to how the body not only breaks down food but also utilizes the nutrients to fuel various functions.

Fostering a Deep Level of Comprehension from the Audience

Achieving this requires making the data accessible and resonanting with the audience. This is where subject matter experts (SMEs) play a crucial role. By involving SMEs early in the analysis process, their domain knowledge can guide the framing and interpretation of the data, ensuring that the analysis aligns with real-world contexts and is presented in a way that is meaningful to the intended audience.

Another key strategy to enhance digestibility is the implementation of a ‘stage-gate’ process, involving regular check-ins and updates with the stakeholder or receiving team. This approach avoids overwhelming them with a bulk of complex information at the end of the study. Instead, stakeholders are brought along on the journey, allowing them to assimilate new insights gradually. It also opens avenues for continuous feedback, ensuring that the analysis remains aligned with the evolving needs and expectations of the audience.

Imagine you are in a large organization implementing a new data-driven strategy. If the data team only presents the final analysis without prior engagement, stakeholders may find it challenging to grasp the nuances or see its relevance to their specific contexts. However, by involving these stakeholders at regular intervals — through periodic presentations or workshops — they become more familiar with the data and its implications. They can offer valuable feedback, steering the analysis towards areas most pertinent to them, thus ensuring that the final output is not just understandable but immediately actionable and tailored to their needs.

Enabling The Audience to Apply the Insights

Actionability revolves around translating this deep comprehension into real-world applications or decisions. It’s about ensuring that the audience can effectively utilize the insights to drive tangible results. It is about really thinking about the “last mile” between your analysis and real-life impact, and how you can help remove any friction to adopt your insights.

For instance, if you are working on a project where you goal is to define user churn — making your study more digestible might include you creating a dashboard allowing your business stakeholders to understand what concretely your results looks like.

Other ideas include running workshops, developing interactive visualizations, etc. — anything to make it easier for the team to hit the ground running.

In summary — the digestibility of a data analysis project is significantly enhanced by involving SMEs from the outset and maintaining ongoing communication with stakeholders. This collaborative approach ensures that the study is not only comprehensible but also directly relevant and valuable to those it is intended to benefit.

Successful data analysis is an amalgamation of technical proficiency, strategic alignment, and practical applicability. It’s not just about following a set of steps but understanding and adapting those steps to the unique context of each project. Timeliness, proper execution, and addressing real organizational needs are the pillars that support the bridge connecting data analysis with organizational success. The ultimate goal is to transform data into actionable insights that drive value and inform strategic decision-making.

Hope you enjoyed reading this piece! Do you have any tips you’d want to share? Let everyone know in the comment section!

PS: This article was cross-posted to Analytics Explained, a newsletter where I distill what I learned at various analytical roles (from Singaporean startups to SF big tech), and answer reader questions about analytics, growth, and career.