Real-time experience of a practitioner

Analytics Project | not only what we do, but also how we do it

Sarit Maitra

Jul 19, 2020·9 min read

Analytics projects often comes with uncertainty and a high implementation risk and thus demand different approach in project management. There is more uncertainty around a typical analytics project comparatively. So it certainly takes some special skills to execute and deliver analytics project. Analytics projects fail to achieve desired outcome more frequently than we would like to admit. Enough time needs to be spent on understanding the exact business problem and then converting this business problem into an analytics problem that can be solved with data.

Most key stakeholders within an organization will have at least an elementary understanding of the Project Management life-cycle. They probably do not have much exposure to the typical analytics project life-cycle. Here I will cover some of the finer points of analytics project which gave me success. However, there are failures too which has helped me to learn.

Despite the great excitement about big data, better analytics tools and the vast resources that many organizations are investing in growing their teams and technology, multiple surveys of data analytics groups report that most analytics projects fail to provide real business value. Max Henrion

Here are some of the failure statistics:

• July 2019: VentureBeat AI reports 87% of data science projects never make it into production
• Jan 2019: NewVantage survey reports 77% of businesses report that “business adoption” of big data and AI initiatives continues to represent a big challenge for business.
• Jan 2019: Gartner says 80% of analytics insights will not deliver business outcomes through 2022 and 80% of AI projects will “remain alchemy, run by wizards” through 2020.
• Nov. 2017: Gartner says 60% of bigdata projects fail to move past preliminary stages.

Well, there are many of such analysis available; but the objective here is not to discuss the failure stories but to think how can we make an analytics initiative into a positive outcome. How out-of-the-box thinking can help to sail through the difficult times of change management.

The processes such as scientific approach, defining problem statement, devising hypothesis which most of the time begins with a question and subsequently data analysis, model development, hyper-parameter optimization, defining accuracy metrics, presentation of outcome etc. are necessary steps and skills. But are these enough to see our developed model going into production or these are just the beginning of our journey to an unknown territory ?

Challenges

There are four critical challenges that most often I came across so far in analytics projects. These are:

1. The models for meeting the uncertain requirements are often ambiguous; they may be new to the analytics team, or they may not even exist. This adds to the exploratory nature of analytics projects.
2. To add more to the trouble, users expect responsiveness; the expectation is that the applications should be highly responsive to user interaction. So, a balance between responsiveness and robustness here.
3. Analytics models are inherently statistically complex with mathematical rigor involved. Therefore, users find it difficult to put their faith in quantitative modeling.
4. Change management has always been challenging given the general level of discomfort among people with mathematics and math-based solutions.

Only 20% of the data science and analytics models actually get implemented

Doug Gray has written a nice article on this from his vast experience and I would agree to most of it. In fact, many analytics projects pass the pilot test but fail to earn wide adoption. Sometimes analytics projects fail for reasons similar to traditional IT projects, and sometimes they fail for very different reasons that are more domain-specific. Doug has shared top 10 reasons for failure of analytics project which are true in real-sense:

1. Insufficient, Incorrect, or Conflicting Data
2. Failure to Understand the Real Business Problem
3. Misapplication of the Analytics | Data Science Model
4. Solving a Problem No One Cares About
5. Poor Communication | Business Interpretation of Results
6. Change is Disruptive & Not Handled Well
7. Unrealistic Expectations
8. Poor Project Management
9. Excessive Focus on the Model, Technique, or Technology
10. Lack of Empathy

Analytics Project Management

As I see that, there are four broad phases of Project Management in Analytics application which involve:

1. Project phases,
2. Work breakdown structure (WBS),
3. Risk management &
4. Communication plan.

Important here is that, effective Project Management ensures open communication and agreement at all phases of the analytics project. Analytics projects often require a tailored approaches considering uniqueness for each project. Data which is the core of any analytics project and mostly an issue in-terms of availability and quality. Though PMI offers a hand-full of resources in regard to effective Project Management, but what always helped me is, to gain experience from other experienced project managers.

Analytics Project Management

I will discuss few approaches here. Practice of these help me in daily course of time while working with the stakeholders and team members.

Methodological approach

In methodology, I would always like to follow agile Project Management with scrum meetings which is an iterative, adaptive approach. This works through multiple iterations of testing, and saves time by making sure the work done throughout the project is aligned to what the business needs; it also takes care of any business changes.

Moreover, in real business case, it is unlikely to have a complete grasp problem statement at one go and of we need to plan for several intermediate feedback sessions for better understanding. Additionally, I always allocate some time for myself to revisit learning from previous projects and also for research activity. The Analytics and Predictive Modeling field is not matured yet and constantly evolving and thereby study and research activity helps me in understanding of new modeling techniques most of which appears in academic journals.

Realistic approach

In business, most of the times, the problems and objectives are not well defined. As an example, the goal is defined at a very high level like “what are the factors that bank is losing revenue or clients?” This kind of high level goal comes with a hidden expectation of a super or hyper-intelligent algorithm to provide a quick answer. It is always advisable to break-down the high level problem statement into multiple easy to handle questions and fit algorithms as accordingly. Moreover, a seasoned Project Manager would be able to foresee the issue and set the expectation by defining the clear goal which can be accomplished withing the given constraint of budget and time. As MIT Solan Management Review has indicated in one of their blog post that, throwing the toughest challenge of most strategic importance at the analyst might not be the easiest path to success.

Modeling is Iterative

How long it will take to build a model?

It is a simple question and I have come across to many experienced analytics consultants who come up with their smart answers with some numbers. But is there really a quick answer available to that question?

We are talking about an analytic modeling (predictive modeling) and there are many dependencies associated with it. The model building stage is iterative in nature that we discussed. We may need to build multiple models, use different techniques, and try different kinds of data transformations before finalizing on the final approach. Project Manager plays a crucial role here in ensuring the trade-off between model accuracy and time spent is done sensibly.

“All models are wrong, but some are useful,” G.E. Box.

However, going back to the question that we are facing, I would probably insist the business owner to allocate a certain time budget to develop such model and accept the resulting quality as the best within the given time. We directly test underlying assumptions behind relationships among exogenous and endogenous variables. We indicate model predictions and test for consistency of our evaluated parameters. By conducting empirical study and using real world data, we describe statistical significance of our tested parameters. We compare our experimentally obtained results with the specified model prediction(s) and decide on the latter’s validity.

A Business Model is a representation of the reality ( of a set of business activities) where the modelling exercise is to faithfully abstract the core activities into a model which is an abbreviated but accurate representation of the reality.

So, how long is not really the problem, but what level of model accuracy are we willing to compromise on. Moreover, model accuracy will be effected by data availability and data quality. Prior to exploratory analysis, we will have no clue on that.

Monitoring and Controlling

Monitoring and controlling an analytics project is equally important as planning for successful execution. Moreover, track and understand model performance in production is important from both a data science and operational perspective. A smart Project Manager would be able create an environment where issues can be raised without the risk of judgment. The sooner an issue is realized, the sooner that issue can be addressed.

In this context, I would like to highlight that, although academic ML stared researching somewhere from 1980s; however, other than a few exceptions, most tech companies have only been doing ML/AI at scale for a few years, and many are only just beginning the long journey. This means that:

• The challenges are often unclear or not fully understood.
• Data science and ML are rapidly changing and so the frameworks and tooling; best practices are often grey.
• The regulatory requirements are changing all the time.

Project Estimation

Here, I would sometime shadow a seasoned professional in the team to take the advantage of his/her experience. I have no qualms about asking help or taking help from others. Sometime this involves conducting workshop which requires preparation too apart from follow-up. So, all these have to go into the estimation process.

Data collection & exploration

Data for an analytics project could be coming in from multiple sources. Therefore, closely working with data analysts is crucial here who will need to identify the various sources of data. There are various dependency involves here, e.g. IT team to extract this data for analysis and other teams. Moreover, the effort to extract data can vary considerably from one data source to another. This makes it hard to estimate the exact time required for this step and the Project Manager deals with a lot of uncertainty here.

Moreover, through data exploration, we get a better understanding of the information available. Often, we identify data issues at this stage — missing data, anomalies, incorrect data etc. Therefore,sufficient time needs to be budgeted for this stage to avoid re-work later.

When to give up?

Not every analytics project reveals brilliant insights. Sometimes, it just doesn’t happen. It could be because of lack of relevant data, data inconsistencies or a variety of other reasons. The project manager’s role can be crucial here in knowing when to stop, when to put an end to the project and what to go back to the business with. Therefore, expectation management plays an important role here.

Customer engagement

So, we are aware that, we have the required technical skills to clean and analyze the data, and apply mathematical rigor & statistical models, ML methods, optimization tools and create compelling visualizations. Is it possible that we are spending too much time trying to understand the data and building models, and too little trying to understand our clients? We already have seen 10 key reasons for a project to fail.

Customer or stakeholders engagement is crucial from a consultative approach. But unfortunately, we tend to spend less time with them to understand and clarify their objectives and decisions or brainstorming session with them. Many a time we get sucked into organizing poorly structured data and debugging complex spreadsheets. Client engagement is really crucial to develop client confidence and to communicate the real issue which are often appreciated. So, relationship management plays an important role here. So, here I play a balance between hard and soft skills.

Key takeaways

Conclusion

Most of the analytics projects are in experimental phase. The technology is not matured yet and constantly evolving. Often someone’s claim of knowing everything comes with ignorance of many thing. Early planning may not be a good idea for experimentation-based approach considering experimental and evolutionary design goes through significant learning and change. Therefore, I would say that, delivery is more important than focusing on the original plan. The motivation to me, when I see the real use of the developed model. It gives me some kind of satisfaction realizing that I have contributed something positive to the business. If the business processes are not aligned with the developed model, or if the business doesn’t understand the definitions used, the models often do not see the light of the day.

I can be reached here.