A recent post by Alastair Adam on LinkedIn discusses the disastrously high churn rate of data scientists, notes Gary Allemann, MD at Master Data Management.The post asserts that most data scientists change jobs every 19 months, on average. Around 12 months is spent getting up to speed – getting to know the data landscape, building relationships with key stakeholders and some understanding of available data.
According to Niel Burge, “In those seven months when they are hitting their stride, it is known that they spend 60% of their time hunting data (Monday-Wednesday), 20% cleaning and organising data (Thursday) and only have 20% to do real work (Friday). So, you get just over 30 days of productive value from each, for 19 months of salary!”
Alastair suggests that most data scientists leave because of the disconnect between their expectations and reality. While data scientists are told they will be “changing the business” through ground-breaking Artificial Intelligence (AI) and Machine Learning (ML) capabilities, the reality is that much of their time is spent finding and preparing data. It’s this disconnect that breeds discontent.
While organisations seem to be happy to invest in advanced analytics and data science skills and tools, very few are making adequate investments in the supporting capabilities – data governance, data quality and metadata management – that provide business context and reduce the frustrations experienced.
When data preparation work is required, the right tools make it quicker and easier to share and reuse knowledge and capabilities. These tools also support collaborative approaches. Like DataOps, which enables the data scientist to receive support from data engineers and other stakeholders, again increasing productivity.
If we can move from 30 days of productive time over 19 months to 60, or 90, that equates to a 200-300% improvement. More importantly, by making their lives easier and increasing the amount of time they are doing actual data science, maybe we will keep them engaged longer?
Of course, technology is not a silver bullet and it requires discipline to work. But, surely the right technology choices must be part of the solution?