Why is Data Science such a bad career?
Dec 16, 2020
201 Comments
Media keeps saying Data Science is the sexiest career of 21st century, turns out it is the worst career choice for many. Every DS position is inundated with applications and it is so hard to get any DS position in the US
comments
Lower career ceiling than SWE (DS maxes at 300K)
Non-transferable domain specializations
Then there are the true Data science jobs where true ML, math, quant, algorithms are used, outputs are produced, dollars are saved, and ML pipelines are put in PROD. These are very few. This is the sexiest job - not the former one.
90% of the Data science projects are run on laptops, that’s where they end - only 10% end up in PROD servers. They don’t even make it to dev data platforms.
The thing that gets distorted by ENG's that become DS vs SCI to DS is that there aren't textbook solutions that you can plan to "just work" on the shit data that no one knows is shit, or represents the wrong business concept.
Keep in mind OP that the 90% failure rate of projects is because you need to have a scientific approach to the fundamental business problem, which PHds and ENGs don't appreciate at first. Also going to a company that has a proper ML platform like metaflow working in their environment makes a huge difference for making ENG to DS happy with trying to match their traditional productivity expectations.
1) Being the enemy/Life of disruption. DS is expected to find opportunity from the data. In reality, most change is a win-lose proposition, that usually means exposing something broken. Only the very strongest people can repeatedly message the need for change, creating losers in the company, and survive and it takes masterful communication and stakeholder management skills. It's incredible how change-avoidant most business leaders are and it's for very, very good reason---change turns the boys club where not much gets done, but boy does everyone sure get along, into a battleground with winners and losers. Most mid-senior management consider their #1 job to be "don't rock the boat".
2) Position of weakness. You are constantly searching for opportunities, but never get to own a problem with pre-existing buyin like a SWE or PM. A DS is expected to find problems, influence others to solve them, and move on---this doesn't scale or imply a need for headcount. On the other hand, a SWE org would expand linearly as each problem needs a SWE owner. Hence product (SWE/PM) empires are easy to create and leaders are hard to remove---they have a position of strength, owning the "means of production" (CODE and ORG) as opposed to the DS position of weakness where you own nothing.
3) The 'measurement' career is not measurable / you don't keep what you kill. Most SWE at big companies realize sharing ideas is a loser proposition as credit-stealing is pervasive in most corporate environments. They overcome this by only generating enough ideas for their own or team scope---and they own delivery of solutions. They "keep what they kill". As a DS, you have to structure your interaction with teams extremely clearly to leave enough of a paper trail to document influence---think of a million post its saying "I thought of that". A year later, when the solution lands, will anyone remember? You don't "keep what you kill".