Data Science vs Software Engineering?

Hotwire LuaghOn
Apr 15, 2019 22 Comments

I'm working as a software engineer for almost 3-4 years now. I've been working with data and like the data engineering part - transforming data in Storm, Spark, etc.. I feel the growth is very limited though. Both internally and externally. Firstly, there's little I can do about the quality of work that we get to do and opportunities for me to learn and upgrade my skills. Moreover, I find it's very often about the no of years of experience you have and the people whom you might know, than about your talent or abilities. It's about being in the right place with the right people, more than about knowing the right thing to do. Also, I don't like going on call over the weekends at all. So, I'm wondering if is should quit my job and focus on learning something more valuable, like data science, which I imagine, would be higher paying, not require weekend on call and more rewarding for my career overall. Any thoughts or suggestions? I'm not that microservice/service/full stack/core Java/dependency injection guy and I don't want to be.


Want to comment? LOG IN or SIGN UP
TOP 22 Comments
  • Google


    I once served 5TB
    TC higher as SWE
    Apr 15, 2019 13
    • Amazon / Eng DynamoDB
      It's not just gathering though. Yes storage is a problem, but anything you do on your local machine (downloading, processing, metrics, etc) - there is an analogous problem when dealing with it at scale. That's distributed systems. And then ofc the shitty network problems come in too
      Apr 16, 2019
    • Hotwire LuaghOn
      Yeah, it's basically one category of problems in different variations. Each data set is different.
      Apr 16, 2019
  • Two Sigma not nice
    I’m a hybrid SWE+DS type. First, don’t use the phrase DS. It means a lot of different things to different people. My ideal definition is someone who is strong at SWE, stats/ML/math, and has domain knowledge so that they can analyze complex data sets and produce high level insights. However, most people interpret it as someone who can do basic model fitting in python or R and knows what a histogram is.

    Second, based on what you described as your interest, being a SWE is probably the best role for you, but you also need to find a job that enables you to grow in the more analytical direction. There are a ton of opportunities out there for people with hybrid SWE+DS skills right now. Most of them are labeled as ML roles.

    Two Sigma has need for people like this, but so do a lot of other places. Having Storm and Spark experience will help you get an interview, but you’ll probably also need to be solid at leetcode to pass the SWE interviews.

    Good luck!
    Apr 16, 2019 1
    • Hotwire LuaghOn
      Thanks. I'll check out the opportunities.
      Apr 16, 2019
  • Seagate toti420
    You imagine DS would be higher paying than SW? What universe do you live in? 😂
    Apr 15, 2019 5
    • Hotwire LuaghOn
      Lol. By data scientist I'm referring to the guy who knows and needs to know various machine learning approaches and the math behind them. He know supervised, unsupervised learning and the myriad of other such algorithms and techniques, which is way more than a typical software engineer build ant jobs in Java or some other framework. Typically DS positions require a masters or Ph.D. in some related discipline and they need to know so many DS algorithms. Similarly, with data engineering, you need to know the right tools and techniques to use, because you can't build a distributed system from scratch for your use case. SWE is typically solving such problems at a small scale. Like building a web application usually that would communicate with a backend. There's such a lack of depth to SWE that very often all you can boast about is SWE while DS is an evolving field requiring you to know more and more. Similarly, data engineers have to keep up with the scale of the data and forming the backbone of data science.
      Apr 15, 2019
    • Seagate toti420
      There are data scientists like most of the folks in FB’s Analytics team, then there are folks like Yann LeCun. There are software engineers like most SWs at FB, who save passwords in text files, then there are folks like Peter Norvig. You’re probably thinking of the latter examples. Yes, you would need decades to get there.
      Apr 15, 2019


    Real time salary information from verified employees