Data Science vs Software Engineering?

Hotwire LuaghOn
Apr 16 22 Comments

I'm working as a software engineer for almost 3-4 years now. I've been working with data and like the data engineering part - transforming data in Storm, Spark, etc.. I feel the growth is very limited though. Both internally and externally. Firstly, there's little I can do about the quality of work that we get to do and opportunities for me to learn and upgrade my skills. Moreover, I find it's very often about the no of years of experience you have and the people whom you might know, than about your talent or abilities. It's about being in the right place with the right people, more than about knowing the right thing to do. Also, I don't like going on call over the weekends at all. So, I'm wondering if is should quit my job and focus on learning something more valuable, like data science, which I imagine, would be higher paying, not require weekend on call and more rewarding for my career overall. Any thoughts or suggestions? I'm not that microservice/service/full stack/core Java/dependency injection guy and I don't want to be.

Facebook Google Expedia Group
22

comments

Want to comment? LOG IN or SIGN UP
  1. TOP
  2. RECENT
  3. OLD
 TOP 22 Comments
  • Google
    AIMLOK

    Google

    BIO
    I once served 5TB
     AIMLOKmore
    TC higher as SWE
    Apr 16 13
    • Google
      AIMLOK

      Google

      BIO
      I once served 5TB
       AIMLOKmore
      Wasn't data one of the reasons why they developed distributed systems? Nah... They did it because of perf
      Apr 16
    • Amazon / Eng DynamoDB
      'we have already found and sorted' Not fully. I'm in AWS and while our systems are decent, there's still always improvement to be made in architecture. And with the constant growing of data, a good distributed systems architecture is very important!

      It's also the answer to network problems. Fault tolerance is also difficult to achieve across all systems.
      Apr 16
    • Hotwire LuaghOn
      OP
      Yeah but data is the juice. I find harnessing the data more appealing than gathering it at a large scale.
      Apr 16
    • Amazon / Eng DynamoDB
      It's not just gathering though. Yes storage is a problem, but anything you do on your local machine (downloading, processing, metrics, etc) - there is an analogous problem when dealing with it at scale. That's distributed systems. And then ofc the shitty network problems come in too
      Apr 16
    • Hotwire LuaghOn
      OP
      Yeah, it's basically one category of problems in different variations. Each data set is different.
      Apr 16
  • Two Sigma not nice
    I’m a hybrid SWE+DS type. First, don’t use the phrase DS. It means a lot of different things to different people. My ideal definition is someone who is strong at SWE, stats/ML/math, and has domain knowledge so that they can analyze complex data sets and produce high level insights. However, most people interpret it as someone who can do basic model fitting in python or R and knows what a histogram is.

    Second, based on what you described as your interest, being a SWE is probably the best role for you, but you also need to find a job that enables you to grow in the more analytical direction. There are a ton of opportunities out there for people with hybrid SWE+DS skills right now. Most of them are labeled as ML roles.

    Two Sigma has need for people like this, but so do a lot of other places. Having Storm and Spark experience will help you get an interview, but you’ll probably also need to be solid at leetcode to pass the SWE interviews.

    Good luck!
    Apr 16 1
    • Hotwire LuaghOn
      OP
      Thanks. I'll check out the opportunities.
      Apr 16
  • Seagate toti420
    You imagine DS would be higher paying than SW? What universe do you live in? 😂
    Apr 16 5
    • Hotwire LuaghOn
      OP
      Because you become DS after studying SWE. Right?
      Apr 16
    • Google
      AIMLOK

      Google

      BIO
      I once served 5TB
       AIMLOKmore
      Not really true.
      Apr 16
    • Seagate toti420
      No! Your typical data scientist has background in some analytical field, is familiar with basic probability and statistics, and can code in say Python and/or R. They can perform data mining and data analysis. They can (should be able to) perform some basic statistical learning like regression, classification, clustering and so on. They can (should be able to) write simple to intermediate SQL queries. A data engineer is more SW oriented, but neither (talking topical cases here) can rightfully sell themselves as SDEs. I.e, they lack fundamental knowledge in data structures and algorithms, design, OS, OO, etc.
      FYI - I am a data scientist, PhD+MS, reasonably versed in stats and can write decent code in Python, have a good knowledge of data structures, some familiarity with algorithms, but I’d totally fail a SWE interview. Even medium LC problems pose a challenge to me.
      Apr 16
    • Hotwire LuaghOn
      OP
      Lol. By data scientist I'm referring to the guy who knows and needs to know various machine learning approaches and the math behind them. He know supervised, unsupervised learning and the myriad of other such algorithms and techniques, which is way more than a typical software engineer build ant jobs in Java or some other framework. Typically DS positions require a masters or Ph.D. in some related discipline and they need to know so many DS algorithms. Similarly, with data engineering, you need to know the right tools and techniques to use, because you can't build a distributed system from scratch for your use case. SWE is typically solving such problems at a small scale. Like building a web application usually that would communicate with a backend. There's such a lack of depth to SWE that very often all you can boast about is SWE while DS is an evolving field requiring you to know more and more. Similarly, data engineers have to keep up with the scale of the data and forming the backbone of data science.
      Apr 16
    • Seagate toti420
      There are data scientists like most of the folks in FB’s Analytics team, then there are folks like Yann LeCun. There are software engineers like most SWs at FB, who save passwords in text files, then there are folks like Peter Norvig. You’re probably thinking of the latter examples. Yes, you would need decades to get there.
      Apr 16