Big Data Engineer vs Data engineer

Fidelity Investments N@N
Oct 4 7 Comments

I started my career as a java developer and moved to working with big data development using map reduce( later spark), nosql etc and then using some kind of rest framework(spring/play) to develop and expose the results via api. I did my MS in CS with a focus on parallel data processing. During this whole time I felt that what I am doing is part of core software development. But lately I have been seeing a lot of job posting for data engineers( even in big 4) which require just sql skill set or just the knowledge of using big data tools or other tools. Also sometimes I see that people assume that data engineers are not software devs. This whole experience has made me wonder if I should move from looking at the title of data engineers and instead focus solely on software devs role.

comments

Want to comment? LOG IN or SIGN UP
TOP 7 Comments
  • Cisco / Eng greedyengr
    Always, consider yourself as a Software Engineer.

    All the other narrow definitions are just HR created necessities.

    I understand that we do have a sub domain in Computer Science as Big Data but look at it more like tools, techniques to solve engineering problems. Don't limit yourself to being a specific type of engineer.
    Oct 4 2
    • Fidelity Investments N@N
      OP
      Yes, thats what my point is writing map reduce or spark jobs require me to use more computer science concepts compared to when I develop rest apis. But people think of working with big data as using some kind of tools like maybe alteryx or informatica.
      Oct 4
    • Cisco / Eng greedyengr
      Correct. There is an implicit bad rep when engineers call themselves Big Data developers.

      That's because, Big Data Engineering typically involves writing scripts, and few M/R jobs. And this doesn't create any visible value towards the product or the user.

      User facing teams get more credit (and scrutiny) than other teams. One of the major reason why I also don't like working on internal tools. These tools create very little impact on overall revenue of the product and company.
      Oct 4
  • Databricks / Data
    data4u

    Databricks Data

    PRE
    500 Startups
    data4umore
    It’s just Semantics in title, role is essentially the same, what’s more important is the scope of the role? Are you mainly doing the data related clean or infra work and data pipelining?
    Oct 13 2
    • Fidelity Investments N@N
      OP
      Its a mix for me right now. Creating data pipelines as well writing analysis job using spark. I have to write apis sometime to expose the results to other teams. Its not the work in current organization which bothers me, its the general scope that other companies have for data engineer which revolves mostly around sql. I would love to do some more low level programming or building a backend of application rather than writing sql query or doing just data analysis using spark wherever I move.
      Oct 14
    • Databricks / Data
      data4u

      Databricks Data

      PRE
      500 Startups
      data4umore
      Find a role we’re it’s SWE,DAta or something of the sort.
      Oct 14
  • Indeed / Eng uPrj01
    Agree .. so much bs around...
    Oct 4 0

Salary
Comparison

    Real time salary information from verified employees