I started my career as a java developer and moved to working with big data development using map reduce( later spark), nosql etc and then using some kind of rest framework(spring/play) to develop and expose the results via api. I did my MS in CS with a focus on parallel data processing. During this whole time I felt that what I am doing is part of core software development. But lately I have been seeing a lot of job posting for data engineers( even in big 4) which require just sql skill set or just the knowledge of using big data tools or other tools. Also sometimes I see that people assume that data engineers are not software devs. This whole experience has made me wonder if I should move from looking at the title of data engineers and instead focus solely on software devs role.
comments
All the other narrow definitions are just HR created necessities.
I understand that we do have a sub domain in Computer Science as Big Data but look at it more like tools, techniques to solve engineering problems. Don't limit yourself to being a specific type of engineer.
That's because, Big Data Engineering typically involves writing scripts, and few M/R jobs. And this doesn't create any visible value towards the product or the user.
User facing teams get more credit (and scrutiny) than other teams. One of the major reason why I also don't like working on internal tools. These tools create very little impact on overall revenue of the product and company.