I started my career as a java developer and moved to working with big data development using map reduce( later spark), nosql etc and then using some kind of rest framework(spring/play) to develop and expose the results via api. I did my MS in CS with a focus on parallel data processing. During this whole time I felt that what I am doing is part of core software development. But lately I have been seeing a lot of job posting for data engineers( even in big 4) which require just sql skill set or just the knowledge of using big data tools or other tools. Also sometimes I see that people assume that data engineers are not software devs. This whole experience has made me wonder if I should move from looking at the title of data engineers and instead focus solely on software devs role.
Agree .. so much bs around...
It’s just Semantics in title, role is essentially the same, what’s more important is the scope of the role? Are you mainly doing the data related clean or infra work and data pipelining?
Its a mix for me right now. Creating data pipelines as well writing analysis job using spark. I have to write apis sometime to expose the results to other teams. Its not the work in current organization which bothers me, its the general scope that other companies have for data engineer which revolves mostly around sql. I would love to do some more low level programming or building a backend of application rather than writing sql query or doing just data analysis using spark wherever I move.
Find a role we’re it’s SWE,DAta or something of the sort.
Layoffs
11h
7863
Google CFO confirms "large-scale" layoffs today (Apr 17)
Health & Wellness
3h
460
Is Seattle to Bay Area worth at 550k TC?
2024 Presidential Election
15h
1374
If you vote for Biden this year, don't complain about layoffs
Tech Industry
15h
1651
What is the appeal to NVIDIA?
Software Engineering Career
4h
2243
28 terrorist worshipping idiots just got themselves fired and I've never been prouder to work at Google.
Always, consider yourself as a Software Engineer. All the other narrow definitions are just HR created necessities. I understand that we do have a sub domain in Computer Science as Big Data but look at it more like tools, techniques to solve engineering problems. Don't limit yourself to being a specific type of engineer.
Yes, thats what my point is writing map reduce or spark jobs require me to use more computer science concepts compared to when I develop rest apis. But people think of working with big data as using some kind of tools like maybe alteryx or informatica.
Correct. There is an implicit bad rep when engineers call themselves Big Data developers. That's because, Big Data Engineering typically involves writing scripts, and few M/R jobs. And this doesn't create any visible value towards the product or the user. User facing teams get more credit (and scrutiny) than other teams. One of the major reason why I also don't like working on internal tools. These tools create very little impact on overall revenue of the product and company.