I read articles that DE is hot and in demand. but I also see that many companies are paying data engineers less than backend SWEs and less than data scientists. So is this just an artificial shortage because the pay is too low? Like McDonald's "can't hire" enough people... at $15/hr in a HCOL area. Those of you who are data engineers, do you love the work? Is it more interesting than being a backend SWE? or is it that the interviews are easier or more specialized? Can you get a DE job without the leetcode and system design required for a SWE, and without the stats and modeling case studies required for a DS? TC 480, L7, YoE: 15. I'm a DS at a company which pays SWE > DS > DE; I do DS and not backend SWE because I enjoy it more even though it pays less. #datascience #dataengineer
From what I've seen, in general DE pays more than DS and at some companies as much as SWE
Do you like capone?
I do yes. Relatively new but it's been a positive experience thus far for sure
I have been working as a DE for 1.5 years. It might be very lucrative but I am done with this. If choosing backend SWE is wrong, I don’t wanna be right.
Could you elaborate please? I’m considering this career option
My simple take: DE is superior to both SWE and DS, the reasoning also simple: No Data, No Dice. Everything sounds great in theory, and you can "refactor" code until you are blue in the face. Push comes to shove, rubber meets the road, data is everything. The data itself is the money. That's why DEs are the most important engineers in the data stack.
By that logic farmers are superior to everyone since we’d all be dead without them
I wouldn't disagree with that logic. They operate on a much lower level of the pyramid, and you are quite right, without them we'd all be dead within 120 days. We received said demonstration about two months or so ago when you couldn't get normal volume of food at stores, and many people did go without food because of that combined with price hikes. Thankfully it was only a partial shortage, because a full shortage would be catastrophic on the level of what many cities in Ukraine or Yemen are currently experiencing.
I am a DS. Personally I think going to data engineering would be terrible, then you just have people like me mad at you all the time. Easier to deal with the business as a DS rather than tech as a DE
This is accurate...unfortunately...
Pardon my ignorance, but why would you be mad at data engineers? I do data engineering (after coming from DS) and I don’t want anyone to be angry with me 😇
Some DE works as a backend software engineer with specialization in data, some DE are analytics engineers writing dbt and sql
This ^
Essentially, the title only refers to the engineer responsible for getting the data from place to place. The role varies by what tasks, tools, and knowledge is required to accomplish that goal. If you have to build API’s (even for internal consumption), pay will be closer to typical backend. Add ML knowledge, even more $$$. If you’re supporting data science by building out tooling to they don’t have to, it’s usually typical SWE salaries. If you’re doing “vanilla” data engineering, it’s less, but more experience doing vanilla DE can get you the roles listed above (but you may have difficulty dealing with DS).
What falls under vanilla DE?
I would consider it the evolved DBA roles - SQL heavy, supporting more corporate application development teams. Closer to IT and Infrastructure. Does that make sense? I can look up some examples if you’d like
The role of a DE varies between companies. Some data engineers are responsible for moving data from one place to another. In these roles the majority of the skill depends on how well the engineer understands the context and domain of the data they are working with. IMO the engineering aspect here is very limited. Then there are data engineers that build data platforms and tools for DS and ML engineers to build pipelines, data models, dashboards etc. Here the focus is on engineering rather than domain knowledge. I would say if your day to day job will be dealing with writing pipelines then you're better off as an SWE.
DE here. The main problem is the title is watered down by analytics people who don't write any code. Data scientists who use power bi or even excel. Business analysts who use grafana. In my day to day I write just as much code as an SWE, the difference is it's for an ETL pipeline and proportionally more SQL. I list my title as "Data Engineer (Infrastructure)" on my resume to clearly indicate that I'm actually building software and not messing around with some analytics tool. Title dilution also skews salary statistics because true DEs are compensated similarly to SWEs. Funny thing is Leetcode is actually good at filtering out the analysts with DE titles. My only warning is be extra sceptical when considering a DE role at no name companies because you don't want to waste your time interviewing for an analytics role. Overall, actual DE is an excellent role, and should be thought of as an SWE with emphasis on data processing. Experience in a DE role should be transferable to an SWE role.
To this day, have no idea what data engineers do. Because in my opinion people who use all those big data technologies like spark, beam etc are software engineers.
There’s huge overlap. A data engineer is either: A. software engineer with focus on databases or B. A bi dev who can do ETL If a company is paying their DEs less, they’re probably in category B.
The post above me is tru :(