SWE to Data Engineering
I am trying to get into Data engineering from a SWE, because I think data engineers have a better future due to all the AI/ML shift. I am getting an opportunity to move to data engineering, but in a smaller company (800 people public company), with 20% pay cut. Is it worth it?
Don’t. It will be a career suicide. Be a software engineer focused on building ML applications or worse get into ML engineering. Don’t get into tangential areas or take short cuts.
ML eng is different than Data eng, however working in data eng does provide a easier path to ML. Most SWEs have no idea how data is managed, scaled and processed. I was shocked to see how many of them had zero knowledge into this domain. That said data eng still is very new field and is loosely used by companies and job can be drastically different from one company to other. Focus on data eng that’s closer to SWE i.e. you are building tooling for data and AI/ML and not just a sql monkey.
Correct. SWEs don’t think in data. Data and ML engineers think in data first.
great perspective. so the conclusion is that as long as you do programming to solve system or data problems in big data system environment , not sql all year long then you are fine.
Ex
We really have no idea what will happen in future. Yes, AI/ML will probably increase in demand, but then that being said, data eng is not AI/ML even if it is related. If you truly want to get into AI/ML, then data eng is just a stepping stone, and you'd have to stay in this role for at least 2 years for a jump to AI/ML to be possible.
Agree about the uncertainties. Sometimes things fall in your laps, but that doesn't mean you should be sitting waiting for it to happen. You got to try, you may fail, but you can succeed as well.
As a data engineer, I suggest you stick to SWE. DE can be cool when new development comes around, but this rarely happens. So it is very likely you will spend all your time bug fixing terrible code (SQL + Python more likely) as most DE's have no software development best practices/experience OR reuploading the data again and again. Personally, I find data quite interesting, but most companies you will be relegated to a third role (providing the data for some analyst to show). P.S: Feel free to DM if you have any specific questions.
The bugs happen because of edge cases in systems most of the times or some arbitrary data coming from Software systems. It need not be due to lack of software development best practices. For ex in one of the companies I was working at, I was processing weblogs that engineering agreed to log in with some specific manner. But they don't test it in all web browsers . It's acceptable for launch when they support for few web browsers. But from reporting we needed to work with data coming from all browsers as we make money from every web browser. There are several such practical difficulties that a data engineer has to deal with. This field seems so easy yet so difficult. On top of it they won't be appreciated for their efforts and several companies pay them less than SWE. Eventually, good engineers either move to Software engineering or get into manager roles. This is the reason why you are seeing bad practices like reuploading data all the time.
@Lyft, you have pretty much write down my extra thoughts. Actually, I am in the situation that you mentioned of moving to SWE (better pay and "a bit" less operational with the upside of more visibility and higher salaries).
I'm curious how you got the Principal role without having prior DE experience (I assume based on your post). I'm a DE and I would expect a Principal to be able to know their way around how to properly distribute big data loads across a cluster for complicated processes, knowing extremely well frameworks that backend engineers typically don't work with like Spark, more advanced SQL/database internals knowledge, data pipeline orchestration tools, how to design a database architecture that can only come with experience, and other things. I'm just wondering, and not trying to put you down.
As a SWE, I have some experience with databases. I learned a few things like Spark, Data lake, etc on my own. I have failed many interview attempts, as interviewers find out that my knowledge in some areas is theoretical. But this one opportunity, I somehow succeeded. I know I will need to work extra hard to justify the role, which is the part I am excited about. Taking a 20% TC cut is something I need to stomach.
Data Engineering isn’t real engineering. Data Science isn’t real science. It’s all computer science and software engineering. For ML start calculus. It’s all curve fitting area under curve and optimization. You can’t go anywhere without it. Learning Spark means nothing. It’s just another big spicy distributed query execution system
"I think data engineers have a better future..." I cannot disagree more. There has been a global shortage of SWEs for nigh on half a century and its only getting worse (for employers). I can't predict how AI will play out, but if history is any guide the hype is overblown vis-a-vis replacing all our jobs. There may even be *more* jobs generated by this tech, and if there is, most of them are going to be SWE. In any case SWE is such a general toolset to have, it can be applied to almost any problem domain. DE seems if anything a subset that would limit your future career choices and flexibility. SWE ftw 4 eva.
Relax, SWE isn't going anywhere. Doesn't help to diversify of course. If you want to get ahead of the curve you'll want to look into "AI integration", there'll be a huge demand for engineers in the coming years who can integrate AI into company workflows.
Is modern AI/ML work a continuation of data engineering or just an adjacent path? I know there are lots of intersections between these two but ultimately quite different What is that small company? If you wanted to, can you make any use of that private equity? Based on TC alone I’d stick with Walmart
There is a good possibility of getting into ML through this job. It's not private equity, it's a publicity traded company, just much smaller (~800 people) than Walmart.
Can you not get some formal ML training which Walmart would sponsor to put on your CV? Use that to get a relevant position later on