In which big tech companies Data Engineer = SWE?
TLDR: I want to transition from Date Engineering. In which big tech companies Date Engineer are closer to being Software Engineer? From what I've heard: - Meta's DEs are not DEs but more of SQL developers - Amazon's DEs might be in some cases Data Platform engineers which is pretty close to what I'm looking for - Apple's DEs are not DEs but mostly Data Analysts with some SRE/DevOps mixed in, roughly the time breakdown might look like so: 60% DA, 30% SRE/DevOps, 9% DE, 1% SWE - The rest ?????? please comment The background, if you are interested: I don't see any bright future for me as DE, except maybe if I transition to management but I'm not sure if I want that. I barely write any code, I barely make technical / design decisions, I have not learnt any new technical skills in years, it seems to me that I'm losing competence as an engineer. I listen to what other DEs say and it seems it is a pretty common theme in the industry. IMO too many people were not good enough to be regular SWE/EMs/EPMs so they became the same in data world. Because of H1B role restrictions I'm supposed to have DE title. I think I'm capable as SWE, I contribute to some large opensource frameworks we use in my free time for example, I can solve most of LeetCode medium in 20 mins, hards are 50/50. I also consider Machine Learning Engineering and getting higher up to Staff DEs level but that is a topic for another discussion. TC: 330
Amazon DEs are a split. Many create pipelines using internal tooling mostly using SQL, and many others are more towards the SDE side, with a focus on Python/Scala/Spark type of stuff. All should know data architectural practices.
Do you think if ask about that on the interview, will I get a straight answer?
Yes, I would expect anyone from the hiring team who is on the loop or phone screen to generally give an honest answer of what type of tools and languages you’d be using if you ask. Not all of your interviewers will be from the hiring team though.
The closest you’ll get is to be on a data platform team which will contains real engineering challenges and problems rather than just know where data is and doing ETL 99% of the time. That’ll open up the pathway for you to transition to SWE
yeah, that's my understanding too, though not sure which companies have such teams, in Apple for example our data platform team consist of SWEs
None. If you want the full swe on data infra/data platform, switch to swe DE at Meta are paid less than DS (which are paid the same as SWE in most cases).
> None. Is it based on experience? Have you interviewed with many or how you have concluded that? As other people pointed out here: DE are sometimes mixed in under the same umbrella, especially when developing data platforms.
DE can code. They should. They can write pipelines in Python/Java. But in my 20 years of working in tech and a out 5 years in data engineering in the past, my limited experience tells me you are not going to get the full swe experience. There is a high chance the manager or the tpm will pass some “data” tickets to you because the swe are building the platform or writing reactjs codd. That being said, you are responsible for what you make out of your job. You can choose to be someone who writes sql query or someone wjo work with the team and automate your workflow using aws services, big query, some k8s jobs etc However, bear in mind, DE is generally paid less and therefore you need to calculate the input/hourly paid out ratio. I still recommend trying to land with a swe title. It helps you in the long run.
Why not interview for SWE data role in some mid tier company or startup ? Startups usually have DEs under SW engineering umbrella so you will have opportunities to write backend code there. Also Netflix is one company where DEs are expected to code a lot ( Scala code ).
I will have to do H1B transfer, not sure that there are many mid tier companies or startups will be ready to do that.
thanks for the tip with Netflix heard some good feedback about them, it seems like they provide much more freedom in data related problems, at least in comparison to Apple