They generally do very different things. Software engineering is about delivering an experience or solution. Data engineering is about ensuring we can learn from that solution.
A data engineer will either work with a software engineer to come up with a schema and data delivery/logging solution, or they will create one from the existing logs the engineers create. The data engineer will figure out how to store the data and make it readable and searchable by data analysts or data visualization teams.
In other words, as a data engineer you are thinking about how we will learn from products vs building products
I think this is an insightful distinction. I would say more simply, data engineers write ETL pipelines into the data warehouse and software engineers... Well, they create software products for stakeholder use. There are times at Netflix where the ETL can be off pattern enough (can't be handled with the normal ETL tools/patterns) that data engineers have to write more code - like SWEs.
They generally do very different things. Software engineering is about delivering an experience or solution. Data engineering is about ensuring we can learn from that solution. A data engineer will either work with a software engineer to come up with a schema and data delivery/logging solution, or they will create one from the existing logs the engineers create. The data engineer will figure out how to store the data and make it readable and searchable by data analysts or data visualization teams. In other words, as a data engineer you are thinking about how we will learn from products vs building products
I think this is an insightful distinction. I would say more simply, data engineers write ETL pipelines into the data warehouse and software engineers... Well, they create software products for stakeholder use. There are times at Netflix where the ETL can be off pattern enough (can't be handled with the normal ETL tools/patterns) that data engineers have to write more code - like SWEs.