which data engineering technology to focus today?
Back in the old days only Oracle & MSFT was the major player in database tech. interested to know which data-engineering/management tech to focus now esp with outburst of so many niche players and has made their mark like MongoDB, Snowflake, Hadoop, AWS, GCP etc. Please share your thougts and also adapt other tech to compliment it more? Exp - 10+ years
Add a comment
Google onetechguy Jul 25, 2019
Google Cloud Dataflow is widely used at G for MR pipelines. Its open source API is Apache Beam. Before joining Google I had no idea it existed since I was focused on AWS offerings.
Amazon prague Jul 25, 2019
Python,sql,airflow and Spark. However these things change every few years
SolarWinds AnEngineer Jul 25, 2019
ClickHouse
I think more than individual pieces of technology it’s more important to understand the idea behind these systems and what trade offs each system has made over the other. If you have a good understanding of these concepts, then picking up an implementation is not much harder.