Senior Data Engineer - Snowflake vs Hadoop
Career help I've been working on Hadoop stack for 2years and building ETL pipelines using Spark is really great area to be. I've been recently offered a position where company uses Snowflake datawarehouse. Should I stick to my Hadoop profile or take up new opportunity where I will use snowflake (I will be involved in designing dw infrastructure and etl pipelines )
Are you sure they aren't using Spark with Snowflake? As long as you are still doing spark who cares if Hive/Hbase/HDFS vs Snowflake
Good point. I will read up on Snowflake using Spark. https://docs.databricks.com/_static/notebooks/snowflake-python.html
Um snowflake has its own pipeline which a monkey can set up , that’s what you pay them $s for . Also there are a ton of companies automating ETL anyway for cheap . Don’t leave open source unless huge jump in TC
SF is purely SQL with minimal ability to configure/customize, which is a strength and weakness. I like their technology, but I would definitely not want to hook my wagon exclusively to a proprietary technology like that, especially when no "real" programming languages are involved. If however SF work is just one part of the process and you still keep using other tools like Spark/Databricks/etc, then that sounds great. A data warehouse really can't do everything, so it'd be a definite red flag if you were exclusively on SF.
Yes but you can use a relational dB, one of the 10-20 ETL companies that work with snowflake and you are done .
Did u take snowflake offer?
Nope. I stayed with my Hadoop stack.
How many shares did snowflake offer u?
Depends in your tc. What is it
Company is willing to offer me the position but we havent reached to TC part yet. Would it hurt my resume in future, if I leave my Hadoop profile and switch completely to Snowflake ?