I'm working as a software engineer for almost 3-4 years now. I've been working with data and like the data engineering part - transforming data in Storm, Spark, etc.. I feel the growth is very limited though. Both internally and externally. Firstly, there's little I can do about the quality of work that we get to do and opportunities for me to learn and upgrade my skills. Moreover, I find it's very often about the no of years of experience you have and the people whom you might know, than about your talent or abilities. It's about being in the right place with the right people, more than about knowing the right thing to do. Also, I don't like going on call over the weekends at all. So, I'm wondering if is should quit my job and focus on learning something more valuable, like data science, which I imagine, would be higher paying, not require weekend on call and more rewarding for my career overall. Any thoughts or suggestions? I'm not that microservice/service/full stack/core Java/dependency injection guy and I don't want to be.
TC higher as SWE