Getting into ML on HW based roles

I am currently working as a system software engineer (at Vmware) with close to 3 years of total experience (just 1 year at Vmware). My background has mainly been in OS, embedded systems, computer architecture and device drivers. I have recently realized that I might not enjoy system software for too long and have started looking into other areas. I have long been fascinated by how Machine Learning and Deep Learning models are implemented on specialized hardware (GPU, TPU, other accelerators) by optimizing them for power, performance, cache, memory etc. and would like to go in that direction. I have been looking at software based roles which involve implementing and optimizing DL based libraries and frameworks on specialized hardwares, and it has piqued my interest quite a bit. I have seen these roles in hardware based companies like Nvidia, apple, intel etc. My understanding is that these roles involve optimizing various libraries (like pytorch, tensorflow etc) on various new hardwares and involve extensive knowledge of system software, computer architecture and ML, which I found really fascinating. I am very much interested in roles which involve writing software while fully understanding the underlying hardware’s architecture. People working in these kind of roles, please provide your thoughts. Also I don’t have any formal experience in ML, DL apart from a course on ML I took in grad school (which was very theoretical). I am open to taking an online course with actual projects, or doing an online masters to get good hands on experience in ML and DL. Combined with my experience in system software would this be a good approach for getting into this area? Yoe- 3 TC- 180k