Ex MLE engineer deleted all code & models in Prod
We had an amazing ML engineer in the team. A new DS manager joined our team and started pointing errors in the code practice and tried to change the deployment process and architecture. There was so much pressure as there was only 1 ML engineer and he messed up a prod model deployment due to time constraints and it was very very complicated due to sudden architecture changes. The manager humiliated him and was telling you are unfit to be a part of the team etc... This really hit him and he resigned the same day. Today we came to know there are no models or scripts in the prod server and there is no backup as well. We debugged and saw that he had written a cron job that executed yesterday which deleted all models and script he had developed from past 2 years. Wow what a revenge 😳 We have no clue what to do now.. Edit: reg backup/ cloud / github No ours was a small team supporting handfull of ML use case.. which had no cloud , no github , everything was managed with just 2 on premises server. I know it's hard to believe but data science was something new to the current org so they didn't invest much in the team or stack until the value was shown to the business. Tc 180 #tech #machinelearning
legend
Dary! 🤙
Lol that's why you don't disrespect the only person that knows the system
Nobody keeping backups/snapshots of that stuff? Lol
Yeah lol, definitely company's fault
It's company's security/infrastructure team's fault because they didn't have adequate security measures or backups. If you need scapegoats- 1. Fire your infrastructure lead or manager alongside security manager. 2. Fire your manager for not having this in security measures in place or asking for help of infrastructure/security teams A company should be prepared for these things.
One person on my team is like this and I am so scared one day that this will happen
Make backups of their code.
Usually employment contracts cover this kind of behavior. Likely the company can sue him for damages and maybe put his name in public domain for this incident. Imagine looking for a new job with a lawsuit bc you nuked your previous employer's prod servers.
Surprised Samsung would not have these basic safeguards in place for prod.
It's a shit show.. stay away from Samsung
Your servers are backed up - just restore from a backup. Also code should be in source control - you do have that, right ??😀
No it's a small team which had no cloud , no github , everything was managed with just 2 on premises server. I know it's hard to believe but data science was something new to the current org so they didn't invest much in the team or stack until the value was shown to the business.
How come two years of work is not backed up? Shit posting?
Wow! Is that even legal?!
It's not. It's also probably made up.
First AMAZING. Going out with a BANG Second, I’m also wondering if that is even legal as well 🤔