As an engineer do you think is it possible to build a system that doesn’t need sev-0 on call?
Let’s just consider about backend systems that serves WEB and mobile requests. In 2022, we can do auto scale up and down with current technology. For migrations and stuff we need engineers as back and those are not sev-0. Do you think we can build systems that doesn’t require on call? I’m not talking about complex services or systems that supports spaceX kind of shit. Just a normal web app like Zillow serving data from DB via BE micro services to web and mobile requests. I’m not including ML services like recommendation engines etc. #tech
Trivial systems maybe, but bugs are unpredictable.
That means the team didn’t do TDD and QA wasn’t great enough.
If a system reaches a certain level of complexity (and the bar isn't that high) no test suite can cover all combinations.
Automotive ECUs are very complicated systems that are designed to never need external input/debugging
Easy to test though due to very well-defined input ranges. And there's a shit ton of ECU-related vehicle recalls, some of them high profile.
Possible? Yes. Practical? No. There are trade offs you have to make in the real world, especially with respect to time.
More iterations more chances to bugs, stop dev and you will make it super robust very soon
of course it is possible. just not at a cost you can afford
As soon as you start adding features, it would start needing an oncall.
I know of a system that doesn't have constant OnCall for developers. They have operations folks that I think do have OnCall but they can only restart services or do other basic things, can't fix actual bugs. Business critical enterprise software, they have SLAs and refund clauses. But no 24/7 OnCall for devs
The more complex the system gets the more likely it will break in some spectacular way. Best case you will need someone to soothe a big user panicking over misinterpreting some metric or forgetting to do something they need to do
Even spaceships blow up. Your web system can go down any time for an unprecedented cause.
Yes