"tech companies" - a misnomer?
Interviewed at a few places and they seem to have architectures/systems in place which are not on par with what I have seen in financial space. These are public or about to go public firms with billions in revenue operating microservices based products out of single availability zone or single DC with very little thought given to everything else. Folks are surprised to know that services can run without downtime for months at a time. Agreed these are platform services running on multi dc mainframes with redundancy built in across infrastructure and applications for availability. And am ignoring the lessons learnt from operating and evolving the platform for over a decade or that the law of averages will catch up with us at some point. But it should not be all that surprising should it? Think Visa/Mastercard or stock exchanges or banks around the world- how many debilitating outages have they had over the years?
tech is more about pushing new features than reliability or quality.
Because those financial companies can’t tolerate any downtime whatsoever. So for them it is critical to invest in availability regardless of the cost and tradeoffs. For most companies a short downtime is tolerable every once in a while, their customers won’t be strongly affected. So for them it’s more important to hit a high development velocity and get to the market faster, and react faster when needed. Or to just keep engineering headcount in check.
I understand this trade off. But do folks from these companies understand these factors at play? A bank being down will end with someone on the chopping block. Standard bank south Africa, TSB CEOs had to step down. Not to mention the scrutiny from regulatory authorities that these outages attract.
Still I see financial services such as Fidelity and Schwab down for scheduled maintenance over weekend!
Yes, scheduled maintenance does exist even today even for the biggies!
My bad. I meant zero unplanned downtime. And we pick a window where the traffic is very low or none.
Scale of big tech companies is much larger which would also lead to more instability and downtime AWS may have had a lot of outages but it's load is ridiculously high and also extremely variable Remember what happened to brokers' servers when GME happened? Most of the stable ones whose servers didn't explode were the more tech focused ones like Wealthsimple and TastyTrade
When you say big tech - I assume you are referring to FAANG? The scale they see meant they had to invest in those areas. I am referring to other tech firms. Visa with over a half a billion transactions per day has a different setup than say a smaller credit union. Not denying that. Didn't know about issues with tech when GME happened. Wasn't it due to liquidity?
Not just FAANG. There are certain companies like Uber and Zoom that deal with insane scale too I'm not sure how Visa card transactions work but it seems like it'd just be a simple requests and response. Whereas compare that to cloud computing services, map routing, streaming, messaging, video calling, etc. All these are much more computation heavy and complex So Robinhood blew up because of liquidity but a bunch of brokers had troubles logging everybody into their servers that wanted to buy GME when the market would open and people had to constantly relog for 10-60 minutes to get in
Depends on what your availability sla is.. Not all services need 5 9s. If you don’t need it why build it
NASA sends rockets and devices into space that use tech from 20 years ago. It does it because it is tested and dependable even if slow. Car manufacturers have issues with chip supply right now and no manufacturer is creating a foundry to make their antiquated chips. But again they are reliable. Reliability is not the only measure of progress though right?
Exactly my thoughts. Each industry and each company dictates the engineering culture. All I am pointing out is the lack of that awareness and the holier than thou attitude I have encountered in the interview process. And this from leaders of those organizations.
You are confusing availability vs agility
I don't think I am. It takes investment of time, effort and money to have availability or agility. You don't get them out of the box. And organizations choose a balance that works for them. What surprised me was that individuals from these companies could not grasp some of the elements that I was talking about. For instance in one of the interviews the interviewer got stuck at no unplanned downtime for close to two years and went down the rabbit hole of measuring downtime etc. Barely leaving time for other talking points.
You’re saying financial companies are more technical than tech companies themselves?
I am not talking about the likes of Amazon, Google, Facebook and others that have done amazing stuff. IBM, Oracle and others did too.
If you think about it finance, telephone networks, cable networks are some of the earliest distributed systems if not the largest.