What's the scoop? Must be a pretty big infrastructure outage of some sort
We have a central point of failure and it failed. Every customer affected. Outage started around 9am - 4pm, people did not have access to their bank, credit card, and investment accounts. It took down every environment QA and Production. There is so much infighting within the top leadership, that teams don’t work together and when smart people bring up there concerns it’s ignored. No one has a big picture view of how the entire architecture works because 1) the people who do are smart enough to quit asap and 2) the leadership that has been around for a while, don’t actually do the hard work to understand the systems. This bug came up months ago and people just worked around it cause they were too lazy to fix it. What’s so frustrating about the entire situation is that it could have been avoided by using better architecture. We didn’t use blockheads, so failures were not contained and they took down the entire system. We also don’t use circuit breakers. If you are processing rabbit mq messages use timeouts, and if exceptions are thrown do not send your dead lettered messages to the front of the queue to be reprocessed, if you’ve already retried them. Send the dead lettered messages to S3 while you figure your shit out.
That's nuts, I contracted for CapOne at their Richmond campus a couple of years back and it seemed they could never make up their mind of they were going to outsource everything 100% or in source everything 100%, so it was always one or the other every couple of years
Good info, it’s time for me to cancel my accounts then, haven’t used them much anyway
I’ve been trying to fucking warn people about another huge bug they’re going to release for the last two weeks, but honestly no one gives a fuck. This is the worst company I’ve ever worked for.
meow!
Wtf 🐱 meow?
Quit and jumped to tech last month, best decision ever
Same company? If not where?
Take me with you
What do you expect from them?!!!!
Don’t they run on your cloud? ;)
Unless this was an AWS outage that's irrelevant. We provide the tools, yes, but any company can build a gun and shoot themselves with it
Exactly what I expect from a company run by east coast, out of touch retards employing people whose only goal is employment.
Hahaha so sad and true. Thank you!
+1 excellent summary of the east coast
Thank you cap one people, sadly I now never wanna work there, but you have my sympathy.
“Alexa, play Deplorables on Fire One” “Deploying Fire to Prod” “Alexa! Stop!” “Sorry, I’m having trouble understanding right now. Try again later”
Lol
When AWS fails? ;)
Should have used Monero
Both the mobile app and creditwise are up since 3.30~