Production releases question

Farmers Insurance / Eng
sicario

Farmers Insurance Eng

BIO
Devops, CI/CD/CD, automation, AWS, Jenkins, Docker, Wireshark, no bullshit.
sicariomore
Aug 17, 2018 11 Comments

Where I work our leadership is completely fucked up, people stuck here for decades with no clue of how a SDLC and IT operations should operate. We are fake AGILE, work with other full waterfall teams, complete lack of ownership of the deployment process to the point that we don't have a staging environment so we put production in maintenance mode during the weekend to do UAT! No red/black or blue/green operations, we also have to literally e-mail out artifacts to a dude to copy them to production. I was wondering how you guys do deployments? Care to share some experience? We have full weekends fucked up with tons of people involved plus a shitload of Indian contractors who sometimes barely speak English. In my previous company we literally deployed in 10 minutes anytime we needed. Here we have indeed a more complex ecosystem but I talked to IT execs at MetLife, Liberty Mutual and they have even more complex architectures however they were able to turnaround a fully containerized CD system. I feel we are living in the stone age of IT.
Thanks for sharing.

comments

Want to comment? LOG IN or SIGN UP
TOP 11 Comments
  • T-Mobile EKVe82
    A lot of it starts from application architecture. If all the apps are tightly coupled with strict dependencies it's going to be hard because you can't change everything at once.

    Do you have IaaS including firewall, dns, load balancers available?

    Anyone running a PaaS or other container orchestration platform like Kubernetes or CloudFoundry?

    Any public cloud use, or ability to use it?

    Pick an app to start with that is stateless and does not require any direct disk access. Containerize it. Get artifactory deployed and a CI tool like Jenkins if you don't have one.

    Artifactory can also serve as your container registry. Get Clair hooked up to it so it can do static security scanning of containers.

    Setting up a proper container orchestration platform can be tricky, so you can use simple docker hosts (or swarm) as a better way to deploy your apps. Replace existing hosts with new ones running docker, and use it for versioned installs of your containerized app that do not have dependency on the underlying OS. The existing hosts ips should already have load balancer and fw rules set up. Avoid the use of mounted volumes if you can. Putting Config in environment variables or a Config Server repo is good.

    Pipeline building semantic versioned artifact of your app triggered by merge to master. Pipeline build of your docker image so you can quickly fix it when a critical CVE is discovered for the stuff you put into your container.

    You'll have a reliable way to package your app in a container which can be deployed in seconds on a host. Move quickly and you can roll back to previous version if something goes wrong.
    Aug 17, 2018 3
    • Farmers Insurance / Eng
      sicario

      Farmers Insurance Eng

      BIO
      Devops, CI/CD/CD, automation, AWS, Jenkins, Docker, Wireshark, no bullshit.
      sicariomore
      OP
      Awesome tips, thanks! We have all on premises and operations people don't even know what a Docker container is or what IaC is . To keep it short, we are light years from a normal situation. The parent company that own us, has a different opinion of what a modern infrastructure should look like. We don't have any real Devops team, literally the people who are suppose to run the show are recycled from other roles, we lack expert, skilled and experienced engineers and the good one who happen to come, leave as soon as they realize the SNAFU situation.
      Aug 17, 2018
    • I joined a team once bc of my experience with distributed systems and agile development. I was to change the team. It’s a up hill climb. Can’t teach old dogs new tricks, ignorance is abundant. For me it wasn’t worth the battle. I left and couldn’t be happier. Working with smart like mind people on new technologies, plus making a lot more.
      Aug 17, 2018
    • T-Mobile EKVe82
      Definitely an up hill climb, but it can be worth it. We've gone from 7 months down to same day deployments for new code going to production.
      Aug 17, 2018
  • Facebook / Mgmt AsdfFdsa?
    Wow...unfortunately not even sure where to begin. I would start by getting your deployment process in order and automated.
    Aug 17, 2018 1
    • Farmers Insurance / Eng
      sicario

      Farmers Insurance Eng

      BIO
      Devops, CI/CD/CD, automation, AWS, Jenkins, Docker, Wireshark, no bullshit.
      sicariomore
      OP
      Thanks! My is more of a rant, when you have managers being working for the same company for 20-30 years without a clue of how it's done in the real world, I guess it's like to explain quantum physics to an ant!
      Aug 17, 2018
  • Tango Health / Eng WaterColor
    Not surprised. A shit ton of companies do software the old school way. The energy companies, banking firms.

    The guys from stone age have no clue how the industry has transformed in the last few years. In one of my ex-companies a deployment to Prod has a ticketing process that can take weeks to roll to the first 100 on day 1. And if for some reason there is a roll back the cycle repeats in about a week. By the time the last user is upgraded it would have been a few months. Oh by the way we used to do a desktop app.

    They still live in denial mode and will never age out.
    Aug 17, 2018 1
    • Farmers Insurance / Eng
      sicario

      Farmers Insurance Eng

      BIO
      Devops, CI/CD/CD, automation, AWS, Jenkins, Docker, Wireshark, no bullshit.
      sicariomore
      OP
      Glad to know I'm not the only dog working in Fuckistan! 😂
      Aug 18, 2018
  • That’s f up. I’ll never use farmers. You should just leave as you won’t learn anything there.
    Aug 17, 2018 1
    • Farmers Insurance / Eng
      sicario

      Farmers Insurance Eng

      BIO
      Devops, CI/CD/CD, automation, AWS, Jenkins, Docker, Wireshark, no bullshit.
      sicariomore
      OP
      I'm working on that. The funny thing they hired me to make it happen, but they forgot to tell me about the politics!
      Aug 17, 2018
  • Slalom Consulting mapman500
    It has to happen with coordinated efforts to transform people, process, and tools. Bring in the right people to build an approach and process to make strides and make progress over time. It won't happen overnight, but can be done. I have experience doing this for companies. PM me.
    Aug 17, 2018 0