Why use Python Flask for backend services?

Feb 16, 2020 24 Comments

Sorry for the n00b question, I have only worked for 1 company so I am genuinely curious about what other companies are doing.

At FB, we have php for web server but all serious backend services are written in cpp or Java, communicate with RPC, but I heard Lyft is using Python Flask for everything, is this true?

It sounds like a web framework, and communication is on REST? Without much investigation, personally I highly doubt its performance, is this a valid concern? Do you guys not have to handle large scale cases?

comments

Want to comment? LOG IN or SIGN UP
TOP 24 Comments
  • Uber
    uPython?

    Go to company page Uber

    uPython?
    The question should be about individual request latency, not number of requests.

    If most of the API is just waiting for an RPC call from another service (including a DB call), then it doesn’t matter what you use as that’s the bottleneck.

    Even vanilla Python can be super high performant. Python has a wide array of tools to run code that would be slow if you wrote it in python and runs it in C/ASM.

    People use python because it’s really fast to develop in and has so many tools because it’s so flexible.

    But given all that, I’ve done Python at 3 different companies and 3 different scales (in terms of project size/number of lines of code) and I would not recommend it if you expect your application to be > 1000 files. It just doesn’t scale well from a development point of view IMO. You start to slow down with each file added. Much much harder IMO to find bugs than a typed language and that’s about to me when the pros don’t out way the cons anymore.
    Feb 16, 2020 3
    • Uber
      uPython?

      Go to company page Uber

      uPython?
      Even “typed” python ain’t truly typed unless they are forking python (big no).

      Where is the expensive processing? What is the operation? Pandas/numpy data manipulation is likely just as fast as your C++/Java implementation.

      HFT is too fast for python, but the difference between 150 and 200ms on the UI is not noticeable (I think threshold is like 75ms).

      I’m not enough of an expert to know, but could estimate, that python applications probably cost more. More time per request and likely more servers needed to balance load.
      Feb 16, 2020
    • Critical UI might need to query multiple very complex backend, just like my example above for feed, ads, search etc, all of them involved with multiple stages of processing, each stage might be done in a separate service, so I think the latency is really important here.
      Feb 16, 2020
  • Amazon
    GTB452M

    Go to company page Amazon

    GTB452M
    Python is getting more popular for backend, because at some point, you'd need to integrate with data science/ML libraries, when that happens, it's just plug and play with python. For other languages, where you'd basically have to spin up another python stack and communicate with it
    Feb 16, 2020 0
  • Lyft
    lyft123..

    Go to company page Lyft

    lyft123..
    Lyft uses go/grpc too. In many cases developer productivity matters much more than small performance differences.
    Feb 16, 2020 4
    • Totally agree with your points. Thanks for the discussion! Do you guys have your own search, storage, MQ solutions etc? Also, I imagine dispatching service is gonna be perf sensitive, right?
      Feb 16, 2020
    • Lyft
      lyft123..

      Go to company page Lyft

      lyft123..
      Some. Size of the company is a factor in amount of internal tooling. And yes, latency matters for dispatch.
      Feb 16, 2020
  • Amazon
    yeeeee

    Go to company page Amazon

    yeeeee
    We use gunicorn + flask for some of our tier1 ML workflows. It can handle millions of transactions per day
    Feb 16, 2020 1
  • Facebook / Eng
    Jon B

    Go to company page Facebook Eng

    Jon B
    Top backend languages at Facebook are cpp and python. Java is not as common.
    Feb 16, 2020 9