Why is Amazon Alexa so much worse than OpenAi in terms of NLP research?
Think about it, they have 120 employees. Focused on risky language ml projects, that are now going to pay off massively. Their new model is a true jump in NLP capabilities. Meanwhile in Alexa, I’m in an org of 120 (out of 5000+ Alexa employees) with a bunch of people trying to solve a problem 1/10th or even 1/50th as complicated. And we’re failing at it, full of technical debt. We have a ton of researchers making 400-600k. The real issue? Alexa NLP orgs are to some degree a facade. Managers and senior engineers making safe incremental improvements to stay on the high salary gravy train. Imagine amazon fired all the overpaid engineers and paid triple (1M+) for world class openai style talent.
Amazon researchers are commonly making 500k+. Especially the ai ones
Not sure about OpenAI but Alexa and Amazon in general extends Applied Scientist offers to people from mediocre PhD and MsC programs. Applied Scientist out of PhD/Msc can easily make >350k with only the bare minimum experience to complete their university programs.
Does Amazon let you spend 5 million on model development?
Amazon is blowing money on Alexa. I bet Alexa spends multiples more than openai
OP let me introduce you to the Frugality LP
Alexa AI had a small scale effort to train their own large language models. That was a tiny effort - maybe 10-15 people involved. And they released it. Prior to recent times, there was little appetite for fundamental work like this, everything had to be "customer obsessed science". That said, there has been good quality research output on incorporating structured knowledge into generative language models, controllability in language generation and so on. We had GPT2-based versions of these going back to 2018/2019, but never productized those pieces of work outside of small niche use cases though. And yes, budget was absolutely one reason for that. Another was that open domain chatbots are interesting but not easily monetizable to scale more resources into. Some recent interactive generative multimodal AI experiences were being launched - cool productization of some real AI work there. But anyway, yes, a lot of applied scientists were going for low hanging fruit incrementally improving intent and entity models, too, but that was hardly the only thing going on. Anyway, the problems at Alexa are 80% business model and monetization, 15% incentives and being unwilling to throw out old systems once they are in production and then doing local optimization on top of old tech debt laden crap, and maybe 5% with limitations on talent.
Right but I think in hindsight, given how fast open ai truly innovated, amazon research chose the wrong direction. Which I would guess is a cultural problem. The product isn’t good enough to monetize
The core problems of entailment, truth and toxicity and training large LMs with human feedback in generated language outputs were being addressed at Alexa AI. There were several papers from the Conversational Modeling team on these topics. InstructGPT and ChatGPT use proximal policy optimization RL in fine tuning/distilling GPT-3 to address these problems, and clearly their approach works well. Interestingly, I see one of my team's papers written in my time at Alexa AI cited in the InstructGPT paper from OpenAI and see several other papers from Alexa AI scientists cited there as well. But I agree with you that until very recently Alexa AI didn't have the infrastructure to train very large language models, so we were working with GPT2 scale models at the time I was there a few years back. A lot came down to budget - Alexa was willing to spend tens of millions retraining intent models, but not to spend tens of millions building the infra to train very large language models from scratch until very late in the game.
There’s a difference between building a customer product, used by everyday people and constrained by lots of safety regulation, and an open source open ended model. ML product and ML research are 2 different things. It’s a good thing and protects us all that product innovation is slower than research innovation.
Terrible management. Too much politics, empire building, greed, lacking patience. Even if they hire open ai level talent, they will not be empowered to succeed.
Empire building
This ^ working at Amazon is about succeeding in the organization not building a product. There are rare exceptions.
It’s really absurd if you think about how little gets accomplished with so many people