Twitter’s new API pricing is interesting - looks like it’ll hit a lot of social intelligence platforms hard (hubspot, brandwatch, etc…) However - Twitter is largely not paywalled/login walled, meaning it’s technically legal to scrape Twitter… you can get all sorts of data - post content, post media, total engagements, engagement entities (e.g. comment parent/child relationships, comment entities), commenter identity… the only think you really can’t do without logging in is grab a list of users who liked/shared the post to build your own user graph, which is only particularly useful for a select set of use cases (e.g. if you’re trying to put together predictive trend analytics, etc…) Question is - did Twitter shoot itself in the foot by adding a pricing model to their API without adding a login wall to tweet content? I know it’s a question if revenue (e.g. you can get more ad revenue if non logged in users can see Twitter content), and I know preventing scraping can somewhat be addressed with IP blocking/request limiting/etc, and/or dynamically loading content with like Angular, but all of these prevention tactics are relatively trivial to overcome. To me, it seems shortsighted that Twitter would go so hard on API pricing without further determining ways to prevent wholesale scraping… what’s to prevent heavy API users from just going the scraping route (other than the threat of civil litigation which hasn’t been terribly successful in the past without a legitimate privacy concern being brought to the table, a la Meta suing scrapers). It seems that small/midsized companies that want to use wholesale Twitter data are even more incentivized to scrape Twitter or used a manage data collector solution (like BrightData) rather than pay for API access/go through the app approval process What are your thoughts?
I fear for the lack of third party investigative research that will arise from this. It will become prohibitively expensive to study trends for academic purposes (but perhaps not for commercial purposes, of course).
I think that's kind of the point TBH. It's a feature, and not an unintentional problem. Only Elon or someone whom he trusts can announce "hate speech is at an all-time low", and not need to provide facts/proof. Or very skewed "proof" that can no longer be validated by academia.
I'd hate to attribute malice but I wouldn't be surprised if you're right. Especially since you work at Twitter.