Startup that depends on web scraping?

New / Eng
xAtC71

New Eng

xAtC71
Nov 1, 2021 5 Comments

I have an idea for a startup that I believe would have a net positive impact on society, in particular, for a vulnerable group of individuals. However, enabling this would require scraping large amounts of metadata from a large social media company.

Nowadays, there are many high-quality scraping services that can overcome a lot of the common scraping prevention methods and I believe it would be technically feasible.

Assuming that it is feasible, is it a good idea to embark on this journey and build an MVP if there is a large risk that this would be shut down at a later point in time or that this would be deemed as uninvestable and would never surpass the MVP stage?

There are past examples of startups that have succeeded by building their businesses on questionable legality (Ex. Airbnb, Uber, Lime, etc..), but I am not sure if considering these examples would just be a survival bias.

Many saw the famous ruling of LinkedIn vs. HiQ as a Golden Ticket for web scraping (HiQ won the case), however, my idea would require logging into the site and likely a breach of Terms of Service.

What are your thoughts? Poll and comment below.

#startup #tech #swe

37 PARTICIPANTS SELECT ONLY ONE ANSWER
VOTE VIEW RESULT

comments

Want to comment? LOG IN or SIGN UP
TOP 5 Comments
  • Google / Eng
    dogfoodie

    Go to company page Google Eng

    dogfoodie
    If you get big enough, you can be sure there is a lawsuit waiting for you.
    Nov 1, 2021 0
  • Apple
    TrjD37

    Go to company page Apple

    TrjD37
    Airbnb Uber and Lyft were never really questionable legally. The laws were implemented after the fact.
    Nov 1, 2021 0
  • Outside of web scraping being insanely hard yeah you can't build around violating tos
    Nov 1, 2021 0
  • New
    a_man_from

    New

    a_man_from
    I developed a scrape system that scrapes anything and everything. You have to respect robots.txt. I personally override it since the data isn’t too sensitive. What are you looking to scrape? Something like FB??..
    Nov 1, 2021 0
  • New
    hugh jarse

    New

    hugh jarse
    false dichotomy, however i say go for it.

    the famous ruling has been remanded back to the lower court and is AFAIK undecided again. i wouldn’t let that stop you
    Nov 2, 2021 0