I’m building a tool for analyzing large amounts of data and querying via different ai/ml techniques. Basically the initial user request will be constructing a large context array of data which first gets scraped from the web then fed through an embedding model, X0,000 records being scraped and fed to open ai embedding service (or an open source model that would be self hosted if it makes sense cost wise, but would probably be a lot slower). The embedding vectors are stored in redis with a ref to the content in its mongo db document. Right now the data is being handed off from node to python scripts and back via intermediate jsonl files. I’m assuming it’s not too much of a hassle to connect the python code directly to the mongo db, but I know that I could also just re write the python bits in js and maybe save myself a headache if I actually have to scale this thing. Is it kosher to mix js and python in node? Has anyone ever done it? #mongodb #nodejs #tech #python #openai
Let me quote you for this gig?