[Task]: Vector Embedding Model Pipeline

[lukasnxyz]

Task Description

Need pipeline to convert data into vectors via openai embedding model and integrate into helixdb. Should be different from running locally vs. managed hosting. Should also offer an embedding model if its run locally.

Priority

Medium

[lukasnxyz]

openai embedding models:

• small dim size: 1536
• large dim size: 3072
• dim reduction is also possible via the api so might have to implement that better with current version of dim reduction in vector_optims branch

https://platform.openai.com/docs/guides/embeddings?lang=python#embedding-models