What's New
New updates and improvements to Shiro
← Back4x faster data embeddings and enhanced security with pgvector
Improvement
We've just made an upgrade to store our vector embeddings in our own PostgreSQL database using pgvector. Previously we had been using a 3rd party hosted vector database with Pinecone.
This enhancement makes it 4 times faster to process embeddings for similarity search. This also improves security since we can now store this data in our own database with our own managed database backups. We no longer have to trust Pinecone to keep this data secure and properly backed up.
Overall the reasons we chose to bring this functionality in-house were:
This enhancement makes it 4 times faster to process embeddings for similarity search. This also improves security since we can now store this data in our own database with our own managed database backups. We no longer have to trust Pinecone to keep this data secure and properly backed up.
Overall the reasons we chose to bring this functionality in-house were:
- Postgres provides a wide range of vector-relevant features, including database backups, row-level security, client library support, and support for Object-Relational Mapping (ORM) in 18 programming languages. It also boasts complete ACID compliance and offers efficient bulk updates and deletes, with metadata updates taking just seconds.
- Consolidating all data within a single Postgres instance reduces the number of roundtrips in a production environment and allows for the convenient running of the entire developer setup on a local machine.
- Reducing the number of external databases reduces operational complexity and lowers the learning curve for managing data.
- Postgres is a battle-tested and robust database system, whereas many specialized vector databases have yet to establish a track record of reliability.