Vibecode Pinecone
track this build5 phases, 10 steps, beginner friendly0%KINDA · weekend project
For anything under a few million vectors, pgvector in the Postgres you already run does the job with a normal index and normal backups. Pinecone earns its fee at scale and for teams who want zero operations; a side project needs neither.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Pinecone. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Pinecone · indie build Semantic search over your own documents with pgvector in the Postgres you already run: chunked documents with embeddings, cosine search with an HNSW index, hybrid ranking with full-text so exact identifiers still win, a localhost search page, and backups that are just pg_dump. Estimated effort: **one sitting**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Database | Postgres 16 with pgvector | rule zero: a vector index is a column type | | Embeddings | A provider behind an interface | OpenAI or a local model via Ollama; swappable by env | | Runtime | Node 22 | the ingest and search scripts | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Everything in this build runs on it: the server, the scripts, the tests. - Get it: Download the LTS installer from nodejs.org, or install with your package manager (brew install node, or nvm install 22). Restart the terminal afterwards. - Verify: node --version prints v22 or higher - [ ] **A terminal and a code editor** · free - Why: Every step below is a command you type or a file you edit. - Get it: VS Code (code.visualstudio.com), Cursor or Zed. Open a folder for the project and use the editor's built-in terminal. - Verify: You can open a folder and run a command in its terminal - [ ] **Git** · free - Why: History for your code, and the way most hosts deploy. - Get it: Install from git-scm.com or with your package manager, then run git init in the project folder once it exists. - Verify: git --version prints a version - [ ] **Postgres 16 with the pgvector extension** · free - Why: The whole store. - Get it: Docker: docker run -d -e POSTGRES_PASSWORD=pw -p 5432:5432 pgvector/pgvector:pg16. Or install pgvector on an existing Postgres. - Verify: CREATE EXTENSION vector; succeeds - [ ] **An embeddings API key, or Ollama locally** · pay per use, or free locally - Why: Phase 1 embeds chunks. - Get it: OpenAI: platform.openai.com/api-keys. Local: install Ollama and ollama pull nomic-embed-text. - [ ] **The documents to search** · free - Why: Markdown, PDFs converted to text, or a folder of notes. - Get it: A folder of files. ## Quick start ```sh mkdir semantic && cd semantic && git init && npm init -y && npm pkg set type=module && npm install pg@8 mkdir -p docs && cp .env.example .env ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Billions of vectors and serverless scale; under a few million this is a column. - serverless scaling to billions of vectors - zero-ops managed indexes - hybrid and sparse search features - the SLA If one of those is essential to you, that is the reason to keep paying for Pinecone, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Pinecone The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me semantic search over my own documents to replace Pinecone. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not write a vector index. pgvector is a Postgres extension with HNSW built in; it is a column and a query. ### Stack (fixed, do not substitute) - Postgres 16 with pgvector. Node 22. An embeddings provider behind an interface (OpenAI or a local model via Ollama). ### Data model (create this before Phase 1) - documents: id, source, title, url, updated_at - chunks: id, document_id, position, text, embedding vector(N), token_count Chunks, not documents, carry the vectors: a 40-page PDF as one vector matches nothing well. ### Phase 1 · Store and embed Build: a chunker (by paragraph, ~500 tokens with overlap) and an embed step behind an interface, writing chunks with their vectors. Idempotent on document hash. Done when: re-ingesting an unchanged document embeds nothing, a changed one re-embeds only its chunks, and the interface swaps providers by env var. Do not build yet: search, UI. ### Phase 2 · Search Build: a query function embedding the question and returning the top k chunks by cosine distance with their documents, using an HNSW index. Done when: a question about a known passage returns it in the top 3 and the query uses the index (EXPLAIN shows it). ### Phase 3 · Hybrid Build: combine vector results with Postgres full-text search via reciprocal rank fusion, because exact product names and error codes are where pure vectors fail. Done when: a query containing an exact identifier ranks the matching chunk first. ### Phase 4 · A page Build: a localhost search page and a JSON endpoint, results with highlighted passages and links to sources. Done when: a search returns in under 300 ms over 100k chunks. ### Phase 5 · Operate Build: migrations, a backup that is just pg_dump, a reindex command, the README. Done when: a restore searches correctly. ### Out of scope (and why) - Billions of vectors and serverless scale. Under a few million, this is a column. ### README must contain - The chunking rule and why. - The provider swap. ===== AGENTS.md ===== # Agent instructions · Pinecone indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Postgres 16 with pgvector, A provider behind an interface, Node 22. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ===== BUILD_PLAN.md ===== # Build plan · Pinecone Semantic search over your own documents with pgvector in the Postgres you already run: chunked documents with embeddings, cosine search with an HNSW index, hybrid ranking with full-text so exact identifiers still win, a localhost search page, and backups that are just pg_dump. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Store and embed Chunks with vectors, idempotent on document hash, provider swappable. ### Steps 1. Create the schema documents (id, source, title, url, updated_at, hash), chunks (id, document_id, position, text, embedding vector(N), token_count). N matches EMBED_MODEL's dimension. Files: `schema.sql` ```sh mkdir semantic && cd semantic && git init && npm init -y && npm pkg set type=module && npm install pg@8 mkdir -p docs && cp .env.example .env ``` 2. Chunk by paragraph around 500 tokens with overlap; embed behind an interface; skip unchanged documents by hash ### Done when - [ ] Re-ingesting an unchanged document embeds nothing - [ ] A changed one re-embeds only its chunks - [ ] Switching EMBED_PROVIDER works ## Phase 2 · Search Top-k by cosine with an HNSW index actually used. ### Steps 1. Create the HNSW index and the query function ```sh CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops); ``` 2. Verify the plan uses the index ```sh EXPLAIN ANALYZE SELECT ... ORDER BY embedding <=> $1 LIMIT 10; ``` ### Done when - [ ] A known passage returns in the top 3 - [ ] EXPLAIN shows the index ## Phase 3 · Hybrid Exact identifiers win: fuse vector and full-text ranks. ### Steps 1. Add a tsvector column and GIN index 2. Reciprocal rank fusion of both result lists ### Done when - [ ] A query with an exact error code ranks the matching chunk first ## Phase 4 · A page Localhost search with highlighted passages and source links. ### Steps 1. A JSON endpoint and a server-rendered page 2. Time it on 100k chunks ### Done when - [ ] Under 300 ms over 100k chunks ## Phase 5 · Operate Migrations, pg_dump, reindex. ### Steps 1. Migrations and a reindex command 2. pg_dump nightly and one restore; README with the chunking rule and provider swap Files: `README.md` ### Done when - [ ] A restore searches correctly ## Not in this build - Billions of vectors and serverless scale; under a few million this is a column. ## After v1, if you want it - Re-ranking with a cross-encoder - Per-user document permissions ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Your Postgres. DATABASE_URL=postgres://postgres:pw@localhost:5432/search # Required. openai or ollama. EMBED_PROVIDER=openai # Optional · secret. For the openai provider. OPENAI_API_KEY=sk-... # Required. Model name; dimension must match the column. EMBED_MODEL=text-embedding-3-small # Required. Where the ingester reads. DOCS_DIR=./docs
You are building a lean indie version of Pinecone. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Pinecone · indie build Semantic search over your own documents with pgvector in the Postgres you already run: chunked documents with embeddings, cosine search with an HNSW index, hybrid ranking with full-text so exact identifiers still win, a localhost search page, and backups that are just pg_dump. Estimated effort: **one sitting**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Database | Postgres 16 with pgvector | rule zero: a vector index is a column type | | Embeddings | A provider behind an interface | OpenAI or a local model via Ollama; swappable by env | | Runtime | Node 22 | the ingest and search scripts | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Everything in this build runs on it: the server, the scripts, the tests. - Get it: Download the LTS installer from nodejs.org, or install with your package manager (brew install node, or nvm install 22). Restart the terminal afterwards. - Verify: node --version prints v22 or higher - [ ] **A terminal and a code editor** · free - Why: Every step below is a command you type or a file you edit. - Get it: VS Code (code.visualstudio.com), Cursor or Zed. Open a folder for the project and use the editor's built-in terminal. - Verify: You can open a folder and run a command in its terminal - [ ] **Git** · free - Why: History for your code, and the way most hosts deploy. - Get it: Install from git-scm.com or with your package manager, then run git init in the project folder once it exists. - Verify: git --version prints a version - [ ] **Postgres 16 with the pgvector extension** · free - Why: The whole store. - Get it: Docker: docker run -d -e POSTGRES_PASSWORD=pw -p 5432:5432 pgvector/pgvector:pg16. Or install pgvector on an existing Postgres. - Verify: CREATE EXTENSION vector; succeeds - [ ] **An embeddings API key, or Ollama locally** · pay per use, or free locally - Why: Phase 1 embeds chunks. - Get it: OpenAI: platform.openai.com/api-keys. Local: install Ollama and ollama pull nomic-embed-text. - [ ] **The documents to search** · free - Why: Markdown, PDFs converted to text, or a folder of notes. - Get it: A folder of files. ## Quick start ```sh mkdir semantic && cd semantic && git init && npm init -y && npm pkg set type=module && npm install pg@8 mkdir -p docs && cp .env.example .env ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Billions of vectors and serverless scale; under a few million this is a column. - serverless scaling to billions of vectors - zero-ops managed indexes - hybrid and sparse search features - the SLA If one of those is essential to you, that is the reason to keep paying for Pinecone, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Pinecone The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me semantic search over my own documents to replace Pinecone. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not write a vector index. pgvector is a Postgres extension with HNSW built in; it is a column and a query. ### Stack (fixed, do not substitute) - Postgres 16 with pgvector. Node 22. An embeddings provider behind an interface (OpenAI or a local model via Ollama). ### Data model (create this before Phase 1) - documents: id, source, title, url, updated_at - chunks: id, document_id, position, text, embedding vector(N), token_count Chunks, not documents, carry the vectors: a 40-page PDF as one vector matches nothing well. ### Phase 1 · Store and embed Build: a chunker (by paragraph, ~500 tokens with overlap) and an embed step behind an interface, writing chunks with their vectors. Idempotent on document hash. Done when: re-ingesting an unchanged document embeds nothing, a changed one re-embeds only its chunks, and the interface swaps providers by env var. Do not build yet: search, UI. ### Phase 2 · Search Build: a query function embedding the question and returning the top k chunks by cosine distance with their documents, using an HNSW index. Done when: a question about a known passage returns it in the top 3 and the query uses the index (EXPLAIN shows it). ### Phase 3 · Hybrid Build: combine vector results with Postgres full-text search via reciprocal rank fusion, because exact product names and error codes are where pure vectors fail. Done when: a query containing an exact identifier ranks the matching chunk first. ### Phase 4 · A page Build: a localhost search page and a JSON endpoint, results with highlighted passages and links to sources. Done when: a search returns in under 300 ms over 100k chunks. ### Phase 5 · Operate Build: migrations, a backup that is just pg_dump, a reindex command, the README. Done when: a restore searches correctly. ### Out of scope (and why) - Billions of vectors and serverless scale. Under a few million, this is a column. ### README must contain - The chunking rule and why. - The provider swap. ===== AGENTS.md ===== # Agent instructions · Pinecone indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Postgres 16 with pgvector, A provider behind an interface, Node 22. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ===== BUILD_PLAN.md ===== # Build plan · Pinecone Semantic search over your own documents with pgvector in the Postgres you already run: chunked documents with embeddings, cosine search with an HNSW index, hybrid ranking with full-text so exact identifiers still win, a localhost search page, and backups that are just pg_dump. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Store and embed Chunks with vectors, idempotent on document hash, provider swappable. ### Steps 1. Create the schema documents (id, source, title, url, updated_at, hash), chunks (id, document_id, position, text, embedding vector(N), token_count). N matches EMBED_MODEL's dimension. Files: `schema.sql` ```sh mkdir semantic && cd semantic && git init && npm init -y && npm pkg set type=module && npm install pg@8 mkdir -p docs && cp .env.example .env ``` 2. Chunk by paragraph around 500 tokens with overlap; embed behind an interface; skip unchanged documents by hash ### Done when - [ ] Re-ingesting an unchanged document embeds nothing - [ ] A changed one re-embeds only its chunks - [ ] Switching EMBED_PROVIDER works ## Phase 2 · Search Top-k by cosine with an HNSW index actually used. ### Steps 1. Create the HNSW index and the query function ```sh CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops); ``` 2. Verify the plan uses the index ```sh EXPLAIN ANALYZE SELECT ... ORDER BY embedding <=> $1 LIMIT 10; ``` ### Done when - [ ] A known passage returns in the top 3 - [ ] EXPLAIN shows the index ## Phase 3 · Hybrid Exact identifiers win: fuse vector and full-text ranks. ### Steps 1. Add a tsvector column and GIN index 2. Reciprocal rank fusion of both result lists ### Done when - [ ] A query with an exact error code ranks the matching chunk first ## Phase 4 · A page Localhost search with highlighted passages and source links. ### Steps 1. A JSON endpoint and a server-rendered page 2. Time it on 100k chunks ### Done when - [ ] Under 300 ms over 100k chunks ## Phase 5 · Operate Migrations, pg_dump, reindex. ### Steps 1. Migrations and a reindex command 2. pg_dump nightly and one restore; README with the chunking rule and provider swap Files: `README.md` ### Done when - [ ] A restore searches correctly ## Not in this build - Billions of vectors and serverless scale; under a few million this is a column. ## After v1, if you want it - Re-ranking with a cross-encoder - Per-user document permissions ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Your Postgres. DATABASE_URL=postgres://postgres:pw@localhost:5432/search # Required. openai or ollama. EMBED_PROVIDER=openai # Optional · secret. For the openai provider. OPENAI_API_KEY=sk-... # Required. Model name; dimension must match the column. EMBED_MODEL=text-embedding-3-small # Required. Where the ingester reads. DOCS_DIR=./docs
You are building a production product version of Pinecone. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Pinecone · product brief ## Problem For anything under a few million vectors, pgvector in the Postgres you already run does the job with a normal index and normal backups. Pinecone earns its fee at scale and for teams who want zero operations; a side project needs neither. ## Product outcome Semantic search as a column in your database, with exact-match behaviour preserved and backups you already know how to do. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Postgres with the pgvector extension - an embeddings API key or a local model ## Explicit non-goals for v1 - Billions of vectors and serverless scale; under a few million this is a column. - serverless scaling to billions of vectors - zero-ops managed indexes - hybrid and sparse search features - the SLA ## Success criteria - Index use verified with EXPLAIN - Hybrid verified on identifiers - Restore verified ===== BRIEF.md ===== # Build brief · Pinecone The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me semantic search over my own documents to replace Pinecone. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not write a vector index. pgvector is a Postgres extension with HNSW built in; it is a column and a query. ### Stack (fixed, do not substitute) - Postgres 16 with pgvector. Node 22. An embeddings provider behind an interface (OpenAI or a local model via Ollama). ### Data model (create this before Phase 1) - documents: id, source, title, url, updated_at - chunks: id, document_id, position, text, embedding vector(N), token_count Chunks, not documents, carry the vectors: a 40-page PDF as one vector matches nothing well. ### Phase 1 · Store and embed Build: a chunker (by paragraph, ~500 tokens with overlap) and an embed step behind an interface, writing chunks with their vectors. Idempotent on document hash. Done when: re-ingesting an unchanged document embeds nothing, a changed one re-embeds only its chunks, and the interface swaps providers by env var. Do not build yet: search, UI. ### Phase 2 · Search Build: a query function embedding the question and returning the top k chunks by cosine distance with their documents, using an HNSW index. Done when: a question about a known passage returns it in the top 3 and the query uses the index (EXPLAIN shows it). ### Phase 3 · Hybrid Build: combine vector results with Postgres full-text search via reciprocal rank fusion, because exact product names and error codes are where pure vectors fail. Done when: a query containing an exact identifier ranks the matching chunk first. ### Phase 4 · A page Build: a localhost search page and a JSON endpoint, results with highlighted passages and links to sources. Done when: a search returns in under 300 ms over 100k chunks. ### Phase 5 · Operate Build: migrations, a backup that is just pg_dump, a reindex command, the README. Done when: a restore searches correctly. ### Out of scope (and why) - Billions of vectors and serverless scale. Under a few million, this is a column. ### README must contain - The chunking rule and why. - The provider swap. ===== ARCHITECTURE.md ===== # Architecture · Pinecone ## Stack | Part | Choice | Why | | --- | --- | --- | | Database | Postgres 16 with pgvector | rule zero: a vector index is a column type | | Embeddings | A provider behind an interface | OpenAI or a local model via Ollama; swappable by env | | Runtime | Node 22 | the ingest and search scripts | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Ingest | chunking and embedding | Any provider | | Store | Postgres and indexes | Qdrant at scale, same chunk shape | | Rank | hybrid fusion | Tune weights | | API | endpoint and page | Any client | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · Your Postgres. - `EMBED_PROVIDER` · required · openai or ollama. - `OPENAI_API_KEY` · optional, secret · For the openai provider. - `EMBED_MODEL` · required · Model name; dimension must match the column. - `DOCS_DIR` · required · Where the ingester reads. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Pinecone product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Postgres 16 with pgvector, A provider behind an interface, Node 22. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ===== MILESTONES.md ===== # Delivery milestones · Pinecone Estimated effort: **one sitting** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Store and embed Chunks with vectors, idempotent on document hash, provider swappable. ### Steps 1. Create the schema documents (id, source, title, url, updated_at, hash), chunks (id, document_id, position, text, embedding vector(N), token_count). N matches EMBED_MODEL's dimension. Files: `schema.sql` ```sh mkdir semantic && cd semantic && git init && npm init -y && npm pkg set type=module && npm install pg@8 mkdir -p docs && cp .env.example .env ``` 2. Chunk by paragraph around 500 tokens with overlap; embed behind an interface; skip unchanged documents by hash ### Done when - [ ] Re-ingesting an unchanged document embeds nothing - [ ] A changed one re-embeds only its chunks - [ ] Switching EMBED_PROVIDER works ## M2 · Search Top-k by cosine with an HNSW index actually used. ### Steps 1. Create the HNSW index and the query function ```sh CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops); ``` 2. Verify the plan uses the index ```sh EXPLAIN ANALYZE SELECT ... ORDER BY embedding <=> $1 LIMIT 10; ``` ### Done when - [ ] A known passage returns in the top 3 - [ ] EXPLAIN shows the index ## M3 · Hybrid Exact identifiers win: fuse vector and full-text ranks. ### Steps 1. Add a tsvector column and GIN index 2. Reciprocal rank fusion of both result lists ### Done when - [ ] A query with an exact error code ranks the matching chunk first ## M4 · A page Localhost search with highlighted passages and source links. ### Steps 1. A JSON endpoint and a server-rendered page 2. Time it on 100k chunks ### Done when - [ ] Under 300 ms over 100k chunks ## M5 · Operate Migrations, pg_dump, reindex. ### Steps 1. Migrations and a reindex command 2. pg_dump nightly and one restore; README with the chunking rule and provider swap Files: `README.md` ### Done when - [ ] A restore searches correctly ## M6 · Serve it to others (production only) Auth, rate limits, and cost visibility. ### Steps 1. Token auth and a rate limit on the JSON endpoint 2. Log embedding calls and estimate monthly cost ### Done when - [ ] Unauthenticated calls are refused - [ ] Cost per day is visible ===== OPERATIONS.md ===== # Operations · Pinecone ## Backup pg_dump nightly. ## Restore pg_restore. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Query latency and embedding spend. ## Incident checklist A leaked key: rotate. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Index use verified with EXPLAIN - [ ] Hybrid verified on identifiers - [ ] Restore verified ## Launch constraint Do not market omitted Pinecone capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Your Postgres. DATABASE_URL=postgres://postgres:pw@localhost:5432/search # Required. openai or ollama. EMBED_PROVIDER=openai # Optional · secret. For the openai provider. OPENAI_API_KEY=sk-... # Required. Model name; dimension must match the column. EMBED_MODEL=text-embedding-3-small # Required. Where the ingester reads. DOCS_DIR=./docs
README.md
# Pinecone · indie build Semantic search over your own documents with pgvector in the Postgres you already run: chunked documents with embeddings, cosine search with an HNSW index, hybrid ranking with full-text so exact identifiers still win, a localhost search page, and backups that are just pg_dump. Estimated effort: **one sitting**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Database | Postgres 16 with pgvector | rule zero: a vector index is a column type | | Embeddings | A provider behind an interface | OpenAI or a local model via Ollama; swappable by env | | Runtime | Node 22 | the ingest and search scripts | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Node.js 22 or newer** · free - Why: Everything in this build runs on it: the server, the scripts, the tests. - Get it: Download the LTS installer from nodejs.org, or install with your package manager (brew install node, or nvm install 22). Restart the terminal afterwards. - Verify: node --version prints v22 or higher - [ ] **A terminal and a code editor** · free - Why: Every step below is a command you type or a file you edit. - Get it: VS Code (code.visualstudio.com), Cursor or Zed. Open a folder for the project and use the editor's built-in terminal. - Verify: You can open a folder and run a command in its terminal - [ ] **Git** · free - Why: History for your code, and the way most hosts deploy. - Get it: Install from git-scm.com or with your package manager, then run git init in the project folder once it exists. - Verify: git --version prints a version - [ ] **Postgres 16 with the pgvector extension** · free - Why: The whole store. - Get it: Docker: docker run -d -e POSTGRES_PASSWORD=pw -p 5432:5432 pgvector/pgvector:pg16. Or install pgvector on an existing Postgres. - Verify: CREATE EXTENSION vector; succeeds - [ ] **An embeddings API key, or Ollama locally** · pay per use, or free locally - Why: Phase 1 embeds chunks. - Get it: OpenAI: platform.openai.com/api-keys. Local: install Ollama and ollama pull nomic-embed-text. - [ ] **The documents to search** · free - Why: Markdown, PDFs converted to text, or a folder of notes. - Get it: A folder of files. ## Quick start ```sh mkdir semantic && cd semantic && git init && npm init -y && npm pkg set type=module && npm install pg@8 mkdir -p docs && cp .env.example .env ``` Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - Billions of vectors and serverless scale; under a few million this is a column. - serverless scaling to billions of vectors - zero-ops managed indexes - hybrid and sparse search features - the SLA If one of those is essential to you, that is the reason to keep paying for Pinecone, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Pinecone The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me semantic search over my own documents to replace Pinecone. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not write a vector index. pgvector is a Postgres extension with HNSW built in; it is a column and a query. ### Stack (fixed, do not substitute) - Postgres 16 with pgvector. Node 22. An embeddings provider behind an interface (OpenAI or a local model via Ollama). ### Data model (create this before Phase 1) - documents: id, source, title, url, updated_at - chunks: id, document_id, position, text, embedding vector(N), token_count Chunks, not documents, carry the vectors: a 40-page PDF as one vector matches nothing well. ### Phase 1 · Store and embed Build: a chunker (by paragraph, ~500 tokens with overlap) and an embed step behind an interface, writing chunks with their vectors. Idempotent on document hash. Done when: re-ingesting an unchanged document embeds nothing, a changed one re-embeds only its chunks, and the interface swaps providers by env var. Do not build yet: search, UI. ### Phase 2 · Search Build: a query function embedding the question and returning the top k chunks by cosine distance with their documents, using an HNSW index. Done when: a question about a known passage returns it in the top 3 and the query uses the index (EXPLAIN shows it). ### Phase 3 · Hybrid Build: combine vector results with Postgres full-text search via reciprocal rank fusion, because exact product names and error codes are where pure vectors fail. Done when: a query containing an exact identifier ranks the matching chunk first. ### Phase 4 · A page Build: a localhost search page and a JSON endpoint, results with highlighted passages and links to sources. Done when: a search returns in under 300 ms over 100k chunks. ### Phase 5 · Operate Build: migrations, a backup that is just pg_dump, a reindex command, the README. Done when: a restore searches correctly. ### Out of scope (and why) - Billions of vectors and serverless scale. Under a few million, this is a column. ### README must contain - The chunking rule and why. - The provider swap.
AGENTS.md
# Agent instructions · Pinecone indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Postgres 16 with pgvector, A provider behind an interface, Node 22. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs".
BUILD_PLAN.md
# Build plan · Pinecone Semantic search over your own documents with pgvector in the Postgres you already run: chunked documents with embeddings, cosine search with an HNSW index, hybrid ranking with full-text so exact identifiers still win, a localhost search page, and backups that are just pg_dump. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Store and embed Chunks with vectors, idempotent on document hash, provider swappable. ### Steps 1. Create the schema documents (id, source, title, url, updated_at, hash), chunks (id, document_id, position, text, embedding vector(N), token_count). N matches EMBED_MODEL's dimension. Files: `schema.sql` ```sh mkdir semantic && cd semantic && git init && npm init -y && npm pkg set type=module && npm install pg@8 mkdir -p docs && cp .env.example .env ``` 2. Chunk by paragraph around 500 tokens with overlap; embed behind an interface; skip unchanged documents by hash ### Done when - [ ] Re-ingesting an unchanged document embeds nothing - [ ] A changed one re-embeds only its chunks - [ ] Switching EMBED_PROVIDER works ## Phase 2 · Search Top-k by cosine with an HNSW index actually used. ### Steps 1. Create the HNSW index and the query function ```sh CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops); ``` 2. Verify the plan uses the index ```sh EXPLAIN ANALYZE SELECT ... ORDER BY embedding <=> $1 LIMIT 10; ``` ### Done when - [ ] A known passage returns in the top 3 - [ ] EXPLAIN shows the index ## Phase 3 · Hybrid Exact identifiers win: fuse vector and full-text ranks. ### Steps 1. Add a tsvector column and GIN index 2. Reciprocal rank fusion of both result lists ### Done when - [ ] A query with an exact error code ranks the matching chunk first ## Phase 4 · A page Localhost search with highlighted passages and source links. ### Steps 1. A JSON endpoint and a server-rendered page 2. Time it on 100k chunks ### Done when - [ ] Under 300 ms over 100k chunks ## Phase 5 · Operate Migrations, pg_dump, reindex. ### Steps 1. Migrations and a reindex command 2. pg_dump nightly and one restore; README with the chunking rule and provider swap Files: `README.md` ### Done when - [ ] A restore searches correctly ## Not in this build - Billions of vectors and serverless scale; under a few million this is a column. ## After v1, if you want it - Re-ranking with a cross-encoder - Per-user document permissions
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Your Postgres. DATABASE_URL=postgres://postgres:pw@localhost:5432/search # Required. openai or ollama. EMBED_PROVIDER=openai # Optional · secret. For the openai provider. OPENAI_API_KEY=sk-... # Required. Model name; dimension must match the column. EMBED_MODEL=text-embedding-3-small # Required. Where the ingester reads. DOCS_DIR=./docs
PRODUCT.md
# Pinecone · product brief ## Problem For anything under a few million vectors, pgvector in the Postgres you already run does the job with a normal index and normal backups. Pinecone earns its fee at scale and for teams who want zero operations; a side project needs neither. ## Product outcome Semantic search as a column in your database, with exact-match behaviour preserved and backups you already know how to do. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Postgres with the pgvector extension - an embeddings API key or a local model ## Explicit non-goals for v1 - Billions of vectors and serverless scale; under a few million this is a column. - serverless scaling to billions of vectors - zero-ops managed indexes - hybrid and sparse search features - the SLA ## Success criteria - Index use verified with EXPLAIN - Hybrid verified on identifiers - Restore verified
BRIEF.md
# Build brief · Pinecone The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me semantic search over my own documents to replace Pinecone. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Rule zero Do not write a vector index. pgvector is a Postgres extension with HNSW built in; it is a column and a query. ### Stack (fixed, do not substitute) - Postgres 16 with pgvector. Node 22. An embeddings provider behind an interface (OpenAI or a local model via Ollama). ### Data model (create this before Phase 1) - documents: id, source, title, url, updated_at - chunks: id, document_id, position, text, embedding vector(N), token_count Chunks, not documents, carry the vectors: a 40-page PDF as one vector matches nothing well. ### Phase 1 · Store and embed Build: a chunker (by paragraph, ~500 tokens with overlap) and an embed step behind an interface, writing chunks with their vectors. Idempotent on document hash. Done when: re-ingesting an unchanged document embeds nothing, a changed one re-embeds only its chunks, and the interface swaps providers by env var. Do not build yet: search, UI. ### Phase 2 · Search Build: a query function embedding the question and returning the top k chunks by cosine distance with their documents, using an HNSW index. Done when: a question about a known passage returns it in the top 3 and the query uses the index (EXPLAIN shows it). ### Phase 3 · Hybrid Build: combine vector results with Postgres full-text search via reciprocal rank fusion, because exact product names and error codes are where pure vectors fail. Done when: a query containing an exact identifier ranks the matching chunk first. ### Phase 4 · A page Build: a localhost search page and a JSON endpoint, results with highlighted passages and links to sources. Done when: a search returns in under 300 ms over 100k chunks. ### Phase 5 · Operate Build: migrations, a backup that is just pg_dump, a reindex command, the README. Done when: a restore searches correctly. ### Out of scope (and why) - Billions of vectors and serverless scale. Under a few million, this is a column. ### README must contain - The chunking rule and why. - The provider swap.
ARCHITECTURE.md
# Architecture · Pinecone ## Stack | Part | Choice | Why | | --- | --- | --- | | Database | Postgres 16 with pgvector | rule zero: a vector index is a column type | | Embeddings | A provider behind an interface | OpenAI or a local model via Ollama; swappable by env | | Runtime | Node 22 | the ingest and search scripts | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Ingest | chunking and embedding | Any provider | | Store | Postgres and indexes | Qdrant at scale, same chunk shape | | Rank | hybrid fusion | Tune weights | | API | endpoint and page | Any client | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `DATABASE_URL` · required, secret · Your Postgres. - `EMBED_PROVIDER` · required · openai or ollama. - `OPENAI_API_KEY` · optional, secret · For the openai provider. - `EMBED_MODEL` · required · Model name; dimension must match the column. - `DOCS_DIR` · required · Where the ingester reads. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Pinecone product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Postgres 16 with pgvector, A provider behind an interface, Node 22. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone.
MILESTONES.md
# Delivery milestones · Pinecone Estimated effort: **one sitting** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Store and embed Chunks with vectors, idempotent on document hash, provider swappable. ### Steps 1. Create the schema documents (id, source, title, url, updated_at, hash), chunks (id, document_id, position, text, embedding vector(N), token_count). N matches EMBED_MODEL's dimension. Files: `schema.sql` ```sh mkdir semantic && cd semantic && git init && npm init -y && npm pkg set type=module && npm install pg@8 mkdir -p docs && cp .env.example .env ``` 2. Chunk by paragraph around 500 tokens with overlap; embed behind an interface; skip unchanged documents by hash ### Done when - [ ] Re-ingesting an unchanged document embeds nothing - [ ] A changed one re-embeds only its chunks - [ ] Switching EMBED_PROVIDER works ## M2 · Search Top-k by cosine with an HNSW index actually used. ### Steps 1. Create the HNSW index and the query function ```sh CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops); ``` 2. Verify the plan uses the index ```sh EXPLAIN ANALYZE SELECT ... ORDER BY embedding <=> $1 LIMIT 10; ``` ### Done when - [ ] A known passage returns in the top 3 - [ ] EXPLAIN shows the index ## M3 · Hybrid Exact identifiers win: fuse vector and full-text ranks. ### Steps 1. Add a tsvector column and GIN index 2. Reciprocal rank fusion of both result lists ### Done when - [ ] A query with an exact error code ranks the matching chunk first ## M4 · A page Localhost search with highlighted passages and source links. ### Steps 1. A JSON endpoint and a server-rendered page 2. Time it on 100k chunks ### Done when - [ ] Under 300 ms over 100k chunks ## M5 · Operate Migrations, pg_dump, reindex. ### Steps 1. Migrations and a reindex command 2. pg_dump nightly and one restore; README with the chunking rule and provider swap Files: `README.md` ### Done when - [ ] A restore searches correctly ## M6 · Serve it to others (production only) Auth, rate limits, and cost visibility. ### Steps 1. Token auth and a rate limit on the JSON endpoint 2. Log embedding calls and estimate monthly cost ### Done when - [ ] Unauthenticated calls are refused - [ ] Cost per day is visible
OPERATIONS.md
# Operations · Pinecone ## Backup pg_dump nightly. ## Restore pg_restore. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Query latency and embedding spend. ## Incident checklist A leaked key: rotate. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Index use verified with EXPLAIN - [ ] Hybrid verified on identifiers - [ ] Restore verified ## Launch constraint Do not market omitted Pinecone capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required · secret. Your Postgres. DATABASE_URL=postgres://postgres:pw@localhost:5432/search # Required. openai or ollama. EMBED_PROVIDER=openai # Optional · secret. For the openai provider. OPENAI_API_KEY=sk-... # Required. Model name; dimension must match the column. EMBED_MODEL=text-embedding-3-small # Required. Where the ingester reads. DOCS_DIR=./docs
$ choose a build depth, inspect the files, then open the complete pack in your agent
Teams pay so vector search is nobody's job. Under a million vectors, it is a column.
xserverless scaling to billions of vectors
xzero-ops managed indexes
xhybrid and sparse search features
xthe SLA
share on X ↗"I just replaced Pinecone ($20/mo) with one prompt"
Pinecone pricing
builder$20/mo · monthly flat · $240/yr
free tierThe free Starter plan covers 2 GB of vectors across 5 indexes with monthly read and write unit caps.
verified 2026-09-04 · source ↗
Is Pinecone free?
The free Starter plan covers 2 GB of vectors across 5 indexes with monthly read and write unit caps. Paid is Builder at $20/mo (checked 2026-09-04).
Vibecode Pinecone
Kinda. The core of Pinecone is buildable in a weekend with the prompt on this page, but there are real gaps: serverless scaling to billions of vectors, zero-ops managed indexes. Read the honest list above before committing.
How much does Pinecone cost?
Pinecone costs about $20/month (Builder, checked 2026-09-04), which is $240 per year.
What do I lose by replacing Pinecone?
Honestly: serverless scaling to billions of vectors; zero-ops managed indexes; hybrid and sparse search features; the SLA. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Pinecone?
Yes: pgvector (vector similarity search for Postgres), Qdrant (open-source vector database). Using prior art is also vibecoding; the prompt is for when you want it exactly your way.