Vibecode Zotero Storage
track this build4 phases, 12 steps, beginner friendly0%KINDA · weekend project
The visible reference sync + storage loop is buildable, but a credible replacement needs more than the first screen. Zotero Storage earns its keep through data, import reliability, so expect a weekend or multi-day build and a narrower personal scope.
the project pack5 files · written for this build, step by step
You are building a lean indie version of Zotero Storage. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Zotero Storage · indie build File sync for a reference library, where the difficulty is that the same paper arrives from three places under three filenames, two machines edit annotations on the same document, and a citation key must stay stable because it is written into someone's manuscript. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Storage | Object storage you control, content addressed | the same paper downloaded twice should cost one copy | | Sync | Change journal per library with per-item resolution | whole-library comparison does not scale past a few thousand items | | Identity | Content hashing plus document identifiers for deduplication | filenames are worthless for identifying a paper | | Annotations | Stored separately from the document and merged per annotation | two machines annotating one paper is the normal case | | Runtime | Python 3.12 with SQLite locally | the library lives on the machine and syncs through storage | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the local library and the sync client. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **Object storage you control** · pennies per month for a personal library - Why: The whole point is that a research library sits on storage you own rather than a service that can change its terms. - Get it: Any provider with a compatible interface, plus a lifecycle policy and versioning decided before you upload anything. - Verify: A file uploads, downloads and is listed by content hash - [ ] **A real library of at least two thousand items** · free - Why: Sync proportionality, deduplication and storage reporting all behave differently at scale, and a hundred items proves nothing. - Get it: Export from your existing reference manager including the attachments, or assemble one from open access papers. - Verify: Two thousand items with attachments exist, including some genuine duplicates - [ ] **Two machines, or two library copies** · free - Why: Every claim about sync and annotation merging is untestable with one, and the interesting cases require both being offline at once. - Get it: A second machine, or two separate library directories with independent device identifiers. - Verify: Both can sync against the same storage and be taken offline independently - [ ] **Two copies of one paper that differ byte for byte** · free - Why: Deduplication by hashing catches identical files, and the real case is a publisher copy and a repository copy of the same paper with different stamping. - Get it: Download one paper from two sources, or add a watermark to a copy. - Verify: Two files with different hashes are identified as the same paper ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - The reference manager and its word processor integration. - Group libraries with shared editing and permissions. - Full text search inside attachments. - Metadata retrieval from publishers. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - licensed scholarly metadata If one of those is essential to you, that is the reason to keep paying for Zotero Storage, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Zotero Storage The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me attachment sync for a research library, across machines, on storage you control to replace Zotero Storage. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Storage: Object storage you control, content addressed · the same paper downloaded twice should cost one copy - Sync: Change journal per library with per-item resolution · whole-library comparison does not scale past a few thousand items - Identity: Content hashing plus document identifiers for deduplication · filenames are worthless for identifying a paper - Annotations: Stored separately from the document and merged per annotation · two machines annotating one paper is the normal case - Runtime: Python 3.12 with SQLite locally · the library lives on the machine and syncs through storage ### Data model (create this before Phase 1) - items(id, citation_key, item_type, title, authors_json, year, doi, created_at, modified_at) - attachments(id, item_id, content_hash, bytes, mime, filename, added_at) · content hash is the identity - blobs(content_hash, storage_path, uploaded_at, refcount) · one copy per distinct file, however many items reference it - annotations(id, attachment_id, page, rect_json, kind, text, colour, device_id, created_at, modified_at, deleted) - journal(id, library_id, entity_kind, entity_id, operation, at, device_id) · the change log sync reads - sync_state(device_id, last_journal_id, last_sync_at) - A citation key is assigned once and never regenerated, because it is written into manuscripts that are not in your control. ### Phase 1 · Knowing when two files are the same paper Build: Content hash every attachment and store one blob per distinct hash. Extract document identifiers where present and use them to link items that differ byte for byte. Report likely duplicates rather than merging automatically. Trap: The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. Done when: The same file added twice stores one blob and two references. Two different files of the same paper are proposed as duplicates via their identifiers, tested with a stamped and an unstamped copy. Merging is proposed, never automatic. Reference counting means deleting one item does not remove a blob another still uses. Do not build yet: No sync, annotations or citation keys. ### Phase 2 · Citation keys that never move Build: Assign a citation key once at item creation and never regenerate it. Handle collisions deterministically. Keep the key when metadata is corrected, and record any key change as an explicit act with the old key retained as an alias. Trap: Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. Done when: Correcting an item's metadata does not change its citation key, tested by editing the year. A collision resolves deterministically and reproducibly. A deliberate key change retains the old key as a resolvable alias. Exported bibliography files use the stable key. Do not build yet: No sync or annotations yet. ### Phase 3 · Two machines, one library Build: A per-library change journal recording every operation. Sync by exchanging journal entries since the last position rather than comparing libraries. Upload and download blobs by content hash. Handle interruption mid-sync. Trap: Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. Done when: Sync cost is proportional to changes rather than library size, verified against a library of five thousand items with three changes. A deletion propagates rather than being reinserted by the other machine. An interrupted sync resumes without duplicating or losing entries. Blob transfer skips content already present. Do not build yet: No annotation merging yet. ### Phase 4 · Annotations from both machines Build: Store annotations separately from the document, keyed to position rather than embedded. Merge per annotation rather than per document. Handle the same annotation edited on both machines by keeping both rather than choosing. Trap: Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. Done when: Annotations made on two machines both survive a sync, tested by annotating the same paper on both while offline. The document file is unchanged by annotating, verified by checksum. The same annotation edited on both machines keeps both versions visibly. Deleting an annotation propagates rather than being reinstated. Do not build yet: No exports or storage management yet. ### Phase 5 · Storage that does not grow forever Build: Report storage use by item and blob. Allow attachments to be kept remotely only, fetched on demand. Export the entire library including files, annotations and keys in a form another tool can read. Trap: Evicting a locally cached attachment while an annotation references a page of it leaves an annotation pointing at nothing when the machine is offline, which is exactly when someone is working on a train. Keep annotations local always, since they are tiny, and evict only the documents. Done when: Storage use is reported per item and per blob. An evicted attachment fetches on demand and its annotations remain readable offline, tested with networking disabled. Export contains metadata, files, annotations and citation keys in a documented format. Deleting an item releases its blob only when no other item references it. Do not build yet: Nothing after this. ### Out of scope (and why) - The reference manager itself, its metadata retrieval and its word processor integration, which is the application this only stores files for. - Group libraries with shared editing and permissions, which is a substantially harder synchronisation and access problem. - Full text search inside attachments, which is a separate indexing project. - Metadata retrieval from publishers, which each expose differently and rate limit. ### README must contain - That attachments are content addressed so the same file stored twice costs one copy - That citation keys are assigned once and never regenerated, because they are in manuscripts - That sync exchanges a journal tail rather than comparing libraries - That annotations live outside the document so merging is per highlight - That eviction removes documents but never annotations, which are needed offline ===== AGENTS.md ===== # Agent instructions · Zotero Storage indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Object storage you control, content addressed, Change journal per library with per-item resolution, Content hashing plus document identifiers for deduplication, Stored separately from the document and merged per annotation, Python 3.12 with SQLite locally. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. - Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. - Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. - Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. ===== BUILD_PLAN.md ===== # Build plan · Zotero Storage File sync for a reference library, where the difficulty is that the same paper arrives from three places under three filenames, two machines edit annotations on the same document, and a citation key must stay stable because it is written into someone's manuscript. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Knowing when two files are the same paper The same paper from three sources is stored once. ### Steps 1. Content address the blobs One copy per distinct file, reference counted. Files: `src/blobs.py` 2. Extract document identifiers Byte differences from stamping mean hashing alone misses real duplicates. Files: `src/identify.py` 3. Propose duplicates, never merge Two items merged wrongly loses annotations and notes. Files: `src/duplicates.py` ### Done when - [ ] The same file twice stores one blob - [ ] A stamped copy is proposed as a duplicate - [ ] Merging is never automatic - [ ] Reference counting protects shared blobs ### Watch out - The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. ## Phase 2 · Citation keys that never move A key written into a manuscript keeps referring to the same thing forever. ### Steps 1. Assign once, never regenerate The key is in someone's manuscript. It is not yours to change. Files: `src/keys.py`, `tests/test_key_stability.py` 2. Resolve collisions deterministically Same inputs, same suffix, every time. Files: `src/keys.py` 3. Alias on a deliberate change The old key must still resolve, because documents still use it. Files: `src/aliases.py` ### Done when - [ ] Metadata correction does not change the key - [ ] Collisions resolve deterministically - [ ] A changed key keeps the old one as an alias - [ ] Exports use the stable key ### Watch out - Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. ## Phase 3 · Two machines, one library Changes made on either machine arrive on the other without loss. ### Steps 1. Journal every operation The change log is what makes sync proportional and deletions possible. Files: `src/journal.py` 2. Exchange the tail since last position Not a comparison. Comparison does not scale and cannot see deletions. Files: `src/sync.py` 3. Resume an interrupted sync Large libraries and unreliable connections. It will be interrupted. Files: `src/sync.py`, `tests/test_resume.py` ### Done when - [ ] Sync cost is proportional to changes - [ ] Deletions propagate rather than resurrecting - [ ] An interrupted sync resumes cleanly - [ ] Existing blobs are not retransferred ### Watch out - Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. ## Phase 4 · Annotations from both machines Highlighting on a laptop and a desktop does not lose either set. ### Steps 1. Keep annotations out of the document The file is content addressed. Writing into it breaks deduplication too. Files: `src/annotations.py` 2. Merge per annotation The document is not the conflict unit. The highlight is. Files: `src/merge.py`, `tests/test_annotation_merge.py` 3. Keep both sides on a true conflict Choosing for the user loses an afternoon of reading. Files: `src/merge.py` ### Done when - [ ] Annotations from two machines both survive - [ ] The document file is unchanged by annotating - [ ] A true conflict keeps both versions - [ ] Annotation deletions propagate ### Watch out - Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. ## Not in this build - The reference manager and its word processor integration. - Group libraries with shared editing and permissions. - Full text search inside attachments. - Metadata retrieval from publishers. ## After v1, if you want it - Full text indexing of attachments, which is the natural next thing once storage is solid - A read-only share of one collection, which needs a permissions model - Conflict review as a queue, once merging has run long enough to produce some ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Metadata, annotations and the change journal. The irreplaceable part. LIBRARY_PATH=~/.local/share/library/library.db # Required. Content-addressed attachments. Versioning on, because a research library is a career's work. BLOB_STORE=s3://my-library/blobs # Required · secret. Scoped to the bucket only. It holds everything you have read. STORAGE_CREDENTIALS=a scoped key # Required. Identifies this machine in the journal. Two machines sharing one breaks merging. DEVICE_ID=laptop # Required. How much of the library is kept locally. Documents evict, annotations never do. LOCAL_CACHE_GB=20 # Required. Present as a constant so the answer is visible in configuration. There is no other value. REGENERATE_CITATION_KEYS=never
You are building a lean indie version of Zotero Storage. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== README.md ===== # Zotero Storage · indie build File sync for a reference library, where the difficulty is that the same paper arrives from three places under three filenames, two machines edit annotations on the same document, and a citation key must stay stable because it is written into someone's manuscript. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Storage | Object storage you control, content addressed | the same paper downloaded twice should cost one copy | | Sync | Change journal per library with per-item resolution | whole-library comparison does not scale past a few thousand items | | Identity | Content hashing plus document identifiers for deduplication | filenames are worthless for identifying a paper | | Annotations | Stored separately from the document and merged per annotation | two machines annotating one paper is the normal case | | Runtime | Python 3.12 with SQLite locally | the library lives on the machine and syncs through storage | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the local library and the sync client. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **Object storage you control** · pennies per month for a personal library - Why: The whole point is that a research library sits on storage you own rather than a service that can change its terms. - Get it: Any provider with a compatible interface, plus a lifecycle policy and versioning decided before you upload anything. - Verify: A file uploads, downloads and is listed by content hash - [ ] **A real library of at least two thousand items** · free - Why: Sync proportionality, deduplication and storage reporting all behave differently at scale, and a hundred items proves nothing. - Get it: Export from your existing reference manager including the attachments, or assemble one from open access papers. - Verify: Two thousand items with attachments exist, including some genuine duplicates - [ ] **Two machines, or two library copies** · free - Why: Every claim about sync and annotation merging is untestable with one, and the interesting cases require both being offline at once. - Get it: A second machine, or two separate library directories with independent device identifiers. - Verify: Both can sync against the same storage and be taken offline independently - [ ] **Two copies of one paper that differ byte for byte** · free - Why: Deduplication by hashing catches identical files, and the real case is a publisher copy and a repository copy of the same paper with different stamping. - Get it: Download one paper from two sources, or add a watermark to a copy. - Verify: Two files with different hashes are identified as the same paper ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - The reference manager and its word processor integration. - Group libraries with shared editing and permissions. - Full text search inside attachments. - Metadata retrieval from publishers. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - licensed scholarly metadata If one of those is essential to you, that is the reason to keep paying for Zotero Storage, and the README should say so rather than pretend. ===== BRIEF.md ===== # Build brief · Zotero Storage The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me attachment sync for a research library, across machines, on storage you control to replace Zotero Storage. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Storage: Object storage you control, content addressed · the same paper downloaded twice should cost one copy - Sync: Change journal per library with per-item resolution · whole-library comparison does not scale past a few thousand items - Identity: Content hashing plus document identifiers for deduplication · filenames are worthless for identifying a paper - Annotations: Stored separately from the document and merged per annotation · two machines annotating one paper is the normal case - Runtime: Python 3.12 with SQLite locally · the library lives on the machine and syncs through storage ### Data model (create this before Phase 1) - items(id, citation_key, item_type, title, authors_json, year, doi, created_at, modified_at) - attachments(id, item_id, content_hash, bytes, mime, filename, added_at) · content hash is the identity - blobs(content_hash, storage_path, uploaded_at, refcount) · one copy per distinct file, however many items reference it - annotations(id, attachment_id, page, rect_json, kind, text, colour, device_id, created_at, modified_at, deleted) - journal(id, library_id, entity_kind, entity_id, operation, at, device_id) · the change log sync reads - sync_state(device_id, last_journal_id, last_sync_at) - A citation key is assigned once and never regenerated, because it is written into manuscripts that are not in your control. ### Phase 1 · Knowing when two files are the same paper Build: Content hash every attachment and store one blob per distinct hash. Extract document identifiers where present and use them to link items that differ byte for byte. Report likely duplicates rather than merging automatically. Trap: The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. Done when: The same file added twice stores one blob and two references. Two different files of the same paper are proposed as duplicates via their identifiers, tested with a stamped and an unstamped copy. Merging is proposed, never automatic. Reference counting means deleting one item does not remove a blob another still uses. Do not build yet: No sync, annotations or citation keys. ### Phase 2 · Citation keys that never move Build: Assign a citation key once at item creation and never regenerate it. Handle collisions deterministically. Keep the key when metadata is corrected, and record any key change as an explicit act with the old key retained as an alias. Trap: Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. Done when: Correcting an item's metadata does not change its citation key, tested by editing the year. A collision resolves deterministically and reproducibly. A deliberate key change retains the old key as a resolvable alias. Exported bibliography files use the stable key. Do not build yet: No sync or annotations yet. ### Phase 3 · Two machines, one library Build: A per-library change journal recording every operation. Sync by exchanging journal entries since the last position rather than comparing libraries. Upload and download blobs by content hash. Handle interruption mid-sync. Trap: Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. Done when: Sync cost is proportional to changes rather than library size, verified against a library of five thousand items with three changes. A deletion propagates rather than being reinserted by the other machine. An interrupted sync resumes without duplicating or losing entries. Blob transfer skips content already present. Do not build yet: No annotation merging yet. ### Phase 4 · Annotations from both machines Build: Store annotations separately from the document, keyed to position rather than embedded. Merge per annotation rather than per document. Handle the same annotation edited on both machines by keeping both rather than choosing. Trap: Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. Done when: Annotations made on two machines both survive a sync, tested by annotating the same paper on both while offline. The document file is unchanged by annotating, verified by checksum. The same annotation edited on both machines keeps both versions visibly. Deleting an annotation propagates rather than being reinstated. Do not build yet: No exports or storage management yet. ### Phase 5 · Storage that does not grow forever Build: Report storage use by item and blob. Allow attachments to be kept remotely only, fetched on demand. Export the entire library including files, annotations and keys in a form another tool can read. Trap: Evicting a locally cached attachment while an annotation references a page of it leaves an annotation pointing at nothing when the machine is offline, which is exactly when someone is working on a train. Keep annotations local always, since they are tiny, and evict only the documents. Done when: Storage use is reported per item and per blob. An evicted attachment fetches on demand and its annotations remain readable offline, tested with networking disabled. Export contains metadata, files, annotations and citation keys in a documented format. Deleting an item releases its blob only when no other item references it. Do not build yet: Nothing after this. ### Out of scope (and why) - The reference manager itself, its metadata retrieval and its word processor integration, which is the application this only stores files for. - Group libraries with shared editing and permissions, which is a substantially harder synchronisation and access problem. - Full text search inside attachments, which is a separate indexing project. - Metadata retrieval from publishers, which each expose differently and rate limit. ### README must contain - That attachments are content addressed so the same file stored twice costs one copy - That citation keys are assigned once and never regenerated, because they are in manuscripts - That sync exchanges a journal tail rather than comparing libraries - That annotations live outside the document so merging is per highlight - That eviction removes documents but never annotations, which are needed offline ===== AGENTS.md ===== # Agent instructions · Zotero Storage indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Object storage you control, content addressed, Change journal per library with per-item resolution, Content hashing plus document identifiers for deduplication, Stored separately from the document and merged per annotation, Python 3.12 with SQLite locally. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. - Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. - Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. - Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. ===== BUILD_PLAN.md ===== # Build plan · Zotero Storage File sync for a reference library, where the difficulty is that the same paper arrives from three places under three filenames, two machines edit annotations on the same document, and a citation key must stay stable because it is written into someone's manuscript. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Knowing when two files are the same paper The same paper from three sources is stored once. ### Steps 1. Content address the blobs One copy per distinct file, reference counted. Files: `src/blobs.py` 2. Extract document identifiers Byte differences from stamping mean hashing alone misses real duplicates. Files: `src/identify.py` 3. Propose duplicates, never merge Two items merged wrongly loses annotations and notes. Files: `src/duplicates.py` ### Done when - [ ] The same file twice stores one blob - [ ] A stamped copy is proposed as a duplicate - [ ] Merging is never automatic - [ ] Reference counting protects shared blobs ### Watch out - The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. ## Phase 2 · Citation keys that never move A key written into a manuscript keeps referring to the same thing forever. ### Steps 1. Assign once, never regenerate The key is in someone's manuscript. It is not yours to change. Files: `src/keys.py`, `tests/test_key_stability.py` 2. Resolve collisions deterministically Same inputs, same suffix, every time. Files: `src/keys.py` 3. Alias on a deliberate change The old key must still resolve, because documents still use it. Files: `src/aliases.py` ### Done when - [ ] Metadata correction does not change the key - [ ] Collisions resolve deterministically - [ ] A changed key keeps the old one as an alias - [ ] Exports use the stable key ### Watch out - Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. ## Phase 3 · Two machines, one library Changes made on either machine arrive on the other without loss. ### Steps 1. Journal every operation The change log is what makes sync proportional and deletions possible. Files: `src/journal.py` 2. Exchange the tail since last position Not a comparison. Comparison does not scale and cannot see deletions. Files: `src/sync.py` 3. Resume an interrupted sync Large libraries and unreliable connections. It will be interrupted. Files: `src/sync.py`, `tests/test_resume.py` ### Done when - [ ] Sync cost is proportional to changes - [ ] Deletions propagate rather than resurrecting - [ ] An interrupted sync resumes cleanly - [ ] Existing blobs are not retransferred ### Watch out - Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. ## Phase 4 · Annotations from both machines Highlighting on a laptop and a desktop does not lose either set. ### Steps 1. Keep annotations out of the document The file is content addressed. Writing into it breaks deduplication too. Files: `src/annotations.py` 2. Merge per annotation The document is not the conflict unit. The highlight is. Files: `src/merge.py`, `tests/test_annotation_merge.py` 3. Keep both sides on a true conflict Choosing for the user loses an afternoon of reading. Files: `src/merge.py` ### Done when - [ ] Annotations from two machines both survive - [ ] The document file is unchanged by annotating - [ ] A true conflict keeps both versions - [ ] Annotation deletions propagate ### Watch out - Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. ## Not in this build - The reference manager and its word processor integration. - Group libraries with shared editing and permissions. - Full text search inside attachments. - Metadata retrieval from publishers. ## After v1, if you want it - Full text indexing of attachments, which is the natural next thing once storage is solid - A read-only share of one collection, which needs a permissions model - Conflict review as a queue, once merging has run long enough to produce some ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Metadata, annotations and the change journal. The irreplaceable part. LIBRARY_PATH=~/.local/share/library/library.db # Required. Content-addressed attachments. Versioning on, because a research library is a career's work. BLOB_STORE=s3://my-library/blobs # Required · secret. Scoped to the bucket only. It holds everything you have read. STORAGE_CREDENTIALS=a scoped key # Required. Identifies this machine in the journal. Two machines sharing one breaks merging. DEVICE_ID=laptop # Required. How much of the library is kept locally. Documents evict, annotations never do. LOCAL_CACHE_GB=20 # Required. Present as a constant so the answer is visible in configuration. There is no other value. REGENERATE_CITATION_KEYS=never
You are building a production product version of Zotero Storage. Create the following project files first, then implement the application by following them. Keep the files updated as decisions change. Do not collapse this into a single README or prompt. ===== PRODUCT.md ===== # Zotero Storage · product brief ## Problem The visible reference sync + storage loop is buildable, but a credible replacement needs more than the first screen. Zotero Storage earns its keep through data, import reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A research library that syncs across machines without losing an annotation, stores each paper once however many times you added it, and keeps citation keys stable because they are already in your manuscripts. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Node.js 22 - browser automation for user-authorized imports - optional OpenAI API key stored in .env ## Explicit non-goals for v1 - The reference manager and its word processor integration. - Group libraries with shared editing and permissions. - Full text search inside attachments. - Metadata retrieval from publishers. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - licensed scholarly metadata ## Success criteria - Metadata correction is confirmed not to change a citation key - Sync cost is confirmed proportional to changes, not library size - Annotations from two machines are confirmed both to survive - The document file is confirmed unchanged by annotating - Annotations are confirmed readable offline after eviction ===== BRIEF.md ===== # Build brief · Zotero Storage The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me attachment sync for a research library, across machines, on storage you control to replace Zotero Storage. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Storage: Object storage you control, content addressed · the same paper downloaded twice should cost one copy - Sync: Change journal per library with per-item resolution · whole-library comparison does not scale past a few thousand items - Identity: Content hashing plus document identifiers for deduplication · filenames are worthless for identifying a paper - Annotations: Stored separately from the document and merged per annotation · two machines annotating one paper is the normal case - Runtime: Python 3.12 with SQLite locally · the library lives on the machine and syncs through storage ### Data model (create this before Phase 1) - items(id, citation_key, item_type, title, authors_json, year, doi, created_at, modified_at) - attachments(id, item_id, content_hash, bytes, mime, filename, added_at) · content hash is the identity - blobs(content_hash, storage_path, uploaded_at, refcount) · one copy per distinct file, however many items reference it - annotations(id, attachment_id, page, rect_json, kind, text, colour, device_id, created_at, modified_at, deleted) - journal(id, library_id, entity_kind, entity_id, operation, at, device_id) · the change log sync reads - sync_state(device_id, last_journal_id, last_sync_at) - A citation key is assigned once and never regenerated, because it is written into manuscripts that are not in your control. ### Phase 1 · Knowing when two files are the same paper Build: Content hash every attachment and store one blob per distinct hash. Extract document identifiers where present and use them to link items that differ byte for byte. Report likely duplicates rather than merging automatically. Trap: The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. Done when: The same file added twice stores one blob and two references. Two different files of the same paper are proposed as duplicates via their identifiers, tested with a stamped and an unstamped copy. Merging is proposed, never automatic. Reference counting means deleting one item does not remove a blob another still uses. Do not build yet: No sync, annotations or citation keys. ### Phase 2 · Citation keys that never move Build: Assign a citation key once at item creation and never regenerate it. Handle collisions deterministically. Keep the key when metadata is corrected, and record any key change as an explicit act with the old key retained as an alias. Trap: Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. Done when: Correcting an item's metadata does not change its citation key, tested by editing the year. A collision resolves deterministically and reproducibly. A deliberate key change retains the old key as a resolvable alias. Exported bibliography files use the stable key. Do not build yet: No sync or annotations yet. ### Phase 3 · Two machines, one library Build: A per-library change journal recording every operation. Sync by exchanging journal entries since the last position rather than comparing libraries. Upload and download blobs by content hash. Handle interruption mid-sync. Trap: Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. Done when: Sync cost is proportional to changes rather than library size, verified against a library of five thousand items with three changes. A deletion propagates rather than being reinserted by the other machine. An interrupted sync resumes without duplicating or losing entries. Blob transfer skips content already present. Do not build yet: No annotation merging yet. ### Phase 4 · Annotations from both machines Build: Store annotations separately from the document, keyed to position rather than embedded. Merge per annotation rather than per document. Handle the same annotation edited on both machines by keeping both rather than choosing. Trap: Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. Done when: Annotations made on two machines both survive a sync, tested by annotating the same paper on both while offline. The document file is unchanged by annotating, verified by checksum. The same annotation edited on both machines keeps both versions visibly. Deleting an annotation propagates rather than being reinstated. Do not build yet: No exports or storage management yet. ### Phase 5 · Storage that does not grow forever Build: Report storage use by item and blob. Allow attachments to be kept remotely only, fetched on demand. Export the entire library including files, annotations and keys in a form another tool can read. Trap: Evicting a locally cached attachment while an annotation references a page of it leaves an annotation pointing at nothing when the machine is offline, which is exactly when someone is working on a train. Keep annotations local always, since they are tiny, and evict only the documents. Done when: Storage use is reported per item and per blob. An evicted attachment fetches on demand and its annotations remain readable offline, tested with networking disabled. Export contains metadata, files, annotations and citation keys in a documented format. Deleting an item releases its blob only when no other item references it. Do not build yet: Nothing after this. ### Out of scope (and why) - The reference manager itself, its metadata retrieval and its word processor integration, which is the application this only stores files for. - Group libraries with shared editing and permissions, which is a substantially harder synchronisation and access problem. - Full text search inside attachments, which is a separate indexing project. - Metadata retrieval from publishers, which each expose differently and rate limit. ### README must contain - That attachments are content addressed so the same file stored twice costs one copy - That citation keys are assigned once and never regenerated, because they are in manuscripts - That sync exchanges a journal tail rather than comparing libraries - That annotations live outside the document so merging is per highlight - That eviction removes documents but never annotations, which are needed offline ===== ARCHITECTURE.md ===== # Architecture · Zotero Storage ## Stack | Part | Choice | Why | | --- | --- | --- | | Storage | Object storage you control, content addressed | the same paper downloaded twice should cost one copy | | Sync | Change journal per library with per-item resolution | whole-library comparison does not scale past a few thousand items | | Identity | Content hashing plus document identifiers for deduplication | filenames are worthless for identifying a paper | | Annotations | Stored separately from the document and merged per annotation | two machines annotating one paper is the normal case | | Runtime | Python 3.12 with SQLite locally | the library lives on the machine and syncs through storage | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Blobs | Content-addressed storage with reference counting. | Any store, provided identity is the content rather than the filename. | | Keys | Stable citation keys and aliases. | Not swappable for regeneration from metadata, which breaks existing manuscripts. | | Journal | The change log sync reads. | Any log, provided sync exchanges a tail rather than comparing libraries. | | Annotations | Records outside the document, merged individually. | Any store, provided annotations never live inside the file. | | Eviction | Local cache management. | Any policy that evicts documents and never annotations. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `LIBRARY_PATH` · required · Metadata, annotations and the change journal. The irreplaceable part. - `BLOB_STORE` · required · Content-addressed attachments. Versioning on, because a research library is a career's work. - `STORAGE_CREDENTIALS` · required, secret · Scoped to the bucket only. It holds everything you have read. - `DEVICE_ID` · required · Identifies this machine in the journal. Two machines sharing one breaks merging. - `LOCAL_CACHE_GB` · required · How much of the library is kept locally. Documents evict, annotations never do. - `REGENERATE_CITATION_KEYS` · required · Present as a constant so the answer is visible in configuration. There is no other value. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it. ===== AGENTS.md ===== # Agent instructions · Zotero Storage product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Object storage you control, content addressed, Change journal per library with per-item resolution, Content hashing plus document identifiers for deduplication, Stored separately from the document and merged per annotation, Python 3.12 with SQLite locally. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. - Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. - Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. - Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. - Evicting a locally cached attachment while an annotation references a page of it leaves an annotation pointing at nothing when the machine is offline, which is exactly when someone is working on a train. Keep annotations local always, since they are tiny, and evict only the documents. ===== MILESTONES.md ===== # Delivery milestones · Zotero Storage Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Knowing when two files are the same paper The same paper from three sources is stored once. ### Steps 1. Content address the blobs One copy per distinct file, reference counted. Files: `src/blobs.py` 2. Extract document identifiers Byte differences from stamping mean hashing alone misses real duplicates. Files: `src/identify.py` 3. Propose duplicates, never merge Two items merged wrongly loses annotations and notes. Files: `src/duplicates.py` ### Done when - [ ] The same file twice stores one blob - [ ] A stamped copy is proposed as a duplicate - [ ] Merging is never automatic - [ ] Reference counting protects shared blobs ### Watch out - The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. ## M2 · Citation keys that never move A key written into a manuscript keeps referring to the same thing forever. ### Steps 1. Assign once, never regenerate The key is in someone's manuscript. It is not yours to change. Files: `src/keys.py`, `tests/test_key_stability.py` 2. Resolve collisions deterministically Same inputs, same suffix, every time. Files: `src/keys.py` 3. Alias on a deliberate change The old key must still resolve, because documents still use it. Files: `src/aliases.py` ### Done when - [ ] Metadata correction does not change the key - [ ] Collisions resolve deterministically - [ ] A changed key keeps the old one as an alias - [ ] Exports use the stable key ### Watch out - Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. ## M3 · Two machines, one library Changes made on either machine arrive on the other without loss. ### Steps 1. Journal every operation The change log is what makes sync proportional and deletions possible. Files: `src/journal.py` 2. Exchange the tail since last position Not a comparison. Comparison does not scale and cannot see deletions. Files: `src/sync.py` 3. Resume an interrupted sync Large libraries and unreliable connections. It will be interrupted. Files: `src/sync.py`, `tests/test_resume.py` ### Done when - [ ] Sync cost is proportional to changes - [ ] Deletions propagate rather than resurrecting - [ ] An interrupted sync resumes cleanly - [ ] Existing blobs are not retransferred ### Watch out - Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. ## M4 · Annotations from both machines Highlighting on a laptop and a desktop does not lose either set. ### Steps 1. Keep annotations out of the document The file is content addressed. Writing into it breaks deduplication too. Files: `src/annotations.py` 2. Merge per annotation The document is not the conflict unit. The highlight is. Files: `src/merge.py`, `tests/test_annotation_merge.py` 3. Keep both sides on a true conflict Choosing for the user loses an afternoon of reading. Files: `src/merge.py` ### Done when - [ ] Annotations from two machines both survive - [ ] The document file is unchanged by annotating - [ ] A true conflict keeps both versions - [ ] Annotation deletions propagate ### Watch out - Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. ## M5 · Storage that does not grow forever (production only) You control what is stored where, and you can leave with everything. ### Steps 1. Report storage by item and blob Deduplication means item size and library size are different questions. Files: `src/usage.py` 2. Evict documents, never annotations Annotations are tiny and needed offline. Documents are large and refetchable. Files: `src/evict.py` 3. Export everything in a readable form A research library is a career's work. It must be able to leave. Files: `src/export.py` ### Done when - [ ] Storage is reported per item and blob - [ ] Evicted attachments fetch on demand - [ ] Annotations stay readable offline - [ ] Export contains metadata, files, annotations and keys ### Watch out - Evicting a locally cached attachment while an annotation references a page of it leaves an annotation pointing at nothing when the machine is offline, which is exactly when someone is working on a train. Keep annotations local always, since they are tiny, and evict only the documents. ===== OPERATIONS.md ===== # Operations · Zotero Storage ## Backup A research library is often a career's work. The metadata database is small and irreplaceable, so back it up frequently. The blob store should have versioning enabled, since content addressing means an accidental deletion is not recoverable from another copy. ## Restore Restore the metadata and confirm citation keys and annotations survived, then sync. A restore that regenerates keys has broken every manuscript citing this library, which is worse than losing the files. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Sync duration relative to change count is the health metric. Duration growing with library size rather than with changes means the journal is not being used properly. ## Incident checklist A lost annotation is the failure that matters, because it represents reading time that cannot be recovered. Annotations merge per record and are never evicted, and both are controls worth checking after any sync change. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Metadata correction is confirmed not to change a citation key - [ ] Sync cost is confirmed proportional to changes, not library size - [ ] Annotations from two machines are confirmed both to survive - [ ] The document file is confirmed unchanged by annotating - [ ] Annotations are confirmed readable offline after eviction ## Launch constraint Do not market omitted Zotero Storage capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered. ===== .env.example ===== # Copy to .env and fill in. Never commit .env; this file documents it. # Required. Metadata, annotations and the change journal. The irreplaceable part. LIBRARY_PATH=~/.local/share/library/library.db # Required. Content-addressed attachments. Versioning on, because a research library is a career's work. BLOB_STORE=s3://my-library/blobs # Required · secret. Scoped to the bucket only. It holds everything you have read. STORAGE_CREDENTIALS=a scoped key # Required. Identifies this machine in the journal. Two machines sharing one breaks merging. DEVICE_ID=laptop # Required. How much of the library is kept locally. Documents evict, annotations never do. LOCAL_CACHE_GB=20 # Required. Present as a constant so the answer is visible in configuration. There is no other value. REGENERATE_CITATION_KEYS=never
README.md
# Zotero Storage · indie build File sync for a reference library, where the difficulty is that the same paper arrives from three places under three filenames, two machines edit annotations on the same document, and a citation key must stay stable because it is written into someone's manuscript. Estimated effort: **multi-day**. Work `BUILD_PLAN.md` top to bottom · every phase ends in a check that has to pass before the next one starts. ## Stack | Part | Choice | Why | | --- | --- | --- | | Storage | Object storage you control, content addressed | the same paper downloaded twice should cost one copy | | Sync | Change journal per library with per-item resolution | whole-library comparison does not scale past a few thousand items | | Identity | Content hashing plus document identifiers for deduplication | filenames are worthless for identifying a paper | | Annotations | Stored separately from the document and merged per annotation | two machines annotating one paper is the normal case | | Runtime | Python 3.12 with SQLite locally | the library lives on the machine and syncs through storage | ## Before you start Have every one of these ready. The plan assumes them from step one. - [ ] **Python 3.12 or newer** · free - Why: Runs the local library and the sync client. - Get it: python.org or your package manager, then a virtual environment. - Verify: python3 --version prints 3.12 or higher - [ ] **Object storage you control** · pennies per month for a personal library - Why: The whole point is that a research library sits on storage you own rather than a service that can change its terms. - Get it: Any provider with a compatible interface, plus a lifecycle policy and versioning decided before you upload anything. - Verify: A file uploads, downloads and is listed by content hash - [ ] **A real library of at least two thousand items** · free - Why: Sync proportionality, deduplication and storage reporting all behave differently at scale, and a hundred items proves nothing. - Get it: Export from your existing reference manager including the attachments, or assemble one from open access papers. - Verify: Two thousand items with attachments exist, including some genuine duplicates - [ ] **Two machines, or two library copies** · free - Why: Every claim about sync and annotation merging is untestable with one, and the interesting cases require both being offline at once. - Get it: A second machine, or two separate library directories with independent device identifiers. - Verify: Both can sync against the same storage and be taken offline independently - [ ] **Two copies of one paper that differ byte for byte** · free - Why: Deduplication by hashing catches identical files, and the real case is a publisher copy and a repository copy of the same paper with different stamping. - Get it: Download one paper from two sources, or add a watermark to a copy. - Verify: Two files with different hashes are identified as the same paper ## Quick start Follow Phase 1 in `BUILD_PLAN.md`; it creates the project. Then copy `.env.example` to `.env` and fill in the values it documents. ## Honest limits This build deliberately does not replace: - The reference manager and its word processor integration. - Group libraries with shared editing and permissions. - Full text search inside attachments. - Metadata retrieval from publishers. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - licensed scholarly metadata If one of those is essential to you, that is the reason to keep paying for Zotero Storage, and the README should say so rather than pretend.
BRIEF.md
# Build brief · Zotero Storage The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me attachment sync for a research library, across machines, on storage you control to replace Zotero Storage. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Storage: Object storage you control, content addressed · the same paper downloaded twice should cost one copy - Sync: Change journal per library with per-item resolution · whole-library comparison does not scale past a few thousand items - Identity: Content hashing plus document identifiers for deduplication · filenames are worthless for identifying a paper - Annotations: Stored separately from the document and merged per annotation · two machines annotating one paper is the normal case - Runtime: Python 3.12 with SQLite locally · the library lives on the machine and syncs through storage ### Data model (create this before Phase 1) - items(id, citation_key, item_type, title, authors_json, year, doi, created_at, modified_at) - attachments(id, item_id, content_hash, bytes, mime, filename, added_at) · content hash is the identity - blobs(content_hash, storage_path, uploaded_at, refcount) · one copy per distinct file, however many items reference it - annotations(id, attachment_id, page, rect_json, kind, text, colour, device_id, created_at, modified_at, deleted) - journal(id, library_id, entity_kind, entity_id, operation, at, device_id) · the change log sync reads - sync_state(device_id, last_journal_id, last_sync_at) - A citation key is assigned once and never regenerated, because it is written into manuscripts that are not in your control. ### Phase 1 · Knowing when two files are the same paper Build: Content hash every attachment and store one blob per distinct hash. Extract document identifiers where present and use them to link items that differ byte for byte. Report likely duplicates rather than merging automatically. Trap: The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. Done when: The same file added twice stores one blob and two references. Two different files of the same paper are proposed as duplicates via their identifiers, tested with a stamped and an unstamped copy. Merging is proposed, never automatic. Reference counting means deleting one item does not remove a blob another still uses. Do not build yet: No sync, annotations or citation keys. ### Phase 2 · Citation keys that never move Build: Assign a citation key once at item creation and never regenerate it. Handle collisions deterministically. Keep the key when metadata is corrected, and record any key change as an explicit act with the old key retained as an alias. Trap: Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. Done when: Correcting an item's metadata does not change its citation key, tested by editing the year. A collision resolves deterministically and reproducibly. A deliberate key change retains the old key as a resolvable alias. Exported bibliography files use the stable key. Do not build yet: No sync or annotations yet. ### Phase 3 · Two machines, one library Build: A per-library change journal recording every operation. Sync by exchanging journal entries since the last position rather than comparing libraries. Upload and download blobs by content hash. Handle interruption mid-sync. Trap: Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. Done when: Sync cost is proportional to changes rather than library size, verified against a library of five thousand items with three changes. A deletion propagates rather than being reinserted by the other machine. An interrupted sync resumes without duplicating or losing entries. Blob transfer skips content already present. Do not build yet: No annotation merging yet. ### Phase 4 · Annotations from both machines Build: Store annotations separately from the document, keyed to position rather than embedded. Merge per annotation rather than per document. Handle the same annotation edited on both machines by keeping both rather than choosing. Trap: Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. Done when: Annotations made on two machines both survive a sync, tested by annotating the same paper on both while offline. The document file is unchanged by annotating, verified by checksum. The same annotation edited on both machines keeps both versions visibly. Deleting an annotation propagates rather than being reinstated. Do not build yet: No exports or storage management yet. ### Phase 5 · Storage that does not grow forever Build: Report storage use by item and blob. Allow attachments to be kept remotely only, fetched on demand. Export the entire library including files, annotations and keys in a form another tool can read. Trap: Evicting a locally cached attachment while an annotation references a page of it leaves an annotation pointing at nothing when the machine is offline, which is exactly when someone is working on a train. Keep annotations local always, since they are tiny, and evict only the documents. Done when: Storage use is reported per item and per blob. An evicted attachment fetches on demand and its annotations remain readable offline, tested with networking disabled. Export contains metadata, files, annotations and citation keys in a documented format. Deleting an item releases its blob only when no other item references it. Do not build yet: Nothing after this. ### Out of scope (and why) - The reference manager itself, its metadata retrieval and its word processor integration, which is the application this only stores files for. - Group libraries with shared editing and permissions, which is a substantially harder synchronisation and access problem. - Full text search inside attachments, which is a separate indexing project. - Metadata retrieval from publishers, which each expose differently and rate limit. ### README must contain - That attachments are content addressed so the same file stored twice costs one copy - That citation keys are assigned once and never regenerated, because they are in manuscripts - That sync exchanges a journal tail rather than comparing libraries - That annotations live outside the document so merging is per highlight - That eviction removes documents but never annotations, which are needed offline
AGENTS.md
# Agent instructions · Zotero Storage indie build - Read `README.md` and `BUILD_PLAN.md` before writing code. The stack is fixed: Object storage you control, content addressed, Change journal per library with per-item resolution, Content hashing plus document identifiers for deduplication, Stored separately from the document and merged per annotation, Python 3.12 with SQLite locally. Do not substitute. - Work one phase at a time, in order. Do not start a phase until every "Done when" item of the previous one passes. - Prefer the fewest moving parts that satisfy the step. No frameworks, services or dependencies the plan does not name. - Secrets live in `.env`, never in source or logs. Keep `.env.example` current when a variable is introduced. - Do not invent cryptography, security guarantees, APIs or compliance claims. - Add a focused test for every destructive, security-sensitive or data-loss path the plan names. - Run the project checks before declaring a phase complete, and record any deliberate shortcut in the README under "Tradeoffs". ## Known traps - The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. - Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. - Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. - Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight.
BUILD_PLAN.md
# Build plan · Zotero Storage File sync for a reference library, where the difficulty is that the same paper arrives from three places under three filenames, two machines edit annotations on the same document, and a citation key must stay stable because it is written into someone's manuscript. Phases are in dependency order. Each ends in a "Done when" list; treat an unticked item as a blocker, not a note. ## Phase 1 · Knowing when two files are the same paper The same paper from three sources is stored once. ### Steps 1. Content address the blobs One copy per distinct file, reference counted. Files: `src/blobs.py` 2. Extract document identifiers Byte differences from stamping mean hashing alone misses real duplicates. Files: `src/identify.py` 3. Propose duplicates, never merge Two items merged wrongly loses annotations and notes. Files: `src/duplicates.py` ### Done when - [ ] The same file twice stores one blob - [ ] A stamped copy is proposed as a duplicate - [ ] Merging is never automatic - [ ] Reference counting protects shared blobs ### Watch out - The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. ## Phase 2 · Citation keys that never move A key written into a manuscript keeps referring to the same thing forever. ### Steps 1. Assign once, never regenerate The key is in someone's manuscript. It is not yours to change. Files: `src/keys.py`, `tests/test_key_stability.py` 2. Resolve collisions deterministically Same inputs, same suffix, every time. Files: `src/keys.py` 3. Alias on a deliberate change The old key must still resolve, because documents still use it. Files: `src/aliases.py` ### Done when - [ ] Metadata correction does not change the key - [ ] Collisions resolve deterministically - [ ] A changed key keeps the old one as an alias - [ ] Exports use the stable key ### Watch out - Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. ## Phase 3 · Two machines, one library Changes made on either machine arrive on the other without loss. ### Steps 1. Journal every operation The change log is what makes sync proportional and deletions possible. Files: `src/journal.py` 2. Exchange the tail since last position Not a comparison. Comparison does not scale and cannot see deletions. Files: `src/sync.py` 3. Resume an interrupted sync Large libraries and unreliable connections. It will be interrupted. Files: `src/sync.py`, `tests/test_resume.py` ### Done when - [ ] Sync cost is proportional to changes - [ ] Deletions propagate rather than resurrecting - [ ] An interrupted sync resumes cleanly - [ ] Existing blobs are not retransferred ### Watch out - Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. ## Phase 4 · Annotations from both machines Highlighting on a laptop and a desktop does not lose either set. ### Steps 1. Keep annotations out of the document The file is content addressed. Writing into it breaks deduplication too. Files: `src/annotations.py` 2. Merge per annotation The document is not the conflict unit. The highlight is. Files: `src/merge.py`, `tests/test_annotation_merge.py` 3. Keep both sides on a true conflict Choosing for the user loses an afternoon of reading. Files: `src/merge.py` ### Done when - [ ] Annotations from two machines both survive - [ ] The document file is unchanged by annotating - [ ] A true conflict keeps both versions - [ ] Annotation deletions propagate ### Watch out - Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. ## Not in this build - The reference manager and its word processor integration. - Group libraries with shared editing and permissions. - Full text search inside attachments. - Metadata retrieval from publishers. ## After v1, if you want it - Full text indexing of attachments, which is the natural next thing once storage is solid - A read-only share of one collection, which needs a permissions model - Conflict review as a queue, once merging has run long enough to produce some
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Metadata, annotations and the change journal. The irreplaceable part. LIBRARY_PATH=~/.local/share/library/library.db # Required. Content-addressed attachments. Versioning on, because a research library is a career's work. BLOB_STORE=s3://my-library/blobs # Required · secret. Scoped to the bucket only. It holds everything you have read. STORAGE_CREDENTIALS=a scoped key # Required. Identifies this machine in the journal. Two machines sharing one breaks merging. DEVICE_ID=laptop # Required. How much of the library is kept locally. Documents evict, annotations never do. LOCAL_CACHE_GB=20 # Required. Present as a constant so the answer is visible in configuration. There is no other value. REGENERATE_CITATION_KEYS=never
PRODUCT.md
# Zotero Storage · product brief ## Problem The visible reference sync + storage loop is buildable, but a credible replacement needs more than the first screen. Zotero Storage earns its keep through data, import reliability, so expect a weekend or multi-day build and a narrower personal scope. ## Product outcome A research library that syncs across machines without losing an annotation, stores each paper once however many times you added it, and keeps citation keys stable because they are already in your manuscripts. ## Target user A builder who needs a maintainable product foundation, not a one-off demo. ## Required capabilities - Node.js 22 - browser automation for user-authorized imports - optional OpenAI API key stored in .env ## Explicit non-goals for v1 - The reference manager and its word processor integration. - Group libraries with shared editing and permissions. - Full text search inside attachments. - Metadata retrieval from publishers. - publisher-specific import reliability - citation graph scale - team libraries and institutional access - licensed scholarly metadata ## Success criteria - Metadata correction is confirmed not to change a citation key - Sync cost is confirmed proportional to changes, not library size - Annotations from two machines are confirmed both to survive - The document file is confirmed unchanged by annotating - Annotations are confirmed readable offline after eviction
BRIEF.md
# Build brief · Zotero Storage The one-shot brief this plan expands. `BUILD_PLAN.md` (or `MILESTONES.md`) is the same sequence broken into steps and checks; where the two disagree, the plan wins. Build me attachment sync for a research library, across machines, on storage you control to replace Zotero Storage. Build it in phases, in the order below. Do not write the whole thing in one pass. Finish a phase, run its "Done when" check, fix what fails, and only then start the next phase. ### Stack (fixed, do not substitute) - Storage: Object storage you control, content addressed · the same paper downloaded twice should cost one copy - Sync: Change journal per library with per-item resolution · whole-library comparison does not scale past a few thousand items - Identity: Content hashing plus document identifiers for deduplication · filenames are worthless for identifying a paper - Annotations: Stored separately from the document and merged per annotation · two machines annotating one paper is the normal case - Runtime: Python 3.12 with SQLite locally · the library lives on the machine and syncs through storage ### Data model (create this before Phase 1) - items(id, citation_key, item_type, title, authors_json, year, doi, created_at, modified_at) - attachments(id, item_id, content_hash, bytes, mime, filename, added_at) · content hash is the identity - blobs(content_hash, storage_path, uploaded_at, refcount) · one copy per distinct file, however many items reference it - annotations(id, attachment_id, page, rect_json, kind, text, colour, device_id, created_at, modified_at, deleted) - journal(id, library_id, entity_kind, entity_id, operation, at, device_id) · the change log sync reads - sync_state(device_id, last_journal_id, last_sync_at) - A citation key is assigned once and never regenerated, because it is written into manuscripts that are not in your control. ### Phase 1 · Knowing when two files are the same paper Build: Content hash every attachment and store one blob per distinct hash. Extract document identifiers where present and use them to link items that differ byte for byte. Report likely duplicates rather than merging automatically. Trap: The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. Done when: The same file added twice stores one blob and two references. Two different files of the same paper are proposed as duplicates via their identifiers, tested with a stamped and an unstamped copy. Merging is proposed, never automatic. Reference counting means deleting one item does not remove a blob another still uses. Do not build yet: No sync, annotations or citation keys. ### Phase 2 · Citation keys that never move Build: Assign a citation key once at item creation and never regenerate it. Handle collisions deterministically. Keep the key when metadata is corrected, and record any key change as an explicit act with the old key retained as an alias. Trap: Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. Done when: Correcting an item's metadata does not change its citation key, tested by editing the year. A collision resolves deterministically and reproducibly. A deliberate key change retains the old key as a resolvable alias. Exported bibliography files use the stable key. Do not build yet: No sync or annotations yet. ### Phase 3 · Two machines, one library Build: A per-library change journal recording every operation. Sync by exchanging journal entries since the last position rather than comparing libraries. Upload and download blobs by content hash. Handle interruption mid-sync. Trap: Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. Done when: Sync cost is proportional to changes rather than library size, verified against a library of five thousand items with three changes. A deletion propagates rather than being reinserted by the other machine. An interrupted sync resumes without duplicating or losing entries. Blob transfer skips content already present. Do not build yet: No annotation merging yet. ### Phase 4 · Annotations from both machines Build: Store annotations separately from the document, keyed to position rather than embedded. Merge per annotation rather than per document. Handle the same annotation edited on both machines by keeping both rather than choosing. Trap: Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. Done when: Annotations made on two machines both survive a sync, tested by annotating the same paper on both while offline. The document file is unchanged by annotating, verified by checksum. The same annotation edited on both machines keeps both versions visibly. Deleting an annotation propagates rather than being reinstated. Do not build yet: No exports or storage management yet. ### Phase 5 · Storage that does not grow forever Build: Report storage use by item and blob. Allow attachments to be kept remotely only, fetched on demand. Export the entire library including files, annotations and keys in a form another tool can read. Trap: Evicting a locally cached attachment while an annotation references a page of it leaves an annotation pointing at nothing when the machine is offline, which is exactly when someone is working on a train. Keep annotations local always, since they are tiny, and evict only the documents. Done when: Storage use is reported per item and per blob. An evicted attachment fetches on demand and its annotations remain readable offline, tested with networking disabled. Export contains metadata, files, annotations and citation keys in a documented format. Deleting an item releases its blob only when no other item references it. Do not build yet: Nothing after this. ### Out of scope (and why) - The reference manager itself, its metadata retrieval and its word processor integration, which is the application this only stores files for. - Group libraries with shared editing and permissions, which is a substantially harder synchronisation and access problem. - Full text search inside attachments, which is a separate indexing project. - Metadata retrieval from publishers, which each expose differently and rate limit. ### README must contain - That attachments are content addressed so the same file stored twice costs one copy - That citation keys are assigned once and never regenerated, because they are in manuscripts - That sync exchanges a journal tail rather than comparing libraries - That annotations live outside the document so merging is per highlight - That eviction removes documents but never annotations, which are needed offline
ARCHITECTURE.md
# Architecture · Zotero Storage ## Stack | Part | Choice | Why | | --- | --- | --- | | Storage | Object storage you control, content addressed | the same paper downloaded twice should cost one copy | | Sync | Change journal per library with per-item resolution | whole-library comparison does not scale past a few thousand items | | Identity | Content hashing plus document identifiers for deduplication | filenames are worthless for identifying a paper | | Annotations | Stored separately from the document and merged per annotation | two machines annotating one paper is the normal case | | Runtime | Python 3.12 with SQLite locally | the library lives on the machine and syncs through storage | ## Modules Each module has one owner concern and a documented way to replace it. | Module | Owns | How to replace it | | --- | --- | --- | | Blobs | Content-addressed storage with reference counting. | Any store, provided identity is the content rather than the filename. | | Keys | Stable citation keys and aliases. | Not swappable for regeneration from metadata, which breaks existing manuscripts. | | Journal | The change log sync reads. | Any log, provided sync exchanges a tail rather than comparing libraries. | | Annotations | Records outside the document, merged individually. | Any store, provided annotations never live inside the file. | | Eviction | Local cache management. | Any policy that evicts documents and never annotations. | ## Configuration Every runtime setting is an environment variable documented in `.env.example`, validated at startup, with a safe local default wherever one exists. - `LIBRARY_PATH` · required · Metadata, annotations and the change journal. The irreplaceable part. - `BLOB_STORE` · required · Content-addressed attachments. Versioning on, because a research library is a career's work. - `STORAGE_CREDENTIALS` · required, secret · Scoped to the bucket only. It holds everything you have read. - `DEVICE_ID` · required · Identifies this machine in the journal. Two machines sharing one breaks merging. - `LOCAL_CACHE_GB` · required · How much of the library is kept locally. Documents evict, annotations never do. - `REGENERATE_CITATION_KEYS` · required · Present as a constant so the answer is visible in configuration. There is no other value. ## Production baseline - Security: least privilege, input validation at every boundary, secret redaction in logs, rate limits on abuse-prone paths, no invented security primitives. - Data: explicit schema and migrations, transactional writes where integrity matters, backup and restore procedures that have been exercised. - Integrations: adapters around third-party providers, idempotent webhook or job processing, bounded retries, timeouts. - Observability: structured logs with request or operation ids, an error-tracking hook, and health and readiness checks where a server exists. - Quality: unit tests for domain rules, integration tests at module boundaries, one end-to-end test of the critical path. ## Decision records For each dependency in the stack table, keep a short note: why it was chosen, its failure mode, and how it is replaced. Do not add infrastructure until a requirement in `PRODUCT.md` justifies it.
AGENTS.md
# Agent instructions · Zotero Storage product build - Read `PRODUCT.md` and `ARCHITECTURE.md` before changing code. The stack is fixed: Object storage you control, content addressed, Change journal per library with per-item resolution, Content hashing plus document identifiers for deduplication, Stored separately from the document and merged per annotation, Python 3.12 with SQLite locally. - Implement milestone by milestone from `MILESTONES.md`; keep each change reviewable and leave the application runnable at every commit. - Treat authentication, payments, encryption, imports, webhooks and destructive actions as high-risk boundaries when present. - Never invent cryptography or silently weaken a requirement to make a check pass. - Put every external service behind an interface with a deterministic fake for tests. - Add migrations and rollback or recovery notes for every persistent data change. - Log useful operational context without credentials, tokens, passwords or personal data. - Update documentation and run every check before completing a milestone. ## Known traps - The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. - Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. - Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. - Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. - Evicting a locally cached attachment while an annotation references a page of it leaves an annotation pointing at nothing when the machine is offline, which is exactly when someone is working on a train. Keep annotations local always, since they are tiny, and evict only the documents.
MILESTONES.md
# Delivery milestones · Zotero Storage Estimated effort: **multi-day** for the indie phases; the production-only milestones add the trust and operability layer. ## M1 · Knowing when two files are the same paper The same paper from three sources is stored once. ### Steps 1. Content address the blobs One copy per distinct file, reference counted. Files: `src/blobs.py` 2. Extract document identifiers Byte differences from stamping mean hashing alone misses real duplicates. Files: `src/identify.py` 3. Propose duplicates, never merge Two items merged wrongly loses annotations and notes. Files: `src/duplicates.py` ### Done when - [ ] The same file twice stores one blob - [ ] A stamped copy is proposed as a duplicate - [ ] Merging is never automatic - [ ] Reference counting protects shared blobs ### Watch out - The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge. ## M2 · Citation keys that never move A key written into a manuscript keeps referring to the same thing forever. ### Steps 1. Assign once, never regenerate The key is in someone's manuscript. It is not yours to change. Files: `src/keys.py`, `tests/test_key_stability.py` 2. Resolve collisions deterministically Same inputs, same suffix, every time. Files: `src/keys.py` 3. Alias on a deliberate change The old key must still resolve, because documents still use it. Files: `src/aliases.py` ### Done when - [ ] Metadata correction does not change the key - [ ] Collisions resolve deterministically - [ ] A changed key keeps the old one as an alias - [ ] Exports use the stable key ### Watch out - Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed. ## M3 · Two machines, one library Changes made on either machine arrive on the other without loss. ### Steps 1. Journal every operation The change log is what makes sync proportional and deletions possible. Files: `src/journal.py` 2. Exchange the tail since last position Not a comparison. Comparison does not scale and cannot see deletions. Files: `src/sync.py` 3. Resume an interrupted sync Large libraries and unreliable connections. It will be interrupted. Files: `src/sync.py`, `tests/test_resume.py` ### Done when - [ ] Sync cost is proportional to changes - [ ] Deletions propagate rather than resurrecting - [ ] An interrupted sync resumes cleanly - [ ] Existing blobs are not retransferred ### Watch out - Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed. ## M4 · Annotations from both machines Highlighting on a laptop and a desktop does not lose either set. ### Steps 1. Keep annotations out of the document The file is content addressed. Writing into it breaks deduplication too. Files: `src/annotations.py` 2. Merge per annotation The document is not the conflict unit. The highlight is. Files: `src/merge.py`, `tests/test_annotation_merge.py` 3. Keep both sides on a true conflict Choosing for the user loses an afternoon of reading. Files: `src/merge.py` ### Done when - [ ] Annotations from two machines both survive - [ ] The document file is unchanged by annotating - [ ] A true conflict keeps both versions - [ ] Annotation deletions propagate ### Watch out - Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight. ## M5 · Storage that does not grow forever (production only) You control what is stored where, and you can leave with everything. ### Steps 1. Report storage by item and blob Deduplication means item size and library size are different questions. Files: `src/usage.py` 2. Evict documents, never annotations Annotations are tiny and needed offline. Documents are large and refetchable. Files: `src/evict.py` 3. Export everything in a readable form A research library is a career's work. It must be able to leave. Files: `src/export.py` ### Done when - [ ] Storage is reported per item and blob - [ ] Evicted attachments fetch on demand - [ ] Annotations stay readable offline - [ ] Export contains metadata, files, annotations and keys ### Watch out - Evicting a locally cached attachment while an annotation references a page of it leaves an annotation pointing at nothing when the machine is offline, which is exactly when someone is working on a train. Keep annotations local always, since they are tiny, and evict only the documents.
OPERATIONS.md
# Operations · Zotero Storage ## Backup A research library is often a career's work. The metadata database is small and irreplaceable, so back it up frequently. The blob store should have versioning enabled, since content addressing means an accidental deletion is not recoverable from another copy. ## Restore Restore the metadata and confirm citation keys and annotations survived, then sync. A restore that regenerates keys has broken every manuscript citing this library, which is worse than losing the files. Do a restore drill before the first real user, and write the date here when it passes. ## Monitoring Sync duration relative to change count is the health metric. Duration growing with library size rather than with changes means the journal is not being used properly. ## Incident checklist A lost annotation is the failure that matters, because it represents reading time that cannot be recovered. Annotations merge per record and are never evicted, and both are controls worth checking after any sync change. 1. Contain the issue without destroying evidence or user data. 2. Record the timeline and affected scope. 3. Rotate exposed secrets and revoke compromised sessions or credentials. 4. Restore from a verified backup when needed. 5. Document the root cause, the remediation and the regression test. ## Release gate - [ ] Metadata correction is confirmed not to change a citation key - [ ] Sync cost is confirmed proportional to changes, not library size - [ ] Annotations from two machines are confirmed both to survive - [ ] The document file is confirmed unchanged by annotating - [ ] Annotations are confirmed readable offline after eviction ## Launch constraint Do not market omitted Zotero Storage capabilities as implemented. The non-goals in `PRODUCT.md` remain user-visible limitations until they are deliberately delivered.
.env.example
# Copy to .env and fill in. Never commit .env; this file documents it. # Required. Metadata, annotations and the change journal. The irreplaceable part. LIBRARY_PATH=~/.local/share/library/library.db # Required. Content-addressed attachments. Versioning on, because a research library is a career's work. BLOB_STORE=s3://my-library/blobs # Required · secret. Scoped to the bucket only. It holds everything you have read. STORAGE_CREDENTIALS=a scoped key # Required. Identifies this machine in the journal. Two machines sharing one breaks merging. DEVICE_ID=laptop # Required. How much of the library is kept locally. Documents evict, annotations never do. LOCAL_CACHE_GB=20 # Required. Present as a constant so the answer is visible in configuration. There is no other value. REGENERATE_CITATION_KEYS=never
$ choose a build depth, inspect the files, then open the complete pack in your agent
Zotero Storage: Researchers pay for correct metadata, resilient importers, citation coverage, and workflows that survive publisher and browser changes.
xpublisher-specific import reliability
xcitation graph scale
xteam libraries and institutional access
xlicensed scholarly metadata
share on X ↗"I just replaced Zotero Storage with one prompt"
Don't feel like building it? These folks already made it free.
NextcloudUnlimited Zotero WebDAV if you already wanted a server to babysit.open source↗
KoofrTen free gigabytes of WebDAV that Zotero can use without a storage subscription.free↗no votes, no pay-to-list · just what's real
Zotero Storage pricing
| plan | monthly | annual (per mo) | what you get |
|---|---|---|---|
| free | $0 | $0 | 300 MB attachment storage. |
| 2 gb | — | $1.67 | 2 GB attachment storage. |
| 6 gb | — | $5 | 6 GB attachment storage. |
| unlimited | — | $10 | Unlimited attachment storage. |
free tier300 MB attachment storage
billingannual only for paid storage; no month-to-month plan
hidden costsgroup-library files count against the storage quota of the group owner; institutional storage is priced separately
verified 2026-08-13 · source ↗
Vibecode Zotero Storage
Kinda. The core of Zotero Storage is buildable in a weekend with the prompt on this page, but there are real gaps: publisher-specific import reliability, citation graph scale. Read the honest list above before committing.
How much does Zotero Storage cost?
Zotero Storage's pricing is usage-based or varies by plan · Zotero itself is free. This row compares paid file storage; check the logged-in storage page before publishing..
What do I lose by replacing Zotero Storage?
Honestly: publisher-specific import reliability; citation graph scale; team libraries and institutional access; licensed scholarly metadata. If any of those are load-bearing for you, keep paying.
Is there an open-source alternative to Zotero Storage?
Yes: Nextcloud (Unlimited Zotero WebDAV if you already wanted a server to babysit.) Koofr (Ten free gigabytes of WebDAV that Zotero can use without a storage subscription.) The prompt is for when you want it exactly your way.