Build Zotero Storage

0%0 of 17 items done

Saved on this device only. Tick prerequisites first, then work the phases in order · do not start one until the checks above it pass.

File sync for a reference library, where the difficulty is that the same paper arrives from three places under three filenames, two machines edit annotations on the same document, and a citation key must stay stable because it is written into someone's manuscript.

estimated effort multi-daythe files for this build are in the project pack

StorageObject storage you control, content addressedSyncChange journal per library with per-item resolutionIdentityContent hashing plus document identifiers for deduplicationAnnotationsStored separately from the document and merged per annotationRuntimePython 3.12 with SQLite locally

Before step 1

Everything below is assumed from the first step. Tick each one when you actually have it, not when you plan to.

installfree
Why Runs the local library and the sync client.
Get it python.org or your package manager, then a virtual environment.
Verify python3 --version prints 3.12 or higher
accountpennies per month for a personal library
Why The whole point is that a research library sits on storage you own rather than a service that can change its terms.
Get it Any provider with a compatible interface, plus a lifecycle policy and versioning decided before you upload anything.
Verify A file uploads, downloads and is listed by content hash
have readyfree
Why Sync proportionality, deduplication and storage reporting all behave differently at scale, and a hundred items proves nothing.
Get it Export from your existing reference manager including the attachments, or assemble one from open access papers.
Verify Two thousand items with attachments exist, including some genuine duplicates
have readyfree
Why Every claim about sync and annotation merging is untestable with one, and the interesting cases require both being offline at once.
Get it A second machine, or two separate library directories with independent device identifiers.
Verify Both can sync against the same storage and be taken offline independently
have readyfree
Why Deduplication by hashing catches identical files, and the real case is a publisher copy and a repository copy of the same paper with different stamping.
Get it Download one paper from two sources, or add a watermark to a copy.
Verify Two files with different hashes are identified as the same paper

Data model

Create these before the first phase that stores anything. Changing a table later is the expensive kind of change.

- items(id, citation_key, item_type, title, authors_json, year, doi, created_at, modified_at)
- attachments(id, item_id, content_hash, bytes, mime, filename, added_at) · content hash is the identity
- blobs(content_hash, storage_path, uploaded_at, refcount) · one copy per distinct file, however many items reference it
- annotations(id, attachment_id, page, rect_json, kind, text, colour, device_id, created_at, modified_at, deleted)
- journal(id, library_id, entity_kind, entity_id, operation, at, device_id) · the change log sync reads
- sync_state(device_id, last_journal_id, last_sync_at)
- A citation key is assigned once and never regenerated, because it is written into manuscripts that are not in your control.

Environment variables

These go in a .env file the app reads at startup. The pack's .env.example is this table as a file · copy it, never commit the filled-in version.

Variable	Needed	Example	Where the value comes from
`LIBRARY_PATH`	required	`~/.local/share/library/library.db`	Metadata, annotations and the change journal. The irreplaceable part.
`BLOB_STORE`	required	`s3://my-library/blobs`	Content-addressed attachments. Versioning on, because a research library is a career's work.
`STORAGE_CREDENTIALS`secret	required	`a scoped key`	Scoped to the bucket only. It holds everything you have read.
`DEVICE_ID`	required	`laptop`	Identifies this machine in the journal. Two machines sharing one breaks merging.
`LOCAL_CACHE_GB`	required	`20`	How much of the library is kept locally. Documents evict, annotations never do.
`REGENERATE_CITATION_KEYS`	required	`never`	Present as a constant so the answer is visible in configuration. There is no other value.

The build, in order

1
Knowing when two files are the same paper
The same paper from three sources is stored once.
1. One copy per distinct file, reference counted.
  Files src/blobs.py
2. Byte differences from stamping mean hashing alone misses real duplicates.
  Files src/identify.py
3. Two items merged wrongly loses annotations and notes.
  Files src/duplicates.py
done when · tick each as it passes
watch out
- The same paper downloaded from a publisher and from a repository differs byte for byte because of watermarks, stamps and metadata, so content hashing alone will not catch it, and filename matching catches almost nothing. Hash for exact duplicates, extract identifiers for the rest, and propose rather than merge.
next phase
2
Citation keys that never move
A key written into a manuscript keeps referring to the same thing forever.
1. The key is in someone's manuscript. It is not yours to change.
  Files src/keys.pytests/test_key_stability.py
2. Same inputs, same suffix, every time.
  Files src/keys.py
3. The old key must still resolve, because documents still use it.
  Files src/aliases.py
done when · tick each as it passes
watch out
- Regenerating citation keys from metadata means correcting an author's name or a publication year silently changes the key, and every manuscript already citing it now has a broken reference that the author will discover during submission. Assign once, keep forever, and alias if a change is genuinely needed.
next phase
3
Two machines, one library
Changes made on either machine arrive on the other without loss.
1. The change log is what makes sync proportional and deletions possible.
  Files src/journal.py
2. Not a comparison. Comparison does not scale and cannot see deletions.
  Files src/sync.py
3. Large libraries and unreliable connections. It will be interrupted.
  Files src/sync.pytests/test_resume.py
done when · tick each as it passes
watch out
- Comparing whole libraries to find differences takes longer than the sync is worth once there are a few thousand items, and it also cannot tell a deletion from an item that never arrived. Journal the operations and exchange the tail, so sync cost is proportional to what changed.
next phase
4
Annotations from both machines
Highlighting on a laptop and a desktop does not lose either set.
1. The file is content addressed. Writing into it breaks deduplication too.
  Files src/annotations.py
2. The document is not the conflict unit. The highlight is.
  Files src/merge.pytests/test_annotation_merge.py
3. Choosing for the user loses an afternoon of reading.
  Files src/merge.py
done when · tick each as it passes
watch out
- Writing annotations into the document file makes the file itself the conflict unit, so two machines annotating one paper produce two divergent files and one of them wins, taking an afternoon's reading with it. Keep annotations as separate records so the merge is per highlight.
5
Storage that does not grow foreverproduct builder
You control what is stored where, and you can leave with everything.
1. Deduplication means item size and library size are different questions.
  Files src/usage.py
2. Annotations are tiny and needed offline. Documents are large and refetchable.
  Files src/evict.py
3. A research library is a career's work. It must be able to leave.
  Files src/export.py
done when · tick each as it passes
watch out
- Evicting a locally cached attachment while an annotation references a page of it leaves an annotation pointing at nothing when the machine is offline, which is exactly when someone is working on a train. Keep annotations local always, since they are tiny, and evict only the documents.

what this build does not replace

The reference manager and its word processor integration.
Group libraries with shared editing and permissions.
Full text search inside attachments.
Metadata retrieval from publishers.
publisher-specific import reliability
citation graph scale
team libraries and institutional access
licensed scholarly metadata

after v1, if you want it

Full text indexing of attachments, which is the natural next thing once storage is solid
A read-only share of one collection, which needs a permissions model
Conflict review as a queue, once merging has run long enough to produce some

Need the files? The project pack on the verdict page hands your agent the whole brief · more read it later.