mirror of
https://github.com/spacedriveapp/spacedrive.git
synced 2025-12-11 20:15:30 +01:00
110 lines
4.8 KiB
Plaintext
110 lines
4.8 KiB
Plaintext
---
|
|
title: Virtual Sidecar System (VSS)
|
|
sidebarTitle: Virtual Sidecars
|
|
---
|
|
|
|
The Virtual Sidecar System (VSS) manages derivative data associated with your files - thumbnails, OCR text, video transcripts, embeddings, and other metadata-rich artifacts.
|
|
|
|
## Key Principles
|
|
|
|
**Content-Scoped**: Sidecars are linked to file content, not paths. Multiple copies of the same file share one set of sidecars.
|
|
|
|
**Non-Destructive**: Spacedrive never modifies your original files during indexing.
|
|
|
|
**Portable**: All managed sidecars live in `.sdlibrary/sidecars`, making your entire setup portable.
|
|
|
|
## Sidecar Types
|
|
|
|
The VSS handles two types of sidecars:
|
|
|
|
**Managed Sidecars** - Spacedrive-generated derivatives stored in `.sdlibrary/sidecars/`:
|
|
- Thumbnails (`.webp`)
|
|
- Video proxies (`.mp4`)
|
|
- OCR text (`.json`)
|
|
- Subtitles and lyrics (`.srt`)
|
|
|
|
**Reference Sidecars** - Pre-existing files treated as sidecars:
|
|
- Live Photo video components
|
|
- RAW + JPEG pairs
|
|
- Tracked by `source_entry_id` in database
|
|
- Files stay in original location until user converts them to managed sidecars
|
|
|
|
### Filesystem Layout
|
|
|
|
All sidecars are stored in a `sidecars` directory within the `.sdlibrary`. The path to a sidecar is deterministic and is derived from the content UUID of the file it is associated with, the kind of sidecar, and a variant.
|
|
|
|
A typical path looks like this:
|
|
|
|
```
|
|
.sdlibrary/
|
|
sidecars/
|
|
content/
|
|
{h0}/{h1}/{content_uuid}/
|
|
thumbs/{variant}.webp
|
|
proxies/{profile}.mp4
|
|
transcript/{variant}.srt
|
|
ocr/default.json
|
|
```
|
|
|
|
- `{h0}/{h1}`: These are the first two byte-pairs of the content UUID, used for sharding the directory structure to ensure good filesystem performance at scale.
|
|
- `{content_uuid}`: The unique identifier of the content the sidecar is associated with.
|
|
- `thumbs/{variant}.webp`: An example of a thumbnail sidecar.
|
|
|
|
### Database Schema
|
|
|
|
The VSS uses two tables in the Spacedrive database:
|
|
|
|
- `sidecars`: This table tracks all the sidecars that have been generated or are pending generation. It stores information like the content UUID, kind, variant, format, path, size, and status.
|
|
- `sidecar_availability`: This table tracks which devices in your library have a copy of a particular sidecar. This is used to avoid regenerating sidecars that already exist on another device.
|
|
|
|
### Sidecar Lifecycle
|
|
|
|
**1. Identification**
|
|
- **Managed**: Indexer queues generation job, creates "pending" record in `sidecars` table
|
|
- **Reference**: Indexer creates `sidecar` record with `source_entry_id` pointing to original file
|
|
|
|
**2. Generation**
|
|
- **Managed**: Background job generates derivative, stores in deterministic path
|
|
- **Reference**: No generation - already exists
|
|
|
|
**3. Recording**
|
|
- **Managed**: Job marks sidecar "ready", records size and metadata
|
|
- **Reference**: Created as "ready" immediately
|
|
- Both update `sidecar_availability` table for current device
|
|
|
|
**4. Syncing**
|
|
- Devices exchange availability information
|
|
- Missing sidecars can be transferred from peers instead of regenerating
|
|
|
|
## Current Implementation Status
|
|
|
|
The VSS is partially implemented. Here is a summary of the current status:
|
|
|
|
- **Implemented:**
|
|
- The core `SidecarManager` service.
|
|
- Filesystem layout and path generation.
|
|
- Database schema and queries for `sidecars` and `sidecar_availability`.
|
|
- Enqueueing of sidecar generation jobs.
|
|
- A `bootstrap_scan` function to synchronize the database with the filesystem.
|
|
- Reference sidecars, which allow tracking files as sidecars without moving them.
|
|
- **To-Do:**
|
|
- **Job Dispatch and Execution:** The system for actually executing the generation jobs is not yet implemented. This is the most critical missing piece.
|
|
- **Filesystem Watcher:** A watcher for the `sidecars` directory is needed for real-time updates.
|
|
- **Checksumming:** Checksumming of sidecar files for integrity verification is not yet implemented.
|
|
|
|
## How to Use the VSS (API)
|
|
|
|
The `SidecarManager` provides a set of APIs for interacting with the VSS. These are primarily for internal use by other Spacedrive services, but they can also be used for development and debugging.
|
|
|
|
- `sidecars.presence(content_uuids, kind, variants)`: Checks for the presence of sidecars for a given set of content UUIDs.
|
|
- `sidecars.path(content_uuid, kind, variant)`: Gets the local path to a sidecar, or enqueues it for generation if it doesn't exist.
|
|
- `sidecars.reconcile()`: Triggers a bootstrap scan to reconcile the database with the filesystem.
|
|
|
|
## Benefits
|
|
|
|
- **Non-Destructive**: Reference sidecars preserve original file locations
|
|
- **Portable**: All managed sidecars in `.sdlibrary` can be moved/backed up as one unit
|
|
- **Efficient**: One sidecar per content hash, shared across devices
|
|
- **Extensible**: New sidecar kinds can be added without schema changes
|
|
- **AI-Ready**: Provides structured data for semantic search and intelligent organization
|