- Enhanced the LibraryConfig and LibrarySettings structs with additional indexer settings for better file filtering during indexing.
- Implemented a new rules system for the indexer, allowing for customizable file handling based on user-defined criteria.
- Updated the discovery phase to leverage the new rules engine, improving indexing efficiency.
These changes aim to modernize the indexing functionality and enhance project organization.
- Removed the outdated combine_tasks.sh script to streamline the project structure.
- Enhanced the LibraryConfig and LibrarySettings structs with new indexer settings for improved file filtering during indexing.
- Introduced a new rules system for the indexer, allowing for more flexible and customizable file handling based on user-defined rules.
- Updated the discovery phase to utilize the new rules engine for filtering entries, improving the efficiency of the indexing process.
These changes aim to modernize the indexing functionality and improve the overall organization of the project.
- Removed outdated scripts for combining documentation and Rust files, as well as the combined documentation file, to streamline the project structure.
- Introduced a new generic script for combining files from various directories, enhancing flexibility and usability.
- Updated task tracking documentation to include new filtering and sorting options for better task management.
These changes aim to simplify the documentation process and improve the overall organization of task management within the project.
- Deleted several markdown files related to the synchronization epic, including task descriptions for the sync relationship schema, sync engine service, and VDFS diffing logic.
- This cleanup reflects a shift in focus away from the previous synchronization design, streamlining the project structure and preparing for new implementations aligned with the updated architecture.
- Introduced new markdown files for the AI epic (AI-000) and its associated AI agent task (AI-001), outlining features for an intelligent data management system.
- Added resource management epic (RES-000) and its first task on adaptive throttling (RES-001) to ensure efficient resource usage on mobile devices.
- Created security tasks for implementing SQLCipher encryption (SEC-002) and a cryptographic audit log (SEC-003), enhancing data security measures.
- Established synchronization tasks, including the sync relationship schema (SYNC-001) and the sync engine service (SYNC-002), to support VDFS-powered synchronization.
These additions lay the groundwork for critical features in AI, resource management, security, and synchronization within the project.
- Changed implementation notes in various task markdown files to future tense, indicating planned features and implementations.
- Updated `Cargo.toml` in the task-validator to include new dependencies for enhanced functionality.
- Refactored the `main.rs` file in the task-validator to introduce a command-line interface for listing and validating tasks, improving usability and organization.
These changes aim to clarify the status of tasks and enhance the task validation process within the project.
- Introduced a new `task-validator` package to validate task files against a defined schema.
- Added a validation script that checks for proper YAML front matter in task markdown files, ensuring data integrity before commits.
- Updated `Cargo.toml` to include the new workspace member `task-validator`.
- Created a `crush.json` configuration for service providers and added a combined Rust files document for better organization.
These changes enhance the development workflow by enforcing task file validation and improving project structure.
Replace polling-based job monitoring with event-driven approach using EventBus.
The tests now subscribe to JobProgress and JobCompleted events instead of
repeatedly querying job status, making them more robust and idiomatic.
- Updated the `handle_command` method to retrieve remote device addresses from the active connection map, improving accuracy in device connection updates.
- Implemented logging for cases where active connections are not found, aiding in troubleshooting.
- Enhanced the `get_node_addr` method to wait for address discovery if none are available, improving the reliability of device communication.
- Added comprehensive logging for address discovery attempts, providing better visibility into the networking process.
These changes aim to improve the robustness and clarity of device connection management within the networking service.
- Introduced a data directory configuration for persistent pairing sessions, enhancing device communication reliability.
- Improved error messages for pairing code validation, providing clearer guidance for users.
- Enhanced the handling of device connections by including address information, improving the accuracy of device state updates.
- Added robust logging for connection and pairing processes, aiding in troubleshooting and monitoring.
These changes aim to streamline the networking setup and enhance the reliability of the pairing protocol within the system.
- Introduces a new, dated sync design document (`SYNC_DESIGN_2025_08_19.md`) to reflect a more robust and refined architecture.
- The new design incorporates several key improvements based on a detailed analysis of the V2 codebase and data models.
Key changes in the new design include:
- A clear architectural distinction between finite "Jobs" (e.g., Backfill) and long-running "Services" (e.g., LiveSync).
- Explicit clarification that derived data, such as closure tables, will not be synced and must be rebuilt locally on each device.
- A new requirement for log redaction and compaction on the leader to ensure data privacy and efficient storage.
- Also updates the original `SYNC_DESIGN.md` to include the clarification about not syncing closure tables.
- Removed the `start_networking` method call from the daemon initialization process, as it was redundant after `init_networking`.
- Commented out the volume detection initialization code for future review, improving code clarity.
- Enhanced the pairing protocol by establishing a persistent connection immediately after successful pairing, ensuring better device communication.
- Updated the handling of pairing messages to maintain a continuous stream for message exchanges, improving responsiveness.
- Improved logging throughout the networking and pairing processes for better visibility into operations and error handling.
These changes aim to streamline the networking setup and enhance the reliability of the pairing protocol within the system.
- Refactored the `test_database_creation_and_migration` function in `database_migration_test.rs` for improved readability and structure.
- Updated the whitepaper to replace the outdated indexing duration summary with a new performance benchmarks description, including testing details on hardware specifications.
- Removed deprecated CSV inputs from the LaTeX document and adjusted related figures to enhance clarity in performance reporting.
These changes aim to improve the maintainability of the test code and enhance the accuracy of performance insights in the whitepaper.
- Introduced a `data_dir` field in `DeviceManager` to allow specification of a custom data directory for device configurations and keys.
- Updated `DeviceKeyManager` to support fallback file paths for master keys, ensuring consistent key management across different data directories.
- Enhanced the `DevicePersistence` and `DeviceRegistry` implementations to utilize the new data directory structure, improving the reliability of device connection persistence.
- Added comprehensive logging and retry mechanisms in the networking service to improve connection stability and visibility during device interactions.
These changes aim to enhance the flexibility and robustness of device management and key handling within the system.
- Added functionality to immediately mark the initiator device as connected upon successful pairing completion, ensuring accurate device status even if the completion message fails.
- Implemented logging for both successful and failed attempts to mark the device as connected, improving visibility into the pairing process.
- Updated file transfer tests to display full checksums instead of truncated versions, enhancing clarity in verification outputs.
These changes aim to improve the reliability of device connection status during pairing and enhance the clarity of file transfer verification.
- Introduced a new `LocationAdded` variant in the `ActionOutput` enum to encapsulate the location ID and optional job ID upon successful location addition.
- Updated the `handle_location_command` function to handle the new response format from the daemon, including improved error handling for unexpected responses.
- Modified the `LocationHandler` to extract and return the location ID and job ID from the action output, ensuring proper response structure.
- Refactored the `ActionHandler` for `LocationAddHandler` to utilize the new `LocationAdded` output format, enhancing clarity in action results.
These changes aim to improve the handling of location addition actions and provide clearer feedback in the system's output.
- Introduced a new `build.rs` file to automate the generation of build metadata, including Git SHA, commit timestamp, and branch information.
- Updated `Cargo.toml` to include new dependencies such as `comfy-table`, `dialoguer`, `indicatif`, `owo-colors`, `supports-color`, `console`, and `colored` for improved CLI output and user interaction.
- Added a new shell script `update_spacedrive.sh` for streamlined updates of the Spacedrive CLI, including Git pull and project build processes.
- Refactored benchmark scenarios by renaming and consolidating related functionality, enhancing clarity and maintainability.
These changes aim to improve the build process and enhance the user experience within the Spacedrive system.
- Introduced new JSON files for shape_large, shape_medium, and shape_small content identification benchmarks, detailing performance metrics and hardware specifications.
- Updated the whitepaper metrics CSV to include the new benchmark results, enhancing the reporting structure for better analysis.
- Enhanced the LaTeX document to incorporate new tables and figures summarizing indexing durations and performance comparisons across different hardware setups.
These changes aim to improve the clarity and comprehensiveness of performance insights within the Spacedrive system.
- Updated the CSV structure for whitepaper metrics to include a new 'Duration_s' column, providing insights into the time taken for each phase of the benchmarking process.
- Modified the CSV reporting logic in the Rust code to calculate and include phase-specific durations, improving the clarity and usability of performance data.
- Adjusted the LaTeX document to reflect the updated CSV format, ensuring accurate representation of benchmark results in the whitepaper.
These changes aim to improve the analysis of performance metrics within the Spacedrive system.
- Deleted obsolete Rust test files related to file copy and device persistence actions to streamline the codebase.
- Introduced new benchmark result JSON files for various scenarios, including shape_large and shape_medium, enhancing performance metrics and hardware details.
- Updated the CSV reporting structure to reflect the latest benchmark results, improving clarity and usability for analysis.
These changes aim to optimize the codebase and provide more accurate performance insights within the Spacedrive system.
- Revised benchmark result JSON files for shape_small and shape_medium scenarios, significantly improving performance metrics and updating hardware details.
- Enhanced the CSV file for whitepaper metrics to include additional metrics for better analysis and reporting.
- Refactored CLI commands to support multiple scenario executions and improved hardware detection, increasing usability and flexibility in benchmarking tasks.
These changes aim to provide more accurate performance insights and streamline the benchmarking process within the Spacedrive system.
- Updated benchmark result JSON files for shape_small and shape_medium scenarios, improving performance metrics with new hardware details and refined durations.
- Introduced a new CSV file for whitepaper metrics, consolidating benchmark results for better reporting and analysis.
- Refactored CLI commands to support multiple scenario executions and improved hardware detection, enhancing usability and flexibility in benchmarking tasks.
These changes aim to provide more accurate performance insights and streamline the benchmarking process within the Spacedrive system.
- Added a new dependency on `sysinfo` for improved hardware label detection in benchmark results.
- Updated benchmark result JSON files to include a more structured format with metadata and durations for better clarity.
- Refactored the CLI commands to utilize the new `BenchmarkRun` structure, enhancing the output of benchmark summaries.
- Removed outdated benchmark result files to streamline the reporting process.
These changes aim to improve the accuracy and usability of benchmark reporting within the Spacedrive system.
- Improved sidecar management by adding new database entities and migration scripts to support enhanced functionality.
- Updated benchmark YAML files to include additional scenarios, expanding performance testing coverage.
- Enhanced CLI commands with a new `recipe_filter` option for better control over recipe file selection.
These changes aim to strengthen the sidecar management system and improve the benchmarking framework's capabilities within the Spacedrive system.
- Updated `Cargo.toml` to modify the `sqlx` dependency, removing the `sqlite-fts5` feature for improved compatibility.
- Introduced new benchmark YAML files for `shape_large`, `shape_medium`, and `shape_small` scenarios, expanding performance testing coverage.
- Added a new `test_daemon.sh` script for testing the daemon's functionality, including logging and error handling.
- Implemented a sidecar management system with new database entities and migration scripts to support sidecar functionality.
- Enhanced the CLI commands to include a `recipe_filter` option for better control over recipe file selection.
These changes aim to improve the benchmarking framework's capabilities, enhance sidecar management, and streamline the testing process within the Spacedrive system.
- Revised the `rel_path` description to clarify the inclusion of sharding prefixes in the path structure.
- Updated directory structure examples to reflect the new two-level hex sharding under `content/`, improving organization and scalability.
- Enhanced the bootstrap scan description to specify the handling of the sharded directory tree, ensuring consistency between the database and filesystem.
These changes aim to improve clarity and understanding of the sidecar system's structure and its operational efficiency at scale.
- Modified the `sqlx` dependency in `Cargo.toml` to include the `sqlite-fts5` feature, enabling full-text search capabilities.
- Added an implementation note in the `SEARCH_DESIGN.md` documentation to clarify the execution of raw SQL statements for FTS5 using SeaORM's APIs, emphasizing the need for the `rusqlite` driver to be compiled with FTS5 support.
These changes aim to improve the functionality and documentation surrounding full-text search implementation within the system.
- Revised the `SearchMode` enum to include `Normal` and `Full`, replacing the previous modes for improved clarity.
- Enhanced the search workflow documentation, detailing the steps for indexing, sidecar generation, and progressive search execution.
- Clarified the asynchronous nature of job processing and the UI update mechanism during search execution.
These changes aim to streamline the search process and improve user understanding of the search functionality within the system.
- Updated the `SearchMode` enum to simplify search modes, replacing the previous definitions with three primary modes: `Fast`, `Normal`, and `Full`.
- Enhanced documentation to clarify the purpose and functionality of each search mode, improving understanding of the search optimization process.
These changes aim to streamline the search mode implementation and enhance the overall search experience within the system.
- Introduced a new optional argument `dataset_root` to the `Mkdata`, `MkdataAll`, `Run`, and `RunAll` commands, allowing users to prefix relative recipe locations with a specified path.
- Updated the `mkdata`, `mkdata_all`, `run_scenario`, and `run_all` functions to accommodate the new `dataset_root` parameter, improving flexibility in dataset generation and scenario execution.
- Enhanced documentation to reflect the changes in command options and usage, ensuring clarity for users.
These modifications aim to improve the usability of the CLI by providing more control over recipe path resolution within the Spacedrive system.
- Updated the benchmark results CSV to include additional metrics such as Files_per_s, GB_per_s, Files, Dirs, GB, Errors, and Recipe for improved clarity and detail.
- Refactored the CLI command to generate a results table that now accommodates the new output format, enhancing usability for users generating performance reports.
- Adjusted the whitepaper documentation to reflect the updated metrics and ensure consistency in reporting.
These changes aim to provide a more comprehensive view of benchmark performance and improve the overall reporting capabilities within the Spacedrive system.
- Introduced new benchmark result JSON files for various scenarios, including HDD and NVMe aggregation and content identification, to expand performance testing coverage.
- Added a new CLI command to generate a summary table from JSON results, allowing users to output results in markdown or CSV format for better reporting.
- Updated documentation to reflect the new command and its usage, improving clarity for users on how to generate and format benchmark results.
These changes aim to enhance the benchmarking framework's capabilities and improve usability for generating performance reports within the Spacedrive system.
- Updated the benchmarks documentation to improve clarity and consistency in command options, specifically changing `--out_json` to `--out-json`.
- Added a comprehensive roadmap outlining future benchmarks, including indexing pipeline, file operations, duplicates detection, search and querying, media pipeline, networking, and system metrics.
- Introduced enhancements for data generation, metrics telemetry, reporting, CLI ergonomics, and CI integration to improve the benchmarking framework's capabilities.
These changes aim to provide a clearer direction for future development and enhance the usability of the benchmarking tools within the Spacedrive system.
- Added new dependencies in `Cargo.toml` for `humantime` and `blake3` to support content generation features.
- Updated benchmark YAML files to include new `content_gen` settings for partial content generation and magic header writing.
- Refactored the dataset generation logic to incorporate content generation options, improving the fidelity of generated datasets.
- Modified benchmark result JSON files to reflect updated performance metrics and additional timing information.
These changes aim to enhance the accuracy and flexibility of the benchmarking framework within the Spacedrive system.
- Added new dependencies in `Cargo.toml` for `async-trait` and `blake3` to support asynchronous operations and hashing functionalities.
- Introduced a new library structure in `src/lib.rs` and created a new binary entry point in `src/bin/sd-bench-new.rs` for the benchmarking harness.
- Updated benchmark result JSON files for HDD and NVMe scenarios, reflecting changes in performance metrics and structure.
- Implemented a modular approach for dataset generation and scenario execution, improving the overall organization and maintainability of the benchmarking code.
These changes aim to improve the benchmarking capabilities and enhance the accuracy of performance metrics within the Spacedrive system.
- Updated benchmark JSON files for HDD and NVMe scenarios, reflecting improved performance metrics in `hdd_large-indexing-discovery.json`, `hdd_medium-indexing-discovery.json`, `nvme_mixed-indexing-discovery.json`, `nvme_small-indexing-discovery.json`, and `nvme_tiny-indexing-discovery.json`.
- Refactored entry processing logic in `entry.rs` to support bulk insertion of related rows, improving transaction efficiency and clarity.
- Enhanced the processing phase in `processing.rs` to accumulate entries for bulk insert, optimizing the indexing process.
These changes aim to improve the accuracy of benchmark results and streamline entry processing within the Spacedrive system.
- Updated benchmark JSON files for HDD and NVMe scenarios, reflecting improved performance metrics in `hdd_large-indexing-discovery.json`, `hdd_medium-indexing-discovery.json`, `nvme_mixed-indexing-discovery.json`, `nvme_small-indexing-discovery.json`, and `nvme_tiny-indexing-discovery.json`.
- Refactored entry processing logic in `entry.rs` to improve transaction handling and clarity, including the introduction of a new method for creating entries with a provided connection.
- Enhanced the processing phase in `processing.rs` to utilize a single transaction for batch entry creations, improving efficiency during the indexing process.
These changes aim to optimize the indexing discovery phase and ensure accurate benchmark results within the Spacedrive system.
- Introduced a new document outlining the Indexing Discovery Throughput Plan, detailing objectives, current baselines, measurement plans, design changes, and expected outcomes to improve discovery throughput and scalability.
- Enhanced the database connection handling in `mod.rs` by applying SQLite PRAGMAs for better write throughput, including settings for journal mode, synchronous behavior, temporary storage, cache size, and memory mapping.
- Updated the indexing state management to include new configuration parameters for discovery concurrency and channel capacities, facilitating improved performance during the indexing process.
These changes aim to optimize the indexing discovery phase and improve database performance within the Spacedrive system.
- Updated benchmark JSON files for HDD and NVMe scenarios with new metrics, including `hdd_large-indexing-discovery.json`, `hdd_medium-indexing-discovery.json`, `nvme_mixed-indexing-discovery.json`, `nvme_small-indexing-discovery.json`, and `nvme_tiny-indexing-discovery.json`.
- Refactored path resolution logic in the `addressing.rs` and `path_resolver.rs` files to improve efficiency and clarity, including enhanced batching for database queries.
- Improved the handling of directory paths and entry aggregation in the indexing phase, ensuring better performance and reliability.
These changes aim to enhance the accuracy of benchmark results and streamline path resolution processes within the Spacedrive system.
- Updated `nvme_tiny.yaml` to improve formatting consistency in file size buckets.
- Introduced new JSON result files for various HDD and NVMe scenarios, including `hdd_large-indexing-discovery.json`, `hdd_medium-indexing-discovery.json`, `nvme_mixed-indexing-discovery.json`, `nvme_small-indexing-discovery.json`, and `nvme_tiny-indexing-discovery.json`.
- Expanded the CLI with a new `RunAll` command to execute all recipes in a directory sequentially, generating JSON summaries for each run.
These changes aim to improve the organization and output of benchmark results, facilitating better analysis and reporting within the Spacedrive system.
- Added `benchdata` to `.gitignore` to exclude benchmark data from version control.
- Updated `Cargo.toml` to include the `benchmarks` workspace member, facilitating better organization of benchmark-related code.
- Enhanced the whitepaper by adding new sections and citations related to database schema optimization, testing frameworks, and performance benchmarks, improving the comprehensiveness of the documentation.
- Updated various auxiliary files to reflect changes in the whitepaper structure and content.
These changes aim to improve project organization and documentation clarity, supporting ongoing development and benchmarking efforts within the Spacedrive system.
- Refactored the `run_processing_phase` function to improve readability and maintainability.
- Reorganized imports for better clarity and consistency.
- Implemented a more efficient batching mechanism for processing entries, ensuring that entries are sorted by depth before processing.
- Enhanced logging to provide clearer insights into the processing steps and outcomes.
- Updated change detection logic to handle new, modified, and moved entries more effectively, improving overall performance during the indexing phase.
These changes aim to streamline the processing phase, making it more efficient and easier to understand, thereby enhancing the overall indexing functionality within the Spacedrive system.
- Introduced job logging capabilities, allowing detailed tracking of job execution and progress.
- Added configuration options for enabling job logging, including log directory, maximum file size, and debug log inclusion.
- Enhanced job context to support file logging, ensuring that job-related events are recorded in separate log files.
- Updated core initialization and job management to utilize the new logging features, improving observability and debugging capabilities.
These changes aim to provide better insights into job execution, facilitating easier troubleshooting and performance monitoring within the Spacedrive system.
