363 Commits

Author SHA1 Message Date
Jamie Pine
4ef5d59ebe feat: Add library validation and selection logic to Context
- Implemented `validate_and_fix_library` method in the Context struct to ensure the current library selection is valid.
- If the stored library ID is not found, the method selects the first available library.
- Updated the main execution flow to call this new method, enhancing library management during command execution.
2025-10-09 05:45:47 -07:00
Jamie Pine
705249019c feat: Implement job registration system for WASM extensions
- Introduced a new `ExtensionJobRegistry` to manage custom job types for extensions at runtime.
- Added `host_register_job` function to facilitate job registration from WASM extensions.
- Updated `PluginEnv` and `PluginManager` to include job registry functionality.
- Enhanced the `job` macro to support automatic job registration during plugin initialization.
- Updated documentation and tests to reflect the new job registration capabilities.
2025-10-09 05:18:35 -07:00
Jamie Pine
741bb7ce0d refactor: Rename job macros and update documentation
- Renamed `spacedrive_job` macro to `job` for clarity and consistency across the SDK.
- Updated references in the SDK and test extension to reflect the new macro name.
- Enhanced comments in the test extension to better describe functionality and usage.
- Added "checkpointing" to the cspell project words for improved spell checking.
2025-10-09 04:07:12 -07:00
Jamie Pine
49c3ff9f5e feat: Introduce Spacedrive SDK for building WASM extensions
- Added a new SDK for developing Spacedrive WASM extensions, featuring a type-safe API and comprehensive documentation.
- Implemented core components including job context management, FFI bindings, and error handling mechanisms.
- Created macros for simplifying extension and job definitions, enhancing developer experience.
- Included detailed README and status documentation to guide users in utilizing the SDK effectively.
- Updated Cargo configurations for the SDK and its macros, ensuring proper dependency management and build processes.
2025-10-09 04:00:12 -07:00
Jamie Pine
4ade103d38 feat: Complete implementation of leaderless hybrid sync system
- Finalized the core synchronization infrastructure, enabling device-independent writes without leader checks.
- Implemented critical components including `TransactionManager`, `SyncProtocolHandler`, and `PeerSync` with enhanced broadcast capabilities.
- Achieved full message routing for state and shared changes, ensuring robust error handling and logging.
- Registered new models in the async-safe registry, facilitating state-based and log-based synchronization.
- Documented implementation progress and architecture changes, highlighting the transition to a leaderless model.
- Prepared for integration testing with remaining tasks outlined for backfill and retry queue implementation.
2025-10-09 00:03:27 -07:00
Jamie Pine
95295b8303 refactor: Transition to leaderless hybrid sync architecture
- Removed leadership-related components from the sync infrastructure, including `LeadershipManager` and `sync_leadership` fields across various models.
- Implemented a new peer-to-peer sync model utilizing Hybrid Logical Clocks (HLC) for shared resources and state-based sync for device-owned data.
- Updated the `Syncable` trait and related modules to reflect the new architecture, ensuring seamless integration of state and log-based synchronization.
- Introduced `PeerLog` for managing device-specific changes and `PeerSync` for handling synchronization in the leaderless environment.
- Revised documentation to outline the new sync architecture and its implications for device synchronization, emphasizing the benefits of a leaderless approach.
2025-10-08 19:18:23 -07:00
Jamie Pine
910dce67f5 feat: Add new documentation and enhance CLI functionality
- Introduced three new markdown files: CLI_LIBRARY_SYNC_COMPLETE.md, IMPLEMENTATION_COMPLETE.md, and LIBRARY_SYNC_SETUP_IMPLEMENTATION.md for comprehensive documentation.
- Updated various CLI domain modules to improve argument handling and output formatting.
- Enhanced device, index, job, library, location, network, and search modules for better integration and user experience.
- Refactored code across multiple domains to improve maintainability and clarity.
2025-10-04 21:31:47 -07:00
Jamie Pine
b4dd9a9454 refactor: Update event handling and type definitions in Swift client
- Removed manual type definitions in favor of generated types from SpacedriveClient, enhancing type safety.
- Refactored event handling in DaemonConnector to utilize a new type-safe Event enum for better clarity and maintainability.
- Updated job management logic to align with the new event structure, improving real-time job tracking and state management.
- Enhanced serialization and deserialization processes for various data structures to ensure compatibility with the Rust daemon output.
- Removed obsolete schema generation binary and updated Cargo.toml to reflect changes in dependencies and project structure.
2025-09-22 00:06:06 -07:00
Jamie Pine
ae19c0cd97 refactor: Update type handling and schema generation for Swift client
- Removed the `codegen` module and its associated files to streamline the codebase.
- Introduced a new binary for generating Swift types using Specta, enhancing type safety and integration.
- Updated various data structures to utilize the `specta` crate for type generation.
- Refactored existing types to remove `JsonSchema` derives in favor of `specta::Type`, improving compatibility with the new generation system.
- Enhanced the `Cargo.toml` to exclude unnecessary crates and improve workspace organization.
2025-09-21 19:13:38 -07:00
Jamie Pine
6e25fbfe41 feat: Introduce persistence completion handling in job management
- Added a new `persistence_complete_tx` channel to facilitate signaling when job state persistence is complete.
- Updated `JobExecutor` and `JobManager` to manage the persistence completion channel, ensuring proper signaling during job execution and state management.
- Enhanced the `ChangeDetector` to include a cache for file existence checks, improving performance and accuracy in detecting changes.
- Refactored related methods to streamline job state handling and improve logging for persistence operations.
2025-09-20 17:40:29 -07:00
Jamie Pine
f83ac9f829 refactor: Clean up emoji handling in Rust files and improve output formatting
- Refactored `remove_emojis.sh` to enhance the emoji removal process by using a more efficient Python script for detecting and removing 3D/colorful emojis followed by spaces.
- Updated various Rust files to remove emoji characters from log messages and print statements, resulting in a cleaner output.
- Improved user experience with color-coded progress and results during the emoji removal process.
- Ensured that the script counts processed files and reports modifications accurately.
2025-09-19 18:40:19 -07:00
Jamie Pine
ecdcf04066 Refactor logging output to remove emojis and improve consistency
- Updated various print statements across the codebase to remove emoji indicators, ensuring a more uniform logging style.
- Adjusted output messages in CLI, job handling, and event processing to enhance clarity and maintainability.
- Refactored job-related logging to provide clearer status updates without emoji clutter.

Co-authored-by: ijamespine <ijamespine@me.com>
2025-09-18 22:44:10 -07:00
Jamie Pine
469fc7cbe2 refactor: complete migration to modular action system and remove ActionOutput enum
- Eliminated the centralized `ActionOutput` enum, achieving true modularity in action handling.
- Introduced the `ActionType` trait, allowing each action to define its own output type directly.
- Updated multiple actions to utilize native output types, enhancing type safety and reducing complexity.
- Enhanced the `ActionManager` with a generic `dispatch_action` method for improved action execution.
- Preserved existing infrastructure for validation, audit logging, and error handling throughout the migration.

These changes significantly improve the modularity and maintainability of the action system while ensuring a consistent API surface.
2025-09-09 02:03:17 -04:00
Jamie Pine
d90b486865 refactor: update project dependencies and remove unused components
- Removed the `prisma-client-rust` and `prisma-client-rust-sdk` dependencies from `Cargo.toml` to streamline the project.
- Deleted the `spacedrive-jobs-derive` crate, consolidating job registration functionality into the main job system.
- Updated the `.gitattributes` file to reflect changes in generated code tracking.
- Adjusted the `package.json` scripts for improved build processes and removed unnecessary commands.
- Enhanced the `combine.sh` script to correctly reference documentation paths.

These changes aim to simplify the project structure and improve maintainability.
2025-09-08 17:26:31 -04:00
Jamie Pine
a8a2c732ef more refactor 2025-09-07 01:43:57 -04:00
Jamie Pine
29a87a91ce huge refactor 2025-09-06 21:00:37 -04:00
Arnab Chakraborty
dcb78f079e Working Sync & Thumbnail Sync (#2875)
* First draft on new p2p design

* Test sync

* Sync works

* Update version

* Fix merge conflicts

* More stuff for thumbnail sync

* Get the thumbnail

* Update mod.rs

* Working thumbnail sync

* Format

* Update thumbnails.rs

* Update runner.rs

* Update runner.rs

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2025-03-20 07:34:59 +03:00
Julian Braha
d51f4eb14d Optimize Option<Flash>::from(FlashValue) using bitwise AND (#2843) 2025-02-12 01:25:08 +03:00
Arnab Chakraborty
58e7151e19 Move Auth Tokens storage from Local Storage to custom encrypted .sdks file (#2831)
* Initial encryption functions

* Storage of encrypted tokens & working login

* Add customized `tauri-plugin-cors-fetch`

We need to have it in our crates folder due to the dependency of `sd-crypto` for the handling of cookies.

* Lint
2024-12-14 19:49:50 -05:00
Ericson Soares
dd56820898 Rustfmt 2024-12-10 12:37:14 -03:00
0xBA5E64
d4f40e5cf0 Update CONTRIBUTING.md (#2820)
* docs: Remove extranious `clean` commands, `pnpm clean` already runs all three.

* docs: Update recommended Rust version to reflect current `rust-toolchain.toml`

* docs: Refactor the Desktop clients developer setup guides

* docs: Refactor the Troubleshooting section for developer setup

* docs: `dev:web` is deprecated, according to @Rocky43007

* docs: Specify that the Tauri app doesn't play with `sd-server`

* docs: Nevermind! `dev:web` is still on the menu!

* docs: GitHub.md "Alerts" don't like indentation.

* fix: "You're just like the team now, just committing random fixes to whatever branch" - @rocky43007

* Apply suggestions from code review

Thanks Lynx!

Co-authored-by: Lynx <141365347+iLynxcat@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Lynx <141365347+iLynxcat@users.noreply.github.com>

* docs: Completely rewritten Mobile App section in `CONTRIBUTING.md`

* docs: Polish polish polish

* docs: Poleish?

* docs: Trimming & Trimming

* docs: Add final step clarification for running Android app

* docs: Try fixing the tip line embed

* Update CONTRIBUTING.md

Co-authored-by: Philipp <34287258+Phippe@users.noreply.github.com>

* Fix minor grammar & spelling mistakes

* Fix Github Docs Contributing link

---------

Co-authored-by: Lynx <141365347+iLynxcat@users.noreply.github.com>
Co-authored-by: Philipp <34287258+Phippe@users.noreply.github.com>
Co-authored-by: Arnab Chakraborty <11457760+Rocky43007@users.noreply.github.com>
2024-11-30 16:38:37 -05:00
Ericson Soares
31f954f38f Optimize and fix attempt at cloud ingester 2024-10-21 22:27:09 -03:00
Vítor Vasconcellos
eb098ebec2 Update iroh and sd-cloud-schema
- Fix usage of deprecated functions
 - Implement clippy beta suggestions
2024-10-21 19:44:08 -03:00
Vítor Vasconcellos
7e7dd8fbde Merge remote-tracking branch 'origin/main' into eng-1828-migration-to-new-cloud-api-system 2024-10-21 14:40:59 -03:00
Vítor Vasconcellos
cc7c9d5793 Improve file thumbnails and Quick Preview (+ some code clean-up and rust deps update) (#2758)
* Update rspc, prisma-client-rust, axum and tanstack-query
 - Deleted some unused examples and fully commented out frontend code
 - Implement many changes required due to the updates
 - Update most rust dependencies

* Re-enable p2p

* Fix server

* Auto format

* Fix injected script format
 - Update some github actions
 - Update pnpm lock file

* Fix devtools showing up when app opens
 - Fix million complaining about Sparkles component

* Fix sd-server

* Fix and improve thumbnails rendering
 - Fix core always saying a new thumbnail was generated even for files that it skiped thumbnail generation
 - Rewrite FileThumb and improve related components

* Ignore tmp files when running prettier

* Improve FileThumb component performance
 - Rework useExplorerDraggable and useExplorerItemData hooks due to reduce unecessary re-renders

* More fixes for thumb component
 - A couple of minor performance improvements to frontend code

* auto format

* Fix Thumbnail and QuickPreview

* Fix logic for when to show 'fail to load original' error message in QuickPreview
 - Updated prisma-client-rust, libp2p, tauri, tauri-specta, rspc and hyper

* Fix type checking
 - Format scripts

* Add script prettier config

* Fix serde missing feature
 - Use rust-libp2p spacedrive fork again
 - Update rspc

* Autoformat + fix pnpm lock

* Fix thumbnail first load again

* Autoformat

* autoformat

* Fix rust-libp2p fork url again?

* Remove usePathsInfiniteQuery hook

* Update tauri 2.0.6
2024-10-21 15:47:40 +00:00
Ericson Soares
f414e549f8 Some tweaks on sync messages send and compression 2024-10-20 22:26:06 -03:00
Ericson Soares
6a69f5f8ef Update sync messages push and pull
Also fix scalability issues on sync design
2024-10-19 02:48:27 -03:00
Ericson Soares
2bb06d6317 Many tweaks to improve sync responsiveness and observability 2024-10-14 23:39:52 -03:00
James Pine
0ac05f46b3 increase max file size for thubm generation 2024-10-12 00:59:27 -07:00
Ericson Soares
375c111305 Some tweaks on sync messages encryption and decryption 2024-10-09 22:04:49 -03:00
Ericson Soares
3208ef5dd2 Change database to use device id on everything instead of pub_id 2024-10-07 22:38:45 -03:00
Vítor Vasconcellos
a223fa0a8b Merge remote-tracking branch 'origin/main' into eng-1828-migration-to-new-cloud-api-system 2024-10-06 23:36:06 -03:00
Vítor Vasconcellos
cdc50140b0 [ENG-1928] Update to tauri 2.0.1 stable (#2752)
* Update to tauri 2.0.0 stable
 - Update a some othe dependencies
 - Autoformat rust and toml files

* Fix clippy complain

* Update tauri 2.0.1
 - Update rspc
 - Update some frontend files due changes in rspc
 - Update some frontend build dependencies

* Revert babel update

* Fix code typing typing and style + a couple of bugs

* More type fixes
2024-10-05 21:17:30 +00:00
Arnab Chakraborty
2c0d94abc4 Merge branch 'main' into eng-1828-migration-to-new-cloud-api-system 2024-10-03 22:23:31 -04:00
Consoli
8ce9f30121 Remove dependency on once_cell and use std types (#2746)
* Remove dependency on once_cell and use std types

* Bump MSRV across the project to 1.81

* Cargo fmt

* Update rust-toolchain to 1.81
2024-09-30 21:19:59 +00:00
Ericson Soares
70ea030c97 Rust fmt 2024-09-29 02:08:59 -03:00
Ericson "Fogo" Soares
3a97ede562 Update toolchain and fix a bunch of warnings 2024-09-27 00:38:48 -03:00
Ericson Soares
d0e52a4342 Fix identifier bug 2024-09-19 22:02:32 -03:00
Arnab Chakraborty
ac07f001ac Merge branch 'main' into eng-1828-migration-to-new-cloud-api-system 2024-09-18 22:30:54 -04:00
Consoli
874a268735 Update pdfium-render version (#2701)
* Revert "Fix pdf thumbnails (#2666)"

This reverts commit c240047e0538f72a572fdf16f5143edf11bf1894.

* Bump pdfium-render dep

* Bring back custom fork
 - Bump the minimum pdfium lib api version (updated bundled lib in native-deps)
 - Custom fork is still required because upstream added a hard limit on chono version 0.4.31, which is incompatible with some of our dependencies which require newer versions

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-09-17 14:44:03 +00:00
Ericson Soares
03e9de497b Fix key size mismatch between opaque and sd_crypto 2024-09-13 12:54:36 -03:00
Ericson Soares
9f8005c479 Relating tables to device 2024-09-06 23:53:35 -03:00
Arnab Chakraborty
02e4bf4d5b Iterator for CompressedCRDTOperationsPerModel 2024-09-06 21:01:54 -04:00
Ericson Soares
0ac38b6845 Tweak crdt ops compression and use it on cloud sync 2024-09-06 21:18:22 -03:00
Ericson Soares
872ca1deec Receiver actor for cloud sync 2024-09-04 01:50:22 -03:00
Ericson Soares
25f0f4fb1a Sender actor for cloud sync messages 2024-08-30 03:34:03 -03:00
Ericson Soares
76cba5226d More tweaks on sync subcrate and other fixes 2024-08-27 23:45:05 -03:00
Ericson Soares
4bf8db02f7 Make core compile again 2024-08-27 17:01:42 -03:00
Ericson Soares
a068ca7c7b Some tweks to simplify sync subcrate
Initial work to port cloud sync system to new cloud services
2024-08-23 23:42:51 -03:00
Ericson Soares
35c4b1549c Cloud P2P abstraction 2024-08-23 15:43:43 -03:00