- Added a new "Open Library" menu item to the application, allowing users to select a library folder.
- Implemented the backend logic to handle library opening, including JSON-RPC requests to the daemon.
- Introduced a filesystem watcher in the LibraryManager to monitor library directory changes, enabling automatic opening and closing of libraries based on filesystem events.
- Enhanced error handling for library operations and added relevant logging for better traceability.
- Updated the LibraryError enum to include filesystem watcher errors, improving error reporting.
- Introduced a new constructor for the HLC struct to allow explicit initialization with timestamp, counter, and device ID.
- Updated indexing tests to improve error handling for entry IDs and added a comprehensive test for reindexing after offline changes, ensuring all original and new files are detected correctly.
- Introduced a new ProxyPolicy struct to manage video scrubbing settings, including options for enabling and regenerating proxy files.
- Updated JobPolicies to include the new proxy policy with default values.
- Enhanced FileInspector to filter batch events by resource ID for improved event handling.
- Modified useNormalizedCache to skip path validation checks when resourceId is provided, optimizing resource handling.
- Updated the LocationUpdateAction to clone the location before building the ActiveModel, ensuring data integrity during updates.
- Implemented a new event emission for ResourceChanged, providing real-time updates to the UI upon location changes.
- Enhanced error handling for missing entry IDs and directory paths, improving robustness in the update process.
- Refactored the LocationInspector component to include new quick actions for reindexing locations, enhancing user interaction.
- Removed redundant isGlobalList flags from various components to streamline the codebase.
- Added critical safety checks to validate that indexing paths are within specified location boundaries, preventing potential data loss from incorrect event routing.
- Improved event routing logic in the LocationWatcher to match events to the longest valid location path, ensuring accurate event handling in scenarios with multiple locations.
- Enhanced logging for successful path validation and event routing, providing better insights into the processing flow.
- Introduced unit tests to verify path matching functionality across multiple locations, ensuring robustness in event handling.
- Implemented a migration step in the LibraryManager to ensure the database schema is up to date before proceeding with operations.
- This change enhances the reliability of the application by preventing potential schema-related issues during runtime.
- Updated the subproject commit reference to indicate a dirty state.
- Added blank lines to various files for improved readability and consistency across the codebase.
- Marked volumes as user-visible based on specific criteria to prevent redundant or non-useful system volumes from being displayed.
- Enhanced the volume tracking actions to ensure only user-visible volumes are included in tracking and untracking operations.
- Updated the storage overview component to filter and display only user-visible volumes, improving user experience and clarity.
- Refactored related code for better maintainability and readability, ensuring consistent handling of volume visibility across the application.
- Introduced a new `unique_bytes` field in the volume model to track deduplicated content size.
- Implemented a volume refresh action to recalculate unique bytes for all volumes owned by the device, emitting a cache refresh event to invalidate frontend caches.
- Updated the storage overview component to display unique bytes and other volume statistics.
- Enhanced the event system to handle cache invalidation events, ensuring real-time updates across the application.
- Added necessary migrations and database updates to support the new volume tracking features.
- Refactored the event monitoring logic in the CLI to improve readability and maintainability.
- Updated the event summarization function to enhance formatting and clarity of event messages.
- Introduced a dedicated log streaming mechanism in the core, separating log handling from event processing.
- Enhanced the RPC server to support real-time log subscriptions, allowing for more efficient log message delivery.
- Cleaned up legacy log event handling, ensuring a clearer distinction between events and logs in the system.
- Added a new LogBus to separate log message streaming from the main event bus, reducing overhead and preventing event pollution.
- Updated the Core struct to include a logs field for the new LogBus.
- Modified the RPC server to register the LogBus for real-time log message broadcasting.
- Enhanced the log emitter to utilize the new LogBus, ensuring efficient log message handling and subscriber management.
- Cleaned up related code and comments for clarity and maintainability.
- Introduced a new Events domain for monitoring events in real-time, allowing users to filter events by type, library ID, job ID, and device ID.
- Updated the CLI to include an Events command, enabling users to access the new monitoring features.
- Enhanced the Cargo.lock and related modules to support the new functionality, ensuring seamless integration with existing systems.
- Added necessary arguments and command handling for the Events command in the CLI.
- Introduced new modules and actions for generating media proxies and thumbstrips, improving media handling capabilities.
- Added configuration, error handling, and processing logic for both proxy and thumbstrip generation.
- Updated existing media operations to integrate new functionalities, ensuring seamless media processing and user experience.
- Enhanced tests to cover new features and ensure reliability in media operations.
The StateSyncHandler was a leftover from an earlier architecture where
protocol handlers were meant to be separate. The current implementation
bypasses it entirely by having the network protocol handler directly
call PeerSync::get_device_state() which properly uses the registry.
Backfill has been working fine because it completely avoids this code path.
Changes:
- Remove StateSyncHandler struct and all its methods from protocol_handler.rs
- Remove state_handler field from BackfillManager
- Update exports in mod.rs
- Clean up imports in backfill.rs
This eliminates the confusing TODO comments about serialization that
were already implemented elsewhere.
- Added logic to exclude the current device from sync partners, ensuring that only other paired devices are considered.
- Enhanced the filtering process to ensure devices have valid NodeId mappings, improving the accuracy of synchronization partners.
- Implemented concurrency limiting for broadcast tasks to prevent network overload, enhancing overall performance and reliability.
- Improved logging to provide detailed insights into the computed sync partners and broadcasting status.
- Introduced a new `.ignore` file to prevent auto-reload on core directory changes.
- Updated foreign key mapping logic to mark records with missing dependencies for retry, ensuring parent relationships are preserved.
- Enhanced watermark management by migrating to a dual watermark schema, allowing for better tracking of validated states and recovery points.
- Added content hash computation for device-owned resources to facilitate update detection during synchronization.
- Improved logging to provide insights into filtered records and watermark promotions, enhancing overall synchronization accuracy.
- Added logic to extract and parse a timestamp from change data, ensuring that state change messages include accurate timing information.
- Updated the BufferedUpdate structure to accommodate the new timestamp field, enhancing the clarity and utility of synchronization events.
- Updated the sync event bus capacity from 10,000 to 100,000 events to better handle bulk indexing and stress scenarios.
- Enhanced the state change broadcasting logic by grouping changes by model type, allowing for more efficient processing and reducing network load through batched StateBatch messages.
- Improved logging to reflect the new capacity and efficiency gains in the broadcasting process.
- Removed outdated sync readiness rules from the entry model, simplifying the logic for determining synchronization eligibility.
- Enhanced the backfill manager's method for handling shared resources, improving code clarity and maintainability.
- Updated subproject commit reference to indicate a dirty state, reflecting recent changes.
- Replaced complex subquery logic with raw SQL for filtering entries based on device ownership, improving query performance and readability.
- Enhanced error handling for UUID synchronization to ensure data consistency.
- Streamlined code by consolidating delete and count operations in the entry closure rebuild process.
- Improved logging in the backfill manager to provide better insights into state changes and record processing.
- Introduced a sentinel value "SKIP_SHARED" to allow surgical recovery that skips shared backfill, focusing on device-owned resources only.
- Updated the backfill manager to handle the new logic, improving efficiency in synchronization processes.
- Enhanced logging to clarify the conditions under which shared backfill is skipped, providing better insights into the recovery process.
- Moved the initialization of mismatched resource tracking outside the block for improved clarity and efficiency in watermark comparison checks.
- Enhanced code organization by reducing redundancy in the mismatch handling logic, ensuring better maintainability and readability.
- Improved comments to clarify the purpose of tracking mismatched resources during synchronization processes.
- Updated the catch-up logic to skip watermark comparisons when count validation confirms synchronization, preventing unnecessary catch-up operations.
- Enhanced comments to clarify the rationale behind the changes, emphasizing the importance of accurate state management in synchronization processes.
- Improved logging to provide insights into the decision-making process regarding watermark-based catch-up.
- Changed the iteration over model types to use a reference, improving performance by avoiding unnecessary cloning.
- Enhanced code clarity by reducing potential overhead in the synchronization process.
- Updated the `query_for_sync` method to accept an optional `device_id` parameter, allowing for filtering of entries based on device ownership.
- Implemented logic to retrieve and filter entries by their associated locations, ensuring only relevant data is synchronized for the specified device.
- Improved logging to clarify the filtering process and its importance for device-owned data synchronization.
- Updated the catch-up logic to trigger only when our resource count is less than the peer's actual count, ensuring efficient synchronization.
- Enhanced comments to clarify the rationale behind the count comparison, emphasizing the importance of accurate state management in synchronization processes.
- Updated the backfill logic to parse HLC from string watermarks, allowing for incremental backfill of shared resources instead of a full backfill.
- Enhanced the catch-up process in PeerSync to preserve the shared watermark, ensuring efficient synchronization by avoiding unnecessary backfills of shared resources.
- Improved logging to clarify the nature of the backfill operations, emphasizing surgical backfill for device-owned resources while maintaining shared state integrity.
- Added logic to determine the initial synchronization state based on the presence of watermarks in sync.db, setting the state to Ready if watermarks exist and Uninitialized if not.
- Enhanced logging to provide clear insights into the initial state determination process, improving visibility for debugging and monitoring synchronization behavior.
- Updated the logic for detecting count mismatches to prioritize count validation over watermark comparisons, ensuring more reliable synchronization.
- Improved logging to provide clearer insights into count discrepancies and the associated watermark differences.
- Simplified the mismatch reporting format for better readability and understanding of synchronization issues.
- Modified the `start` and `start_periodic_watermark_check` methods to accept `self` as an `Arc<Self>`, enhancing memory management and preventing potential dangling references.
- Updated logging messages to reflect changes in watermark exchange handling, ensuring clarity in the synchronization process.
- Improved the handling of connected sync partners by utilizing the upgraded weak reference pattern for better stability during periodic checks.
- Simplified the processing of WatermarkExchangeResponse messages to ensure they are only handled via the bidirectional stream.
- Added a warning log for unexpected responses, improving visibility into potential misuse of the sync request mechanism.
- Removed unnecessary parameters from the response handling, enhancing code clarity and maintainability.
- Updated the watermark exchange process to utilize a request/response pattern, enhancing synchronization reliability.
- Enhanced logging to provide clearer insights into the request and response lifecycle during watermark exchanges.
- Added error handling for unexpected message types in the response, improving robustness.
- Clarified comments to better describe the asynchronous nature of the watermark exchange process.
- Revised the catch-up logic to make the sync loop passive, driven by watermark exchanges and reconnection events.
- Enhanced comments to clarify the new approach, focusing on edge cases where retry states indicate a stuck condition.
- Removed the previous time-based sync checks in favor of a more robust failure tracking mechanism.
- Updated the entry count query to utilize the entry_closure table for more accurate counting of entries associated with device locations.
- Enhanced comments to clarify the purpose of the query and the use of closure tables in the synchronization process.
- Changed logging from info to debug for various library operations to reduce log verbosity.
- Updated the periodic watermark check interval from 5 minutes to 1 minute for more frequent synchronization checks.
- Commented out the ContentBreakdown component in the Overview route for potential future use.
- Updated the watermark exchange methods to accept a database connection, enabling retrieval of peer resource counts during synchronization.
- Enhanced logging to provide visibility into the resource counts being exchanged, improving debugging and monitoring capabilities.
- Addressed potential issues with resource count availability by ensuring counts are fetched correctly during static exchanges.
- Introduced new functionality to validate resource counts during watermark exchanges, improving synchronization accuracy.
- Added methods to retrieve device-owned resource counts for gap detection, ensuring that discrepancies are identified and addressed.
- Updated the `SyncMessage` structure to include actual resource counts, facilitating better tracking of synchronization state.
- Enhanced the `PeerSync` implementation to clear watermarks for mismatched resources, allowing for surgical recovery in case of count mismatches.
- Improved logging for better visibility into synchronization processes and potential issues.
- Introduced a new `DataAvailableNotification` message type to notify peers of new data availability, triggering immediate watermark exchanges for synchronization.
- Enhanced the `PeerSync` implementation to handle notifications and manage real-time activity tracking per peer, improving synchronization efficiency.
- Updated the `SyncProtocolHandler` to process the new notification type and initiate catch-up logic accordingly.
- Implemented periodic watermark checks to ensure timely synchronization and prevent missed events during idle periods.
- Added support for per-resource watermarks in the synchronization process, allowing for fine-grained comparison and catch-up logic.
- Implemented new configuration options for real-time batching, including maximum entries before flush and flush interval in milliseconds.
- Enhanced the `apply_state_change` function to check for tombstoned records, preventing the resurrection of deleted entries.
- Updated various components to utilize the new watermark and batching features, improving synchronization efficiency and robustness.
- Updated the submodule reference in the landing app.
- Added a statistics recalculation listener in the LibraryManager to monitor resource changes.
- Included clipboard permissions in the capabilities configuration for enhanced functionality.
- Improved the LibraryInfoOutput struct to implement the Identifiable trait for better resource management.
- Added clipboard permissions for reading and writing text in the default capabilities configuration.
- Enhanced the sync test harness to capture job lifecycle events, improving tracking of indexing completion and orphaned file management.
- Implemented checks for orphaned files during synchronization, ensuring robust handling of incomplete sync scenarios.
- Updated the submodule reference in `workbench`.
- Added debug logging in `ResourceManager` to track resource event emissions and virtual resource mappings.
- Improved the `PeerSync` implementation to emit resource events in a non-blocking manner, enhancing UI reactivity and ensuring proper resource formatting.
- Updated integration tests to include new resource types and refined synchronization checks for stability.
- Updated the foreign key mapping logic in `fk_mapper.rs` to return an error when a referenced record is missing, instead of setting the foreign key to NULL. This change ensures that sync dependencies are properly managed and allows for retrying when dependencies are resolved.
- Enhanced the error handling in `peer.rs` to buffer state changes that depend on missing foreign keys, improving the robustness of the synchronization process and preventing data loss during updates.
- Updated the `Library` implementation to generate HLCs, append to the peer log, and emit real-time events for instant synchronization.
- Introduced a mechanism to collect entries for real-time broadcasts, ensuring immediate updates are sent to peers.
- Added a yield point in the content phase to prevent foreign key orphaning by ensuring content identity events are emitted before entry updates.
- Enhanced the `NetworkingService` to cache active connections, allowing for reuse of existing connections and reducing the overhead of establishing new ones.
- Implemented logic to check the status of cached connections before creating new ones, improving efficiency during synchronization and request handling.
- Added detailed logging for connection reuse and creation, aiding in monitoring and debugging network interactions.
- Introduced a batching mechanism in `PeerSync` to accumulate state change events, improving network efficiency by processing up to 100 entries at once or flushing every 50ms.
- Added a new method `flush_state_change_batch` to handle the processing of batched state changes, enhancing overall synchronization performance.
- Updated event handling logic to support batching, reducing the frequency of individual state change processing and improving throughput.
