Resource Synchronization & Addition

When users collaborate on shared documents, they need two distinct synchronization processes: syncing changes to existing shared resources, and adding new resources to existing collaborations. Both processes ensure that collaborators maintain consistent views of their shared workspace while preserving cryptographic security.

Resource State Synchronization

Resource synchronization handles keeping shared documents up-to-date between collaborators. After user sync completes with manifest comparison, peers begin parallel state synchronization for all resources that need updating.

Vector Clock Change Detection

Vector clocks serve as lightweight “dirty bits” to determine which resources need synchronization. Each document maintains a simple vector clock that increments when changes occur, allowing peers to quickly identify resources that have diverged without examining detailed content.

Yjs State Vector Exchange Protocol

The actual synchronization uses Yjs state vectors, which provide precise CRDT state information rather than simple version numbers. When a resource needs syncing:

1. State Vector Request: One peer sends its Yjs state vectors for the resource - this contains the complete state summary of all CRDT operations applied locally.

2. Update Calculation: The receiving peer compares these state vectors with its local CRDT state to identify exactly which operations the sender is missing.

3. Bidirectional Response: The peer sends back both the calculated update operations AND its own state vectors, enabling bidirectional synchronization in a single round trip.

4. Apply and Validate: The original sender applies the updates, compares the result with the received state vectors, and returns any remaining differences to ensure perfect convergence.

This process leverages Yjs’s built-in synchronization algorithm, which guarantees that both peers converge to identical CRDT states regardless of the order of operations or network conditions.

Parallel Synchronization Architecture

After user sync manifest comparison completes, all resource synchronization happens in parallel:

Resource Addition: Missing resources are sent concurrently using ResourceAddRequest messages with UCAN tokens and re-encrypted AES keys.

State Synchronization: Resources that need updating begin state vector exchanges simultaneously, with each resource sync operating independently.

Chunked Transfer: All data transfers use the 8KB chunked protocol over QUIC bi-directional streams, ensuring reliable delivery of large payloads.

Error Isolation: Failed sync operations for one resource don’t affect synchronization of other resources, providing resilient parallel processing.

Adding Resources to Collaborations

When users create new documents or want to share existing ones with their collaboration network, the resource addition process ensures all appropriate collaborators gain access while maintaining cryptographic security.

New Resource Creation and Distribution

When Alice creates a new document and shares it with existing collaborators, the resource propagates through the network during the parallel sync phase:

Integration with User Sync Cycles

Resource addition happens after user network synchronization completes:

Post-Network Sync: After user sync exchanges what each side is missing and both sides know what to expect, resource synchronization begins.

Known Expectations: Both peers understand exactly which resources they need to send and receive based on the completed manifest comparison.

Initiator-Driven: The sync initiator sends the first StateVectorRequest to begin the resource update process.

Bidirectional Exchange: Both sides send missing resources to each other as determined during the user sync phase.

Message Protocol and Reliability

Resource Synchronization Messages

The protocol defines specific message types for resource operations:

pub enum ResourceUpdateMsg {
    // Initial message with state vector
    StateVectorRequest {
        resource_id: String,
        state_vectors: String,
        ucan_token: String,
    },
    // Response with updates and state vector
    UpdatesResponse {
        resource_id: String,
        updates: String,
        ucan_token: String,
    },
    FinalUpdateMerge {
        resource_id: String,
        updates: String,
        vector_clocks: Vec<ResourceVectorClock>,
        share_records: Vec<ShareRecord>,
        resource_keys: Vec<ResourceKey>,
    },
    // Acknowledgment that sync is complete
    VectorClockResponse {
        resource_id: String,
        update_clock: Vec<ResourceVectorClock>,
        add_clock: Vec<ResourceVectorClock>,
        share_records: Vec<ShareRecord>,
        resource_keys: Vec<ResourceKey>,
    },
}

Chunked Transfer and Error Handling

All messages use the established 8KB chunked transfer protocol over QUIC:

Reliable Delivery: QUIC’s built-in reliability ensures chunks arrive in order without loss.

Large Payload Support: CRDT states and updates can be arbitrarily large, handled transparently by the chunking mechanism.

Parallel Streams: Each resource sync uses its own QUIC bi-directional stream, enabling true parallelism.

Error Recovery: Failed transfers are retried with exponential backoff, and partial failures don’t affect other concurrent operations.

State Validation and Consistency

After synchronization operations complete, the system validates consistency:

CRDT State Integrity: Validates that merged Yjs states are well-formed and internally consistent.

Vector Clock Alignment: Ensures local vector clocks accurately reflect the new synchronized state.

Permission Validation: Confirms that all resource access attempts use valid UCAN tokens with appropriate capabilities.

Content Decryption: Verifies that shared AES keys successfully decrypt document content and produce valid data.

This comprehensive synchronization approach ensures that both the discovery network and the content network remain consistent and secure while maximizing throughput through parallel processing and reliable transfer protocols.