Website Synchronization

Sthalam’s website synchronization protocol ensures viewers stay up-to-date with publisher content while maintaining efficient network usage and respecting permission boundaries. The sync protocol differs significantly from Livnote’s user synchronization, focusing on content distribution rather than collaborative editing.

Synchronization Types

Initial Sync

First-time connection when viewer subscribes to publisher:

• Complete folder and resource transfer
• All existing content delivered
• Viewer marked as first_sync = true

Incremental Sync

Subsequent connections to receive updates:

• Detect new resources added to folder
• Sync changes to existing resources
• Efficient state vector-based reconciliation

User Sync vs. Website Sync

User Sync (Livnote - Publisher ↔ Node)

Purpose: Synchronize shared user context between collaborating peers

Flow:

• Publisher and node exchange manifests
• Compare known users and devices
• Share delegated connection tokens
• Enable transitive peer discovery

Use Case: Building peer-to-peer collaboration networks

Website Sync (Sthalam - Node ↔ Viewer)

Purpose: Distribute published content to viewers with specific permissions

Flow:

• Viewer sends resource manifest
• Node detects new/updated resources
• Node validates UCAN permissions
• Sends content with granular permissions

Use Case: Sovereign content publishing and distribution

Incremental Sync Protocol

Step 1: Viewer Reconnection

When a viewer reconnects (e.g., comes back online):

1. Handshake: WebsiteHandshake exchange to verify identities
2. Check First Sync: Database query shows first_sync = true
3. Trigger Incremental Sync: Call perform_incremental_sync()

Step 2: Folder Resource Info Exchange

Viewer Builds Manifest (network/src/p2p/website_handler.rs:426-512):

// Get all folders viewer has access to
let folder_manifest = services::get_viewer_folder_manifest(
    &local_user.id,
    repo_ctx
).await?;

// For each folder
for folder_info in folder_manifest {
    // Send folder ID + resource IDs + folder UCAN
    let message = Message::Website(WebsiteMessage::FolderResourceInfo {
        folder_id: folder_info.folder_id,
        resource_ids: folder_info.resource_ids,  // What viewer currently has
        folder_ucan: folder_info.folder_ucan,
    });
}

Node Validates and Compares (network/src/p2p/website_handler.rs:514-669):

// 1. Validate folder UCAN
let ucan = crypto_utils::ucan_utils::validate_structure(&folder_ucan).await?;

// 2. Extract and verify folder_id
let ucan_folder_id = crypto_utils::ucan_utils::extract_folder_id_from_ucan(&ucan)?;
assert_eq!(ucan_folder_id, folder_id);

// 3. Get node's resource list for this folder
let node_resource_ids = repo_ctx
    .resource_repo
    .get_resource_ids_by_folder_id(&folder_id)
    .await?;

// 4. Find new resources (on node but not on viewer)
let new_resource_ids: Vec<String> = node_resource_ids
    .into_iter()
    .filter(|id| !viewer_resource_ids.contains(id))
    .collect();

Step 3: New Resource Distribution

For each new resource the viewer doesn’t have:

Resource Preparation:

// Get resource type
let resource = repo_ctx
    .resource_repo
    .find_by_id(&resource_id, &local_user.id)
    .await?;

// Prepare for viewer with resource-specific permissions
let resource_sync_data = services::prepare_resource_for_viewer(
    &resource_id,
    &resource.resource_type,
    &viewer_user,
    &local_user,
    repo_ctx,
    &crypto_utils,
).await?;

Permission Generation (services/src/node_service.rs:217-234):

// Generate 3-document permissions
let permissions = vec![
    (
        format!("sthalam:resource:{}:blocksuite_doc", resource_id),
        "crud/read".to_string(),  // Main content: read-only
    ),
    (
        format!("sthalam:resource:{}:thread_comments_doc", resource_id),
        "crud/write".to_string(),  // Comments: bidirectional
    ),
    (
        format!("sthalam:resource:{}:form_submissions_doc", resource_id),
        "crud/append".to_string(),  // Submissions: append-only
    ),
];

Distribution:

// Send new resource with viewer-specific UCAN
self.send_message(Message::ResourceAdditionRequest(resource_sync_data)).await?;

Step 4: Existing Resource Sync

For resources viewer already has, sync state changes:

Viewer Sends State Vectors:

// Get Yjs state vectors for each document
let sync_info = services::get_resource_sync_info(
    &resource_id,
    &local_user.id,
    repo_ctx,
    &crypto_utils,
).await?;

// Send incremental sync request
let message = Message::Website(WebsiteMessage::IncrementalSyncRequest {
    resource_id: sync_info.resource_id,
    resource_ucan: sync_info.resource_ucan,  // Resource-specific UCAN
    sync_data: sync_info.sync_data,          // Yjs state vectors
});

Node Validates and Processes (network/src/p2p/website_handler.rs:673-770):

// 1. Validate resource UCAN
let ucan = crypto_utils::ucan_utils::validate_structure(&resource_ucan).await?;

// 2. Extract and verify resource_id
let ucan_resource_id = crypto_utils::ucan_utils::extract_resource_id_from_ucan(&ucan)?;
assert_eq!(ucan_resource_id, resource_id);

// 3. Process sync and generate updates
let response_sync_data = services::process_incremental_resource_sync(
    &resource_id,
    &local_user.id,
    &sync_data,  // Viewer's state vectors
    repo_ctx,
    &crypto_utils,
).await?;

// 4. Send response with missing updates
self.send_message(Message::Website(WebsiteMessage::IncrementalSyncResponse {
    resource_id,
    sync_data: response_sync_data,
})).await?;

Viewer Applies Updates and Sends Comments (network/src/p2p/website_handler.rs:772-862):

// Apply node's updates to local state
let viewer_updates = services::apply_and_generate_viewer_updates(
    &resource_id,
    &local_user.id,
    &sync_data,  // Node's updates
    repo_ctx,
    &crypto_utils,
).await?;

// Emit to frontend for UI update
self.event_emitter.emit(P2PEvent::UpdatesEvent {
    resource_id,
    updates: sync_data,
    client_id: 0,
});

// Send viewer's comment updates back to node
self.send_message(Message::Website(WebsiteMessage::ViewerCommentsUpdate {
    resource_id,
    resource_ucan,
    sync_data: viewer_updates,  // Only comments, not main content
})).await?;

Message Types

FolderResourceInfo

Viewer sends current state:

• folder_id: Which folder to sync
• resource_ids: List of resources viewer currently has
• folder_ucan: Folder access token for validation

IncrementalSyncRequest

Viewer requests updates for existing resource:

• resource_id: Which resource to sync
• resource_ucan: Resource-specific token
• sync_data: Yjs state vectors (what viewer currently has)

IncrementalSyncResponse

Node sends missing updates:

• resource_id: Resource being synced
• sync_data: Yjs CRDT updates (what viewer is missing)

ViewerCommentsUpdate

Viewer sends comment changes back:

• resource_id: Resource with comments
• resource_ucan: Proof of write permission
• sync_data: Comment thread updates only

UCAN Token Validation

Every sync message includes and validates UCAN tokens:

Folder-Level (network/src/p2p/website_handler.rs:530-555):

// Extract folder_id from UCAN capabilities
let ucan_folder_id = extract_folder_id_from_ucan(&ucan)?;

// Verify it matches the requested folder
if ucan_folder_id != folder_id {
    return Err("Folder ID mismatch");
}

Resource-Level (network/src/p2p/website_handler.rs:686-710):

// Extract resource_id from UCAN capabilities
let ucan_resource_id = extract_resource_id_from_ucan(&ucan)?;

// Supports multiple formats:
// - "domain:resource:id"
// - "domain:resource:id:doc_type"  (3-doc architecture)

if ucan_resource_id != resource_id {
    return Err("Resource ID mismatch");
}

This validation ensures viewers can only sync resources they have permissions for.

Sync Efficiency

State Vector-Based:

• Only missing CRDT operations are transferred
• No redundant data transmission
• Efficient bandwidth usage

Permission-Aware:

• crud/read → Viewer receives updates, cannot send
• crud/write → Bidirectional sync for comments
• crud/append → Viewer can add, not modify existing

Incremental:

• Only new resources and changes are synced
• Complete re-sync not required on reconnection
• Scales efficiently with growing content libraries

Real-Time Update Flow

When publishers create new content, viewers can receive updates by actively polling the node through the incremental sync protocol:

Publisher Creates New Content

1. Create Resource: Publisher designs new website, blog post, or newsletter
2. Publish to Network: Content encrypted and filed on P2P network
3. Sync to Node: Publisher’s sovereign node receives the new resource via ResourceAdditionRequest

Viewer Polls and Receives Updates

1. Viewer Reconnects: Comes back online and actively initiates incremental sync
2. Send Folder Manifest: Viewer sends FolderResourceInfo with current resource list
3. Node Detects New Resources: Compares viewer’s list with node’s list
4. Node Responds with Resources: Node includes new resources in sync response via ResourceAdditionRequest (part of the response to viewer’s pull request)
5. Viewer Stores: New resources validated and stored locally

Important: The viewer initiates the sync request (pull). The node responds by including new resources in that response. The node never initiates connections or pushes data to viewers.

Key Distinction: Request vs Response

• Connection: Viewer always initiates (node never initiates connections)
• New Resources: Node includes them in sync response when viewer requests sync
• Existing Resource Updates: Viewer requests via state vector sync, node responds with diffs

Viewer Access Flow

Once viewer polls and receives new resources:

• Local First: Viewer opens content, loads from local database instantly
• Pull-Based Model: Viewer must actively poll node to receive future content publisher adds to folder
• No Push Notifications: Node doesn’t track viewers or send notifications - viewers control polling frequency

What Happens During Sync

1. Viewer Initiates: Viewer reconnects and establishes connection to node
2. Manifest Exchange: Viewer sends list of known resources (viewer pulls)
3. New Resources: Node responds with any resources viewer doesn’t have (in sync response)
4. State Reconciliation: Viewer requests updates for existing resources, node responds with Yjs state vectors
5. Update Application: Both sides apply missing CRDT operations
6. Comment Sync: Viewer sends comments back to node
7. Completion: Viewer is fully up-to-date with publisher’s content (until next poll)

The website sync protocol enables efficient, permission-aware content distribution while maintaining the sovereignty of both publishers and viewers.