Skip to main content

COG-4: Grid Lattice

Status:      Draft (Work in Progress)
Version:     0.1
Created:     2025-01-23
Authors:     Mike Anderson
Work in Progress

This specification is under active development. Structure and details may change significantly based on implementation experience and community feedback.

This standard specifies the lattice-based state management system for the Covia Grid, enabling conflict-free replication and synchronisation of venue state across the distributed network.

Purpose​

The Grid requires a robust mechanism for managing distributed state across autonomous venues that may operate independently, experience network partitions, and need to reconcile state after periods of disconnection. Traditional database synchronisation approaches are inadequate for this use case because they typically require:

  • Central coordination or consensus mechanisms
  • Strict ordering of operations
  • Complex conflict resolution logic

The Grid Lattice provides an alternative approach based on Conflict-free Replicated Data Types (CRDTs), which guarantee eventual consistency without coordination. This enables venues to:

  • Operate autonomously during network partitions
  • Merge state from multiple sources without conflicts
  • Maintain cryptographic integrity of shared data
  • Support efficient peer-to-peer state synchronisation

Key Benefits​

Eventual Consistency Without Coordination​

Lattice-based state management ensures that all venues converge to the same state regardless of the order in which updates are received. This is achieved through mathematically-defined merge operations that satisfy the CRDT properties:

  • Commutative: merge(a, b) = merge(b, a)
  • Associative: merge(merge(a, b), c) = merge(a, merge(b, c))
  • Idempotent: merge(a, a) = a

These properties guarantee that venues can exchange state in any order, through any topology, and arrive at consistent results.

Content-Addressable Metadata​

Asset metadata on the Grid is content-addressable, meaning the identifier (Asset ID) is derived from the SHA256 hash of the metadata itself. This design ensures:

  • Integrity verification: Any modification to metadata changes its hash, making tampering detectable
  • Deduplication: Identical metadata across venues shares the same identifier
  • Safe sharing: Content-addressed data can be freely replicated without risk of conflicts

Hierarchical State Organisation​

The lattice is organised hierarchically, allowing different merge semantics at each level:

  • Grid-level state manages the collection of venues
  • Venue-level state manages assets and jobs independently
  • Asset and job entries use appropriate merge strategies for their data types

Autonomous Venue Operation​

Each venue maintains sovereignty over its own state while participating in the broader Grid. The lattice structure ensures that:

  • Venues can operate during network partitions
  • State from multiple venues can be merged without data loss
  • No central authority is required for state reconciliation

Specification​

Lattice Structure​

The Grid Lattice follows a hierarchical structure rooted at the :grid keyword:

:grid                                    ; GridLattice
  :venues                                ; Index<DID-String, VenueLattice>
    <venue-did-string>                   ; VenueLattice
      :assets                            ; Index<Hash, AssetRecord>
      :jobs                              ; Index<JobID, JobRecord>
      :storage                           ; Index<Hash, Blob> (content-addressed blob storage)
  :meta                                  ; Index<Hash, JSON-String>

Note: The :venues collection uses Index rather than Map for efficient sorted access with blob-like DID string keys.

Root Lattice​

The root lattice (Covia.ROOT) is a KeyedLattice that maps the :grid keyword to the GridLattice instance.

Implementations MUST use the :grid keyword as the root key for Grid state.

Grid Lattice​

The Grid Lattice manages global state with two primary components:

Venues Index (:venues)​

The :venues field contains a sorted index from venue DID strings to venue state:

  • Type: Index<AString, VenueLattice> (sorted by DID string)
  • Key: Venue DID as a string (e.g., "did:web:venue-1.covia.ai")
  • Value: Venue state managed by VenueLattice
  • Merge semantics: Per-venue merge using VenueLattice rules

The use of Index rather than Map provides efficient sorted access and better performance for blob-like DID string keys.

Implementations MUST apply VenueLattice merge semantics when merging venue entries with the same DID.

Shared Metadata (:meta)​

The :meta field contains content-addressable metadata shared across venues:

  • Key: SHA256 hash of the metadata JSON string
  • Value: Metadata as a JSON string
  • Merge semantics: Union merge (content-addressed, same hash = same content)

Implementations MUST use SHA256 hashing for metadata keys.

Implementations SHOULD NOT modify existing metadata entries (they are immutable by design).

Venue Lattice​

Each venue's state is managed by a VenueLattice with the following structure:

Assets (:assets)​

The :assets field contains the venue's asset registry:

  • Key: Asset ID (SHA256 hash of asset metadata)
  • Value: Asset record (implementation-defined structure)
  • Merge semantics: Union merge (content-addressed assets are immutable)

Implementations MUST use union merge for assets, accepting all entries from both states.

Jobs (:jobs)​

The :jobs field tracks job execution state:

  • Key: Job ID (unique identifier)
  • Value: Job record containing status, timestamps, and results
  • Merge semantics: Timestamp-based merge (newer status wins)

Implementations MUST include an updated timestamp field in job records.

Implementations MUST use the updated timestamp to resolve merge conflicts, preferring the newer record.

Storage (:storage)​

The :storage field provides content-addressable blob storage for the venue:

  • Type: Index<Hash, Blob>
  • Key: SHA256 hash of the content
  • Value: Binary blob data
  • Merge semantics: Union merge (content-addressed, same hash = same content)

This storage is used for asset content, attachments, and other binary data associated with the venue. The content-addressed design ensures:

  • Deduplication: Identical content is stored only once
  • Integrity: Content can be verified against its hash
  • Safe replication: Content can be freely shared without conflicts

Implementations MAY use alternative storage backends (file system, DLFS, etc.) while maintaining the content-addressed semantics in the lattice.

Venue Identification​

Venues on the Grid are identified using Decentralised Identifiers (DIDs) as specified in COG-2.

A venue's DID serves as its unique key in the :venues map. This design ensures:

  • Global uniqueness: DIDs provide globally unique identification
  • Self-sovereignty: Venues control their own identity
  • Verifiability: DID resolution enables verification of venue authenticity

Implementations MUST use the venue's canonical DID string as the key in the :venues map.

A venue MAY have multiple equivalent DIDs, but MUST consistently use its canonical DID as the lattice key.

Merge Operations​

Null Handling​

The following rules apply when merging with null values:

  • merge(value, null) = value
  • merge(null, value) = value
  • merge(null, null) = null

Identity Element​

The zero (identity) element for each lattice level:

  • Grid Lattice: {:venues (empty-index), :meta (empty-index)}
  • Venue Lattice: {:assets {}, :jobs {}, :storage (empty-index)}

Implementations MUST ensure that merge(value, zero) = value for all valid states.

State Synchronisation​

Venues MAY synchronise state using any transport mechanism that preserves the integrity of lattice values.

When synchronising, venues SHOULD:

  1. Exchange lattice state or deltas
  2. Apply the merge operation to combine local and remote state
  3. Persist the merged result

Implementations MUST NOT require strict ordering of synchronisation messages.

Implementations SHOULD support incremental synchronisation to minimise bandwidth.

Security Considerations​

Metadata Integrity​

Content-addressable metadata provides built-in integrity verification. Implementations SHOULD verify that metadata hashes match their content before accepting updates.

Venue Authority​

Only the venue identified by a DID should be authoritative for its state under that DID in the :venues map. Implementations SHOULD implement mechanisms to prevent unauthorised modification of venue state.

Timestamp Manipulation​

Timestamp-based merge for jobs is vulnerable to manipulation by actors with incorrect or malicious clocks. Implementations MAY implement additional validation or use signed timestamps for critical job state.

Reference Implementation​

The reference implementation is available in the Covia venue module:

  • covia.lattice.Covia - Root lattice definition
  • covia.lattice.GridLattice - Grid-level lattice implementation
  • covia.lattice.VenueLattice - Venue-level lattice implementation
  • covia.lattice.CASLattice - Content-addressable storage lattice
  • covia.venue.storage.LatticeStorage - Storage backed by lattice cursor
  • covia.venue.storage.FileStorage - Storage backed by filesystem (supports DLFS)
  • covia.venue.Config - Configuration constants for storage and venue setup