New architecture makes neural network merging conflict-free across 26 strategies

A new paper from researcher Ryan Gillespie tackles a fundamental flaw in neural network model merging: all 26 existing merge strategies — including weight averaging, SLERP, TIES, DARE, Fisher merging, and evolutionary approaches — fail to satisfy the commutative, associative, and idempotent properties required for conflict-free replicated data types (CRDTs). This means that when multiple replicas merge model updates in different orders, they can produce different results, breaking consistency in distributed training or federated learning settings. The paper proves this failure is structural: normalization-based merges cannot simultaneously satisfy all three properties.

To solve this, Gillespie presents CRDTMergeState, a two-layer architecture that wraps any merge strategy in a CRDT-compliant layer. Layer 1 manages contributions via an OR-Set CRDT, where the merge operation is set union — trivially commutative, associative, and idempotent. Layer 2 applies the actual merge strategy as a deterministic pure function over a canonically-ordered contribution set, with randomness seeded from a Merkle root. This separation guarantees Strong Eventual Consistency: all replicas receiving the same contributions compute identical merged models, regardless of message ordering. Empirical validation covers controlled 4x4 tensors (104/104 tests passed), production-scale models up to 7.24B parameters (208 strategy-level tests, 43,368 layer-level property checks), and multi-node convergence under gossip and partition healing (100 nodes, 20 orderings), with CRDT overhead below 0.5 ms. The reference implementation is available as crdt-merge v0.9.4.