Federated Learning Over P2P Communication

The FL Communication Bottleneck

A typical federated learning round works like this: each client trains a local model on its private data, computes gradients (or updated model weights), sends them to a central aggregation server, and receives the aggregated global model. The cycle repeats for hundreds or thousands of rounds.

The bottleneck is structural. Every client must upload its full gradient update to the same server. With a ResNet-50 model (about 100 MB of parameters) and 100 clients, the server must receive 10 GB of data per round. With a large language model at 7 billion parameters (roughly 14 GB in FP16), the numbers become absurd: 100 clients uploading 14 GB each means 1.4 TB of inbound traffic per round.

The server's inbound bandwidth becomes the chokepoint. Clients queue up waiting to upload. The slowest client (the straggler) determines when the round completes. And the server itself is a single point of failure -- if it goes down, all clients halt.

Gradient compression and quantization help reduce the per-client payload, but they do not fix the topology. The star topology -- every client talks to one server -- is the root cause. What you actually want is for clients to exchange gradients directly, peer to peer, and aggregate locally. This is decentralized federated learning, and its unsolved problem has always been communication: how do clients find each other, traverse NAT, and exchange data securely?

Why gRPC Falls Short

gRPC is the default transport for federated learning frameworks like Flower, PySyft, and TensorFlow Federated. It works well in controlled environments (all nodes in the same data center, no NAT, pre-configured TLS), but it breaks down in realistic FL deployments.

The 2 GB message size limit. gRPC defaults to a maximum message size of 4 MB, and even when you increase it, the practical ceiling is roughly 2 GB due to protobuf serialization constraints. A 7B parameter model in FP16 is 14 GB. You have to manually chunk, stream, reassemble, and handle partial failures. This is doable, but it is custom plumbing that every FL framework reimplements slightly differently.

No NAT traversal. gRPC requires the server to be reachable on a known IP and port. When FL clients are hospitals behind corporate firewalls, IoT devices on cellular networks, or laptops on residential WiFi, they cannot accept inbound connections. The standard workaround is a cloud-hosted relay server, which reintroduces the central bottleneck you were trying to avoid.

Server infrastructure required. Even for decentralized FL algorithms like gossip-based aggregation, you still need every node to run a gRPC server that other nodes can reach. This means public IPs, open firewall ports, and TLS certificate management for each participant. The operational burden scales linearly with the number of nodes, and most FL research teams do not want to become infrastructure teams.

No peer discovery. gRPC has no built-in mechanism for nodes to find each other. You need an external service registry, hardcoded peer lists, or a coordination server. In dynamic FL where nodes join and leave between rounds, this becomes its own distributed systems problem.

Pilot as FL Transport

Pilot Protocol is an overlay network for AI agents that runs over UDP. Each node gets a 48-bit virtual address, an Ed25519 identity, and automatic NAT traversal. For federated learning, it provides exactly the pieces that gRPC lacks: peer-to-peer connectivity through NAT, encrypted data exchange without per-node servers, and hostname lookup through the registry.

The architecture is straightforward. Each FL client runs a Pilot daemon alongside its training process. The daemon handles networking: NAT traversal via STUN, hole-punching, or relay (tried in that order until one works; the daemon does not classify the NAT type), encrypted tunnels using X25519 key exchange and AES-256-GCM, and peer discovery through a lightweight registry.

Gradient exchange happens over port 1001 (data exchange). This port supports file transfer and arbitrary message passing. A gradient tensor serialized to a file can be sent directly to a peer with a single CLI command or API call. send-file streams the file in acknowledged chunks and checks a SHA-256 over the whole file; the receiving daemon keeps at most 2 GiB of received files (including unfinished transfers) by default.

# Node A sends its gradient file to Node B
pilotctl send-file 0:0000.0002.0003 ./gradients_round_42.pt

# Node B sends its gradient file to Node A
pilotctl send-file 0:0000.0001.0001 ./gradients_round_42.pt

Once the two nodes trust each other (see the security section), both commands work whatever the NAT type. If both nodes are behind NAT, Pilot automatically negotiates hole-punching or falls back to relay through the beacon server. The gradient data is encrypted in transit with AES-256-GCM. No TLS certificates, no firewall rules, no cloud relay to configure.

Architecture: FL Nodes on Pilot

Here is the architecture for a decentralized FL system with N nodes, no central aggregation server, and gossip-based gradient exchange.

Each FL node consists of three components:

1. Training process -- PyTorch, TensorFlow, or JAX. Computes local gradients on private data.
2. Pilot daemon -- handles networking and runs in the background.
3. FL coordinator script -- orchestrates the round: exports gradients, sends them to peers, receives peer gradients, aggregates, and updates the local model.

The round proceeds as follows:

1. Each node trains on its local data and serializes gradients to a file.
2. Each node resolves its peers' addresses from their hostnames (see the discovery section).
3. Each node sends its gradient file to K peers (where K is a configurable fan-out).
4. Each node receives gradient files from its peers.
5. Each node locally averages the received gradients with its own.
6. Repeat from step 1.

There is no central server. The aggregation is fully distributed. Each node converges toward the global average through repeated local averaging, which is mathematically equivalent to centralized FedAvg under standard assumptions.

Code Example: Python FL Node with Gradient Exchange

Here is a Python FL node that trains a model, exchanges gradients with peers via Pilot, and aggregates. It uses subprocess to call pilotctl for file transfer and address lookup.

import subprocess
import json
import torch
import torch.nn as nn
import os
import time

# Simple model for demonstration
model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Linear(256, 10),
)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

# Participants in this FL task, agreed out of band (one hostname per node)
PARTICIPANTS = ["fl-hospital-a", "fl-hospital-b", "fl-clinic-c", "fl-clinic-d"]
MY_HOSTNAME = "fl-hospital-a"

def discover_peers():
    """Resolve the other participants' addresses through the registry."""
    peers = []
    for host in PARTICIPANTS:
        if host == MY_HOSTNAME:
            continue
        result = subprocess.run(
            ["pilotctl", "--json", "find", host],
            capture_output=True, text=True,
        )
        if result.returncode == 0:
            peers.append(json.loads(result.stdout)["data"]["address"])
    return peers

def send_gradients(peer_address, round_num):
    """Send gradient file to a peer over Pilot port 1001."""
    grad_path = f"/tmp/gradients_round_{round_num}.pt"
    torch.save(
        {k: v.grad for k, v in model.named_parameters() if v.grad is not None},
        grad_path,
    )
    subprocess.run([
        "pilotctl", "send-file", peer_address, grad_path,
    ])

def receive_gradients(round_num, expected, timeout=30):
    """Wait for this round's gradient files from peers."""
    # Files arrive in ~/.pilot/received/ as <name>-<timestamp>-<seq>.pt
    recv_dir = os.path.expanduser("~/.pilot/received/")
    prefix = f"gradients_round_{round_num}-"
    deadline = time.time() + timeout
    while True:
        names = []
        if os.path.isdir(recv_dir):
            names = [f for f in os.listdir(recv_dir)
                     if f.startswith(prefix) and f.endswith(".pt")]
        if len(names) >= expected or time.time() > deadline:
            return [torch.load(os.path.join(recv_dir, f)) for f in names]
        time.sleep(1)

def aggregate(local_grads, peer_grads_list):
    """Average local gradients with peer gradients (FedAvg)."""
    n = len(peer_grads_list) + 1
    averaged = {}
    for key in local_grads:
        total = local_grads[key].clone()
        for peer_grads in peer_grads_list:
            if key in peer_grads:
                total += peer_grads[key]
        averaged[key] = total / n
    return averaged

# Main FL loop
NUM_ROUNDS = 100
FAN_OUT = 3  # Send to 3 random peers per round

for round_num in range(NUM_ROUNDS):
    # Step 1: Local training
    train_one_epoch(model, optimizer, local_dataloader)

    # Step 2: Discover peers
    peers = discover_peers()
    selected = peers[:FAN_OUT]

    # Step 3: Exchange gradients
    for peer in selected:
        send_gradients(peer, round_num)

    peer_grads = receive_gradients(round_num, expected=len(selected))

    # Step 4: Aggregate
    local_grads = {
        k: v.grad for k, v in model.named_parameters()
        if v.grad is not None
    }
    averaged = aggregate(local_grads, peer_grads)

    # Step 5: Apply averaged gradients
    with torch.no_grad():
        for name, param in model.named_parameters():
            if name in averaged:
                param.grad = averaged[name]
    optimizer.step()

    print(f"Round {round_num}: aggregated with {len(peer_grads)} peers")

This is a simplified example. A production FL system would add round synchronization, gradient compression, Byzantine fault tolerance for malicious peers, and differential privacy noise injection. But the transport layer -- resolving peers, sending gradient files, receiving them, and doing this through NAT -- is handled by Pilot.

Discovery: FL Nodes Find Each Other by Hostname

In centralized FL, the server knows all clients because they registered with it. In decentralized FL, each node still needs to know who its peers are. With Pilot, every node registers a hostname, the participants in a task agree on the list of hostnames, and the registry turns each hostname into the node's current virtual address. Nodes can also carry up to three tags: short lowercase labels that the registry returns with a node's lookup. Tags describe a node; there is no search by tag.

# Give this node a stable name and label it
pilotctl set-hostname fl-hospital-a
pilotctl extras set-tags fl-node vision-model

# Resolve another participant
pilotctl find fl-hospital-b

find prints the peer's node ID, address and visibility:

Hostname:  fl-hospital-b
Node ID:   48213
Address:   0:0000.0000.BC55
Visible:   private

A private node resolves only for peers that trust it or share a network with it, so the trust handshake below comes first. If you run your own rendezvous, an operator can put each FL task's nodes in their own network; pilotctl network members <id> then lists every member, and its --json output includes each member's hostname and tags. Separate tasks use separate hostname lists or separate networks.

This replaces the registration server and the manual endpoint management that decentralized FL frameworks typically require: addresses stay the same when a node changes IP or network.

Security: Encrypted Gradient Exchange

Gradient leakage is a well-documented attack vector in federated learning. Research has shown that raw gradients can be inverted to reconstruct training data -- including images, text, and tabular records. If gradients are transmitted in plaintext over the network, any observer on the path can capture them and attempt reconstruction.

Pilot encrypts all data in transit using X25519 key exchange and AES-256-GCM. When two nodes establish a connection, they perform an ECDH handshake to derive a shared secret, then use that secret to encrypt every byte of payload data. This happens automatically, with no configuration.

Beyond encryption, Pilot's trust model adds access control. Nodes are private by default at the application connectivity layer: directory metadata can support discovery, while private endpoints and application traffic remain gated. For FL, nodes establish peer trust or share a network before gradient exchange begins.

# Node A requests trust with Node B (includes justification)
pilotctl handshake 0:0000.0002.0003 "FL gradient exchange for vision-model task"

# Node B approves (after verifying the request)
pilotctl approve 0:0000.0001.0001

Once trust is established, it persists across sessions. If a node is compromised or behaves maliciously (sending poisoned gradients), any peer can instantly revoke trust:

# Revoke trust from a suspected malicious node
pilotctl untrust 0:0000.0002.0003

Revocation is immediate. The revoked node can no longer send data to or receive data from the revoking peer. This provides a practical defense against model poisoning attacks at the transport layer: if your Byzantine fault detection algorithm flags a node, you can cut it off at the network level without waiting for the next aggregation round.

Comparison: Pilot vs gRPC vs Custom P2P

How three transport options compare for exchanging gradient files between two nodes that are both behind NAT:

The gRPC relay path sends every byte through a third server instead of directly between peers. When hole-punching succeeds, Pilot's path is direct and avoids that extra hop; when it does not, Pilot relays through the beacon too. A custom UDP approach can avoid both, but it needs its own reliability, congestion control, and NAT traversal code -- the code that Pilot already provides.

For large models (7B+ parameters, 14+ GB), gRPC requires chunking and streaming logic that adds complexity and failure modes. Pilot's send-file already streams in acknowledged chunks, verifies a SHA-256 over the whole file, and resumes where it stopped when a dropped transfer is sent again. Because a receiver holds at most 2 GiB of received files by default, split full checkpoints of that size into shards and clear the received directory between rounds (pilotctl received --clear).

Getting Started

To set up an FL node with Pilot transport:

# Install Pilot on each FL node
curl -fsSL https://pilotprotocol.network/install.sh | sh

# Start the daemon
pilotctl daemon start --email fl-node@example.com

# Name and tag this node as an FL participant
pilotctl set-hostname fl-hospital-a
pilotctl extras set-tags fl-node vision-model

# Ask another FL node for trust (its operator runs pilotctl approve)
pilotctl handshake 0:0000.0002.0003 "FL gradient exchange for vision-model task"

# Verify connectivity to that node
pilotctl ping 0:0000.0002.0003

# Exchange a test file
pilotctl send-file 0:0000.0002.0003 ./test_gradients.pt

On the public backbone, nodes talk to each other directly once they trust each other; no network needs to be joined. Private networks are created by a registry operator, so to group FL nodes into one, run your own rendezvous. The daemon handles NAT traversal automatically. You do not need to configure firewall rules, open ports, or set up relay servers. If both nodes are behind symmetric NAT (the hardest case), Pilot falls back to relay through the beacon server transparently.

For a full decentralized FL system, integrate the pilotctl commands into your training loop as shown in the Python example above, or use the Go driver and dataexchange packages directly for tighter integration:

import (
    "fmt"
    "os"
    "path/filepath"

    "github.com/pilot-protocol/common/driver"
    "github.com/pilot-protocol/common/protocol"
    "github.com/pilot-protocol/dataexchange"
)

func exchangeGradients(peerAddr string, gradFile string) error {
    d, err := driver.Connect("") // default daemon socket
    if err != nil {
        return err
    }
    defer d.Close()

    addr, err := protocol.ParseAddr(peerAddr)
    if err != nil {
        return err
    }
    // Dial the peer's data-exchange service (port 1001)
    c, err := dataexchange.Dial(d, addr)
    if err != nil {
        return err
    }
    defer c.Close()

    f, err := os.Open(gradFile)
    if err != nil {
        return err
    }
    defer f.Close()
    st, err := f.Stat()
    if err != nil {
        return err
    }

    // Chunked, acknowledged, SHA-256-verified transfer; a retry resumes
    res, err := c.SendFileStream(filepath.Base(gradFile), f, st.Size(), 0)
    if err != nil {
        return err
    }
    if !res.OK {
        return fmt.Errorf("receiver refused the file: %s", res.Message)
    }
    return nil
}

Communication overhead is the dominant bottleneck in federated learning. Removing the central server from the communication path, enabling direct peer-to-peer gradient exchange, and handling NAT traversal automatically removes the central server's inbound link as the bottleneck. Pilot provides the transport layer so you can focus on the learning algorithms instead of the networking plumbing. Once nodes can reach each other directly, the next question is how they find and verify each other in the first place — see the AI agent discovery process on P2P networks for a step-by-step walkthrough of capability announcement and trust verification.