Operate: day 2

Day-2 work splits in two: on-chain changes to your verifier contracts, run through the on-chain kit's make targets, and off-chain operations on the running cells (your Helm releases). This page maps what those operations are and where each is documented; the kits carry the exact commands.

Sign on-chain changes: Safe or EOA

The make targets that change on-chain state take OUTPUT_MODE, set to SAFE or EOA (there is no default).

  • SAFE, recommended for production: the target writes a Safe transaction batch under out/safe/ instead of broadcasting, and your multisig imports and executes it, so governance stays the owner of the contracts. See Safe batches.
  • EOA: the target broadcasts immediately, signed by whatever the SIGNER variable holds. SIGNER passes straight to forge script, so any Foundry wallet works: AWS KMS (--aws, the default), GCP KMS (--gcp), an encrypted local keystore (--account <name> or --keystore <path>), a hardware wallet, or --private-key for local use. Prefer a keystore or KMS over a raw private key. See Signing.

The deploy targets (factory, resolver, verifier) are EOA-only and always broadcast.

On-chain operations

Run these through the make targets in either mode above. The generated command reference lists every target with its required caller and variables, and the operations and governance pages walk the workflows.

  • Manage lanes. Add or update a lane's remote config on the source verifier (apply-remote-config), set per-lane sender allowlists (apply-allowlists), wire the resolver's inbound and outbound maps when you add a lane or a new verifier version (apply-inbound, apply-outbound), and pause a lane (pause-lane, which sets the router to zero for that destination on the source verifier and writes the change into the lane's config file to commit).
  • Manage the committee. Add, remove, or rotate committee signers on the destination verifier (apply-signature-configs), and set the allowed-finality cap (set-finality-config).
  • Manage roles and ownership. Set the verifier's fee aggregator and allowlist admin (set-dynamic-config), update its storage locations (update-storage-locations), and hand over control through the two-step transfers for the owner (transfer-owner, then accept-owner from the incoming signer) and the storage-locations admin (transfer-sla, then accept-sla).
  • Manage fees. Withdraw accrued fee-token balances to the aggregators (sweep-fees, permissionless); read balances first with balance-report.
  • Check for drift. Snapshot the live on-chain roles and settings (snapshot), and run the config and governance drift check on a schedule (drift).

Off-chain operations

These act on the running cells, not the contracts.

  • Upgrade and roll back. helm upgrade with a changed config or image tag restarts the pods (the chart stamps config and secret checksums) and the containers run their database migrations on startup, so expect brief downtime while the single pod is recreated. helm rollback restores the previous release but does not reverse migrations, so back up Postgres before you upgrade. Choose the image tag from the public ECR gallery.
  • Scale the committee. Add fault tolerance by running more cells, each its own Helm release with its own Postgres, secrets, and key custody, never by adding replicas (both StatefulSets are fixed at replicas: 1). See Scale to a committee.
  • Add or remove a cell. After you add or remove an aggregator, exchange the peer credentials (below) and re-send the aggregator endpoint list to the indexer; see Onboard to the CCIP indexer.
  • Exchange or rotate peer credentials. Every verifier-to-aggregator pair in the committee uses one HMAC key pair, so when you add or rotate a member you generate and distribute the pairs. The off-chain kit's peer information and credential exchange documentation has the layout.
  • Rotate the signing key. Follow Signer key custody and KMS for the Postgres keystore or cloud KMS.
  • Manage Postgres. Provision storage with headroom and alert on disk; the cell does not size the database for you. With the Postgres keystore, backups contain the signing key, so encrypt them and restrict who can read them.
  • Monitor. Collect metrics and set alerts; see Logging and monitoring.

Troubleshooting

The rows are ordered by where they surface, from deploy through onboarding.

SymptomLikely causeWhere to look
Green pods, no attestation everAn address field holds the CommitteeVerifier, not the resolverDeploy your first cell, step 2
Verifier crash-loops on startup, no chain sourcescommittee_verifier_addresses and on_ramp_addresses are not both set for the same chain selectorDeploy your first cell, step 2
Aggregator crash-loops on startupquorumConfigs or destinationVerifiers is emptyDeploy your first cell, step 2
Pods stuck in ContainerCreating, PermissionDenied (GCP) or AccessDenied (AWS)The ServiceAccount cannot read the secret; workload identity is not wiredDeploy your first cell, step 3
Pods stuck in ContainerCreating, serviceAccount.tokens not providedSecrets Store CSI driver installed without the tokenRequests audience (AWS)Deploy your first cell, step 3
Verifier crash-loops, AccessDenied on kms:Sign or kms:GetPublicKeyKMS keystore missing a grant; it needs Sign, GetPublicKey, and (on AWS) DescribeKey on the keySigner key custody and KMS
connection refused past a minute after deployAggregator not listening on port 50051, often a config with no [server] addressLogging and monitoring
Every request 503, Gateway Programmed: True, pods 1/1Load balancer health check on the default GET /, not readinessExpose the aggregator
Only Healthy in the logs, a message takes ~13 minutes (looks hung)finality_depth: 0 waits for the source chain's finality tag; nothing is wrongLogging and monitoring
CCIP API reports the verifier as UNKNOWNCommittee not onboarded yetOnboard to the CCIP indexer
policy_unavailable rising, messages not signedPolicy-hook endpoint down or timing outLogging and monitoring
Dropping task - policy hook returned FAIL in the logYour endpoint rejected the messageAdd a custom policy hook
Onboarded, but a message was never executedAll aggregators unreachable for over an hourOnboard to the CCIP indexer

What's next

Get the latest Chainlink content straight to your inbox.