FortiGate ADVPN Failover: Why an Available Backup Tunnel Did Not Restore the Site

QCS's public account describes a referred site-down incident in an ADVPN 2.0 network with two hubs, multiple IPsec overlays, SD-WAN and BGP peering over loopbacks. A backup tunnel was available, yet the branch did not recover during WAN failover. The investigation compared SLA/member health, BGP neighbors, kernel entries and BGP egress.

It found that the probe destination and hub BGP peer address were the same. The account reports recovery after moving the probe to a separate hub-side loopback.

That is the reported outcome, not a new measured result from this article. The public post does not disclose the customer's exact FortiOS build, complete topology, routing policy, timers or convergence measurements. No FortiGate, GNS3 or networking lab was operated for this guide.

The diagrams are original conceptual teaching aids, not a reconstruction of the customer's two-hub design. The operational worksheets and proposed checks below add practical structure without pretending those missing details are known. [1]

Fortinet's staff guidance for FortiOS 7.0 and later explains a specific BGP-over-loopback and SD-WAN interaction. Probe destinations receive kernel routes over the participating members. Those routes remain for probing even when a health check fails. Reusing that destination for the hub BGP peer can therefore leave BGP using its previous tunnel rather than the available alternative. The documented design separates the health-check destination from the peer address.

A useful mental model has three questions. Is there usable transport? Can the routing peers communicate on the intended path? Can the required application complete its transaction? Passing one question does not answer the next. This is a troubleshooting framework, not a universal internal packet-processing order.

Do not generalize the finding into 'loopbacks are unsafe' or 'ADVPN 2.0 is broken'. A loopback is a stable identity, not a promise of reachability. This case concerns the relationship between two destination roles. The same outward symptom could also arise from a different peer source, routing policy, an unavailable application or another dependency. Test the hypothesis against the actual evidence. [2][5]

Step 1: Confirm authorization, incident owner and an affected business transaction. Record branch, hub, VDOM/VRF, exact build and the time of the failure. Name the original and alternate overlays without publishing customer addresses. Record whether management access traverses the same network being investigated.

Step 2: Preserve a supported configuration backup and an independent recovery path. Agree a maintenance window before deliberately interrupting any link. Do not disable interfaces, clear tunnels or reset BGP merely to collect a more convenient screenshot. If the current outage already provides the failure condition, collect observations without adding another disruption.

Step 3: Build a destination-role worksheet. For each hub, record the BGP neighbor address, local peer source, probe destination, probe protocol, source address and participating SD-WAN members. Compare the actual values, not just similarly named objects. Mark whether two roles share one address. Distinguish what is configured from what is currently observed.

Step 4: Create a timestamped baseline for tunnel state, member/SLA status, BGP state, route toward the peer and an approved application check. Store evidence privately and label gaps. Missing data must stay 'not observed', not become a zero-loss or successful-failover claim. [2][3]

Use the authorized FortiGate console, not a desktop command prompt, and confirm the administrative context first. The command cards below are inspection examples supported by the linked documentation. Syntax and available output vary by release. Use the installed build's help and stop on an unavailable command or permission error; do not grant yourself wider access.

Step 5: Compare health-check results with BGP state and the route toward the actual peer. A live member does not prove the intended SLA rule or application route is selected. BGP establishment alone does not establish that the required prefix is present. Examine the particular destination and service named in your incident record.

Step 6: If authorized, capture a bounded sample of BGP traffic with a narrow peer filter. Interface labels and source/destination addresses help test the egress hypothesis. A packet capture is observation, not a reason to reset a working peer. Never leave a capture running indefinitely while waiting for traffic. Treat addresses, topology and packet metadata as sensitive operational information.

For a large route table, prefer the target-specific routing query first. Review the kernel view only when it is needed to distinguish protocol routing from installed probe-related entries. Save the relevant excerpt privately; do not paste the entire routing table into a public ticket or online tool. [4][7][6]

The shared-destination hypothesis is credible when the configured peer and probe destination match, the observed failure is consistent with the vendor-described path behavior, and the route/capture evidence supports that relationship. If the addresses already differ or BGP is using the expected healthy overlay, investigate another cause rather than making a cosmetic loopback change.

Check peer source and interface constraints as a separate branch of the investigation. Fortinet documents that update-source chooses the source address while an interface setting can constrain where a neighbor is expected. Inspect these settings against the approved design; do not simply remove them. A received packet on an unexpected interface may be important evidence, but is not an automatic instruction to relax the configuration.

Write the working conclusion as a falsifiable statement: which observation would contradict it, and what would remain broken after the proposed correction? For example, restoring BGP but not the required application would mean the incident is not fully resolved. This discipline prevents a clean control-plane screenshot from hiding an ongoing service outage. [5][7]

For a confirmed shared-target condition, design a distinct reachable probe endpoint appropriate to the intended test. QCS's source reports a separate hub-side loopback; that is one implementation, not a requirement to copy an undisclosed customer address. Do not replace the probe with an arbitrary public host that measures a different service path. Keep the BGP peer identity unchanged unless the approved design separately requires its migration.

Step 7: Before changing the probe destination, verify ownership of the new address, protocol response, source selection and return reachability over each intended overlay. For an unnumbered tunnel, source selection needs particular attention; the vendor example explicitly supplies the source for its probes. Do not copy its sample addresses into production. Check the applicable access controls without opening unrestricted management access.

Step 8: Inventory every health check and SD-WAN rule that depends on the object. Approve a scoped edit, record the old values and preserve the unrelated members, thresholds and routing policy. Use the exact release's supported workflow, read back the saved result, and verify the new endpoint before broader rollout. A valid configuration is not proof of a responsive target.

For dual hubs, repeat the role and reachability checks for each hub independently. A passing test toward one hub does not validate the other hub or a spoke-to-spoke shortcut. The diagram below intentionally shows destination roles for one logical hub only. [2][3]

Agree an explicit test matrix and outage budget with the service owner. Cover the healthy baseline, each approved single-overlay failure, restoration and any separately approved hub failure. Do not cause a second fault while the first has not recovered. Assign an observer, recovery owner and stop condition for each test. Planned tests are not evidence until they have actually been executed.

For each row, record health-check behavior, BGP neighbor state, route toward the peer, observed BGP egress and the required application result in both directions. Check return-path behavior from the appropriate other end. A tunnel-up indicator alone is insufficient. Confirm the intended branch prefixes and policy behavior rather than assuming that an established session carries all required routes.

Measure interruption only with real timestamps and a defined transaction. Distinguish a new application connection from a pre-existing session, and record either as untested when appropriate. Do not publish a convergence-time promise based on this case; no timing measurement is provided. Probe routes may legitimately remain after failure. Their mere existence is not the criterion for failure: evaluate their destination role and the observed path.

After restoration, verify that the network returns to its intended steady state without flapping. Keep the evidence matrix and residual limitations with the change record. If only the control connection has recovered, leave application validation open. [4][2][1]

Define recovery before editing. Preserve the old configuration, but do not automatically reinstate a known harmful shared destination during an outage. If the new probe is unreachable or service health worsens, stop further rollout and let the incident owner select the safest supported recovery using the recorded baseline. Restore only the agreed scope; a full-device restore can overwrite unrelated work.

If the evidence does not match this mechanism, broaden the investigation with the responsible engineer. A source-address mismatch, different route selection, tunnel negotiation problem or application-side failure needs its own proof. The diagnostic cards deliberately exclude tunnel resets, BGP clears and continuous debug. Those are not beginner-safe substitutes for a reasoned hypothesis.

Escalate with exact builds, a sanitized topology, destination-role worksheet, relevant route and capture excerpts, the tested failure condition and the observed result. Keep credentials and complete customer configuration out of public channels. The reusable lesson is to validate transport, control plane and service separately. QCS can assist with the investigation, but neither a new loopback nor a single command guarantees recovery in every ADVPN design. [7][6]