Troubleshooting Wi-Fi at Black Hat USA 2026 with ThousandEyes

Published On: September 7, 2026

Troubleshooting Wi-Fi at Black Hat USA 2026 with ThousandEyes

The Community Operations Middle at Black Hat USA 2026 is a collaborative expertise between a number of enterprise companions to convey one of the best of community efficiency, optimization, and safety to the workers and attendees in the course of the coaching and convention days in Las Vegas, Nevada. Over 150 Wi-Fi 7 Entry Factors have been deployed onsite to offer normal convention wi-fi and particular person wi-fi service for the trade trainings. A big enterprise-scale community like this was supporting over 20,000 attendees; for such a system community monitoring was wanted to confirm that the community was working easily and that attendees and college students are in a position to entry core enterprise providers corresponding to AWS, Azure, and GCP.

To deal with this process, Cisco and Black Hat deployed bespoke ThousandEyes brokers to observe the wi-fi resolution. These consisted of greater than 30 small community screens that have been deployed across the Mandalay Bay Conference Middle and Enterprise Corridor in strategic areas the place site visitors could be highest and the community reliability most necessary.

These gadgets, together with a customized dashboard resolution, offered important perception into the efficiency of core capabilities corresponding to DNS, throughput, cloud response time and extra. This allowed our crew on the NOC to behave proactively on wi-fi and wired issues earlier than customers reported points.

As a part of this technique, a few of our troubleshooting crossed a number of totally different merchandise from totally different firms. This resulted in new troubleshooting strategies and procedures that improved our monitoring resolution of prior years.

Roaming Points

For the primary three days of the convention, roaming was not an excessive amount of of a consideration for the ThousandEyes nodes, as they have been every related to an SSID on a singular AP. On this situation, roaming shouldn’t be attainable since there are usually not a number of BSSIDs to roam to. Shifting onto the ultimate three days of Briefings, we dismantled the classroom SSID design and positioned the ThousandEyes nodes onto the final Wi-Fi community.

Throughout this time, it was noticed that one of many nodes was experiencing degraded obtain speeds. We might see in ThousandEyes that the shopper had a pointy drop-off in throughput on one of many obtain checks.

At this level we would have liked further metrics to find out why the shopper would immediately be experiencing points with throughput. To triage additional, further perception was wanted from the Arista aspect. Logging into the Arista CloudVision dashboard (by way of Duo Listing, the NOC id supplier) we might lookup the shopper by hostname or MAC deal with. Right here we discovered that the shopper had made a roaming resolution and moved to a special Entry Level.

This prompted us to research the ThousandEyes agent itself and led us to our first lesson discovered. Upon reviewing the logs, we discovered that the shopper had a beacon loss occasion; it was not in a position to hear its related AP for a time frame, so it moved to an AP with higher sign high quality.

Aug 04 10:17:20 wlan0: CTRL-EVENT-BEACON-LOSS
Aug 04 10:17:25 wlan0: CTRL-EVENT-BEACON-LOSS
Aug 04 10:17:26 wlan0: CTRL-EVENT-DISCONNECTED bssid=XX:XX:XX:XX:6f:12 motive=4 locally_generated=1
Aug 04 10:17:34 wlan0: SME: Attempting to authenticate with XX:XX:XX:XX:57:52 (SSID='SSID' freq=5520 MHz)
Aug 04 10:17:34 wlan0: CTRL-EVENT-CONNECTED - Connection to XX:XX:XX:XX:57:52 accomplished

The Arista dashboard corroborated this discovering with an uptick in site visitors quantity and influence to the information charge and RSSI (Obtained Sign Power Indicator) on the similar time. So, at the moment, one thing occurred that made the BSSID XX:XX:XX:XX:6f:12 unavailable to the ThousandEyes node. By the point we ran this scan, that BSSID was wholesome once more at -45 dBm, in keeping with a transient occasion on the entry level.

On the TE Agent, we might see that though there have been extra optimum Entry Factors obtainable, it was caught to a comparatively weak BSSID.

te-user@te-agent:/var/log$ sudo iw dev wlan0 scan | awk -v need="SSID" '
/^BSS/{bssid=$2}
/freq:/{fr=$2}
/sign:/{sig=$2}
/SSID:/{ssid=substr($0,index($0,"SSID: ")+6);
if (ssid==need) printf "%-8s dBm ch/%-5s %-18s %sn", sig, fr, bssid, ssid}'

-50.00 dBm ch/2462 XX:XX:XX:XX:6e:f1(on SSID
-65.00 dBm ch/2462 XX:XX:XX:XX:57:31(on SSID
-75.00 dBm ch/5600 XX:XX:XX:XX:65:b2(on SSID
-79.00 dBm ch/5785 XX:XX:XX:XX:00:71(on SSID
-69.00 dBm ch/5520 XX:XX:XX:XX:57:52(on SSID << Joined AP
-78.00 dBm ch/5520 XX:XX:XX:XX:51:62(on SSID
-70.00 dBm ch/5580 XX:XX:XX:XX:6b:52(on SSID
-70.00 dBm ch/5580 XX:XX:XX:XX:a4:82(on SSID
-45.00 dBm ch/5200 XX:XX:XX:XX:6f:12(on SSID << Optimum AP
-65.00 dBm ch/2462 XX:XX:XX:XX:6b:31(on SSID

A 24 dB distinction is much past any threshold a shopper would usually must justify a roam. The rationale it stayed put is that it by no means went trying.

The ThousandEyes brokers ship with the variable bgscan=”easy:30:-70:86400″ configured. bgscan is a wpa_supplicant module that NetworkManager units on our brokers. The values on this variable are as follows:

Easy is the background scanning module getting used. It should describe the values following this.
30 is how usually a scan will happen if the minimal threshold shouldn’t be met. So, if a sign is acquired at -71 dBm or weaker, wpa_supplicant will scan the air each 30 seconds in search of a greater BSSID to roam to.
-70 dBm is the edge to set off a frequent seek for a brand new AP. If the RSSI of the AP is acquired at -70 dBm or stronger, we are going to scan each 86400 seconds or each 24 hours.
86400 seconds is how usually the wi-fi will scan whereas the sign is stronger than -70 dBm.

We are able to see within the CLI output above that we’re simply barely sitting above this threshold at -69 dBm. Which means we is not going to roam to a extra optimum AP inside a 24-hour window.

Our lesson discovered is to lift the sign threshold, shifting -70 dBm to -65 dBm within the bgscan string, so the agent begins scanning earlier than it will get caught on a marginal BSSID.

Wired Pace Negotiation on the Entry Level

For an prolonged time frame, we have been seeing throughput degradation for the Wi-fi Shoppers in one of many school rooms within the Mandalay Bay Conference Middle within the ThousandEyes Dashboard. This was a case the place there was just one AP and one BSSID being broadcast for a coaching.

The ThousandEyes shopper and different purchasers on the AP had all seen appreciable efficiency degradation within the latter half of the afternoon. On website, we have been in a position to verify that they have been unable to succeed in anticipated speeds.

Utilizing the Arista CloudVision dashboard, the AP’s switchport particulars confirmed the hyperlink had negotiated at 100 Mbps as a substitute of 1 Gbps.

As soon as the Arista crew was notified, they have been in a position to act; a fast cable swap resolved the difficulty for the scholars and restored full service.

Conclusion

Neither of those issues would have registered as an outage. The roaming agent held a steady affiliation with good sign and a 0.06% retry charge; the classroom entry level reported an lively hyperlink, profitable authentication, and negotiated PoE. On the infrastructure aspect, every thing was inexperienced. Solely the throughput measured from the shopper’s place informed a special story.

That’s the case for artificial monitoring at an occasion like this. Infrastructure telemetry tells you a tool is working. Steady testing from the ground tells you the community is working for the folks standing on it. As soon as ThousandEyes informed us which shopper and when, our colleagues at Arista might inform us why, and neither half of that reply was price a lot with out the opposite.

We’re carrying two modifications ahead: tune background-scan thresholds on stationary probes, and alert on negotiated hyperlink pace fairly than hyperlink state alone.

Our due to the Arista Networks crew within the Black Hat NOC for his or her partnership all through the week.

You may learn the opposite blogs from our colleagues at Black Hat USA.

About Black Hat

Black Hat is the cybersecurity trade’s most established and in-depth safety occasion sequence. Based in 1997, these annual, multi-day occasions present attendees with the newest in cybersecurity analysis, growth, and tendencies. Pushed by the wants of the group, Black Hat occasions showcase content material immediately from the group by means of Briefings displays, Trainings programs, Summits, and extra. Because the occasion sequence the place all profession ranges and tutorial disciplines convene to collaborate, community, and focus on the cybersecurity subjects that matter most to them, attendees can discover Black Hat occasions in the US, Canada, Europe, Center East and Africa, and Asia. For extra info, please go to the Black Hat web site.

Supply hyperlink