Â̾ÞÈËÊÓƵ

Spring and Summer 2023: Preparing for a Wi-Fi system overhaul

Â̾ÞÈËÊÓƵ has forÌýmany yearsÌýprovided fast, reliable internet connectivity to the campus community in all buildings.Ìý In early 2023, our wireless system was still providing speeds thatÌýmet and exceeded the requirements ofÌýthe campus community.Ìý However, the equipment being used on-campus was soon to beÌýdiscontinued by its vendor, meaning that Â̾ÞÈËÊÓƵ would need to transition to a new Wi-Fi system by the end of the year to avoid security issues (possible security holesÌýthat would no longer be resolved by the Wi-Fi system's manufacturer) and reliability issues (lack of availability of replacement parts).Ìý As such, IT Services initiated the process of searching among the availableÌýWi-Fi solutions on the marketÌýto begin the process of finding a suitable replacement Wi-Fi system.
Ìý

Defining our needs

Although the existing Wi-Fi system (based on what is known as "Wi-Fi 5" technology)Ìýwas meeting the current needs of the community, we looked for opportunities to implement a cutting-edge system that would meet not only the current needs of the community, but that would meet the needs of theÌýequipment and applications that would be used over the upcoming years.Ìý In addition to the features that our current system offered, we therefore wished to add:

• Wi-Fi 6E compatibility
  • Wi-Fi 6E is a version of Wi-Fi 6 that uses an extra band of radio frequencies in the 6 GHz range, which would allow for much better performance than the existing system that used only the 5 GHz and 2.4 GHz bands.Ìý This allows much faster speeds, and will be noticeable as more Wi-Fi 6E-compatible devices appear on our campus.
• Increased number of access points for better signal
  • Although IT Services has added extra Wi-Fi access points in the past when members of our community have had trouble connecting in certain areas, we proactively scouted the campus for areas with weaker-than-optimal signal, and determined how many access points we'd need to add to cover all of these zones.
  • In the end, we determined a need for just over 80Ìýaccess points.Ìý Â̾ÞÈËÊÓƵ's wireless deployment is relatively dense, with almost every public space and classroom havingÌýits own access point (or more than one, in the case of some lobby areas where demand is high).
• "Fast Roaming"
  • 802.11r "Fast Roaming" technology allows compatible devices to roam from one Wi-Fi access point to another with virtually no drop in the connection, which provides a major experience improvement over the older system where roaming between access points could cause a connection drop lasting severalÌýseconds.
    Ìý

Contacting vendors, and performing tests

Having determined the specifications we needed to provide our community with an excellent Wi-Fi experience, we contacted four market-leading vendors, detailed our needs, looked over the solutions that were offered.Ìý We pursued conversations with three vendors and ended up getting equipment from two of those vendors that we could test.

Tests were performed in the busiest location on campus during the summer semester, the Amex Café, allowing us to see how the access points would behave with a high amount of users.Ìý After working through some technical issues with some of the test devices, we found a solution that performed very well and suited our needs.
Ìý

Late summer 2023: Replacing the Wi-Fi system

With the assistance of Â̾ÞÈËÊÓƵ's internalÌýmaintenance team, IT ServicesÌýreplaced the entire Wi-Fi system in mid-August, mounting over 80 access points throughout the campus's seven buildings.Ìý After initial tests, everything seemed to be working very well: speeds and connection quality were excellent.
Ìý

Fall 2023:ÌýAlong with the crowds came the trouble

When students and faculty arrived on campus, we were thrilled to be able to offer a new, upgraded Wi-Fi experience, after having tested the new equipment in a busy location during the summer semester and having no problems before the arrival of students.Ìý However, after the arrival of the entire student body on-campus, we encountered two issues:

• Four access points would randomly restart, taking 2-3 minutes to reboot.
• We received complaints of generally unreliable network connectivity on-campus, but were unable to pin down the origin of the problem, considering that the system had been functioning perfectly just a few days before - and we could not usually make the problem occur on-demand.
  Ìý

Access points restarting - what was the cause?

We opened a support request with the equipment supplier to ask why the access points were restarting.Ìý After all, this was a brand-new system, and even a high amount of stress on the equipment shouldn't make the device restart.Ìý The vendor was slow to respond, so we did investigations on our own, looking into what might cause this:

• Basic checks?
  • All cables were checked, power supplies were added, even network infrastructure was changed. Nothing we changed improved the situation, eliminating the possibility that it was a silly, obvious problem like a bad cable.
• Defective access points?
  • Next was the suspicion that the four access points being used were victims of a manufacturing defect. We replaced all four access points with spares, and they all continued to have the same issues.
• A bug in their firmware?
  • Originally, our Wi-Fi equipment had not worked well with some of our existing network equipment, but the issue was resolved by an update from the manufacturer.Ìý We suspected that perhaps the bug hadn't been properly fixed, and moved around equipment in our wired network to test this hypothesis, but it didn't seem to make a difference.
• Location?
  • We swapped the locations ofÌýan in-service access point that did not suffer any issuesÌýwith one that was suffering from random reboots... and the issue followed the installation location itself, not the access point.Ìý With this information, we suspected that the access points were having an issue with something in the specific affected locations.

We provided ongoing updates to the vendor, but their support was not helpful.Ìý Their support did confirm, after several weeks of data collection, that they discovered that the software in the access points was access points wasÌýcrashing on their own, and that it was not a fault in our network that caused the issue.Ìý Although this provided welcome reassurance that we had not incorrectly configured the system, it didn't fix this annoying problem.
Ìý

General performance issues - or, "the difficulty in finding a problem that you can't recreate"

We received complaints of bad connectivity in many areas (mainly in the Quai d'Orsay and Combes buildings), and although we would occasionally see the issues ourselves when walking around the buildings, we could not reliably recreate them, and usually the students and staff who raised these issues with us could not recreate the issues on demand.Ìý We noticed that issues mostly disappeared when few people were in the buildings, but that even with lots of people present, speeds were often excellent - when things worked.Ìý We opened a second ticket with the Wi-Fi equipment vendor, but they also had difficultyÌýdiscovering the origin of the problem, which seemed toÌýpop up and disappear at any moment.Ìý However, it was clear that there was an issue, due to the numerous complaints we received about the new system.

The vendor's support team investigated possible issues in our existing network backbone that might cause the problems about which we were hearing, but did not find anything wrong, leaving us in a situation where even the equipment's manufacturer confirmed that nothing was set up incorrectly yet could not find the origin of the issues.
Ìý

Finally, progress!

After over a month of receiving disappointing non-responses on both of the support requests that we'd raised, the issue was finally escalated to someone working for the equipment vendor who was able to push theirÌýengineering team to work on identifyingÌýthe issues.Ìý After a few weeks, they were able to find what was causing the four wireless access points to crash (and we later found, what was also causing the general performance issues).
Ìý

So, what was the problem?

Technical Background: IPv6

Devices on the internet need a way to speak with each other, which they do using the Internet Protocol version 4 (IPv4) and an accompanying "IP address".Ìý Traditionally, each device was assigned a temporary or permanentÌýIPv4 addressÌýconsisting of four numbersÌýranging from 0-255, separated by decimals: for example, 193.49.37.66Ìýis an IPv4 address that could be used by a server on the internet.Ìý However, because IPv4 addresses by nature only allowÌýfor a maximum of just over 4 billion unique addresses, which seemed like a lot in the 1980sÌýbut is entirely insufficient for the growth of the modern Internet, a newer Internet Protocol called IPv6Ìýwas introduced, with a new, longer address.Ìý An IPv6 address is represented as 32 hexadecimal characters (0-9, a-f) divided into groups of four that are separated by colons: 2001:067c:2ebc:abcd:face:1967:b33f:cafe, for example, would be an example of a validÌýIPv6 address. This new format, allowing for 16^32 (16 to the power of 32) addresses, or about 340 undecillionÌýaddresses, providesÌýplenty of room for the internet to grow.Ìý However, until almost every device on the internet can use IPv6, new devices with IPv6 addressesÌýgenerally need to use both older IPv4 addresses and newer IPv6 addresses to communicate.Ìý We are currently partway through a decades-long transition period where online services are made available using both IPv4 and IPv6, as Internet providers and corporate/education networks slowly activate IPv6, and eventually will deactivate IPv4.

Major services such as Google, Facebook, and LinkedIn have been available using both the older IPv4 addresses and the newerÌýIPv6 addressesÌýfor over a decade.Ìý Around 70 percent of French internet connections support IPv6 (the third-highest ranking in the world in the world), and more than two-thirds of Internet traffic on IPv6-enabled networks (where both IPv6 and IPv4 addresses are available) is routed using IPv6.Ìý However,Ìýcorporate and educational networks are notorious for being very slow to implement this new IPv6 connectivityÌýand continue to rely only on the older IPv4 protocol, often due to a lack of resources or training.Ìý As a result, equipment that is provided by major networking companies is often better-tested with IPv4 than with IPv6, even if the equipment is certified to work with both.
Ìý

Context: IPv6 and Â̾ÞÈËÊÓƵ

Â̾ÞÈËÊÓƵ activated IPv6 connections on its network in the first part of 2023, providing every non-wireless device with both IPv4 and IPv6 addresses.Ìý This allowedÌýour Internet connection to be more reliable, decreasedÌýlatency, and allowed us to double our available bandwidth to the campus.Ìý IPv6 was not activated on our Wi-Fi networkÌýat theÌýtime, because our old Wi-Fi system did not support IPv6 addresses. When we installed the new Wi-Fi system, IPv6 became instantly available on Wi-Fi, allowing usÌýto offer complete IPv6 and IPv4 connectivity to our community.Ìý Over the course of the summer no issues were reported, and our tests showed that the system was working as expected.
Ìý

The source of our problems: codeÌýin the Wi-Fi access points' firmware

After almost two months of applying pressure on the Wi-Fi equipment vendor, they notified us that they had made progress.Ìý They noticed that something about IPv6 (reminder, a protocol that is not yet widely implemented with their customers) was causing the Wi-Fi access points to crash and have performance issues.Ìý They thenÌýcreated a network in their lab that was similar to ours, and started transferring large amounts of data over it using the IPv6 protocol, until they found a configuration where their test equipment started crashing the way that ours did.Ìý The culprit?Ìý A piece of software in the device, tasked with converting IPv6 "multicast" packets to IPv6 "unicast" packets, was simply not able to keep up with the high IPv6 traffic level on the network that started when students arrived on campus for the Fall semester.Ìý Given that this software is essential to our IPv6 network configuration and cannot be disabled, their suggestion was to temporarily block IPv6 traffic on the Wi-Fi network, to effectively work around the issue in their Wi-Fi system.
Ìý

Temporary fix: disabling IPv6 on our Wi-Fi networks

After temporarily disabling IPv6 traffic on our Wi-Fi access points, as the vendor recommended:

• The access points that had been crashing started working reliably.
• Complaints about sporadic Wi-Fi connections stopped.
• The connection speed improved in many high-traffic locations.

Because virtually all online services are still offered using the old IPv4 protocol, we made the decision that until the vendor fixes their devices' handling of IPv6, we will leave IPv6 deactivated, offering a reliable, fast connection using only IPv4.Ìý This is the current state of the network, and it has been reliable: an IPv4-only network, similar to what we had prior to summer of 2023, but with better performance and connection stability than our old Wi-Fi system that had been replaced.
Ìý

Moving forward

Because implementing IPv6 has technical advantages, we continue to hold conversations with the vendor regarding the development of a permanent solution, and have had regular meetings with their US-based Director of Product Management to receive updates on their progress.Ìý As the vendor works to resolveÌýthis issue,Ìýwe will keep IPv6 connectivity turned off on our Wi-Fi networks, to maintain a high quality of service using the IPv4 protocol.Ìý When the vendor provides us with a technical solution that will allow us to re-activate IPv6 on the Wi-Fi network, we will deploy it slowly,Ìýwith extensive testing, until we are satisfied that they have fixed the issue in such a way that we can reactivate IPv6 campus-wide.

We would like to thank the Â̾ÞÈËÊÓƵ community for their patience as we worked toÌýidentify the problem and to resolve the issues that people have experienced.Ìý We encourage all members of the community who have any questions or concerns relating to Wi-FiÌýat Â̾ÞÈËÊÓƵ to visit the IT Services office in room Q-A201, or to e-mailÌýitservicesaup.edu.