UNC Charlotte experienced five significant internet outages across campus within a week due to a firewall issue and software bug.
The main issue regarding the Wi-Fi stems from a network firewall problem. The firewall is the network's security system that monitors and controls incoming and outgoing network traffic.
Dr. Michael Carlin, vice chancellor for OneIT and chief information officer (CIO), and the University's IT department worked to find and solve the issue.
"[The team] spent a lot of time validating and proving where the issue was, working behind the scenes to identify the issue," Carlin said. The team worked for over eight hours one night alone to resolve the issue.
The University has an extensive network, so the firewalls can back up to 100 million individual connections, Carlin said
"On an average day on campus, we have well over 50,000 devices connected at any given time," Carlin said.
"This summer, normal replacements were made," Carlin said. Two of the school's firewalls were replaced, which are directly connected to the internet provider. These new firewalls ran all summer with great efficiency, Carlin said.
The first outage occurred on Tuesday, Aug. 23, at 1 p.m. Users of the on-campus Wi-Fi experienced a complete shutdown of their electronic devices.
The second and third outages occurred on Thursday, Aug. 25, at 10 a.m. and again at 2:45 p.m.
The last outage occurred on Tuesday, Aug. 30, at 10 a.m. and again at 11:06 a.m.
When the firewall started to malfunction during the first week of classes, the IT department first believed the outages might have been caused by the number of incoming traffic to the University, Carlin said.
However, due to another outage at 3 a.m., the team ruled out the assumption because most students and faculty were not using the internet at 3 a.m. In addition, students in on-campus dorms did not experience this outage, as the dorms use a private provider.
Carlin said that the IT team determined the issue was not a hardware problem because the likelihood of having two firewalls with the same hardware malfunction was improbable.
The team checked on the software of the firewalls, which were updated this summer with the vendor. The vendor is the internet provider. The team discovered that the vendor's software had a bug, which was the primary concern.
Once the issue was discovered, the team tested the firewall using packet capturing, which is a way to help identify network performance issues. Packet capturing allows the IT team to look for details about what is happening inside the firewall to send to the vendor.
While waiting for the vendor to come up with a diagnosis, the IT team reverted the software to its previous version. However, like updating a cell phone, using an older version of the software can have its issues; however, it is not a long-term solution.
While the IT department waits to hear from the vendor for a long-term solution, Carlin said, "The team at Charlotte has done everything they can do to resolve the issues, and now it is back in the vendor's court."
The IT department is unsure if this is an isolated issue or if the vendor is receiving similar calls from other universities.
The University hopes to receive a concrete answer from the vendor soon. In the meantime, the Wi-Fi is using the old software, allowing students, faculty and staff an internet connection.