Outage Alerts

Jan. 16 Service Outage

Update, Jan. 23: OIT’s networking equipment vendor has verified that a bug within a core network switch resulted in the widespread network outage.  Unfortunately the internal redundancy mechanisms built into the switch did not initiate as expected due to the nature of the bug.  The vendor has documented this issue and has provided us with a patch to correct the issue. 

While it does appear that this outage was caused by a vendor software issue, OIT is using the event as a learning experience for how we can better prepare and respond to future outage incidents.  Our teams have met to document areas for further research and/or improvement based upon the lessons learned from the event. 

Original Post, Jan. 17: Thurs. Jan. 16, OIT experienced a networking hardware failure that impacted many services and applications on campus.

The networking hardware is designed with dual switches for redundancy; however, the faulty switch appeared to only partially fail, preventing the automatic failover to the redundant switch. 

A full root cause analysis is being performed by OIT, and we have also engaged the hardware vendor to assist in hardware diagnostics.  Additional details for the root cause of this outage will be provided as they become available.

Jan. 7, 2020 Service Outage

Tuesday, Jan. 7 OIT experienced a vendor-acknowledged bug in firmware that caused some UA services to be unavailable. These services included myBama, DegreeWorks, Banner and other services. The 29-minute outage was remediated, and additional measures were put in place later that evening to mitigate the risk of future outages.

Sept. 14 myBama Login Issue After Update

After the myBama patch upgrade was performed Saturday, Sept. 14 at 8am, some users reported that they were unable to login to myBama. When the patch upgrade was applied, an authentication database unexpectedly reverted to an older copy from June 2019, which was the date of the last myBama upgrade. This caused users created after June 1, 2019 to not be able to login to myBama. The issue was quickly corrected by restoring the backup copy of the most recent authentication data.

myBama, Banner and Other Services Unavailable

Update 6/11 9:40 a.m. – Monday, June 10 at 7:15 p.m., OIT experienced a shortage in encrypted storage while a scheduled process was running. This affected databases and caused an outage with the following applications.

  • Appworx
  • Banner (and Banner-related like Workflow, eProcurement, etc.)
  • PCR360
  • FRM
  • myBama
  • DegreeWorks
  • Sitefile
  • Crystal Reports Server
  • Eprint
  • Digital Signage
  • Sequoia
  • StarRez
  • SharePoint aka Tideshare

The scheduled process was terminated and services were re-established by 8:50 p.m. OIT is actively working to increase our encrypted storage amount while also evaluating systems and procedures to determine what data requires encryption in order to appropriately manage storage capacity.

Update 8:50pm – Services are now available. A root cause analysis will be posted here when available.

Original post – 7:20 p.m. – myBama, Banner and some other services are currently unavailable. OIT is investigating and will provide updates.

VMWare Hardware Failure

4:25pm 5.1.19 Update & Root Cause – Services are now restored, the failed hardware has been removed from production.

3:50pm 5.1.19 – A VMWare host in the OIT Data Center has experienced a hardware failure. OIT teams are working to resolve the issue and determine affected applications.

Network Outage – Rose Administration

Root Cause Update 5.1.19 – Tuesday, April 30, Rose Administration and a few surrounding buildings experienced a network outage. The root cause of this outage was loss of power to the UPS supporting the network in that area. When power was restored, services resumed. Redundant campus network connectivity limited the outage to only seven of the 60+ buildings in that area of campus.

Original Post 4.30.19 – OIT began investigating a network outage in Rose Administration and surrounding buildings at 4:00 p.m. Tuesday, April 30. Services were restored by 4:45pm. Root cause will be posted here when available.

April 4 – Temporary Service Degradation

RCA Update 4.12.19 – Thursday, April 4, 2019, OIT experienced a temporary service degradation that impacted users’ ability to login to some systems. LDAP indexing changes intended for a new test server were inadvertently applied in production, causing loss of critical indexes and immediate performance degradation. To remediate the issue, some LDAP servers were restarted at 9:30 a.m., which created temporarily failed connections, but ultimately resolved the issue by 10:45 a.m.

Original Post 4.4.19 – Thursday, April 4, OIT experienced a temporary service degradation that impacted users ability to login to systems such as myBama. OIT is investigating the issue and will provide more information and a root cause analysis when available.

March 18 Voicemail Outage

Monday, March 18, 2019, voicemail service was unavailable from 9:27 a.m. until 9:45 a.m. The server was rebooted at 9:31 a.m., and service resumed shortly after. We apologize for any inconvenience. OIT is working with the vendor, Cisco, to investigate the outage.

Feb. 13, 2019 VMWare Host Outage

At 6:50 a.m. Wednesday, Feb. 13, a VMWare host in the OIT data center became unresponsive. This affected Banner, myBama and several other applications in the VM hosting environment. We have removed the host from service pending investigation. Thank you for your patience as our staff worked to resolve this outage.