UMS Blackboard News

Blackboard Planned Maintenance - Sunday, November 24 - 5am to 6am

posted Nov 22, 2013, 1:27 PM by Calvin Bishop II   [ updated Nov 22, 2013, 1:32 PM ]

All Blackboard services will be unavailable this Sunday for maintenance.  We need to make the same changes to our back office servers that were made this past week to the public facing servers.  In order to make changes to the back office servers, the public facing servers need to be shutdown during this time, resulting in the outage.

Scope of Maintenance: All Blackboard Services.
Maintenance Start: November 24, 2013 5am.
Maintenance End: November 24, 2013 6am.

Blackboard Service Summary

posted Nov 20, 2013, 1:32 PM by Calvin Bishop II

Given the state of Blackboard over the past couple of days, we thought we would offer some technical details on what has happened, the actions performed, and current state. As of this afternoon, while there have been reports of a few isolated incidents, we have not heard of the environment acting slow or folks not being able to log in.  These incidents do not appear to be as widespread as they were the past few days.  On a positive note, in the days following the upgrade, no issues or bugs were discovered in the system itself. Feedback regarding the minor user interface changes has been positive.


A summary of the events and actions taken to correct them are below:


Early Friday evening:

Blackboard removed from service for the upgrade to Service pack 13.


Saturday afternoon:

Environment available for folks to use.


Late Saturday afternoon:

Discovered that queued tasks were not running.


Sunday morning:

Queued tasks running again following a correction to a mismapped hostname.


Early Monday afternoon:

Blackboard running slowly.

Blackboard support contacted to help diagnose.

Blackboard recommended that we change the amount of memory that is allocated to the application.


Late Monday afternoon:

While trying to apply the memory allocation changes, we ran into a problem with loading the content management system, which manages content related to courses and student submissions.

Blackboard support contacted to help diagnose the problem.


Monday Evening:

Corrected the problem with the misconfiguration on the system which then enabled the content system to load.

Memory allocation configuration changes were applied and the new servers were made available to the public.

Service appeared to be restored.


Tuesday Morning:

As user load increased on the environment, one of the application servers became unresponsive.

The unresponsive server was removed from public use.


Early Tuesday Afternoon:

Blackboard became slow and folks having trouble logging in, much as they did the previous day.

Blackboard support contacted to help diagnose the issue.

Blackboard support recommended that we configure the environment to have more application servers that serve fewer folks each.


Late Tuesday afternoon.

Five new servers were created with all of the Blackboard recommended configuration changes from the past couple of days.


Early Tuesday evening:

New application servers were made available to the public.

Old servers that did not match the new configuration were removed from public access.

Folks would have seamlessly been transferred from the old servers to the new ones.


Late Tuesday evening:

The servers that were removed from public access were reconfigured to match the now current running environment.

We now have thirteen application servers that are running per Blackboard’s recommendation.


Wednesday:

Monitoring of services reveals no widespread problems that had been experienced previously.

We expect that the current configuration is stable and operating normally.

Blackboard Service Restored

posted Nov 19, 2013, 4:08 PM by Calvin Bishop II

We made another configuration change a few moments ago to the Blackboard environment which we believe to resolve the problems that we have experienced over the past few days.  Please let us know if you continue to see any problems with folks logging in or pages not loading.  Thank you for your patience during these past couple of days.

Blackboard Service Disruption

posted Nov 19, 2013, 10:32 AM by Calvin Bishop II

We are experiencing problems with folks logging in and pages taking longer to load than normal.  This is much like what we saw yesterday morning prior to the actual outage.  We are working to resolve the problem.  Sorry yet again for the disruption. 

Partial Blackboard Outage

posted Nov 19, 2013, 9:51 AM by Calvin Bishop II

One of our nine application servers is not responding.  Folks that were on that server will automatically be redirected to another available application server.  We are looking into the problem to see if it is related to yesterdays outage.

Blackboard Service Restored

posted Nov 18, 2013, 5:18 PM by Calvin Bishop II

Blackboard service is now restored.

Blackboard Service Outage Update

posted Nov 18, 2013, 2:52 PM by Calvin Bishop II

We are encountering difficulties with the new configuration that Blackboard has provided us.  Rest assured that we will continue to work with Blackboard to resolve this issue.

Blackboard Service Outage Update

posted Nov 18, 2013, 1:44 PM by Calvin Bishop II

We are still working on the configuration changes required to make Blackboard stable. Sorry for the long delay in returning service.  

Blackboard Service Outage Update

posted Nov 18, 2013, 12:33 PM by Calvin Bishop II

Over the past hour, we have continued to worked with Blackboard to determine the cause for the high load on the application servers.  Blackboard has made some more recommendations and we are applying the changes to the environment right now.  Once complete, the new servers will be placed in service, which should resolve the issues folks are currently seeing with the environment.

Blackboard Service Outage Update

posted Nov 18, 2013, 11:22 AM by Calvin Bishop II

As it has been about an hour, we would like to offer you an update.  We discovered that the application servers needed to have a configuration change.  We have some servers that have not yet been put into service and are now configuring them to handle the load better. Once complete, they will be put in service and the others that were not operating correctly will be removed.  Once this happens service should be restored and we will send out another note.  Thank you again for your patience. 

1-10 of 123