Given the state of Blackboard over the past couple of days, we thought we would offer some technical details on what has happened, the actions performed, and current state. As of this afternoon, while there have been reports of a few isolated incidents, we have not heard of the environment acting slow or folks not being able to log in.  These incidents do not appear to be as widespread as they were the past few days.  On a positive note, in the days following the upgrade, no issues or bugs were discovered in the system itself. Feedback regarding the minor user interface changes has been positive.

A summary of the events and actions taken to correct them are below:

Early Friday evening:

Blackboard removed from service for the upgrade to Service pack 13.

Saturday afternoon:

Environment available for folks to use.

Late Saturday afternoon:

Discovered that queued tasks were not running.

Sunday morning:

Queued tasks running again following a correction to a mismapped hostname.

Early Monday afternoon:

Blackboard running slowly.

Blackboard support contacted to help diagnose.

Blackboard recommended that we change the amount of memory that is allocated to the application.

Late Monday afternoon:

While trying to apply the memory allocation changes, we ran into a problem with loading the content management system, which manages content related to courses and student submissions.

Blackboard support contacted to help diagnose the problem.

Monday Evening:

Corrected the problem with the misconfiguration on the system which then enabled the content system to load.

Memory allocation configuration changes were applied and the new servers were made available to the public.

Service appeared to be restored.

Tuesday Morning:

As user load increased on the environment, one of the application servers became unresponsive.

The unresponsive server was removed from public use.

Early Tuesday Afternoon:

Blackboard became slow and folks having trouble logging in, much as they did the previous day.

Blackboard support contacted to help diagnose the issue.

Blackboard support recommended that we configure the environment to have more application servers that serve fewer folks each.

Late Tuesday afternoon.

Five new servers were created with all of the Blackboard recommended configuration changes from the past couple of days.

Early Tuesday evening:

New application servers were made available to the public.

Old servers that did not match the new configuration were removed from public access.

Folks would have seamlessly been transferred from the old servers to the new ones.

Late Tuesday evening:

The servers that were removed from public access were reconfigured to match the now current running environment.

We now have thirteen application servers that are running per Blackboard’s recommendation.

Wednesday:

Monitoring of services reveals no widespread problems that had been experienced previously.