CloudLinux - CloudLinux Blog - MySQL 5.5 for MySQL Governor Issue Follow Up

MySQL 5.5 for MySQL Governor Issue Follow Up

As many of you might be aware, we have released MySQL 5.5 update that broke mysqli compatibility for many customers. Here I want to explain what happens, and the steps we are planning to take to prevent such things from happening in the future.

Sequence of events:
  • A new patch was applied to MySQL 5.5 that fixed backward compatibility with libmysqlclient16 library experienced by one customer
  • The patch was tested on customer's server
  • MySQL was pushed into MySQL Governor repositories
  • It was automatically downloaded and installed by yum update or upcp running on customer's server
  • The update caused compatibility issues with mysqli php extension, resulting in multiple websites being down
  • Within 30 minutes of us understanding what happened, we have removed new MySQL from repository and provided advisory on how to fix the issue. Yet, significant number of customers had updated by that time.
Here are is what we consider to have caused the problem for so many systems:

  • MySQL package should have gone through beta process first. We consider MySQL governor to be a beta overall, and we beta test most software we release. Beta testing requires administrators to manually install the update version of software. It limits the number of systems where updates are executed, and makes sure that administrator knows what caused the issue. It allows admins with multiple systems to test new software on one or two servers, before it is unleashed on all their servers.
  • Incomplete testing was done on MySQL package. Developer decided that given how small patch is, full testing is not required
  • No announcement went out about the change.
Here is what we plan to do to address the identified issues:

  • We will split MySQL governor repositories into beta and stable repositories
  • Beta repository will be disabled by default. Stable repository will be enabled
  • All new versions of MySQL packages will go into beta repositories, and to update to beta version, user would have to run: --update-mysql-beta
  • Such beta releases will be announced via our blog and mailing list
  • Once we are confident in a new version -- we will move it to stable
  • This process will continue once MySQL Governor will be moved to stable channel itself
  • We will create automated tests for building php using EasyApache that will run on each build
  • We will create php page that will try to use new MySQL build using mysqli and make it part of automated testing
  • All versions of alt-php will be tested using this page as well against new MySQL build.
The ETA for repository changes is 2 weeks
The ETA for full cycle automated testing is 6 week.

I would like to welcome any feedback and recommendations regarding the issue and the approach we are taking to prevent such issues from happening again.
Beta: Updates for lvemanager, lve-utils and lve-st...
CL6 and Hybrid kernel 2.6.32-458.18.1.lve1.2.39 mo...

By accepting you will be accessing a service provided by a third-party external to