Supporting the UW Web

The student Web servers are being upgraded this month:

Student Web Server Upgrade
http://www.washington.edu/computing/web/publishing/server-upgrade.html

It looks like there are still 271 MySQL servers that need to move. They have 13 days to do that. On April 22nd, the DNS name vergil.washington.edu will go to vergil01 instead of vergil21, and Web applications won’t connect to those MySQL servers running on vergil21 anymore, assuming they are connecting to vergil.washington.edu.

Here are current numbers of MySQL servers on vergil and vergil01:

Current vergil (vergil21.u.washington.edu)
load: 6.37 mysqlds: 271

New vergil (vergil01.u.washington.edu)
load: 0.03 mysqlds: 3

Many more details of what is being upgraded are on the page linked at the top of this post.

• Export the DB to sql statements.
• Search-and-replace “InnoDB” with “MYISAM”.
• Re-import the tables.  Might want to do this into a different DB than the old one, just in case.
• Configure MediaWiki to use the new MyISAM DB.
• Make sure the user MediaWiki is using to connect to MySQL will be able to access the new DB.

Bonus:

• Put the “skip-innodb” line into the .my.cnf file.
• Stop the DB.
• Start the DB.

InnoDB is gone.  Yaaay.

detail:

bin/mysqldump -u root -p --lock-tables --add-drop-database DATABASE_NAME > DATABASE_NAME.sql

/rc22/d77/agraf/bin/rpl  'ENGINE=InnoDB' 'ENGINE=MyISAM' DATABASE_NAME.sql

bin/mysql -u root -p DATABASE_NAME < DATABASE_NAME.sql

Today I updated the documentation for installing some wikis.

• MediaWiki instructions went from version 1.11.0 to 1.12.0
• MoinMoin instructions went from version 1.5.3 to 1.5.8

The new version of MediaWiki has an option for selecting InnoDB or MyISAM, so the previous changes to that documentation don’t seem quite as necessary, but they still make sense, so I’ve left them intact.

I edited the MediaWiki instructions today to include a note that users should disable InnoDB before installing MediaWiki. If people do this, then their wikis will work after MySQL has been automatically restarted by cron after a system restart. If people don’t do this, then I have to fix their databases after a system restart due to stale NFS locks.
I wrote about this topic last Monday.  Since then I read about MediaWiki and MySQL storage engines, and tried an experiment where I converted the InnoDB tables my MediaWiki installation was using to MyISAM tables.  As far as I can tell, MediaWiki works fine using MyISAM.

Here’s a brief mention on the MySQL Reference Manual of problems that InnoDB has when its data or log files are stored on NFS volumes: Restrictions on InnoDB Tables

Warning

It is not a good idea to configure to use datafiles or logfiles on NFS volumes. Otherwise, the files might be locked by other processes and become unavailable for use by MySQL.

The issue in the cases I see commonly seems to be that when the NFS client system is rebooted, the NFS server does not get any notice that locks should be released on the files.  When the NFS client comes back up from the reboot and reconnects to the server, those files are still locked, and the client cannot get a new lock on them.  This causes MySQL to issue errors like the following:

InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.

and this:

InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means ‘Resource temporarily unavailable’.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!

For probably at least a year I have had instructions for installing MediaWiki on the UA Web servers.

Installing MediaWiki
http://www.washington.edu/computing/web/publishing/mediawiki.html

These instructions include a default installation that uses the MediaWiki user and authentication system, as well as an option for using the Pubcookie SSO authentication system that is used for most UW Technology applications at the University of Washington. The problem with the way those instructions say to set things up is that you need to log in with Pubcookie before you can even view the wiki. This isn’t always what’s desired.

A couple weeks ago two users requested a means of setting up MediaWiki so that it could be anonymously read, but users would need to log in using Pubcookie in order to edit a page. I came up with a solution which they were both happy with, and which seems to work. I am not sure if there are ways of breaking MediaWiki when it is configured in this manner, but it seems to be working fine so far.

Here is a copy of MediaWiki configured with the new anonymous read/Pubcookie required for edits configuration:

Test Wiki

The instructions have now been updated to include the new section called “Requiring UW NetID Authentication only for Editing“.

The next thing to work on is changing the MediaWiki documentation to inform users that they should probably use the MyISAM storage engine on MediaWiki installations backed by MySQL on the UA Web servers. This is because InnoDB databases will not come back automatically if ovid is restarted without shutting down MySQL first. The culprit is NFS, which is used to serve all the files in user Web and home directories on the UA system.

I’ve got some instructions up for how to use the C&C-installed MySQL binaries. Noone should have to install MySQL on ovid or vergil anymore, which will save lots of disk space. This also means you shouldn’t have to install upgrades.

I’ve been using this copy of MySQL for running this blog, and any other MySQL applications I happen to be running, for months, and it works for me. It looks like 26 other accounts are using this copy of MySQL already. Good for them!

Using the C&C-Installed MySQL Binaries
http://www.washington.edu/computing/web/publishing/mysql-installed.html

Update:

The old cron script should now work fine, due to changes in the setup script.

The mysql_upgrade script, which checks all tables in all databases for incompatibilities with the current version of MySQL Server, has a bug that doesn’t allow it to process passwords correctly. We are planning to upgrade MySQL to 5.0.45 on Jan 2nd. This version has the mysql_upgrade bug fixed.

This week and next week I’m hoping to get all of the MySQL instructions I currently have at www.washington.edu/computing/web/publishing/ revised to reflect the current systems and relatively recent versions of the software they discuss.

Changes I’ve made so far

• Merged myuw.net MySQL documentation into main docs (still needs some work)
• Updated the warning about the size of the install on the install page
• Removed ovid migration-specific notes from install page

Future MySQL documentation goals

• Create migration and installation instructions for system-wide MySQL installation
• Develop and publish policies governing how this installation will be upgraded and how databases will be migrated to new versions of MySQL

The big move from MySQL 4 on ovid to MySQL 5 on ovid21 is well underway.

Instructions for installing MySQL 5 on ovid21 linked below. Now including notes that might help you move from MySQL 4 on ovid!

Installing and Starting MySQL 5.0
http://www.washington.edu/computing/web/publishing/mysql5-install.html