Supporting the UW Web

I edited the MediaWiki instructions today to include a note that users should disable InnoDB before installing MediaWiki. If people do this, then their wikis will work after MySQL has been automatically restarted by cron after a system restart. If people don’t do this, then I have to fix their databases after a system restart due to stale NFS locks.
I wrote about this topic last Monday.  Since then I read about MediaWiki and MySQL storage engines, and tried an experiment where I converted the InnoDB tables my MediaWiki installation was using to MyISAM tables.  As far as I can tell, MediaWiki works fine using MyISAM.

Here’s a brief mention on the MySQL Reference Manual of problems that InnoDB has when its data or log files are stored on NFS volumes: Restrictions on InnoDB Tables

Warning

It is not a good idea to configure to use datafiles or logfiles on NFS volumes. Otherwise, the files might be locked by other processes and become unavailable for use by MySQL.

The issue in the cases I see commonly seems to be that when the NFS client system is rebooted, the NFS server does not get any notice that locks should be released on the files.  When the NFS client comes back up from the reboot and reconnects to the server, those files are still locked, and the client cannot get a new lock on them.  This causes MySQL to issue errors like the following:

InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.

and this:

InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means ‘Resource temporarily unavailable’.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!

I previously published instructions for setting up a Ruby Gems repository on your own account, but I neglected to include the one special step that will let a Ruby on Rails application use that gem repository.

Now that step has been added to the end of the Ruby Gems on Your Account instructions.

I know we are running an out-of-date installation of Ruby and Rails, but we have requested a new one. Keep your fingers crossed.

Our Web & Web dev hosts
ruby 1.8.4
ruby on rails 1.1.6

Current stable versions
ruby 1.8.6
ruby on rails 2.0.2

The http-authentication plugin required for the fancy Pubcookie authentication instructions for WordPress does not function with the new WordPress 2.5.

I have changed the WordPress instructions to use the old 2.0.11 version of WordPress until the plugin gets fixed.

For probably at least a year I have had instructions for installing MediaWiki on the UA Web servers.

Installing MediaWiki
http://www.washington.edu/computing/web/publishing/mediawiki.html

These instructions include a default installation that uses the MediaWiki user and authentication system, as well as an option for using the Pubcookie SSO authentication system that is used for most UW Technology applications at the University of Washington. The problem with the way those instructions say to set things up is that you need to log in with Pubcookie before you can even view the wiki. This isn’t always what’s desired.

A couple weeks ago two users requested a means of setting up MediaWiki so that it could be anonymously read, but users would need to log in using Pubcookie in order to edit a page. I came up with a solution which they were both happy with, and which seems to work. I am not sure if there are ways of breaking MediaWiki when it is configured in this manner, but it seems to be working fine so far.

Here is a copy of MediaWiki configured with the new anonymous read/Pubcookie required for edits configuration:

Test Wiki

The instructions have now been updated to include the new section called “Requiring UW NetID Authentication only for Editing“.

The next thing to work on is changing the MediaWiki documentation to inform users that they should probably use the MyISAM storage engine on MediaWiki installations backed by MySQL on the UA Web servers. This is because InnoDB databases will not come back automatically if ovid is restarted without shutting down MySQL first. The culprit is NFS, which is used to serve all the files in user Web and home directories on the UA system.

Near the beginning of 2008 we’re planning to upgrade the software on the student, staff, faculty, depts, courses, MyUW.net and www Web servers.

The upgrade will include Apache, PHP, OpenSSL, mod_uwa, and enabling compression.

More detail is available on the public Web page for this upgrade.

Update: This work was completed in March.

I’ve got some instructions up for how to use the C&C-installed MySQL binaries. Noone should have to install MySQL on ovid or vergil anymore, which will save lots of disk space. This also means you shouldn’t have to install upgrades.

I’ve been using this copy of MySQL for running this blog, and any other MySQL applications I happen to be running, for months, and it works for me. It looks like 26 other accounts are using this copy of MySQL already. Good for them!

Using the C&C-Installed MySQL Binaries
http://www.washington.edu/computing/web/publishing/mysql-installed.html

Update:

The old cron script should now work fine, due to changes in the setup script.

The mysql_upgrade script, which checks all tables in all databases for incompatibilities with the current version of MySQL Server, has a bug that doesn’t allow it to process passwords correctly. We are planning to upgrade MySQL to 5.0.45 on Jan 2nd. This version has the mysql_upgrade bug fixed.

This week and next week I’m hoping to get all of the MySQL instructions I currently have at www.washington.edu/computing/web/publishing/ revised to reflect the current systems and relatively recent versions of the software they discuss.

Changes I’ve made so far

• Merged myuw.net MySQL documentation into main docs (still needs some work)
• Updated the warning about the size of the install on the install page
• Removed ovid migration-specific notes from install page

Future MySQL documentation goals

• Create migration and installation instructions for system-wide MySQL installation
• Develop and publish policies governing how this installation will be upgraded and how databases will be migrated to new versions of MySQL

The big move from MySQL 4 on ovid to MySQL 5 on ovid21 is well underway.

Instructions for installing MySQL 5 on ovid21 linked below. Now including notes that might help you move from MySQL 4 on ovid!

Installing and Starting MySQL 5.0
http://www.washington.edu/computing/web/publishing/mysql5-install.html

Today is my first day back from a 2 week vacation wherein I biked down some of the Washington and Oregon coast with some friends. I still have a lot of email to read.

These servers are going to be upgraded from March to May 2007.  This means a lot of planning, communication, testing, and helping users get their pages and applications working on the new operating system, hardware, and software versions.

The web and web development servers will be moving from AIX to Linux.

The following Web-related software will be upgraded:

• Apache Server: 1.3.29 -> 2.2.4
• PHP Hypertext Preprocessor 4.3.10 -> 5.2.1
• perl Scripting Language 5.6.0 -> 5.8.0

New software will be available:

• Ruby on Rails (Rails 1.1.6 on Ruby 1.8.4)
• New PHP capabilities: XML/XSL, multi-byte string functions, openSSL and more
• New perl capabilities: multithreading, better numeric accuracy, new modules

This upgrade will cause all of the MySQL server running on Ovid to stop working, so people running those will need to install new versions of MySQL.

Any other compiled code will stop working, so people using compiled code on ovid or the web servers will also need to either install new versions or recompile their code.

Since all those scripting languages like Perl, PHP, and Python will be upgraded, people using those should make sure their scripts will run with the new versions.

Further information on the system upgrade can be found at:

Web Server Upgrade
http://www.washington.edu/computing/web/publishing/server-upgrade.html

If you have questions about this upgrade that can’t be answered by that web page, please contact me at help@cac.washington.edu.