Supporting the UW Web

So the student Web servers are upgraded as of April 29th. The new machines seem to be running well, and I have not so far received any complaints of Web applications not running properly on the new software.

Student Web Server Upgrade
http://www.washington.edu/computing/web/publishing/server-upgrade.html

I’ve been getting a fair number of questions about migrating MySQL servers from the old vergil to the new one, but it has not been too bad.

The number of MySQL servers on vergil is approaching 100.

New vergil (vergil01.u.washington.edu)
load: 0.03 mysqlds: 92

This has been a model upgrade.

The student Web servers are being upgraded this month:

Student Web Server Upgrade
http://www.washington.edu/computing/web/publishing/server-upgrade.html

It looks like there are still 271 MySQL servers that need to move. They have 13 days to do that. On April 22nd, the DNS name vergil.washington.edu will go to vergil01 instead of vergil21, and Web applications won’t connect to those MySQL servers running on vergil21 anymore, assuming they are connecting to vergil.washington.edu.

Here are current numbers of MySQL servers on vergil and vergil01:

Current vergil (vergil21.u.washington.edu)
load: 6.37 mysqlds: 271

New vergil (vergil01.u.washington.edu)
load: 0.03 mysqlds: 3

Many more details of what is being upgraded are on the page linked at the top of this post.

This happened a long time ago, but I thought I should call out that this thing is working again.  Anyone with a UA account can access this by going to the root of their Web server, but here are links directly to the error log viewer for each account.

students – https://students.washington.edu/adm/errorlog/
staff – https://staff.washington.edu/adm/errorlog/
faculty – https://faculty.washington.edu/adm/errorlog/
courses – https://courses.washington.edu/adm/errorlog/
depts – https://depts.washington.edu/adm/errorlog/

The University of Melbourne has a nice-looking installation of WordPress MU:

University of Melbourne Blogs

They’re running it as a service @ a cost-recovery rate of $200/year.  It doesn’t look like they have too many blogs running there yet, and the process of starting a new blog seems to be a bit manual for my tastes, but it is good to see WordPress MU being used in this capacity by a university.  I’d support that.

Harvard Law School also runs WordPress MU, with a more automatic blog-creation process, but theirs is free.

Weblogs at Harvard Law School

I get a significant number of requests for information on how to do this.  It’s a pretty bad kludge, but it does seem to work.

Basically, you make three directories:

• one containing the content and an .htaccess file denying all access
• one containing an .htaccess file allowing access via basic auth
• one containing an .htaccess file allowing access via pubcookie

The directories with access via the different authentication methods will also contain symbolic links to either files in the content directory(if the files are not going to change much) or to another directory in the content directory so that you can change the files around without redoing the symbolic links.

You can take a look at an example implementation of this scheme in the multi-auth directory on my staff Web site.

• Export the DB to sql statements.
• Search-and-replace “InnoDB” with “MYISAM”.
• Re-import the tables.  Might want to do this into a different DB than the old one, just in case.
• Configure MediaWiki to use the new MyISAM DB.
• Make sure the user MediaWiki is using to connect to MySQL will be able to access the new DB.

Bonus:

• Put the “skip-innodb” line into the .my.cnf file.
• Stop the DB.
• Start the DB.

InnoDB is gone.  Yaaay.

detail:

bin/mysqldump -u root -p --lock-tables --add-drop-database DATABASE_NAME > DATABASE_NAME.sql

/rc22/d77/agraf/bin/rpl  'ENGINE=InnoDB' 'ENGINE=MyISAM' DATABASE_NAME.sql

bin/mysql -u root -p DATABASE_NAME < DATABASE_NAME.sql

WordPress 2.5.1 was released today, which means that I will now edit the WordPress installation documentation to use the latest WordPress release again, since version 2.5.1 works with the new http-authentication 2.0 plugin.

It is a relief to have the WordPress documentation using the latest, more secure version of WordPress again.  Of course this blog is using WordPress 2.5.1 and the new http-authentication plugin now.

Today I updated the documentation for installing some wikis.

• MediaWiki instructions went from version 1.11.0 to 1.12.0
• MoinMoin instructions went from version 1.5.3 to 1.5.8

The new version of MediaWiki has an option for selecting InnoDB or MyISAM, so the previous changes to that documentation don’t seem quite as necessary, but they still make sense, so I’ve left them intact.

Today a couple of things were added to the UA Web servers.  This includes staff.washington.edu, faculty.washington.edu, courses.washington.edu, depts.washington.edu, students.washington.edu, ovid.u.washington.edu, and vergil.u.washington.edu.

• Python was upgraded from 2.5 to 2.5.2
• The sqlite3 (3.5.4) command-line tool was added.
• Python’s sqlite3 module now functions properly
• The MS SQL Server PHP module was added, which allows connections to MS SQL servers using PHP.

I edited the MediaWiki instructions today to include a note that users should disable InnoDB before installing MediaWiki. If people do this, then their wikis will work after MySQL has been automatically restarted by cron after a system restart. If people don’t do this, then I have to fix their databases after a system restart due to stale NFS locks.
I wrote about this topic last Monday.  Since then I read about MediaWiki and MySQL storage engines, and tried an experiment where I converted the InnoDB tables my MediaWiki installation was using to MyISAM tables.  As far as I can tell, MediaWiki works fine using MyISAM.

Here’s a brief mention on the MySQL Reference Manual of problems that InnoDB has when its data or log files are stored on NFS volumes: Restrictions on InnoDB Tables

Warning

It is not a good idea to configure to use datafiles or logfiles on NFS volumes. Otherwise, the files might be locked by other processes and become unavailable for use by MySQL.

The issue in the cases I see commonly seems to be that when the NFS client system is rebooted, the NFS server does not get any notice that locks should be released on the files.  When the NFS client comes back up from the reboot and reconnects to the server, those files are still locked, and the client cannot get a new lock on them.  This causes MySQL to issue errors like the following:

InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.

and this:

InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means ‘Resource temporarily unavailable’.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!