My wife and a developer co-worker were working on a client Plone site the other day when they ran into an anomaly. Melody was checking whether search results showed an icon to indicate pages which should only be available to registered users (using custom code for the site which is different than what normally happens in Plone, private content not showing up at all in the search results). She wasn’t seeing the lock icon, but the developer saw it on his own local copy of the same site. They then proceeded to look for find differences between the two sites.
Everything in the skin looked the same. After changing the skin on each site so it would show the icon in all cases (by removing the conditional), the developer saw the icon on all results but Melody still didn’t see it at all. She didn’t even see extra text that was added to the skin as a test. Her browser was checked to make sure that the page wasn’t being cached, and the page source checked to see if the image was in the HTML and somehow not displayed (not the case). Another browser was also tried.
They confirmed that the results page did have a combination of public and private results, and the private pages really weren’t viewable by non-members. Everything checked proved to be the same on the two sites.
The next twist was that the developer did see the lock icon when he searched on the site Melody was using. Very odd!
Actually, the way I phrased that may give away the actual cause of the problem, but first some comments about trying to find the root of inconsistent behavior.
The progression of testing done by Melody and the developer shows a good way to attack the problem. After looking for differences, try to make changes which should change the results, which they did by trying to make the icon show up for all results and adding test text. Throughout, check that you’re actually looking at the right thing. That may sound trite, but it’s exactly what was being done when the results were confirmed to have private content, when the source and browser cache were checked, and when multiple browsers were tested.
So you see no differences and are sure you’re looking at the right thing. The next step is to look for things which you’re assuming are the same but in reality are different. This can be one of the hardest steps, especially by yourself. They did this by having the developer viewing the site Melody was using rather than his local site, making sure the changes were really done in sync on the two sites, but still they were both deep enough into the details that it made it harder for them to step back to see what was different.
Jon Bentley (of Programming Pearls) writes about one user who could login to a system while sitting down, but was refused access while standing up. As little sense as it made, the behavior was repeatable. An outside observer asked what was different with how the user logged in, and it ended up that the user was a touch typist while sitting, but a hunt-and-peck typer while standing. Examining the keyboard showed a couple keycaps were swapped, causing an incorrect password to be typed. After fixing the keyboard, the user had no problems.
So what ended up being the problem with the lock icon? Two different people, two different methods of using search. The image showed up on the results page when using a custom search form, but didn’t when using the built-in site search. The clue was seen by the fresh eyes of a third party – in addition to the lock icon not showing up, the results page Melody saw also had extra text after the result links, confirming that she was really seeing something that wasn’t being formatted by the skin.
The small difference of how search was activated reinforces how hard it is to reassess one’s assumptions when testing. And when you’re at the point of looking at details, it becomes more difficult to do so; that’s the perfect time to bring in someone else. Could be that person sees something new, or it could be just by explaining the problem something else will occur to you.
I’ve worked with many programmers, some of them with truly extraordinary debugging skills. All of them have had a similar thing happen – spent a lot of time looking for a tree, and only after turning to someone else to take a fresh look could they see the forest.
