Lately, we’ve been going nuts with a problem some of our visitors are having.  They keep complaining to us that our site is unbearably slow.  Here’s the problem.  When we check it out, it’s just fine.  It’s not the fastest site on the planet, but certainly not the slowest either.  So what’s causing the problem?  One clue we have is that the only customers who are complaining use IE6.  Unfortunately, that’s all we know.

So how do we figure out the problem?  Who do you call when your website is having technical problems?  When you dial website 911, who picks up the phone?  For us, the question is a mystery.  We’ve tried posting requests for help at E-lance and Guru.com.  Unfortunately, the responses are confusing and do not inspire us.

We’ve asked friends and colleagues to help as well.  Can analytics help us?  We’ve looked at our stats and found that, yes, IE6 users are less engaged in our website.  But it doesn’t tell us the problem.

One interesting tool we’ve found is Yslow.  This program analyzes your website and tells you where you need to reduce your JavaScript, or DNS calls and offers several suggestions.  Yslow works with Firefox only.  If our problem was with Firefox, we’d have it figured out in no time.

So what do you do when you have these problems?  I’m all ears.

Mature Process Reporting Should Become Seedbed for Discoveries, Patents, and Broad Practice Foundations

With constant data streams passing over the minds of analysts everyday, it seems higher than likely that, over time, the team which deals with this data should start to construct ideas and applications built on consistent experiences. The amount of information which is aggregated about visitor and customer information over time gives a specific picture about who needs what with regard to navigation, structure, and assistance within the website. That information, paired with analysis, should produce a list of items for IT and stakeholders to brainstorm on solutions and short-cuts to help users achieve navigational nirvana.

The idea of the web analytics process is the continued production, analysis, and decision support for the business in which it has been installed; as well as the obligatory feedback from the action which takes root within the agency. Therefore the establishment of process is, at least in my eyes, a goal for online businesses with more recent adoption. In the case of mature and refined analytics process practitioners, this should be a means to success of a single entity, and feed into the larger sphere of markets, industries, and eventually, universally adapted technologies which could feed macro-improvement of the user experience globally.

Look Past Process: Visitors Deserve Your Attention and Devotion

We are all users who are visitors. As a collective group, we can attest to the number of half-baked ideas which are thrown on the net for one reason or another. Many of us who matured working with computers have a few we hide away from the world. Hell, I have ten of my own. We all find sites which we thought were going to meet our expectations and fail miserably. Our arrival on pages often misses the original intent of the search or slightly misdirects the concept. We blame the search results, but, broadly, this is the output from lousy understanding of SEO taxonomy (i.e. Silos) and/or usability issues.

With usability testing, search marketing and SEO information as widely available as they are, the only explanations for a lack of progress in these subjects are:

1. Finances: Marketing budgets fail to consider the possible positive primary outcomes of efforts to improve these aspects of a site, much less the periphery value of the research contributions and application value abroad
2. Hesitation: Generally, businesses in practice tend to be ‘bearish’ when it comes to expenditure on technology which discusses returns. Intent might be pure, but the risk clearly does not merit the restraint.
3. Ego: In contrast to the previous item, trendy businesses based on whimsy and willingness to engage in risk view using data as somehow ‘cheating’ on creativity or lacking in true inspiration. Therefore, they overlook or suppress buy-in through a flurry of logical fallacies.
4. Perceived Complexity: To some, creating an in-house usability program, building a lab, standardizing reporting and inciting discussions on stats and quantified experience data is an ENORMOUS undertaking. It becomes a sociological hurdle for the company and eventually, with enough steam, a self-fulfilling prophecy.

With little or no participation in gathering an understanding of how YOUR site visitors respond to content, you, in essence, neglect their voice. By neglecting them, you inhibit your growth and deprive a community of people dedicated to evolving our world wide web experience of data which can contribute to greater understanding. Gaining this perspective is the key to unlocking the true potential of a process-driven analytics practice and a wildly valuable means to building the new architecture of the world wide web.

A Lofty Prediction with Exponential Considerations

If I had to place a bet on the future of web analytics, it would be on services and solutions. Commercial vendors like Omniture and Coremetrics (and Google Analytics when/if that time comes) will provide a series of broad tools, which, at least Omniture, has started doing. As the analysis and application of insight suggested actions cascades down from those platforms, industries, and markets will eventually respond with vertical groups and practices zones. Some will split specifically into advertising, others into expertise on eCommerce, still others into Social Media or Branding or what have you.

Over time, probably a couple years before enough talented analysts can blossom and contribute, markets and geographies will develop their own identities where conversations and comparing notes will likely drive highly-predictive analytics and scaled systems for buy-in by the most frugal small/medium business ventures. Shared resources will again be the natural progression (outsourcing is a natural evolution of industry). Small but incredibly effective micro-solutions based on research compiled by ambitious teams driven by numbers will dominate each niche feeding more information back into the collective process.

Google Website Optimizer, the multivariate testing suite offered with AdWords, has a few little used or known tricks which a practical analyst might find useful. One which I will discuss here is the ability to manipulate the ‘Goal’ script to prevent overloading conversion goal pages with multiple scripts. Its a really simple fix, and a simple and practical idea for running many tests simultaneously, without creating statistical overlap or technical issues for your most important pages.

After recently completing work on a major MVT undertaking of Global Elements on the CableOrganizer.com site, Paul, Juan and myself had built up a significant cache of tests which we wanted to launch right away. Everyday in our morning web initiatives meeting we found ourselves getting anxious to get on new things right away. Juan was looking to put some of our button calls-to-action to the test as well as our process to the cart. Paul wanted to run a follow-up on the headlines portions of our previous test. I got shivers thinking of how much fun its going to be to test a static old version of a major problem page versus a long-researched strategy to implement our targeted content provided by in-house design and the SiteBrand solution.

PROBLEM: Four tests with the same conversion goal running simultaneously.

SOLUTION: Run test with same goal and a single script alluding to several test designations.

Say your site (or client’s site) is based on a conversion objective of online sales. This means that you probably have a shopping cart solution. Most solutions provide a way to manipulate the code as it is presented in HTML by the server scripting. This means that you’d have to potentially put 1 full script for each multivariate test running to identify a goal based on that objective. However, all parts of the script are not necessary to identify as a goal appropriate to a page individually.

Check it out: Here is a screenshot of all our tests currently running using ‘Add to Cart’ as a conversion goal (for reasons we will explain some other time).

All of the tests above are running on one script with 9 different goal codes referred to in the test provided designation function. Implemented in the appropriate place, your script should take on an appearance similar to this:

The victory here is in the ability to not worry about script-heavy pages essential to conversion (i.e. shopping cart). Added benefits are that the script can be altered or exchanged to meet the needs of the test. Instead of adding 9 lines of code, you can get away with 1. Quick, simple and effective from that point on.

If anyone has any interest in, or problems with this, feel free to write me an email or comment here on thoughts.

Early in my training as a ‘web analyst’, Paul (my boss and the COO from CableOrganizer.com) set out some goals for me which included checkpoints to my actually being considered ‘mature’ in my designation. Those steps included tracking and reporting KPIs and insights on a weekly basis, regularly scheduled usability tests, and the successful completion of a multivariate test. I set my sights on those goals while Paul prepared the road for installing a data decision process in the business.

The purpose of this post is to explain the progression from carbon-based report handler to full scale and skill realized web analyst. It will attempt to point to areas which may help new and developing analysts gain momentum in the perfecting of their craft. Lastly, it will outline a few quick experiments which might help prime the process engine. (If you’re looking for ways to improve your value, I’m handing you the recipe)

For the purpose of support, the testing I will be describing includes:

• A/B or Split Testing Using Google Website Optimizer - testing single variant elements per URL which traffic is directed into by a script. Goal success is then attributed to the page.
• Full Factorial Multivariate Testing with GWO - testing multiple elements per single URL, simultaneously, where the items tested are identified within the page and rendered randomly. Goal success is distributed across the table of elements per variation and aggregated statistically to determine relevance of each element to success and compared to all variations and the original.
• Usability Testing -qualitative test to gain insight into a user session experience on a given site where a professional, random, or representative subject is observed performing tasks and navigating a web site. These can be performed by usability experts in labs, by simple observation and analysis, or through a handful of testing services whom use and relay audio/video files back to the customer.

These experimentation methods, in their simplest form, provide an analyst with a set of tools to validate their assertions. The outcome of GWO tests which are set up correctly and run to completion provide invaluable statistical justification for keeping or replacing elements within a website. Usability testing these same areas and elements should augment the data by providing a qualitative perspective to the findings.
Making experimentation the goal forces developing creative hypotheses. Looking back this seems to be the most essential right of passage into the world of active practice of analytics. Where measurement, assertions, and hypotheses are part of the analytical psychosis; knowing there are systems in place to support or diminish statements forces us to think forward. It is by this means that assertions, testing, and ultimately improvements to the site become more innovative and increase the chances of greater success.

Continued success in testing and measuring site improvements for a primary goal (i.e. - conversion) increases the value of an analyst and their merit as authority among colleagues. It has been my personal experience that the more you test and objectively report complete results, the more weight your contribution is given. Sometimes things do not support your hypothesis and it is equally as important that these results are given to the appropriate people. Should the analyst be lucky enough to be surrounded by highly intelligent peers, the resulting discussion from success or failure from each hypothesis should be equally as fruitful in insights on which to base future hypotheses.

GWO Experiments need not be enormous and complicated from the start. Get into testing by making up four or five alternative headlines for a high traffic page. Try each of the testing methodologies. Here’s a quick test to try just to get the mechanics down:

1. Identify and analyze a page for testing with decent traffic and lackluster performance. (*This will help benchmark performance to understand impact more clearly.)
2. Create four or five suitable alternatives (with at least one marginally poor headline to create divergence).
3. Make an appropriate number of copies of pages to match the number of variant headlines
4. Rename pages of variations and supply new URLs to Google
5. Install GWO provided scripts on original page
6. Install GWO provided script on variant pages
7. Install GWO conversion goal script on goal page
8. Test the scripts
9. Execute test

After a few days, or weeks, depending on the level of traffic and the apparent difference in your variations, you should experience some divergence which can begin to allude to validating and supporting, or, possibly diminishing your claim. (again, regardless of the outcome, so long as you have clear data, the test should be considered a success)

(I’ll publish an edit with some photos and some tips on usability here when I have the time.)

Getting a test under your belt can be an enormous benefit. Just knowing you can perform a test makes you think differently. That aspect of perspective is a huge step in getting to where the real analysis takes place.

If you’re planning on making some movement toward delivering targeted content, and by current estimates of interest in the topic I’d imagine sooner or later you would, you’ll eventually have to make some motion to measure the actions and the return on costs. In the past year, CableOrganizer.com installed a collaborative project to increase our ability to prepare and deliver targeted content based on rules and dispositions for arriving traffic. We used SiteBrand as our solution and measured the results both in their tool and using our Omniture SiteCatalyst interface.

This article should provide insights into the following assertions:

• Targeted Content was successful at minimal application based not solely on statistical regression but in measurement on a per click basis.
• Common and steady revenue returned on investment remains consistently above 500-700%. (Based on our costs by the SiteBrand implementation)
• Time and effort to create and deliver targeted content is worth the resource investment, at first, second and final pass when based on good analysis
• Omniture, though not yet partnered with SiteBrand, can efficiently and effectively measure this solution without extra costs or set up fees.

Targeted Content delivery is, although dynamic and requiring moderate to advanced analysis, not very difficult to implement. The process is relatively simple. You find areas on your landing pages which you would find highly visible and actionable and create alternative code which can be applied. Alright, maybe not quite that simple, but conceptually conceivable.

To determine what needs to be placed in that area, do some analysis. Take a good look at what is bringing people into the landing pages which you’re having problems with. In SiteCatalyst you can harvest a world of data at the page level. The same is true of Google Analytics. Find out why your users are landing on that page and create for them a page which they can relate to and act on based on their arriving disposition. In May 2007 at the eMetrics Summit, a gentleman named Christopher from Microsoft gave an incredible diagram of how to achieve this using dedicated servers. For most of us, this is cost prohibitive. For the rest of us, we can easily break down our 8-10 most important means to arrive at a page and concentrate on that.

Getting in the groove is half the obstacle to making a great, thoughtful, reactive landing page. Don’t sit and wonder for months trying to break into delivery. Sit down, hammer out a bunch of alternative code and images, and get the stuff out there. You will truly be surprised at the value of the pressed effort upon measuring. There is always room to refine and rework problem areas when you get moving. You’re mantra here should be something akin to: “Jimi Hendrix wasn’t born with a guitar and a bandana” or “Michael Jordan was cut from his High School Basketball Team”.

Measure your work; and don’t make one source your text on philosophy. Its a funny thing, but SiteCatalyst was not really set up to measure applications like SiteBrand out of the box. As a result of our content being served offsite, or at least the ‘object’ being served in and out by SiteBrand, we’re able to count these creative ‘Zones’ as campaigns. No eVar necessary. Caution: think about your campaign architecture for this before you input all your tracking codes. I would strongly urge you to consider numbering zones, naming creatives and delegating versions, ownership, and any partner agencies. The more complete and uniform these are when you set them up, the better off you are later in follow-up analysis. The following diagram should show how our zones behave and what type of value we’ve seen returned from our foray in to measured targeted content:

As you can see, there is a fairly solid level of performance in some zones over others. It is interesting to note, that, while the Zone 3 banner shoved into the corner here above is in what most consider a ‘Cold’ area of the site, its since had tremendous success. This feeds interesting insights back into the process. Suddenly, we’ve found ways to make areas of our site previously invisible to the user seem very relevant. Click rates and participation from this zone currently have participatory value in more orders than any two. I guess that means a possible cruise is a powerful ‘click-motivator’.

The big picture is that targeted content is affordable. It can be done with relatively little resource consumption. It can be tested. It can be measured (at least with SiteCatalyst). It can drive facets of the analytics process. It can raise conversion. You have that much information. I would be interested to hear anyone’s accounts of similar experiences.

For more information on any of what is used or discussed above, please feel free to contact me by comment or email. You can contact SiteBrand here. The publication from a Case Study with CableOrganizer.com is available on their site.

Google’s wonderfully free Website Optimizer tool just got a little better. Apparently, they’ve heard enough of us complaining about the problems which existed with the cookie life. Previously this was only available on a session basis. So, no matter how long you set the cookie for, it would expire or disappear with the closure of a browser or tab. In theory, the session could last with a refresh for up to the amount of time specified in the _utimeout function of the script which was placed on the cookie.

As of 7pm, 1 November 2007, this cookie is now set for a specified life of up to 2 years across sessions. Specificity regarding the creation date of the experiment is a consideration. About that Google stated:

“Experiments created prior to October 30, 2007 won’t use these new
cookies — this will include follow-up experiments and copied
experiments created after October 30 that are based on original
experiments that were created prior to October 30. For example, if you
created an original experiment on October 10th 2007, then created a
follow-up experiment based on the original on November 10th, 2007,
that follow-up would not use 2-year cookies. “

You can check out the rest of the details if you want to by clicking into their WebEx Playback here. The original release of this information is also available through the Google Website Optimizer BETA forum.

As always, I love to talk about these types of things. If anyone has any questions or comments about how to deal with GWO, or problems with specific testing scenarios, please feel free to drop a line. Please also remember to sift through some of the testing “tagged” posts if you need any help getting some more advanced testing done. I previously posted a great deal about Taguchi Multivariate Tests, conversion proxies, and cookie manipulation on my old blog. I’ll be migrating parsed versions of these very soon.

Many sites are built around templates or around a system of reproducing common elements throughout the established navigation theme. The components of these site-wide elements are often very important elements contributing to conversion but easy to ignore from a multivariate testing perspective. The purpose of this post is to explain an apparent problem in test development which, when countered with an offered method, may help produce great gains. This methodology was developed for a site in a retail setting and can be reproduced effectively, with sufficient traffic, in a relatively short time.

Google Website Optimizer is my free multivariate tool of choice. Personally, it has not let me down. There are times when it takes some creative leaps to uncover the appropriate methodology, but I have yet to run into a situation where I cannot produce the test I need by the means provided. The testing described here is what I would consider VERY ADVANCED. I say this because it requires a high level of tool familiarity, a solid understanding of the available approaches, site operations/mid-level developer programming skills, and significant pre-test documentation and elements preparation.

Step 1. Choose your elements and variations wisely.
There are lists of items which exist for the purpose of placing relative emphasis on certain global elements of a page which users value and which contribute, in part, to the conversion goal of a site. These usually include graphic and text link calls-to-action, application cues, logos, headlines and taglines, trust elements, brand affiliations, unique selling propositions, and rotational zones. Attach some custom tracking to each of these to quickly uncover the relative value of each of these as they perform in the context of each presentation within the path. Uncover those most highly related to your conversion goal across a variety of paths and plan your test around these.

Step 2: Consult Best Practices
Do the research to find out what are common and best practices for each of the elements which have been designated as having a positive correlation to the conversion metric. Apply several variations of top practices to each of the elements. Isolate them, prepare the variations, install them where they need to be and test them prior to incorporating them into the set up of the test. Ensure that each fits the space exactly as it should be described in code. Copy and paste all the backside code into very comprehensive tables for the purpose of saving time on re-setup as well as helping for organizational and statistical modeling purposes.

Step 3. Set Up Google Optimizer Framework
If you’ve used Google Website Optimizer, you’re probably aware of how the tagging works. For global elements, those which occur on every page throughout the site, you will want to ensure that the script snippets are placed as close to the top and bottom of the OUTPUT HTML as possible. This means they may have to go into your template, or your dynamic page framework as necessary. This may include placing them inside of a PHP file or some other server side application execution file. As long as they appear correctly in the output, this won’t create a problem.

Step 4. Set Conversion Goal
You probably already have a page which you consider your ultimate indicator of final conversion. For a retailer this is usually the thank you page. There you will install your conversion goals tag. In some cases, where traffic might seem deficient or other obstacles exist, it might be advisable to come up with a conversion proxy. Sometimes this discussion sets off purists who feel correlation success is not enough to provide clear relevance to testing outcomes. For our purposes, so long as traffic in UNIQUE VISITORS is high enough to produce statistical validity over 4-5 weeks, the closer to actual conversion the better (try cart additions, or shipping page if not).

Step 5. Install Variation Splices and Build Element-Variable Catalog
Take the time to carefully match up these really sensitive areas in your site code to the variations which you intend to affect. Insert the code for the testing scripts where they need to be. (Remember, they need to be ordered and set correctly based on the OUTPUT of server executed code). Once these splices are in place, Google will present the interface to begin creating variations and give you the opportunity to load in the codes from your table. Name these variations based on some system of organization which will promote quick and simple identification. Test them over and over again to ensure that you’ve not negatively impacted any major feature on your site. (I can speak from experience that these codes can play hell on site-search features when they’re hosted away from your normal code. Ensure that you either can include or exclude this page from your testing for certain. Having to reset is very frustrating).

Step 6. Do a Full Final Review of Every Variation
Look for things like variation duplication, missing or misspelled image location URLs, text variation miscues, color issues, anything that doesn’t look right. Close out each one on your spreadsheet or checklist. Make sure that you have every possibility looked into and have another person come in and look at each. Extra eyes help at every step.

Step 7. Cross Your Fingers and Execute
Once you’ve gone down the list at least a half-dozen times, hit the button and let it live.

Step 8. Restart the test with a copy after making the changes to the items which you overlooked.
Every test which I have set up so far with a medium/high degree of difficulty has required that I immediately stop the test after a rep or associate finds something wrong with a variation. I’ve come to love this process and the Touch of Grey it creates. Set it up again, relaunch and prepare to repeat steps 6-8 until you start getting clean tests with good data.

Within a couple days, you can get the first series of relevance ratings. In a few more, you should start to see divergence. By a couple weeks, a clear winner will get way out ahead. Just sit back and let the data do the work for you. By 40 days or so, depending on the number of combinations and the traffic, you should have a very complete package and incredible insight as to what YOUR customers are doing and relating to as valuable on your site. This should give rise to several other analysis scenarios and Voila. Multivariate testing success.

Hope this is helpful. I know its kind of complicated. Feel free to send in comments or questions. More to follow.

Daniel W. Shields