Search Engine Optimization Featured Image

Recovering from a Website Migration Gone Wrong

This is the story of how a budget web design service tanked a small online business with a website migration gone wrong. This customer came to us after the fact for help.  While Reich Web Consulting has had success in helping them recover, their story serves as a cautionary tale for others.

For the sake of privacy we’ve changed the names of everyone involved in this story. I have no desire to publicly shame the company involved in this story even if they deserve it.

What’s a Website Migration

website migration is when a website’s domain name is switched from pointing to one version of a website to another. This can happen when when a website is redesigned on the same platform. A good example would be when a company rebuilds their site with a new WordPress theme.  This can also happen when a website is rebuilt on a new platform. An example would be when moving a website from SquareSpace to WordPress or vice versa.

Migrating a website requires more planning than launching a new website. Since search engines, social media, and other websites already have accumulated links to your site, you have to ensure that those links are not broken in the process. Otherwise the Internet’s collective trust in your site will plummet.

How a Budget Web Design Service Ruined an Online Business

A few months ago I received a call from the owner of Hand-Forged Widgets Inc. (not their real name)The company is a husband & wife team selling some very attractive iron work they make themselves. They had a flourishing store on a platform that had gone stagnant and chose to move their website to a WordPress/WooCommerce solution.  Green Servers (changed for privacy) is a well-known budget hosting company that also offers web design services. Green Servers sold them on a low-cost all-inclusive package. They would host and migrate their content to a new site on WordPress and WooCommerce.

Green Servers delivered on what they promised. But their promise what a ticking time-bomb, and the client was not SEO savvy enough to see it coming.

A Website Migration Gone Wrong

The moment Green Servers updated DNS to point to the new site, Hand-Forged Widgets business flat-lined.  I don’t mean they experienced a small and temporary reduction in sales. I mean orders stopped coming in completely for months.

Hand-Forged widgets found us through our YouTube Channel and decided to give us a call. As soon as I had access to their website and Google Search Console data the problem was obvious: the new website wasn’t redirecting the old URLs. Whomever built the site failed to account for the hundreds of existing URLs.  Every link to every page of their websites that search engines had indexed was now a 404 error. To make matters worse, their website had been switched from http:// to https:// without a proper redirect between them.

Google lost trust. Fast.

The customer was no longer appearing in either organic or paid search results. Google Search Console was full of hundreds of Crawl Errors, and Google’s crawl rate for their site was flat. This suggested that Google no longer found their site valuable enough to crawl and update in their index. And since Google stopped crawling their site, their new URLs would never make it into search results.

A Successful Website Migration Starts with Planning

Green Servers treated the project as a new website. They didn’t apply any of the planning and care required to successfully migrate a website from one platform to another. They made two simple mistakes:

  1. The designer didn’t collect and redirect all the URLs from the old website.
  2. When they switched the site to SSL-only they didn’t setup a redirect from http:// to https://.

Let’s explore these topics in some detail.

Why Redirecting Old URLs Matters

First let’s understand a few important search engine terms: crawl and index.  Search engines crawl websites. This means that the search engine starts with a list of websites, downloads important information from those websites, and follows any links they contain. Wash, rinse, repeat. Search engines keep an index of websites: basically a massive list of links on the Internet. The search engine uses a collection of algorithms to determine when which indexed pages should display in search results for which search terms.

Existing websites have already been indexed by Google. This means that search engines have already compiled a list of pages on your website. If those pages instantly disappear, what do you think will happen?  Google’s algorithm may consider your content unreliable, and your rankings will suffer.

How to Handle Redirects

You can see that it’s critical to maintain the integrity of your URL structure when you migrate a website.  This process doesn’t have to be difficult, and there are two ways to handle it:

  1. When building the new website, take care to maintain your URL structure. Use a tool like Xenu Link Sleuth to compile a list of existing URLs. Compare them to the new site’s URLs before launch and make sure that all existing URLs will still function.
  2. Use a tool like Xenu Link Sleuth to compile a list of existing URLs on the old website. Use this tool to build a list of 301 (Permanant) redirects from old links to the most appropriate pages on the new website. Import this redirect list into your new website.

Green Servers did none of this. As a result Google, Bing, and others essentially saw Hand-Forged Widgets website disappear instantly. A search engine stays successful by offering high-quality links in their search results. If your links are no longer high-quality there is no reason to serve them up.

The Right Way to Handle a Conversion to a Secure URLs

It’s true: Google offers higher ranking for secure websites versus insecure websites. But the switch to https:// needs to be done strategically. If your site will be served completely over HTTPS you should setup a site-wide permanent redirect from the http:// version of your URLs to their https:// equivalent. These redirects keep links intact and tell Google to use the secure version of the URL rather than the insecure version going forward.

So what happens if you neglect to redirect?  Google and other search engines have a concept they call duplicate content. Duplicate content occurs when two web pages contain identical (or very similar) content.  Search engines hate duplicate content, and penalize it. As far as you’re concerned you probably see the http:// and https:// versions of your website as the same thing.  Search Engines do not, and if both are accessible and serving the same content, they’ll see duplicate content.

So when you setup SSL on your website it’s important to handle it correctly in one of the following ways:

  1. Setup a site-wide 301 redirect. Most CMS software makes this easy, either through a configuration option or a plug-in.
  2. Use rel=canonical to tell search engines which version of your URLs are authoritative. Again, most CMS software makes this pretty simple.

Related Side Note: decide on whether you want to standardize on the www or non-www version of your website and make sure that one version 301 redirects to the other. Otherwise you can face duplicate content issues between the two as well.

The key take-away is to make sure that search engines don’t see multiple versions of your content due to lack of planning and bad configuration.

Can These Mistakes Be Fixed?

If your web designer/developer botches your site migration, can the mistakes be rectified?  Of course. In the case of Hand-Forged Widgets I’ve been working with them for two months, and we’ve made significant progress.

The problem is that the damage done by a website migration gone wrong is long-term.  Discovering and repairing the problems using the advice above isn’t particularly complicated. But improving your site’s reputation with a search engine once’s it’s damaged can be slow, and completely at their discretion. Fixing the technical issues that caused Google to lose trust in your website isn’t enough.  You also need to give Google a reason to start trusting your content again, and give it reasons and signals to start crawling your content at regular intervals.

Ways to Suggest to Google to Crawl your Websites

Here are a few ideas on how to jump-start that process.  I owe a thank you go the guys at the SEO 101 podcast  and the SEO 101 Community at Google+ for fielding my question about this. I had already implemented most of their suggestions before the episode aired, but they did confirm my theories on how to convince Google to crawl the customer’s site.

  1. Fix all of the technical issues described above.
  2. Monitor Google Search Console for crawl errors and fix them as-needed.
  3. Submit a sitemap.  If possible, use the sitemap to specify a high crawl frequency to suggest to Google that they should re-crawl your content sooner than later.
  4. Create new, high-quality content.  Creating new content will give Google a reason to crawl your site.
  5. Get active on social media. Start publishing to social media sites and sharing links to your website. Social media activity can suggest to Google that your site is alive, well, and active.
  6. Paid Search. Putting all your eggs in the organic search basket can lead to disaster. Site changes like those that happened to my customer can tank your ranking. And ranking algorithms change.  Website owners can find their Google ranking tanked through inaction. Google changes ranking factors regularly, and rankings can drop when sites fail to accommodate them. Utilizing paid search is a great way to keep steady traffic to your site through those hard times. And successful interactions from paid search visitors can help Google see that your site is, indeed, worthwhile.

The Moral of this Story

This story should help web designers and developers properly plan website migrations.  It should also help business owners know the right questions to ask when they’re hiring a developer or agency to redesign their website or migrate it to a new platform.  These are the key steps to follow:

  1. Ensure that all old URLs are left intact, or that 301 redirects have been correctly implemented to maintain link integrity.
  2. Make sure that multiple versions of your site don’t result in duplicate content problems with search engines.  Ensure that you have a single “canonical” version of your website (either www or non-www) and that the non-canonical version 301 redirects to the other. If you’re switching to SSL-based URLs make sure http:// is 301 redirected to non-http://, or that you’re utilizing rel=canonical to tell Google which site to consider the primary source of your content.
  3. Utilize tools such as Google Webmaster Tools to monitor your website’s health and crawl statistics.

Six Ways to Find 404 Errors On Your Website

404 Not Found: even if you’re not a web designer or a programmer you’ve probably seen this error before. But if you have been living underground in the disconnected world of the mole people for a few decades, 404 Not Found is the error code on the web that means you’ve tried to access a resource that doesn’t exist.

404 errors need to be dealt with. When a customer experiences a 404 it’s a missed opportunity for you and a bad user experience for them. When a search engine experiences a 404 error the missing resources could be removed from their index, and it could be translated as a signal that your website is unreliable.

But before you can fix a 404 Not Found error you need to know they’re happening. This article explore six easy ways to discover 404 errors on your own website.

1. Find 404 Errors Using Server Logs

One of the easiest ways to discover 404 errors is by utilizing your hosting environment’s access logs and error logs. Every hosting environment is different so unfortunately I can’t explain where to find yours, but a Google search should prove fruitful. Searching for “cpanel raw access logs” turns up a plethora of helpful pages for the CPanel hosting environment.

Your log files may need some massaging to be useful. Most are text files that can be easily opened in Excel and then filtered by HTTP response code.

The Pros: This method should show you all 404 errors that occurred on your site in the time frame covered by the log.

The Cons: Your hosting environment’s log files can be difficult to read and utilize unless you know your way around a spreadsheet.

The raw access logs from this website. These file contain lots of data and need some help to be useful.

The raw access logs from this website. These file contain lots of data and need some help to be useful.

2. Find 404 Errors Using a Spider or Link Scanner

This method doesn’t actually find 404 errors. It discovers broken links on your website so they can be fixed before they generate 404 errors. An ounce of prevention is worth a pound of cure.

An easy way to find potential 404 errors is by scanning your website with a spider or link scanner. A spider indexes your site the same way that a search engine like Google does: it starts with a URL and scans the code for links, and then works through that list recursively. There are lots of programs and online services that can scan your site for free, and will provide a list of broken links that exist on your site.  My two favorites are Xenu Link Sleuth and Screaming Frog SEO.

The Pros:

Using a spider to locate broken links on your website doesn’t actually find 404 errors: it helps prevent them.  By scanning your site, discovering, and fixing broken links you’ll prevent your visitors from visiting URLs that don’t exist and reduce the number of 404 errors that occur on your website in the future.

The Cons:

Anyone can link to your website, and you don’t have control over the URLs that they link to. Just because you fix all of the broken links on your website doesn’t mean that other websites, or even search engines, don’t have active links to broken URLs on your website. You won’t be able to discover or fix those using a spider.

A report from Xenu Link Sleuth. Xenu requests every URL of your site and returns the status code, among other things.

A report from Xenu Link Sleuth. Xenu requests every URL of your site and returns the status code, among other things.

3. Find 404 Errors Using Google Analytics (and Yoast SEO)

This one is a little WordPress-specific, but you can do a similar trick with other content management systems.

If you use the Google Analytics by Yoast plugin, it automatically tags 404 errors so you can find them in Google Analytics using the Content Drilldown tool. Just go to Behavior > Site Content > Content Drilldown and do a search for 404.html.

The Pros: 

It’s right in Google Analytics where you would expect to find this sort of data. Because it’s in Google Analytics you can export your list of 404 errors to do something useful with it, like construct a list of URL to redirect.

The Cons:

It’s WordPress specific. It requires you to install yet another plugin on your website to basically embed a few lines of JavaScript for Google Analytics.

Yoast SEO automatically registers your 404 errors in Google Analytics.

Yoast SEO automatically registers your 404 errors in Google Analytics.

4. Find 404 Errors Using Google Search Console

Google Search Console (formerly Google Webmaster Tools) provides a window into how Google sees your website. Under the Crawl Errors Google Search Console provides a list of all connection errors that occured while Google attempted to index your site.

The Pros:

Google regularly spiders your site and attempts to index any URLs that it finds, as well as any URLs it already had in it’s database. If any URL stops working, whether it’s new or historical, Google Search Console will let you know.  These reports can be exported to CSV, so you can do something useful with them, like create a redirect list. This is also helpful because essentially Google Search Console is discovering broken links before actual humans are, so check regularly and act on it.

The Cons:

If you have any pages that you’ve intentionally prevented search engines from indexing either through robots.txt or through a noindex tag, Google will not index them and, therefore, not check to see if the links still work. Google Search Console only displays 404 errors generated by Google’s crawler and not actual users.

Using Google Analytics to Find 404 Errors

Using Google Analytics to Find 404 Errors

5. Find Errors Using SEO Redirection (or another 404 Management Plugin)

This tip is WordPress specific, but most content management systems have a similar feature or plugin.

There are lots of plugins that can help you discover and fix 404 errors. I use SEO Redirection as well as it’s premium sibling, SEO Redirection Premium. These plugins track 404 errors that occur on your website and help you easily resolve them by redirecting the broken URL to an existing page. Yoast SEO’s premium version also has this feature built in. Yoast SEO Premium will hook up straight to your Google Search Console account, and allow you to redirect 404 errors discovered by Google from within the plugin.

The Pros:

WordPress can become your one-stop shop for discovering and dealing with 404 errors and broken links. And if you’re willing to pay for a premium plugin, you can really automate the process.

The Cons:

WordPress specific, and all of the problems that come with installing plugins. To make the most of this method you should  purchase the premium version of one of these plugins. But trust me: they’re worth it.

404 Errors Caught by SEO Redirection Premium

404 Errors Caught by SEO Redirection Premium

6. Find 404 Errors with Other Services

There are a host of other services that can help you discover 404 errors including and Moz. I only bring these up for the sake of thoroughness. There’s absolutely no reason you need to sign up for a paid service just to discover broken URLs and 404 errors, but these services do a lot more than that and are worth investigating.

The Pros:

Another set of eyes scanning your website for errors can’t be a bad thing. Plus these services offer far more than just 404 error reports.

The Cons:

Services like Moz don’t come cheap. So don’t use them unless you’re looking for more than just a 404 error detector.

Moz 4xx Error Report

Moz 4xx Error Report


So what do I do?  I use a combination of all of the methods above. A link scanner or spider like Xenu helps you proactively discover and fix broken links within your website, but can’t help you with links from other sources like other websites or Google. Google Analytics helps you discover URLs that are actually breaking as people attempt to view them. Google Search Console helps you discover broken links that Google either has in their index, or is trying to index. WordPress plugins like SEO Redirection and Yoast SEO Premium help me easily deal with 404 errors as I discover them. And of course, I already use Moz for other reasons, so I take it’s 404 report into consideration as well.

In another post, I’ll be talking about how to handle 404 errors on your WordPress website once you find them. Stay tuned!