Subscribe to Updates

Click here to subscribe to new posts by email. We use Google FeedBurner to send these notifications.

Laridian Website – Server Migration Post-Mortem

Posted on: August 5th, 2015 by Craig Rairdin 10 Comments

Update – Aug 5, 2015 – We discovered that the sync server wasn’t running, so you weren’t able to sync your user data even after the DNS changes propagated. This has been fixed. If you had trouble synchronizing your notes, highlights, and bookmarks, try again.


 

This article will mean very little to most of you but some of you might find it slightly interesting.

I don’t like to get off a perfectly good horse midstream. Windows Server 2003 and SQL Server 2000 have been serving us well for a very long time. But Microsoft discontinued support for  Windows Server 2003 last month and that means no more security updates. As a result, our next PCI security scan was doomed to fail, which doesn’t mean your personal data would be at risk (any more than passing the PCI scan protects the data), but it doesn’t make for good PR.

(PCI is the Payment Card Industry and the scan is required to meet their card security standards. Laridian doesn’t store your card data so the requirements for us are pretty light. You’re at significantly more risk when you hand your card to the clerk at Target, and we all know from experience that even though they passed all the security scans it didn’t do them much good.)

Upgrading to Windows Server 2012 and SQL Server 2012 meant physically moving the contents of our servers to new hardware, and as long as we were doing that, we decided to shop around for better pricing. We found it at SingleHop.com, which offers the type of dedicated server solutions that we need. And they do it for half the price of the company we had previously been with.

Laridian maintains several large databases. These contain your customer account, your transaction history, all the user-created data you’ve sync’ed to the Laridian Cloud, and the downloadable files that make up our books and Bibles. In addition to those we have a couple others for in-house purposes. Because SQL Server 2000 is past end-of-life, those databases could not be imported directly into SQL Server 2012. We had to first import them into a SQL Server 2008 instance, then import that to SQL Server 2012. The good news was that having done that, the databases functioned exactly the same. In fact, we were able to simply point the old website to the new database, and it would work just fine.

Let me pause for a minute to say this: We cannot build PocketBible 1.0.0 for iOS anymore. Not only would the resulting program not even come close to working on iOS 8, it would fail in the compilation and linking process. PocketBible 1.0.0 was released in September 2009, just six years ago.

Contrast that with the code that runs on our website. In 1998, we contracted with Jomax Technologies — Bob Parsons first Internet company, before he founded GoDaddy — to create our e-commerce site. There is a significant amount of code dating from 1998 still running on our site, especially on our back office site, which is where we generate sales reports, create new product pages, and define priority codes. This code is running unaltered seventeen years later, accessing a database that has been in continuous use for all those years.

When you hear me complaining about the unnecessarily rapid pace of development from Apple (and other companies who drive our industry) and how they create problems due to their lack of regression testing and backwards compatibility, this is what I’m talking about. Because Microsoft knows it would be a huge problem to break millions of websites, they go out of their way to continue to support the technologies on which the internet is built.

But I digress…

This move took place over about a four-week period. (I actually thought it would take twice that long.) The first step was to move the system we use for source code archival (SourceGear Vault). This was necessary because we use Vault to maintain the website. We check code out of Vault, make our changes, and check it back in. Vault populates the website folders from the files we check in. It would be most convenient if we could continue to do this the same way on the new website.

Because we were running Windows Server 2003, we couldn’t upgrade to the latest version of Vault, which requires Windows Server 2008. And because we were running such an old version of Vault (version 5) we couldn’t upgrade to the latest (version 8) directly. We had to first upgrade to version 6, which upgraded our database, then upgrade to version 8, which upgraded it again. With Vault working on the new server, we were able to move software development and book production to the new server within about two weeks.

Prior to moving Vault we had captured a snapshot of the websites and databases and were running those on the new server for testing. This allowed us to do a quick test to verify that the Web pages themselves would run under Windows Server 2012 and IIS 8. They worked just fine.

Next, we knew that during the transition to the new site there would be a period of time while DNS changes were propagating during which we would have to access the new database from the old website. We ran some tests that verified this would work.

Last, we had to build the Laridian Sync Server Service code with Visual Studio 2015 and verify it worked. It did.

At this point we spent a couple days locking down the firewall settings. We had a period of just a few hours when we had an open SMTP (email) server that was exposed to the outside world. I was shocked by how quickly the spammers discovered it (by literally rifling through IP addresses and sniffing for servers). We worked with SingleHop to quickly lock that down.

Now we just needed a procedure to follow for getting a final copy of the database moved to the new host. The problem was that we couldn’t be writing to it while it was being moved, which meant shutting down all commerce, account updates, product registration, and user-data synchronization for some unknown period of time. SingleHop estimated two hours to move the database, but suggested we allow three. I allowed four. (It took five.)

Once we knew what needed to be done, we picked a convenient date and time to do it. We were able to configure much of the code to just shut itself down at 8AM CDT the day of the migration. So, for example, the code to do synchronization of user data from PocketBible for Windows would still be there waiting for connections, but starting at 8AM (7:30, actually) it would return an error code to the client saying that the site was down for maintenance. By automating that process, it meant there were fewer manual actions that needed to be taken in the moments before the migration began. In fact, by the night before the move we were down to where it would take only ten check boxes and about 3-4 button clicks to completely bring Laridian to a stop for the two, three, or four (or five) hours it would take to move the database.

The morning of the move, I discovered SingleHop had left themselves logged into a server, which blocked me from getting in to click on two of my ten check boxes. I asked them to do that for me, which was not a problem. Then I discovered that the person who I was told would be doing the migration hadn’t been told about that fact until 30 minutes after the migration was to have begun. He was rousted out of bed or wherever he was, and started moving the databases.

With that small hurdle overcome, and with the website having automatically stopped processing new transactions to the database, I was able to get in and make three lines of code changes that it took to point the old website to the new database server. That was really all I needed to do during the time while the database was being exported from SQL 2000, imported to SQL 2008, exported from SQL 2008, and imported into SQL 2012.

The last step of the move was to make the DNS changes required to point everyone to the new server. It was at this moment that our registrar (GoDaddy) decided to make buggy changes to their website that kept us from changing our zone file. After two or three hours of attempts, we were finally able to get those changes made.

In the end the move turned out to be easier than I thought and was significantly less time-consuming than I had anticipated. I’m sure we’ll discover small things that are broken, but the major functionality of the new servers appears to be working.

Thank you for your patience during the move.

 

 

Tags:

10 Responses

  1. Lawson Culver says:

    Yep… I definitely found it an interesting read. My web-based side business has been around for right around 9 years. We’ve had three different hosting companies, but five server moves. I’m the sole employee, so I had to do all of that stuff. Even though I’ve done it five times, there’s still consistently things that pop up that I never anticipated.

    My last hosting company change was due to a weeklong outage that they couldn’t guarantee wouldn’t last longer (while I was losing money.) Consider yourself lucky that you got to plan your move in advance. 🙂

    • Craig Rairdin says:

      We haven’t had a significant outage since the shared hosting days (1998-2002 for us). Our last hosting company didn’t have an outage the entire 12-13 years we were with them, and that included a period of several days when the power was out all around them. They had one incident where they fried a RAID controller during a routine drive change. They recovered from a backup, then sent the drives out to recover the database. From there we were able to re-create 2-3 days worth of transactions that were missing from the backup. It took several weeks but the actual outage was very short.

      I considered moving to a local hosting company just because it was local, but they only have one big internet connection. The last place had 9 providers in the data center we were in. 🙂

  2. edh says:

    Yes, very interesting reading.

    So, now that you’ve gone through this, will you do the same thing, i.e. run as long as you can on what you have, or will you try to keep more up to date with the server products so you don’t have to do double/triple upgrades to get to the latest. For example, if you know SQL Server 2023 will not support direct migration from SQL 2012, will you move to SQL 2020 even if it isn’t necessary just so when you are forced to go to SQL 2027 for some reason, it isn’t jumping through the same hoops you had to do to go from 2000 to 2012?

    We are facing similar questions at our office.

    • Craig Rairdin says:

      Upgrading the OS or a major server component like SQL Server can’t be done while the server itself is live, so if there’s an upgrade to SQL Server, we’re talking about moving to a new machine. Same if there’s an IIS upgrade or a Windows Server upgrade. The move to a new machine with a new OS or SQL Server instance takes the same amount of time whether it’s a major update or a minor update. The added time to do the SQL 2008 intermediate step was only about 2-3 hours in a process that took 4 weeks (of calendar time; probably could’ve done it with a dedicated team and 100% focus in less time). So there’s not a huge motive to do every upgrade.

      We run into this in other situations all the time. For example, I mentioned upgrading our Vault SCM. They do constant upgrades and it disrupts business here significantly when they do. I prefer to wait until I absolutely have to before I take the time to debug their latest upgrade.

      In my experience, I might spend 10% more time to do a version X to version X+N upgrade than I would take to do a version X to version X+1 upgrade of just about anything. It’s never worth doing every single upgrade.

      Instead, it’s important to take to your blog and Facebook to complain about the unceasing march of progress. 🙂

  3. Susan says:

    I’m not a programmer or any other type of computer expert, I just like it when things work as expected. And I did like reading about your experience during this transition.

    Note: I tried syncing my data this morning, from both my Android phone and my iPod Touch, but succeeded on neither device. The phone just stops trying after a time, giving no message. The iPod reports “PocketBible Alert No response from the server. Try again later.” Is this expected?

    • Craig Rairdin says:

      It was kind of expected. We expected it to take time for the DNS changes to propagate, but then when it wasn’t working 24 hours later, we dug deeper and found that the sync server hadn’t been restarted after updating it to remove the “temporarily unavailable” message. Should work now.

  4. Anita Smith says:

    Not sure if this is the right place to log these issues. I am getting a few errors. 1) I purchased a couple of dictionaries yesterday. They synced fine with my iPhone, but not showing up on my mac. 2) Books/Cloud Library-getting a certificate issue. 3) PocketBible/Check for Updates-get an error saying it can’t be done now. Not sure if 2 or 3 impact the ability to see the new purchases.

    I enjoyed the post on the move. I work with a technology team and it was very interesting to see you follow very similar best practices!!!

  5. Jeff Crowder says:

    Thought you might like to know that the new server still syncs my ORIGINAL iPad! I have not used it much since I got an android tablet. Picked it up the other day and charged it. I read the blog about the server updates. Mainly out of curiosity, I did a manual sync since I had turned off auto sync while beta testing my android tablet. Within seconds, I had months of devotional readings and sermons on my iPad. Very cool!

Leave a Reply

©2017 Laridian