Subscribe to Updates

Click here to subscribe to new posts by email. We use Google FeedBurner to send these notifications.

Archive for August, 2015

Laridian Website – Server Migration Post-Mortem

Posted on: August 5th, 2015 by Craig Rairdin 10 Comments

Update – Aug 5, 2015 – We discovered that the sync server wasn’t running, so you weren’t able to sync your user data even after the DNS changes propagated. This has been fixed. If you had trouble synchronizing your notes, highlights, and bookmarks, try again.


This article will mean very little to most of you but some of you might find it slightly interesting.

I don’t like to get off a perfectly good horse midstream. Windows Server 2003 and SQL Server 2000 have been serving us well for a very long time. But Microsoft discontinued support for  Windows Server 2003 last month and that means no more security updates. As a result, our next PCI security scan was doomed to fail, which doesn’t mean your personal data would be at risk (any more than passing the PCI scan protects the data), but it doesn’t make for good PR.

(PCI is the Payment Card Industry and the scan is required to meet their card security standards. Laridian doesn’t store your card data so the requirements for us are pretty light. You’re at significantly more risk when you hand your card to the clerk at Target, and we all know from experience that even though they passed all the security scans it didn’t do them much good.)

Upgrading to Windows Server 2012 and SQL Server 2012 meant physically moving the contents of our servers to new hardware, and as long as we were doing that, we decided to shop around for better pricing. We found it at, which offers the type of dedicated server solutions that we need. And they do it for half the price of the company we had previously been with.

Laridian maintains several large databases. These contain your customer account, your transaction history, all the user-created data you’ve sync’ed to the Laridian Cloud, and the downloadable files that make up our books and Bibles. In addition to those we have a couple others for in-house purposes. Because SQL Server 2000 is past end-of-life, those databases could not be imported directly into SQL Server 2012. We had to first import them into a SQL Server 2008 instance, then import that to SQL Server 2012. The good news was that having done that, the databases functioned exactly the same. In fact, we were able to simply point the old website to the new database, and it would work just fine.

Let me pause for a minute to say this: We cannot build PocketBible 1.0.0 for iOS anymore. Not only would the resulting program not even come close to working on iOS 8, it would fail in the compilation and linking process. PocketBible 1.0.0 was released in September 2009, just six years ago.

Contrast that with the code that runs on our website. In 1998, we contracted with Jomax Technologies — Bob Parsons first Internet company, before he founded GoDaddy — to create our e-commerce site. There is a significant amount of code dating from 1998 still running on our site, especially on our back office site, which is where we generate sales reports, create new product pages, and define priority codes. This code is running unaltered seventeen years later, accessing a database that has been in continuous use for all those years.

When you hear me complaining about the unnecessarily rapid pace of development from Apple (and other companies who drive our industry) and how they create problems due to their lack of regression testing and backwards compatibility, this is what I’m talking about. Because Microsoft knows it would be a huge problem to break millions of websites, they go out of their way to continue to support the technologies on which the internet is built.

But I digress…

This move took place over about a four-week period. (I actually thought it would take twice that long.) The first step was to move the system we use for source code archival (SourceGear Vault). This was necessary because we use Vault to maintain the website. We check code out of Vault, make our changes, and check it back in. Vault populates the website folders from the files we check in. It would be most convenient if we could continue to do this the same way on the new website.

Because we were running Windows Server 2003, we couldn’t upgrade to the latest version of Vault, which requires Windows Server 2008. And because we were running such an old version of Vault (version 5) we couldn’t upgrade to the latest (version 8) directly. We had to first upgrade to version 6, which upgraded our database, then upgrade to version 8, which upgraded it again. With Vault working on the new server, we were able to move software development and book production to the new server within about two weeks.

Prior to moving Vault we had captured a snapshot of the websites and databases and were running those on the new server for testing. This allowed us to do a quick test to verify that the Web pages themselves would run under Windows Server 2012 and IIS 8. They worked just fine.

Next, we knew that during the transition to the new site there would be a period of time while DNS changes were propagating during which we would have to access the new database from the old website. We ran some tests that verified this would work.

Last, we had to build the Laridian Sync Server Service code with Visual Studio 2015 and verify it worked. It did.

At this point we spent a couple days locking down the firewall settings. We had a period of just a few hours when we had an open SMTP (email) server that was exposed to the outside world. I was shocked by how quickly the spammers discovered it (by literally rifling through IP addresses and sniffing for servers). We worked with SingleHop to quickly lock that down.

Now we just needed a procedure to follow for getting a final copy of the database moved to the new host. The problem was that we couldn’t be writing to it while it was being moved, which meant shutting down all commerce, account updates, product registration, and user-data synchronization for some unknown period of time. SingleHop estimated two hours to move the database, but suggested we allow three. I allowed four. (It took five.)

Once we knew what needed to be done, we picked a convenient date and time to do it. We were able to configure much of the code to just shut itself down at 8AM CDT the day of the migration. So, for example, the code to do synchronization of user data from PocketBible for Windows would still be there waiting for connections, but starting at 8AM (7:30, actually) it would return an error code to the client saying that the site was down for maintenance. By automating that process, it meant there were fewer manual actions that needed to be taken in the moments before the migration began. In fact, by the night before the move we were down to where it would take only ten check boxes and about 3-4 button clicks to completely bring Laridian to a stop for the two, three, or four (or five) hours it would take to move the database.

The morning of the move, I discovered SingleHop had left themselves logged into a server, which blocked me from getting in to click on two of my ten check boxes. I asked them to do that for me, which was not a problem. Then I discovered that the person who I was told would be doing the migration hadn’t been told about that fact until 30 minutes after the migration was to have begun. He was rousted out of bed or wherever he was, and started moving the databases.

With that small hurdle overcome, and with the website having automatically stopped processing new transactions to the database, I was able to get in and make three lines of code changes that it took to point the old website to the new database server. That was really all I needed to do during the time while the database was being exported from SQL 2000, imported to SQL 2008, exported from SQL 2008, and imported into SQL 2012.

The last step of the move was to make the DNS changes required to point everyone to the new server. It was at this moment that our registrar (GoDaddy) decided to make buggy changes to their website that kept us from changing our zone file. After two or three hours of attempts, we were finally able to get those changes made.

In the end the move turned out to be easier than I thought and was significantly less time-consuming than I had anticipated. I’m sure we’ll discover small things that are broken, but the major functionality of the new servers appears to be working.

Thank you for your patience during the move.



Laridian Website – Planned Outage

Posted on: August 4th, 2015 by Craig Rairdin 3 Comments

The server migration we did this morning (Tuesday, August 4) is complete as far as we can tell. The last phase of it is changing the various DNS records for our various sites so that they point to the new server. That has been done, but it takes time for the changes to DNS to propagate throughout the Internet.

Until that time, you’ll see a yellow marquee banner across the top of pages at and you won’t be able to sync your user data. If you see the yellow banner, the site will be slow because it’s talking across the wire to the database server instead of having it located “right next door” on its own subnet in the same building. Once the DNS change finds its way to your machine, you’ll be back up to full speed at the new site.

I’ll write up a little post-mortem article for the techies among you just for fun.

If you have problems with our site that don’t fix themselves by Wednesday, August 5, drop us an email at

Reason for the Outage

Laridian operates services on a variety of servers located at more than one hosting company. From time to time we move these services to new locations either to enhance their capability or to save money or both. We are generally able to do this in a way that minimizes or eliminates downtime. In this case, we are moving our database server, which stores almost everything of importance at Laridian including your customer account, transaction history, user-created data (notes, highlights, and bookmarks), and all our books.

It wasn’t possible in this case to make this transition without actually stopping all updates to the database, copying the data to the new server, and restarting it at its new location. During this brief time, we couldn’t do any operations that cause the database to change, or we risked losing those changes (i.e. they would get written to the “old” location after the database has been moved to the “new” location).


Once this whole process is complete, we expect enhanced performance of the website, sync service, downloads, and other related services. Security of all of these services will be increased. And despite the more powerful hardware on which this will all be running, our costs will be lower. This will allow us to continue to produce Bibles and reference materials at prices at or below what you’ll find elsewhere.


2:45PM Remaining DNS changes complete.

1:35PM Laridian Cloud sync services are back up. The IP address for synchronization has changed, so you may continue to get the “maintenance” message (or not be able to connect at all) until DNS changes propagate to your server. This could take up to 48 hours but in our experience most of you will see the change within a couple hours of it happening (which was actually a couple hours ago).

12:00PM Domain registrar is up and down. We have been able to make some DNS changes, but not all.

11:15AM Our domain registrar chose this time to go down. Of course. This isn’t a big deal, it just means that the sites will be slower. Once we can make DNS changes, the websites and the database server will be on the same subnet. Until then, the websites have to talk across the wire to the database, which means they’ll be slower. The worst part of this is that user data synchronization (Laridian Cloud) can’t be brought back up without changing DNS. We don’t anticipate this will take long.

10:45AM Commerce, product registration, account updates, and Apple App Store in-app purchase downloads are back online. The only thing currently offline is the Laridian Cloud (user data synchronization).

PocketBible 1.x.x for iOS users: If you got a message while trying to sync that said “You’re running a very old version of PocketBible“, it’s because you’re still using version 1.x and we’re currently on version 3. To upgrade, first sync (not just backup, but sync) your user data with the server after this maintenance is over. Then search the App Store for PocketBible. The program is free. Download and run it. Register using the same customer ID and password as you have been using, then turn on automatic synchronization under Manage My Data in the menu. The program will pick up your notes, highlights, and bookmarks and you’ll just have to download your Bibles and reference books.

8:30AM Migration officially started.

8:00AM Commerce, account update, PocketBible 2.x/3.x sync service, and other related services disabled.

7:30AM PocketBible for Windows and PocketBible 1.x sync services disabled.

©2017 Laridian