Same account name, two different servers

We run a number of servers and a number of domains, and the domains do not map neatly onto the servers. Over the years several were consolidated, split, migrated and consolidated again, which is normal and usually harmless.

Then we spent an evening rebuilding a site and pulled the wrong files three times in a row.

How it happens

Hosting accounts get named after the brand rather than the domain. It reads well. It is the obvious thing to do. It also means a company that owns a dot com and a dot net can end up with two accounts on two different machines carrying exactly the same name, each with a folder at exactly the same path.

So the path you copy from muscle memory is valid on both machines and contains completely different sites. Nothing errors. Nothing warns you. You get a real archive full of real files belonging to the wrong property.

There was no error at any point. Every step succeeded. The files were simply the wrong ones.

What finally caught it

Not the folder name and not the file listing, both of which looked plausible. It was a single line inside one of the files: a canonical URL naming a domain that was not the one we were working on.

The second time, it was the wording of an error page. The page being served did not match the page we had just uploaded, which meant the upload had gone somewhere else or had never landed. That one line of text was worth more than an hour of checking directories.

What we changed

• Confirm the machine before confirming the folder. The path is not identifying. The machine plus the path is.
• Read the file, not the filename. A canonical tag, a page title or a phone number inside the file tells you which property you are actually holding.
• Where a company owns several domains, put the domain in the account name rather than the brand. It is uglier and it is unambiguous.
• When something you just deployed does not appear, suspect the destination before suspecting the deployment.

The broader version

Naming conventions get written when you have three of something. They fail when you have thirty, and they fail quietly, because a convention that produces collisions still produces perfectly valid names. Nothing in the system objects.

If you have been running your own infrastructure for years, spend twenty minutes listing every account across every machine and looking for duplicates. We had two, both entirely reasonable at the moment they were created, and together they cost us an evening.