What a tiler told me about security

Technology Trends
An old tiler once told me that all bathrooms blighted by bad tiling, go wrong before a single tile had been stuck to the walls.

He then showed me how he worked out exactly where to start tiling on each surface, so as to ensure that he didn’t end up with a 1/4” tile cut into a corner somewhere, and that he didn’t stop 1” short of the ceiling and so on. In this fashion, with a few simple steps, he determined the position of every tile he would lay before beginning work. Thereby ensuring that each would have sufficient surface area to support being firmly stuck into place.

I’m reminded of him every time I see developers tearing into a new project without any thought as to placement of the code or security…

Consider a hypothetical greenfield project. Perhaps a web front-end to a database and application server, with a few supporting services hosted on a collection of servers within a datacenter or hosted platform like AWS. With an eye to cost, the servers are running Centos Linux, with the DB perhaps MySql or Maria DB, or some other open source solution.

The point where this project should start – and never does – is with the hardening of the servers. They should be completely locked down to a standard like CISCAT. This will ensure all kinds of good things are implemented, like:

  • Removing execute permissions from /tmp and other areas where code should not be run.
  • Ensuring that known exploits cannot be implemented via lax permissions.
  • User’s accounts are locked down, preferably via LDAP, to ensure that users who fail to understand security aren’t exploited by hackers.
  • Applications run under their own user accounts in directories that they-alone own.

In fact CISCAT lists several hundred things that should be done to harden a vanilla Centos build, and other OS’s. If you don’t mind paying for them, you can buy images off the shelf from them. Once all that good stuff is done, switch on the firewall for good measure and block everything but SSH.

Now you are ready to give it to the developers, who will likely complain that the box is so well locked-down that they can’t do anything.


Now we have forced them to begin in the right place. Want to develop an application? Fine, start by creating the user it will run as and think about what level of access that user will require.

Next, where on the box will it run? Define the directory structure and set the minimum permissions it requires. Does it need execute permissions to read an inbound file? No, so  don’t set execute and write permissions in the inbound directory. Does the application user need a terminal session? No, so set No TTY in the ssh config. Now if someone hacks the user account, they can’t get a command line.

Now, instead of retrospectively realising that you need to secure your cluster, and likely starting an expensive project just to do that, you have a cluster of servers that are each hardened in their own right, in addition to residing behind the usual web firewall and other obstacles to the malicious. If an intruder hacks their way through the web firewall and proxy, they have to do the same again for every single server in the cluster, hacking a different account each time.

Organisations make this mistake time after time, only to belatedly realise that in order to get the security accreditation they need to handle their customer’s credit card transactions, they now have to retrospectively try – and typically fail – to secure their shiny new web site. 

In reality, this hardly ever happens and they reach an uneasy compromise. Most organisations rely on keeping the bad guys out of the cluster via firewalls and DMZ’s etc. It’s not that they don’t want hardened clusters, they just can’t implement them because their infrastructure wasn’t designed that way from the start. Start locking stuff down retrospectively, and things start to fail.

So if you want my advice, bite the bullet and spend a week or so hardening your base server image before starting development. You’ll thank me later and save yourself a lot of pain and money in the long run.

Bradley Atkins


Bradley has extensive software engineering experience ,specifically of *NIX Operating Systems, (Sun, AIX, Centos) to maintain and troubleshoot the teams estate of test and development clusters.