Tunnelling out of corporate networks (Part 5)

Read on to find out our solution in the fight against malware and Rich Internet Applications:

Articles 1 2 3 4 5

For some time now, we have been experimenting with LiveCDs running OpenBSD. The original idea was to use them as one of our disaster recovery solutions. We have also toyed with the idea of using LiveCDs for remote access but more to reduce operating costs than to increase security. One would also think that this would be an ideal solution to be used internally to eliminate infection and the spread of malware. Well, perhaps in some environments this would be the perfect solution but it was too radical, not very practical or feasible as a replacement for user workstations. Yet, in the end we did use LiveCDs for the following.

We extended the life of our old (8 years) Thinkpad laptops by turning them into read only images running OpenBSD, kind of like LiveCDs but using a 1 GB slice of the spinning hard disk. We would bring out the laptops for training sessions where the staff would just remote desktop to their own workstations. We used to use laptops running Windows for training sessions but moving to an OpenBSD LiveCD running off of read-only hard disks has been a huge time saver for us. However, in the end, LiveCDs were not used to prevent infection on the internal network. Our solution is finally detailed below.

Network segregation is one step that you can take to help increase security. A few years ago, we had the opportunity to build a network from scratch as we were moving into a new building. I remembered some advice that I read from the Unix Administration Handbook, "...it is important to design the network with expansion and increased bandwidth in mind. As cable is being installed, especially in out-of-the-way, hard-to-reach places, pull three to four times the number of pairs you actually need. Remember: the majority of installation cost is labor, not materials."

For our new office, we had three CAT5e cables plus a phone cable laid to each cubicle: one for internal (trusted) networks, one for Internet only (untrusted) networks and the third for a future IP phone network. All distinctly differentiated by coloured cables.

After moving into the new office, we would encourage our staff to bring in their laptops to be hooked up to the Internet only network for personal activity. On the internal network, workstation MAC addresses are locked down to each port on their respective switch as workstations rarely move. Yes, I know, it's not too difficult to change MAC addresses but in reality, we have very good physical security, a very good relationship with our staff and besides that, our monitoring system would detect and alert us if a cable was taken out and/or reseated. I should point out that without good physical security or if our staff were given laptops as their main computer, then our solution would have been different and our efforts to achieve the same end goal would have been considerably more challenging.

In hindsight, laying extra cables to each cubicle was one of the best things that we did to help improve security. Even our meeting rooms have Internet only ports for every seat that clients, vendors or guests can use. These physical networks have eliminated any need for wireless networks. I don't know if this is a trend or not but I know that some of the biggest Japanese companies have eliminated wireless networks in their buildings and offices altogether. They've even gone so far as to replace all employee laptops with workstations. This is an interesting and perhaps controversial move but without question a smart decision to increase and better manage security.

Physical and logical network segregation are good for security but unfortunately, they do not prevent infection of trusted hosts on the internal network where staff are able to browse the Internet. The problem was detailed in the second part of this series with the ever increasing threats coming through the browser as Rich Internet Applications. The next little bit will explain what we did to prevent this once and for all.

Physical network separation proved to be effective at solving a few problems. Our next step was to isolate Internet browsing from hosts with access to confidential and private information. There are many ways to do this and what we have done may not be what you would have done but we would be more than happy if others shared their experience in this area.

The main goal for us was to eliminate the potential of malware infection or exploit on internal trusted hosts. However, with users involved browsing the Internet, that would prove to be difficult even with Firefox coupled with NoScript and various other Add-ons. We can't just cut off the Internet. That's unreasonable, if not impossible to do in most companies. How are staff going to twitter or stay in touch and chat with their friends on Facebook, listen to Internet radio? Come to think of it, doing so would kill two birds with one stone. We would increase security and work force productivity at the same time. Realistically though, with any solution that we deemed suitable, we wanted to limit the impact to the staff that it may have.

The solution is simple. We run a pool of Windows Server 2008 Terminal Servers hosted on VMWare ESXi Servers as thin provisioned linked clones. These servers can only connect to the Internet. They are reverted nightly to good clean snapshots ready to be used the very next day. Here's a good article that we used and basically followed to get us set up using linked clones.

Staff are redirected to a pool of Terminal Servers with a little pf rdr round-robin when they connect via RDC. All Internet access from their own workstations is blocked except for a white list of internal applications that require Internet connectivity. This considerably reduces the exposure to these internal trusted workstations from harmful and/or unwanted exploit code or malware coming from their browsers.

The internal workstations are locked down to a great extent and staff are not given Administrator privileges on their own workstations. Since they have dual screens, having a Remote Desktop session on one screen for Internet related activity isn't that much different from having a browser open taking up the same amount of screen real estate. A read-only drive is mounted when a Terminal Server session is up just in case they need to download material available on the Internet for work purpose. Staff are able to retrieve material from the Internet but can't upload anything originating from the internal network. There is absolutely no business reason to do otherwise. We felt that this was a reasonable balance and compromise.

Finding a reasonable balance is important. All solutions need to be as business friendly as possible. Security needs to be transparent to the end user. Using the right tools for the job, questioning the status quo, thinking outside of the box and being as pragmatic as possible has helped us to build and fine tune the solutions presented in this series of articles.

To recap, I started out with why tunnels are bad. Then I explained why relying on end point security is a bad idea. I showed how we deal with this partly through network segregation and firewalls enforcing security policy. Logging, detection and alerting whenever there was a policy violation was made easy after we consolidated and correlated seemingly unrelated bits of information to ensure predictability on the network. Of course, malware detection becomes trivial when you do this but we wanted to go one step further and eliminate malware infection from ever happening in the first place. Now you know how we do this.

Finally, I should mention that most of the planning and work to set all of this up was done by Ryan McBride (mcbride@), CISA, CSA. Many of you may know him as one of the OpenBSD developers but to me, he's also one of the most security minded and professional individuals I've ever had the pleasure of working with. If you liked any part of this series, he's the one to thank. I'm just the messenger.

Mark T. Uemura