Systems Administration Portfolio

First off, let me say I don’t usually refer to myself in the Third Person. That said, looking into a list of qualifications for a Computer Guy such as myself can easily drift off into what sounds like Greek mythology. The point of this is to highlight some of my projects/tasks in a *relatively* easy to read format.

Disaster Recovery

For quick restoration, I developed and maintain a backup / restore procedure for multiple web servers. Full virtual machine Backups using Trilead VM Explorer, partial backups are also run with a script backing up key folders and MySQL databases which are then backed up to a backup server running open source BackupPC.

Web Analytics

Web analytic are key to search engine optimization and building a website. Using tools such as Google Analytics and open source AWStats which read the actual server log files, we are able to analyze traffic and bandwidth.

Windows System updates and Security Patches

Originally, I would run Windows Updates when I noticed that little update shield by the clock. While this might work for 50-80 servers, this method is not very scalable. Thanks to setting up a Windows Update Server, it’s much easier to release updates and see which machines have patches installed, failed or needed. This is not only an incredible time saver but also great for maintaining overall security across multiple machines.

Infrastructure / System / Server Monitoring

Monitoring systems is very important to maintain a 99.9% up time. Various monitoring tools and processes allow this. Programs such as SpiceWorks, a custom website monitor and PSToolsi n addition to others allow for excellent infrastructure stability.

Domain Migration

Built a new 2008 domain server and secondary server and thoroughly tested prior to migrating about 120 web servers from a Windows 2000 Domain with little impact to our e commerce customers other than the reboot.

Virtualization Architecture

Suggested this by a boss at the time, I leaped at the chance to virtualize our E commerce data center using VMWare’s ESXi 3.5. While it was a new technology to me, there is really no reason to be scared of change, especially something so promising as server virtualization. Physical to virtual (P2V) migrations drastically cut rack space and power usage while improving performance in every benchmark. For example, while a 1u server might use around 100watts, our newer 2u servers which can hold over 30 virtual machines, pulls under 300 watts total.

Storage Area Network

You can’t virtualize your server infrastructure without storage. The earliest setup from 2008 uses NFS on a 2003 server and is still used today. Given the budget constraints, I then re purposed an older server as an OpenFiler ISCSI SAN server. This is still working great, partially thanks to the multiple paths allowed with an ISCSI connection. While still limited to 1Gb cables, the multiple paths improve overall throughput over a fairly standard 1Gb network. Furthermore, even if a cable is pulled, the multiple paths keep everything up without so much as a hiccup. More recent servers to the Storage network include 2008 data center edition, RAID 10, ISCSI connections, and of course multiple paths.

VLAN Cabling
Multiple VLANs represented in each color cable
Dell Poweredge SAN running 2008 Data Center edition and Openfiler SAN

Remote Access

You can’t run a data center without remote access. While Windows Remote desktop is OK, it is not as good when multiple people want to connect to the same machines. PCAnywhere was the solution prior to my starting, but when I saw it would be $100 per machine for a program that was just OK, I looked elsewhere. It didn’t take long to see UVNC was the way to go. 6 years later, UVNCis still the remote access of choice and I’m sure the $20,000 a year in licensing fees the company continues to save is nice too.

Central Managed Antivirus

Running a separate antivirus across hundreds of servers doesn’t make sense. With centralized antivirus such as a Trend Micro Server, Antivirus protection is significantly better while being much less time consuming and more reliable.

System Images

Installing Windows takes a long time. By keeping a current image using tools such as Acronis, VMWare Converter and Sysprep, its much quicker and takes most of the hands on portions out allowing time to work on other, possibly more meaningful projects.

IIS Optimization

Through IIS (internet information systems) optimization, noticeable improvements in site performance can be achieved. As one coworker put it “You turbocharged their site!” Even changes in browser caching alone has the affect of dropping page load times from a couple seconds to under 1. While this might seem like much, any edge your store can achieve over another will only help.

MySQL Optimization

Not every server is the same, nor is every customer or site design. As such, MySQL optimization needs to be performed to keep everything running as smooth as possible. Multiple variables are modified including, but not limited to; query cache, table cache, table size, key buffer size, key_read_requests/key_reads and more. Modifying these settings in addition hardware configuration such as RAID level, number of disks, RAM, CPU, etc all go into making a solid, snappy and reliable MySQL server.

Network / Server / Website / Data Center Security

Every Network Administrator needs to stay on their toes. When managing 200 company web servers, there are exponentially more possible threats. Firewalls, PCI Scans, general network scans, security patches, and keeping up with all the ‘script kiddies’ is very important in keeping hundreds of thousands of customer records safe.

Email Server

Moving to a new data center, we found ourselves in need of a reliable email server. A long term distribution of Ubuntu Linux combined with open source PostFix met the needs quite nicely.

DNS

Often referred to as the Internets phone book, the Domain Name System can be rather confusing. Whether is local DNS such as Active Directory or an internet name server, I am well versed in the different records such as A, MX, CNAME, and SPF.

Eliminate Repetitive Tasks through Automation

Managing hundreds of servers effectively can seem like a daunting task. Automation is vital in keeping a data center up to date with software patches and installs. For example, using a combination of software installing batch scripts and PSTools to push it out updates, what used to take hours or days can now be done in a matter of minutes. Another example is with AWStats. By modifying the configuration files and using a powerful editor such as Notepad++, I have cut down the install time from about 2 hours to nearly 15 minutes.

Home Theater PC Complete with DVR

Unsatisfied with offerings from the local cable company, I set out to build my own DVR. Mission accomplished thanks to Beyond TV by Snapstream! With nearly 200 hours of HD recording, anywhere access to our shows, automatic guide updates, and how easy it is to use, this ranks very well with my lovely wife.