Systems Engineer (DevOps Lead) • August 2017 — June 2019
Reporting directly to the VP of Engineering, I helped automate processes, improved security stances, prototyped complex comprehensive foundational solutions (HashiCorp Vault and Consul), and stabilized production systems and engineering workflows.
- Refactored Terraform and Ansible infrastructure-as-code, with community-supported Terraform modules and Asible Roles.
- Deployed AWS IAM Instance Profiles in Terraform, streamlining integration with AWS Platform Services.
- Prototyped Hashicorp Vault (for Secrets Management) and Consul (for DNS-like Service Discovery), in a High-Availability, clustered mode.
- Evangelized Vault for Secrets Management with senior developers.
- Sent pull requests to Vault Terraform Registry modules with improvements.
- Moved infrastructure code to Vagrant-centric testing and validation model, where appropriate.
- Evaluated and implemented Trend Micro's Deep Security as a Service on all AWS hosts.
- Discovered, isolated and remediated a pre-existing security incident on a critical development instance.
- Implemented OpenVPN Access Server for Engineering, for access to AWS VPCs.
- Recommended and configured pfSense Firewall for office network, with OpenVPN access.
- Opened discussions with various service vendors like JumpCloud for LDAP authentication and New Relic.
- Initiated "Best Practices" Epics for Security, AWS, Terraform, Ansible, and others to bring attention to practices and workflows.
- Introduced company to The 12 Factor App
Senior Technical Operations Engineer • June 2015 — February 2016
For a blockchain-enhanced, cryptocurrency-based financial services startup, I worked as lead on projects to automate rapid, reliable deployment.
- Building on experience gained in my previous position, continued my deep dive into Chef and Ruby.
- Utilized Test Kitchen for all code, and implemented code reviews for infrastructure code.
- Transitioned live cluster network of Ripple protocol clients from single bare-metal provider to SoftLayer.
- Began work on implementing Chef Provisioner for machine provisioning with Softlayer and Amazon AWS/EC2.
- Saved them at least $12,000 per month by decommissioning almost one hundred orphaned servers and services.
- Mentored sharp Desktop IT Admin on Operations concepts and career (thanks to the master SysAdmin Tom Limoncelli and crew's books).
Productivity and Systems Engineer • August 2014 — May 2015
Stealth social startup in SF and Palo Alto, venture-backed by Google. I was the first non-Google engineer hired in the Palo Alto office.
- Learned Chef, utilizing Test Kitchen + Vagrant
- Migrated all Hello code to a GitHub Private Repo
- Performed in-depth 3rd-party development and productivity tool evaluations
- Linux SysAdmin/DevOps process introductions, advising development team on infrastructure impacts.
- Transformed in-house build scripts to Travis-CI.com YAML for both mobile client (iOS) builds and backend server builds (Java w/ Ant, yeah, I know, meh).
- Moved production onto RightScale, which included moving to Percona MySQL Server, database replication farm, large storage arrays with GlusterFS, and all on Google Compute Engine.
- Oversaw deployment of New Relic OS Agent on all Linux servers, on Java/Tomcat applications, and on Mobile iOS client. (New Relic captures EVERY crash!)
- Negotiated costs with vendors. LOTS of automation.
Senior Linux Systems Engineer • August 2011 — August 2014
Oversaw legacy site "Codesion.com" aka "CVSDude" transition after acquisition, to become "cloudforge.com".
- Reduced a large amount of technical debt and planned how to pay the remainder.
- Tuned MySQL, stopping database issues cold.
- Converted RedHat Enterprise stack to CentOS *in-place*.
- Restarted package upgrade process, which had been frozen in place by fear for over a year.
- Lead a team of 3 engineers in maintaining 48 physical servers and 5 "cloud" virtualized servers at SoftLayer, in multiple datacenters.
- Learned Puppet and completely revamp existing Puppet legacy infrastructure for much improved reliability, repeatability and code reuse.
System Architect/Systems Administrator/Computer Duster • April 2009 — September 2011
For an IT consulting company dedicated to supporting non-profits with size-appropriate technology, worked with over a dozen Bay Area non-profit clients (and one international client) to put out fires, architect solid solutions, and provide excellent documentation and personable service. Prominent clients included Youth Employment Partnership of Oakland, Inc., ACCESS Women's Health Justice, and over a dozen others on an as-needed basis.
Systems Administrator/IT Manager • March 2008 — October 2008
For a prominent non-profit micro-finance lending web site, supported and improved internal and production systems.
- Built out and migrated to new web and database server architecture, based on CentOS 5.1, Apache 2.2, Zend PHP and MySQL 5, from previous single-host legacy host on CentOS 4.4
- Implemented load-balancing across web hosts
- Implemented MySQL replication, MAATKIT MySQL tools
- Implemented Nagios, NagiosQL, Cacti and custom Cacti HTTP transaction latency scripts and graphs
- Tuned Apache, PHP and MySQL, based on load, informed by benchmarks and organizational goals
- Implemented log file summarizers pflogsumm and logwatch, improving visibility into production system health for development team
- Documented system build out recipes and architecture overviews
- Implemented the use of Subversion for configuration file change control
- Created code deployment scripts
- Initiated alternative co-location vendor communications
- Identified administration and performance problems with locked-down virtualized guest hosts and lack of visibility into system performance metrics
- For office network, implemented Windows Active Directory roaming profiles and JungleDisk corporate backups to Amazon S3, as well as switching virus scanning to Kaspersky AntiVirus for all Microsoft Windows systems
Systems Administrator • September 2007 — March 2008
For one of the oldest weather sites on the Web, supported existing systems and network devices, assisted software developers and assisted the Director of Technology in maintaining three data centers.
- Cleaned main data center of old hardware, including rack-mount servers and network equipment; arranged pickup and reuse by local 501(c)(3) charity (OTX-WEST), which I selected.
- Supported PXELINUX diskless systems (loosely based on Slackware 9.1).
- Implemented NTPd (with multicast updates) across diskless hosts.
- Created more streamlined and tuned diskless-oriented Linux Kernel builds.
- Streamlined BIND configuration files for ease of updates and standardized shared options, and implemented master/slave BIND servers with NOTIFY updates to slaves and to ISP DNS servers.
- Assisted with BIOS reconfigurations to fix boot up problems, and tuning for PCI Express.
- Upgraded internal MediaWiki instance to latest version.
- Documented previously under-documented systems.
- Handled system and application outages.
- Assisted with planned data center power outage.
- Proposed and implemented home-grown Highly Available NFS filer with RAID5 providing 5.4 TB usable storage, using Gentoo Linux, Heartbeat, DRBD, and Areca 1220 RAID controller cards with RAID battery backup unit, on two servers with virtual IP failover. Provided benchmarks, determining better hardware RAID controller models and facilitating kernel tuning.
- Improved communications from Operations to the development and application teams.
- Fixed a variety of problems with NFS, recommended standard client mount options.
- Advised senior staff on system performance and administration process improvements.
- Upgraded Awstats, and rewrote automated log processing Bash shell scripts, using looping constructs to maximize usage of all CPU cores, while minimizing impact on server resources. Initiated use of non-memory intensive and non-disk intensive web server log aggregation tools. Log processing time cut by a two-thirds, and solved inconsistent data representation.
- Coordinated vendor on-site maintenance and upgrades.
- Server upgrades and service migrations (Slackware 9.1 to 12.0), including Sendmail, milter, and Apache, and internal user directories.
- Assisted developers with proper system utility usage, and developer virtual machine development OS instances.
- Racked, cabled and configured servers, and performed general hardware upgrades.
Jumpcut.com (née Yahoo, Inc.), San Francisco, CA
Systems and Network Engineer • July 2006 — April 2007
For a "Web 2.0" video editing, sharing, and community startup, supported systems and network devices, and assisted software development.
- Production applications supported include Apache 2.0.X, PHP 5.X, MySQL 5.X, all running under Gentoo Linux
- Performed testing and configuration optimization
- Polished Cacti and Nagios implementations, implemented teMySQL templates, and created a custom script, data sources and templates for HTTP latency graphing
- Implemented new Apache SSL hosts
- Implemented OpenVPN LAN-to-LAN VPN between office and co-location facility
- Created "chroot jails" for various production applications, using the Jail shell, sudo, and a bit of shell scripting, with an aim to make safer calls from production applications to external applications
- Implemented SQLGrey for managed Postfix Grey-listing
- Added secondary backup "store-and-forward" Postfix MTA with SQLGrey integration
- Worked with CTO on development of applications, including creating development environments using VMWare, WINE and Xorg X11 drivers
- Implemented OpenWrt (http://www.openwrt.org/) on office router
- Managed corporate desktop systems migration to Yahoo's infrastructure
- Configured Yahoo's custom Apache and PHP, and deployed site code to Yahoo! production systems as a part of Yahoo's acquisition of Jumpcut
- Wiki'd documentation of systems and project planning
Verizon Business (née Totality, Inc), San Francisco, CA, via Taos Mountain, Inc.
Senior Systems Engineer • December 2005 — July 2006
Supported Verizon Business Managed Service Provider clients, such as the Gap, Best Buy, American Airlines, and many others, in a Service Level Agreement model:
- Improved production procedures documentation and process
- Supported business-critical Solaris, Red Hat Linux, HP-UX, AIX, and Windows systems and applications. “Unrusted” proprietary UNIX skillset
- Suggested, and helped pilot, Plone/Zope CMS for replacement of aging CVS-updated AFT/HTML documentation system. Worked with Perl programmer to create automated site scraping for deployment of read-only version of Plone site, using Bash, cron and wget
- Resolved client issues alongside database administrators and network engineers
- Trained new systems engineers
- Consistently high marks from management
Systems Administration Consultant • March 2005 — June 2005
Advised on and implemented the following services:
- Assisted the Systems Administrator in closing a backlog of requests
- Implemented a spam flagging system using Postfix and DSPAM
- Upgraded internally-facing wiki (TWiki) to latest stable version
- Implemented a VPN solution with OpenVPN, and added to the OpenVPN codebase
- Documented progress and systems with Bugzilla and TWiki
Deployment Engineer (Consultant) • July 2003 — June 2004
Contracted by former full-time supervisor:
- Packaged and deployed difficult multi-layered software to over 1200 Windows workstations, in a validated, controlled and FDA-regulated environment
- Wrote Perl scripts to automate local configuration changes and systems administration on Windows workstations
- Prototyped and assisted in the evaluation of Gentoo Linux/Zope Content Management Frameworks development and staging servers
- Managed Linux development server relocation to data center
Systems Administrator (Consultant) • October 2002 — November 2003
Contracted to provide the following services for PhD-level researchers and staff:
- Proposed and implemented web-based full-featured shared calendaring system
- Reduced costs by re-purposing older hardware
- Used Red Hat GNU/Linux 7.3., stripped down to HTTP, HTTPS, NTP and SSH network services, with Big Brother 1.9c for monitoring
- Installed and configured Web Calendar
- Used iptables to restrict access and usage to two subnets
- Prototyped web-based survey system Mod_Survey (defunct)
- Created role-based shared folders with Windows 2000/XP "Offline Files" functionality
- Security assessments, using a variety of tools, including Nessus, nmap and Ettercap
- User Training (very high marks!)
Denalii, Inc. (defunct), Burlingame, CA
Senior Systems Engineer • March 2001 — March 2002
Responsible for the following systems and services for heterogeneous Production network:
- Network Applications: BIND 8, ProFTPd, Apache, NTP, OpenSSH/SFTP
- Web Application Platform: Oracle 8, Orion Application Server, Apache, PHP 4, Sawmill web log analyzer
- System Monitoring: Big Brother, Syslog (centralized)
- Hardware: Sun/Intel
- Operating Systems: Solaris 8, Red Hat Linux 6.2 & 7.1
- Network devices: Cisco (3600 Routers, 4006 Smart Switches, ArrowPoint load balancers), Netscreen 100 firewalls
Supervised the following projects:
- Rolled out NTP to all hosts, syncing to both internal servers and external time servers
- Rolled out Big Brother, for host monitoring, role-based paging, and uptime tracking
- Migrated HTML content from Roxen to Apache
- “In-sourced” corporate web site, DNS, and network administration (phased out SiteSmith)
- Performed and managed technical aspects of closing engineering-centered office
- Maintained production Perl environment, including module maintenance, configuration and updates (DBI, DBD::Oracle, MD5, etc.)
- Deployed applications onto Orion J2EE Application Server (1.5.2), resolved Windows-to-UNIX migration issues
- Quickly and thoroughly undid sloppy administration practices of previous Systems Administrator, including correcting bad kernel build from unstable kernel tree, remaking DNS structure and updating many applications for security and bugfixes
- Implemented security policies, including password aging and auditing, regular Nessus scanning, regular network packet sniffing using snoop (Solaris), tcpdump (Linux), Ethereal for capture and analysis (Linux & Windows), and nmap (both external and internal network service scanning)
- Kept documentation up to date. Worked closely with Senior Oracle DBA to both maintain and improve current Oracle administration on Solaris, rolled-out Oracle on Linux, and also trained her on more advanced Systems Administration tasks to act as my backup, and she trained me on more advanced Oracle administration techniques
Helped guide developers on architecture of logging and alerting systems to integrate with existing Open Source tools, made use of Open Source solutions for all appropriate areas, kept costs to a minimum, created generally stronger and more closely held policies and procedures for administration and maintenance of production systems.
Zing Networks, Inc. (defunct), San Francisco, CA
Director of Network Operations (promoted) • October 1999 — December 2000
Responsible for the following systems and services:
- Network Applications: Apache, Squid (HTTP accelerator mode), Sendmail, BoldFish (mass e-mailing server), OpenSSH
- Web Application Platform: Apache, Squid, mod_perl, Mason, Oracle
- Monitoring/Reporting: Aria Web Tracking, Big Brother, MRTG, PIXlog, LogWatch, swatch
- Hardware: Mostly Intel, with Sun for Oracle (E4500 primary, E3500 standby)
- Storage: EMC Symmetrix drive arrays and Celerra File Server, with 3 TB usable storage, Legato Networker with SpectraLogic Bullfrog AIT-1 tape jukebox
- Operating Systems: RedHat Linux 6.2, Solaris 7
- Network devices: Cisco PIX firewalls, Foundry Networks (ServerIron (Layer 7 switch), NetIron (Layer 3 switch), 3Com BayStack 350 (Layer 3 switch), point-to-point T1
- Vendor Relationships: Exodus, VALinux, Sun, SiteSmith, Redapt, GlobalCenter, Envive, Akamai, CDW
Supervised the following projects:
- Rolled out of new EMC Symmetrix node, adding 1.5TB of usable storage for photo & video storage
- Added SiteSmith to maintain high levels of uptime and support
- Planned Globalization of Zing's Network (Japan, Europe and US East Coast)
- Evaluated and recommended move to new data center (GlobalCenter to replace Exodus Communications)
- Supervised install and maintained Aria Web Monitoring system
- Akamized new sites static content
- Supervised two Sr. Unix/Network Administrator contractors
Implemented the following office network systems from the ground up:
- VPN with Compatible System's (now owned by Cisco) IntraPort 2 VPN router, with Win 95/98, NT 4.0, Linux and Mac clients
- MS Exchange 5.5 SP3, with TrendMicro's ScanMail for Exchange 3.0, Outlook Web Access
- Mapped external site e-mails to Exchange Public Folders
- Rolled out Outlook 2000, gave 3 classes for users, with rave reviews
- ARCServeIT 6.61 for NT, using OFA, Exchange Client, Client for UNIX (Solaris), Tape Library Option (with 30-slot SpectraLogic Treefrog tape autoloader with 1 AIT-2 drive) on 5-day Differential rotation
- Reduced PC architecture to 3 major vendors (Dell, Apple, IBM), and 5 models tailored to company needs
- Created PC purchasing forms and procedures
- McAfee's Total Virus Defense, managed from McAfee Network Management Console
- BoldFish mass-mail server for Linux
- Negotiated Microsoft Open License
- Employee Purchase Program with CDW
Supported the following:
- Microsoft DNS & DNS/DHCP/WINS (maintained static routes, migrated DNS to new server)
- Apache 1.3.12 on Linux, IIS 4.0
- 2 NT Domains, with 9 NT 4.0 SP6a servers and 50 desktop and notebook PCs and 6 Macs
- HP EtherJet (installed internal and external print device servers)
- Installed & maintained co-located servers at Exodus Communications in Santa Clara
- BigBrother Network Monitor on Linux
Supervised office Cat 5 Ethernet re-configuration & deployment, negotiated cost-effective vendor relationships, stopped company virus problems COLD, smooth transition from Netscape Messaging Server 3.62 to Exchange 5.5 SP3. Promoted from Systems Administrator (internal) to UNIX Systems Administrator for Production (public web site). Y2K updates. Supported 3 US offices, 1 Israeli office.
University of Pittsburgh
Unattained • 1988 — 1992
Majored in English, but completely unmotivated by academics. Radio station DJ, focused on Blues music and indie rock/experimental sonic underground.
Fox Chapel Area High School
High School Diploma • 1988
AP Computer Science, often called out from class to fix administration computers...
Fixing it (and Making it Faster)
For improvements made to software deployment systems