Experience

BoldAndBusted Consulting

Sole Proprietor • November 2022 — Present

My side-gigs for small and medium-sized shops! Coaching and coding for fun and profit. Some highlights:

  • DevOps/Site Reliability Engineering coaching for the "full org chart", from CEO to front-line developers
  • Advised a CEO on how to organize their engineering department, and how to create a high-performance, high-reliability software by focusing on reducing complexity for all developers, and creating positive feedback loops from hypotheses, fast failure and evaluations.
  • Advised a small environmental reclimation company's CEO on how to approach building an ERP platform themselves, usign Xero and the Xero ecosystem
  • Advised a CEO, their head of engineering on how to improve communications between their "devops engineer" and their "application developers". Coached their frontline engineer through creating a Internal Developer Platform, using Kubernetes, Backstage, and a GitOps approach.

AHEAD, Inc.

Computing Automation Consultant (DevOps/SRE/Cloud Automation) • December 2019 — Present

For a Chicago-based Technical Services company, provide technical oversight, demonstrations, advice and code development to internal technical teams of AHEAD's large corporate clients (Advantasure, Harley-Davidson, H&R Block, WideOpenWest! among others). Also provide assistance to AHEAD for internally-focused technical initiatives. Reading "The Phoenix Project" by Gene Kim, et al. for guidance and scoping. Member of the "DevOps" team, with a focus on various HashiCorp toolsets, from Terraform and Vagrant to Vault and Consul. Introduced "Documentation as Code" workflows and techniques to clients. Some highlights:

  • Deployed and configured Vault Enterprise to on-premise hosts for a major telecommunications company, for use with Kubernetes, using Ansible, and Vagrant as a local testing environment to reduce intra-environment variables.
  • Created a CI/CD and PR-based workflow with GitHub Actions and Terraform on AWS-hosted "Spot Instances" - keeping costs low - no jobs, no cost!
  • Python project to work with PynamoDB (for a more ‘Pythonic’ AWS DynamoDB interface than Boto3's native interface) in AWS Lambda with deployment managed by the Serverless Framework.
  • To enhance empathy for developers, taught infrastructure team at a client how to develop a Python Flask tutorial app, but with Test Driven Development, and pair programming techniques.
  • Guided clients using Miro (https://miro.com/) for client workshops, design sessions, and brainstorming sessions, with solid success.
Certifications gained:
  • Certified Kubernetes Application Developer
  • Certified Kubernetes Administrator
  • Certified Kubernetes Security Specialist
  • HashiCorp Terraform Associate
(Remote position with some travel.)

Abra, Inc.

Systems Engineer (DevOps Lead) • August 2017 — June 2019

Reporting directly to the VP of Engineering, I helped automate processes, improved security stances, prototyped complex comprehensive foundational solutions (HashiCorp Vault and Consul), and stabilized production systems and engineering workflows.

  • Refactored Terraform and Ansible infrastructure-as-code, with community-supported Terraform modules and Asible Roles.
  • Deployed AWS IAM Instance Profiles in Terraform, streamlining integration with AWS Platform Services.
  • Prototyped Hashicorp Vault (for Secrets Management) and Consul (for DNS-like Service Discovery), in a High-Availability, clustered mode.
  • Evangelized Vault for Secrets Management with senior developers.
  • Sent pull requests to Vault Terraform Registry modules with improvements.
  • Moved infrastructure code to Vagrant-centric testing and validation model, where appropriate.
  • Evaluated and implemented Trend Micro's Deep Security as a Service on all AWS hosts.
  • Discovered, isolated and remediated a pre-existing security incident on a critical development instance.
  • Implemented OpenVPN Access Server for Engineering, for access to AWS VPCs.
  • Recommended and configured pfSense Firewall for office network, with OpenVPN access.
  • Opened discussions with various service vendors like JumpCloud for LDAP authentication and New Relic.
  • Initiated "Best Practices" Epics for Security, AWS, Terraform, Ansible, and others to bring attention to practices and workflows.
  • Introduced company to The 12 Factor App

Ripple, Inc.

Senior Technical Operations Engineer • June 2015 — February 2016

For a blockchain-enhanced, cryptocurrency-based financial services startup, I worked as lead on projects to automate rapid, reliable deployment.

  • Building on experience gained in my previous position, continued my deep dive into Chef and Ruby.
  • Utilized Test Kitchen for all code, and implemented code reviews for infrastructure code.
  • Transitioned live cluster network of Ripple protocol clients from single bare-metal provider to SoftLayer.
  • Began work on implementing Chef Provisioner for machine provisioning with Softlayer and Amazon AWS/EC2.
  • Saved them at least $12,000 per month by decommissioning almost one hundred orphaned servers and services.
  • Mentored sharp Desktop IT Admin on Operations concepts and career (thanks to the master SysAdmin Tom Limoncelli and crew's books).

Hello.com

Productivity and Systems Engineer • August 2014 — May 2015

Stealth social startup in SF and Palo Alto, venture-backed by Google. I was the first non-Google engineer hired in the Palo Alto office.

  • Learned Chef, utilizing Test Kitchen + Vagrant
  • Migrated all Hello code to a GitHub Private Repo
  • Performed in-depth 3rd-party development and productivity tool evaluations
  • Linux SysAdmin/DevOps process introductions, advising development team on infrastructure impacts.
  • Transformed in-house build scripts to Travis-CI.com YAML for both mobile client (iOS) builds and backend server builds (Java w/ Ant, yeah, I know, meh).
  • Moved production onto RightScale, which included moving to Percona MySQL Server, database replication farm, large storage arrays with GlusterFS, and all on Google Compute Engine.
  • Oversaw deployment of New Relic OS Agent on all Linux servers, on Java/Tomcat applications, and on Mobile iOS client. (New Relic captures EVERY crash!)
  • Negotiated costs with vendors. LOTS of automation.

Collab.Net, Inc.

Senior Linux Systems Engineer • August 2011 — August 2014

Oversaw legacy site "Codesion.com" aka "CVSDude" transition after acquisition, to become "cloudforge.com".

  • Reduced a large amount of technical debt and planned how to pay the remainder.
  • Tuned MySQL, stopping database issues cold.
  • Converted RedHat Enterprise stack to CentOS *in-place*.
  • Restarted package upgrade process, which had been frozen in place by fear for over a year.
  • Lead a team of 3 engineers in maintaining 48 physical servers and 5 "cloud" virtualized servers at SoftLayer, in multiple datacenters.
  • Learned Puppet and completely revamp existing Puppet legacy infrastructure for much improved reliability, repeatability and code reuse.

Giant Rabbit, LLC, Berkeley, CA (via my now-defunct company Bold and Busted, LLC)'

System Architect/Systems Administrator/Computer Duster • April 2009 — September 2011

For an IT consulting company dedicated to supporting non-profits with size-appropriate technology, worked with over a dozen Bay Area non-profit clients (and one international client) to put out fires, architect solid solutions, and provide excellent documentation and personable service. Prominent clients included Youth Employment Partnership of Oakland, Inc., ACCESS Women's Health Justice, and over a dozen others on an as-needed basis.

Kiva Microfunds, Inc., San Francisco, CA

Systems Administrator/IT Manager • March 2008 — October 2008

For a prominent non-profit micro-finance lending web site, supported and improved internal and production systems.

  • Built out and migrated to new web and database server architecture, based on CentOS 5.1, Apache 2.2, Zend PHP and MySQL 5, from previous single-host legacy host on CentOS 4.4
  • Implemented load-balancing across web hosts
  • Implemented MySQL replication, MAATKIT MySQL tools
  • Implemented Nagios, NagiosQL, Cacti and custom Cacti HTTP transaction latency scripts and graphs
  • Tuned Apache, PHP and MySQL, based on load, informed by benchmarks and organizational goals
  • Implemented log file summarizers pflogsumm and logwatch, improving visibility into production system health for development team
  • Documented system build out recipes and architecture overviews
  • Implemented the use of Subversion for configuration file change control
  • Created code deployment scripts
  • Initiated alternative co-location vendor communications
  • Identified administration and performance problems with locked-down virtualized guest hosts and lack of visibility into system performance metrics
  • For office network, implemented Windows Active Directory roaming profiles and JungleDisk corporate backups to Amazon S3, as well as switching virus scanning to Kaspersky AntiVirus for all Microsoft Windows systems

Weather Underground, Inc., San Francisco, CA

Systems Administrator • September 2007 — March 2008

For one of the oldest weather sites on the Web, supported existing systems and network devices, assisted software developers and assisted the Director of Technology in maintaining three data centers.

  • Cleaned main data center of old hardware, including rack-mount servers and network equipment; arranged pickup and reuse by local 501(c)(3) charity (OTX-WEST), which I selected.
  • Supported PXELINUX diskless systems (loosely based on Slackware 9.1).
  • Implemented NTPd (with multicast updates) across diskless hosts.
  • Created more streamlined and tuned diskless-oriented Linux Kernel builds.
  • Streamlined BIND configuration files for ease of updates and standardized shared options, and implemented master/slave BIND servers with NOTIFY updates to slaves and to ISP DNS servers.
  • Assisted with BIOS reconfigurations to fix boot up problems, and tuning for PCI Express.
  • Upgraded internal MediaWiki instance to latest version.
  • Documented previously under-documented systems.
  • Handled system and application outages.
  • Assisted with planned data center power outage.
  • Proposed and implemented home-grown Highly Available NFS filer with RAID5 providing 5.4 TB usable storage, using Gentoo Linux, Heartbeat, DRBD, and Areca 1220 RAID controller cards with RAID battery backup unit, on two servers with virtual IP failover. Provided benchmarks, determining better hardware RAID controller models and facilitating kernel tuning.
  • Improved communications from Operations to the development and application teams.
  • Fixed a variety of problems with NFS, recommended standard client mount options.
  • Advised senior staff on system performance and administration process improvements.
  • Upgraded Awstats, and rewrote automated log processing Bash shell scripts, using looping constructs to maximize usage of all CPU cores, while minimizing impact on server resources. Initiated use of non-memory intensive and non-disk intensive web server log aggregation tools. Log processing time cut by a two-thirds, and solved inconsistent data representation.
  • Coordinated vendor on-site maintenance and upgrades.
  • Server upgrades and service migrations (Slackware 9.1 to 12.0), including Sendmail, milter, and Apache, and internal user directories.
  • Assisted developers with proper system utility usage, and developer virtual machine development OS instances.
  • Racked, cabled and configured servers, and performed general hardware upgrades.

Jumpcut.com (née Yahoo, Inc.), San Francisco, CA

Systems and Network Engineer • July 2006 — April 2007

For a "Web 2.0" video editing, sharing, and community startup, supported systems and network devices, and assisted software development.

  • Production applications supported include Apache 2.0.X, PHP 5.X, MySQL 5.X, all running under Gentoo Linux
  • Performed testing and configuration optimization
  • Polished Cacti and Nagios implementations, implemented teMySQL templates, and created a custom script, data sources and templates for HTTP latency graphing
  • Implemented new Apache SSL hosts
  • Implemented OpenVPN LAN-to-LAN VPN between office and co-location facility
  • Created "chroot jails" for various production applications, using the Jail shell, sudo, and a bit of shell scripting, with an aim to make safer calls from production applications to external applications
  • Implemented SQLGrey for managed Postfix Grey-listing
  • Added secondary backup "store-and-forward" Postfix MTA with SQLGrey integration
  • Worked with CTO on development of applications, including creating development environments using VMWare, WINE and Xorg X11 drivers
  • Implemented OpenWrt (http://www.openwrt.org/) on office router
  • Managed corporate desktop systems migration to Yahoo's infrastructure
  • Configured Yahoo's custom Apache and PHP, and deployed site code to Yahoo! production systems as a part of Yahoo's acquisition of Jumpcut
  • Wiki'd documentation of systems and project planning

Verizon Business (née Totality, Inc), San Francisco, CA, via Taos Mountain, Inc.

Senior Systems Engineer • December 2005 — July 2006

Supported Verizon Business Managed Service Provider clients, such as the Gap, Best Buy, American Airlines, and many others, in a Service Level Agreement model:

  • Improved production procedures documentation and process
  • Supported business-critical Solaris, Red Hat Linux, HP-UX, AIX, and Windows systems and applications. “Unrusted” proprietary UNIX skillset
  • Suggested, and helped pilot, Plone/Zope CMS for replacement of aging CVS-updated AFT/HTML documentation system. Worked with Perl programmer to create automated site scraping for deployment of read-only version of Plone site, using Bash, cron and wget
  • Resolved client issues alongside database administrators and network engineers
  • Trained new systems engineers
  • Consistently high marks from management

Riverbed, Inc, San Francisco, CA

Systems Administration Consultant • March 2005 — June 2005

Advised on and implemented the following services:

  • Assisted the Systems Administrator in closing a backlog of requests
  • Implemented a spam flagging system using Postfix and DSPAM
  • Upgraded internally-facing wiki (TWiki) to latest stable version
  • Implemented a VPN solution with OpenVPN, and added to the OpenVPN codebase
  • Documented progress and systems with Bugzilla and TWiki

Genentech, Inc., South San Francisco, CA

Deployment Engineer (Consultant) • July 2003 — June 2004

Contracted by former full-time supervisor:

  • Packaged and deployed difficult multi-layered software to over 1200 Windows workstations, in a validated, controlled and FDA-regulated environment
  • Wrote Perl scripts to automate local configuration changes and systems administration on Windows workstations
  • Prototyped and assisted in the evaluation of Gentoo Linux/Zope Content Management Frameworks development and staging servers
  • Managed Linux development server relocation to data center

Center for Reproductive Health Research & Policy, University of California, San Francisco

Systems Administrator (Consultant) • October 2002 — November 2003

Contracted to provide the following services for PhD-level researchers and staff:

  • Proposed and implemented web-based full-featured shared calendaring system
    • Reduced costs by re-purposing older hardware
    • Used Red Hat GNU/Linux 7.3., stripped down to HTTP, HTTPS, NTP and SSH network services, with Big Brother 1.9c for monitoring
    • Installed and configured Web Calendar
    • Used iptables to restrict access and usage to two subnets
  • Prototyped web-based survey system Mod_Survey (defunct)
  • Created role-based shared folders with Windows 2000/XP "Offline Files" functionality
  • Security assessments, using a variety of tools, including Nessus, nmap and Ettercap
  • User Training (very high marks!)

Denalii, Inc. (defunct), Burlingame, CA

Senior Systems Engineer • March 2001 — March 2002

Responsible for the following systems and services for heterogeneous Production network:

  • Network Applications: BIND 8, ProFTPd, Apache, NTP, OpenSSH/SFTP
  • Web Application Platform: Oracle 8, Orion Application Server, Apache, PHP 4, Sawmill web log analyzer
  • System Monitoring: Big Brother, Syslog (centralized)
  • Hardware: Sun/Intel
  • Operating Systems: Solaris 8, Red Hat Linux 6.2 & 7.1
  • Network devices: Cisco (3600 Routers, 4006 Smart Switches, ArrowPoint load balancers), Netscreen 100 firewalls
Supervised the following projects:
  • Rolled out NTP to all hosts, syncing to both internal servers and external time servers
  • Rolled out Big Brother, for host monitoring, role-based paging, and uptime tracking
  • Migrated HTML content from Roxen to Apache
  • “In-sourced” corporate web site, DNS, and network administration (phased out SiteSmith)
  • Performed and managed technical aspects of closing engineering-centered office
  • Maintained production Perl environment, including module maintenance, configuration and updates (DBI, DBD::Oracle, MD5, etc.)
  • Deployed applications onto Orion J2EE Application Server (1.5.2), resolved Windows-to-UNIX migration issues
  • Quickly and thoroughly undid sloppy administration practices of previous Systems Administrator, including correcting bad kernel build from unstable kernel tree, remaking DNS structure and updating many applications for security and bugfixes
  • Implemented security policies, including password aging and auditing, regular Nessus scanning, regular network packet sniffing using snoop (Solaris), tcpdump (Linux), Ethereal for capture and analysis (Linux & Windows), and nmap (both external and internal network service scanning)
  • Kept documentation up to date. Worked closely with Senior Oracle DBA to both maintain and improve current Oracle administration on Solaris, rolled-out Oracle on Linux, and also trained her on more advanced Systems Administration tasks to act as my backup, and she trained me on more advanced Oracle administration techniques
Special Note:Helped guide developers on architecture of logging and alerting systems to integrate with existing Open Source tools, made use of Open Source solutions for all appropriate areas, kept costs to a minimum, created generally stronger and more closely held policies and procedures for administration and maintenance of production systems.

Zing Networks, Inc. (defunct), San Francisco, CA

Director of Network Operations (promoted) • October 1999 — December 2000

Responsible for the following systems and services:

  • Network Applications: Apache, Squid (HTTP accelerator mode), Sendmail, BoldFish (mass e-mailing server), OpenSSH
  • Web Application Platform: Apache, Squid, mod_perl, Mason, Oracle
  • Monitoring/Reporting: Aria Web Tracking, Big Brother, MRTG, PIXlog, LogWatch, swatch
  • Hardware: Mostly Intel, with Sun for Oracle (E4500 primary, E3500 standby)
  • Storage: EMC Symmetrix drive arrays and Celerra File Server, with 3 TB usable storage, Legato Networker with SpectraLogic Bullfrog AIT-1 tape jukebox
  • Operating Systems: RedHat Linux 6.2, Solaris 7
  • Network devices: Cisco PIX firewalls, Foundry Networks (ServerIron (Layer 7 switch), NetIron (Layer 3 switch), 3Com BayStack 350 (Layer 3 switch), point-to-point T1
  • Vendor Relationships: Exodus, VALinux, Sun, SiteSmith, Redapt, GlobalCenter, Envive, Akamai, CDW
Supervised the following projects:
  • Rolled out of new EMC Symmetrix node, adding 1.5TB of usable storage for photo & video storage
  • Added SiteSmith to maintain high levels of uptime and support
  • Planned Globalization of Zing's Network (Japan, Europe and US East Coast)
  • Evaluated and recommended move to new data center (GlobalCenter to replace Exodus Communications)
  • Supervised install and maintained Aria Web Monitoring system
  • Akamized new sites static content
  • Supervised two Sr. Unix/Network Administrator contractors
Implemented the following office network systems from the ground up:
  • VPN with Compatible System's (now owned by Cisco) IntraPort 2 VPN router, with Win 95/98, NT 4.0, Linux and Mac clients
  • MS Exchange 5.5 SP3, with TrendMicro's ScanMail for Exchange 3.0, Outlook Web Access
  • Mapped external site e-mails to Exchange Public Folders
  • Rolled out Outlook 2000, gave 3 classes for users, with rave reviews
  • ARCServeIT 6.61 for NT, using OFA, Exchange Client, Client for UNIX (Solaris), Tape Library Option (with 30-slot SpectraLogic Treefrog tape autoloader with 1 AIT-2 drive) on 5-day Differential rotation
  • Reduced PC architecture to 3 major vendors (Dell, Apple, IBM), and 5 models tailored to company needs
  • Created PC purchasing forms and procedures
  • McAfee's Total Virus Defense, managed from McAfee Network Management Console
  • BoldFish mass-mail server for Linux
  • Negotiated Microsoft Open License
  • Employee Purchase Program with CDW
Supported the following:
  • Microsoft DNS & DNS/DHCP/WINS (maintained static routes, migrated DNS to new server)
  • Apache 1.3.12 on Linux, IIS 4.0
  • 2 NT Domains, with 9 NT 4.0 SP6a servers and 50 desktop and notebook PCs and 6 Macs
  • HP EtherJet (installed internal and external print device servers)
  • Installed & maintained co-located servers at Exodus Communications in Santa Clara
  • BigBrother Network Monitor on Linux
Special Note: Supervised office Cat 5 Ethernet re-configuration & deployment, negotiated cost-effective vendor relationships, stopped company virus problems COLD, smooth transition from Netscape Messaging Server 3.62 to Exchange 5.5 SP3. Promoted from Systems Administrator (internal) to UNIX Systems Administrator for Production (public web site). Y2K updates. Supported 3 US offices, 1 Israeli office.

Education

University of Pittsburgh

Unattained • 1988 — 1992

Majored in English, but completely unmotivated by academics. Radio station DJ, focused on Blues music and indie rock/experimental sonic underground.

Fox Chapel Area High School

High School Diploma • 1988

AP Computer Science, often called out from class to fix administration computers...

Recognition

Certified Kubernetes Administrator

The Linux Foundation • 2021

Certified Kubernetes Security Specialist

The Linux Foundation • 2022

Certified Kubernetes Application Developer

The Linux Foundation • 2021

Fixing it (and Making it Faster)

Genentech, Inc. • 1998

For improvements made to software deployment systems

Associations

Electronic Frontier Foundation

Member • 2001 — Present

Outside Interests

  • Linux Kernel tuning
  • Embedded system experimentation (Raspberry Pi!)
  • Social Democracy/Democratic Socialism
  • Some crazy musics