I’ve done DevOps work on several projects at Slickage, but the biggest one I’ve worked on is Epochtalk “Next generation forum software” (https://github.com/epochtalk, https://github.com/epochtalk/epochtalk, epochtalk.org). It’s been through a lot of changes in architecture and backing technologies. When I started working with it, we were using NodeJS and Angular framework, migrating data from SMF (forum software) MySQL into PostgreSQL. It was running as a Node server on bare metal and was only available in development mode.

When I joined the team, we didn't have a formal revision style, and thus had a lot of trouble collaborating. It was so bad at one point that one of my teammates took all of my commits in a pull request and squashed them down without asking me first; in the process, erasing loads of detail in why I did things a certain way and how I managed to fix bugs or overcome obstacles.

I believe the commit history is an extremely powerful tool in a small development team’s arsenal for tracing and fixing bugs. I use it on a daily basis to compensate for our propensity to occasionally make short-sighted decisions; after all developers are only human, we are far from perfect. I like to get things right the first time as much as possible but overall, it’s much better to make a mistake and go back and fix it than to have fear of mistakes be a limiting factor on progress.

Due to the fact that the “squasher” teammate was the project lead, I went along with his direction for a while. After he left the company, I took charge and switched the team over to using a Pull Request and Merge based collaboration style. My intent was to preserve the commit history and to preserve the information contained in the branches between commits. I explained how branching would encourage us to work on features as units and lead to a more readable history than squash and rebase. I personally even use sub-branches, since I work best by breaking down tasks into the smallest units possible, and I merge them back into my own branch as I complete a feature. I’ve heard that Gerrit has the ability to squash and rebase while maintaining commit history, but I haven’t tried it out yet. It might be something worth looking into if the scope of work relates to how developers collaborate.

I’m always on the lookout for best practices and I love sharing what I find with my team. At the beginning of my DevOps experience, I discovered Heroku’s 12-Factor best practices and have been sticking to them ever since; making sure that all of the projects I work on are easily and dependably deployable. Any time I'm setting up ops for a new project, I re-visit the manifesto.

Epochtalk was the first project I converted to comply with 12-Factor. From consolidating configuration files and routines, to containerizing all the pieces, exposing configurations through the environment, locking down specific dependency versions, implementing automated versioning for internal dependencies, learning about linked containers and debugging their connections, automating builds in CI for testing and deployment, persisting data across restarts, unifying testing staging and production environments to ensure consistency, exposing administration functionality through cli, the list goes on and on.

In the end, it was not such an easy task, but each project since then has gone a lot smoother; especially now that I’ve established some good standards. Each conversion step has its own challenges, and Epochtalk's case required restructuring the project as a whole, touching nearly every file. However, as the Factors were put to work, the project started feeling a lot more polished for deployment - and promoted better development practices as well!

Once all of the pieces were using containers and container networking instead of bare metal deploys, the next step was to create a SAAS deployment strategy. I tried using Kubernetes, but it was still in its early stages of development (they were not directly associated with Docker yet) and documentation was a bit lacking at the time. I contacted Kubernetes’ dev team and joined their Slack, but it was a big headache and in the end I decided that I should build with Ansible instead.

Through some good old shell scripting, and a lot of trial and error, I was able to write playbooks to provision AWS resources (EC2, S3, Route53, RDS, CloudWatch), configure new virtual machines, and run our Dockerized project - all using customer information to create deployment namespaces and DNS entries. It was a bit of a clunky implementation, but it got the job done. On the bright side, I learned a lot about what it takes to get a production deployment up and running.
Additionally, having done all of this through scripting, I feel like I have a deeper understanding of how it all works. I am, however, very much looking forward to using more standardized tools like Kubernetes and Terraform.