The Three Pillars of GitOps: Pipelines, Observability, & Source Control

Written by: , Sep. 24, 2018

Weaveworks Cloud & Flux - Git Version Control for Automated Kubernetes Web Server Configuration

According to Alexis Richardson, co-founder/CEO of Weaveworks, and William Denniss, project manager at Google Cloud Platform, GitOps are "modern best practices for high velocity app development using cloud native tools." With GitOps, web server configuration is regarded as code and subject to version control, where Git operates "as a single source of truth for the whole system." GitOps depends on completely automated software delivery pipelines, with 24/7 monitoring and observability "baked into the beginning," where data security is of "critical importance. Everything has to be version controlled and stored in a single source of truth from which you can recover." In Agile development teams, Terraform, Spinnaker, Puppet, & Ansible are currently popular for web server script automation using Git for disk image prints with Docker & Kubernetes on public cloud hosts for CI/CD requirements in project management. Weaveworks has developed a number of cloud platforms that address the software development & web server hosting pipeline holistically such as Weave Cloud, Weave Flux, and integrated AWS container management products. GitOps allows developers to make hundreds of small changes per day to running websites & mobile applications in live production rather than rolling out monolithic upgrades. YAML is used to build declarative infrastructure statements that can be repeated and implemented as standards for cloud network automation. In GitOps, diff alerts & pull requests are managed through decentralized networks of programmers in Agile teams from any office location using Git, increasing management flexibility in hiring and worker productivity times. GitOps has now become standard in web/mobile app development for enterprise corporations, start-up companies, non-profits, government agencies, & SMEs worldwide.

Weave Cloud DevOps: Embracing Git as a Single Source of Truth for the Whole System

The three pillars of GitOps are pipelines, observability, and control. Pipelines relate to the software development process for websites & mobile applications combined with the cloud hosting requirements of maintaining them in production. Web & mobile apps have the need for continual upgrades of new custom-coded features like GUIs, profiles, shopping carts, themes, etc., with security patches for programming languages, databases, operating systems, and extensions required to be updated on the web server regularly. Agile programming teams need software release pipeline automation that is best accomplished currently through "the use of Git as a source of truth for the desired state of the applications supported in production." Observability can be accomplished through a GitOps repository with one repo per app or service.  It is recommended to use a separate branch for each environment, i.e. staging, production, & development. Agile programmers can push changes to staging branches first, then roll out new code to production as merge requests using Git after the bug testing is completed in a sandbox. This allows for better monitoring, logging, tracing, & visualization of the software code in web hosting, including a holistic view of the real system state depicting user traffic rates, resource usage, & processing requirements. Administrators can use "canaries" via Istio with staged deployments to test code changes, allowing teams to introduce new features with better notation logs and documentation in the source.

GitOps - Operations by Pull Request: "William Denniss is the author of several IETF Internet-Drafts including OAuth 2.0 for Native Apps, and founded AppAuth, the leading open source OAuth client for native apps. Alexis is the co-founder and CEO of Weaveworks. He is also the chairman of the TOC for CNCF, and the co-founder of the Coed:Code meet ups. Previously he was at Pivotal, as head of products for Spring, RabbitMQ, Redis, Apache Tomcat and vFabric." Learn More About the Cloud-Native Foundation.

With GitOps, every microservice should have a unified dashboard for real-time feedback & analysis that programmers & systems administrators can reference. Control can be viewed as developing from operating patterns in service deployments. Using Git, all web server configurations from DevOps are treated as code and subject to version control. The command "kubectl get <object> -o yaml" can be used to extract the configuration of an active server to transfer it to a Git repository for Kubernetes integration.  Sealed secrets allow for the use of encrypted private /public keys stored in Git per programmer as a means of authorization. Git orchestration, using diff & synch commands, is preferred over use of kubectl in Weave Flux for Kubernetes synchronization of containers. The key point of GitOps is whereas all teams of programmers will develop unique workflow and collaboration methods, the use of industry recognized best practices can save time, increase efficiency, and boost productivity in action. Web servers can be further optimized, including increased data & user density per VM or partition, through object-oriented partitioning of the stack software across container instances with Kubernetes/Open Stack/AWS load balancing of packet data requests.

Weave Cloud: "Easily deploy, manage and monitor container-based applications... Fully integrated with CNCF Kubernetes, Docker Swarm/UCP, AWS ECS, Apache Mesos, and Mesosphere DC/OS, Weave Cloud can be added to your development pipeline without requiring you to ‘replatform’ – making setup easy and delivery faster." Learn More About Weave Cloud.

Weave Flux: YAML Code Patterns for Git, Kubectl, & Istio Integration

YAML allows DevOps teams  to build declarative infrastructure that can reproduce cloud network architecture from statements. YAML is used in Kubernetes, Docker, Terraform, Ansible, Spinnaker, Puppet, Bitnami, & GitHub as the logic of devops where "configuration is code." From this basis, programmers and systems administrators can also apply Git version control to YAML files for better organization of project resources with Kubernetes or Docker Swarm automation. The majority of this activity occurs through diff alerts and pull requests, where any developer can join an Agile team through GitHub, using validated security keys for encrypted connections.

The Three Core Principles of GitOps:

  1. Use declarative configuration to define your application and services.
  2. All changes need to go through your Git review process - no one should be using kubectl directly.
  3. Use an operator in the cluster to drive the observed cluster state to the desired state, as declared by your configuration in Git.

Weave Flux allows Agile programming teams to practice CI/CD using container images with Docker & Kubernetes orchestration, with Git version control insuring that the code is able to be reviewed, tested, & reverted where required on restore. Automated deployment of web servers leads to greater efficiency in production taking advantage of new elastic cluster platforms on public cloud hosts. Code changes can be introduced to running apps more quickly by maintenance teams or Agile developers. Troubleshooting & debugging is enhanced by the incremental changes introduced by Git with abundant comments and notation by staff workers. Weave Flux for Kubernetes automation is available on GitHub.

Weave Cloud on AWS: Manage Kubernetes with Git, Terraform, & Ansible

Weaveworks Cloud is a solution for running Kubernetes on AWS that competes with many others in the marketplace in adding unified administration features plus tools & utilities for DevOps requirements. The Weave Cloud platform includes utilities for hardware virtualization, cloud networking, load balancing, data security, ELBs, RDS instances, DynamoDB table support, S3 bucket integration, & IAM roles. Weave Cloud allows Agile teams to use scripting features with Ansible, Jenkins, Terraform, Puppet, & Prometheus with Kubernetes for more options in building a custom cloud hosting solutions for web/mobile apps in production. Weaveworks Cloud uses Git for Kubernetes cluster management on AWS, with Terradiff, Ansiblediff, & Kubediff available to monitor code differences using version control for software updates. These tools are used to debug code changes that may cause breakage or errors in other parts of a website, i.e. a particular CMS or ecommerce script module/theme, or a cloud cluster web server in containerized production.

WeaveWorks & Terraform: "We chose Terraform, from Hashicorp, to provision our infrastructure resources. Terraform allows us to declaratively list the resources we want in a high level configuration language, describe the interdependencies between them, and intelligently apply changes to this config in an order that honors the dependencies. Terraform supports parameterizable, instantiable modules such that we can share the resource declarations for VMs etc between clusters, parameterizing each cluster for number of minions, instance type etc. Terraform even has a 'plan' mode, which shows us if our configuration matches reality. We combine this with prom-run, which runs a command and exports its exit codes to Prometheus in Weave Cloud, to build a terradiff job that we monitor and alert on." Learn More About WeaveWorks on AWS.

Weave Cloud for GKE: "Weave Cloud for Google Kubernetes Engine adds delivery pipelines to Git repositories, release automation, monitoring and observability to enable application operations at scale. Until now, developers and operations staff have had to integrate multiple services to enjoy the benefits of Kubernetes. Setting up a cluster, adding continuous integration (CI) and then continuous deployment (CD), hooking up monitoring, logging and management are critical to running production systems. Weave Cloud for GKE removes this complexity by providing a pre-integrated solution in one place with one coherent set of dashboards. This radically accelerates both the initial experience of getting an application running and simplifies the configuration and management of more advanced solutions. Using the free tier of Weave Cloud gives developers everything they need to explore Kubernetes and start building cloud native container applications without cost or complexity." Learn More About WeaveWorks.

Weaveworks GitOps Tools: Kubernetes Hosting Solutions & Agile Software Development

Weaveworks Cloud is a complete software development pipeline & delivery management platform with many utilities that can be used to script container solutions for automation with Kubernetes using Git for version control. Weaveworks Cloud is designed to meet the CI/CD requirements of modern DevOps teams in enterprise corporations, government agencies, start-ups, and big brands. SMEs will find the Weaveworks GitOps tools easy to use & affordable enough for staff investment with features that rival other solutions in the market. Weaveworks offers production-ready solutions for Kubernetes management on AWS which integrate with all of the popular version control software using Git favored by Agile teams in collaboration and is already widely used by many enterprise data centers in the cloud.  Weaveworks Cloud is a recommended SaaS partner product on AWS that can help programmers & developers simplify their deployment of microservices in containers by adding monitoring & management tools, although it remains to be seen how long the company remains competitive on consumer pricing, i.e. free open source software usage vs. enterprise licensing fees.

Author:
Eliran Ouzan is the Co. Founder and designer of HostAdvice and also owns Moonshot Marketing LTD, a leading web design & development firm and was a member at GreenPeace.

Widely known for his pixel-perfect and high conversion rate web designs. Over the course of his web experience he experimeneted with over 200 web hosting companies and have a superior knowledge on what defines a good hosting company.

Share this post

"The Three Pillars of GitOps: Pipelines, Observability, & Source Control"