Using a GitOps workflow for Kubernetes (PREMIUM)

• Introduced in GitLab 13.7.
• Introduced in GitLab 14.0, the resource_inclusions and resource_exclusions attributes were removed and reconcile_timeout, dry_run_strategy, prune, prune_timeout, prune_propagation_policy, and inventory_policy attributes were added.

With GitOps, you can manage containerized clusters and applications from a Git repository that:

• Is the single source of truth of your system.
• Is the single place where you operate your system.

By combining GitLab, Kubernetes, and GitOps, you can have:

• GitLab as the GitOps operator.
• Kubernetes as the automation and convergence system.
• GitLab CI/CD for Continuous Integration and the agent for Continuous Deployment.
• Built-in automatic drift remediation.
• Resource management with server-side applies for transparent multi-actor field management.

This diagram shows the repositories and main actors in a GitOps deployment:

sequenceDiagram
  participant D as Developer
  participant A as Application code repository
  participant M as Manifest repository
  participant K as GitLab agent
  participant C as Agent configuration repository
  loop Regularly
    K-->>C: Grab the configuration
  end
  D->>+A: Pushing code changes
  A->>M: Updating manifest
  loop Regularly
    K-->>M: Watching changes
    M-->>K: Pulling and applying changes
  end

For details, view the architecture documentation.

GitOps workflow steps

To update a Kubernetes cluster by using GitOps, complete the following steps.

1. Ensure you have a working Kubernetes cluster, and that the manifests are in a GitLab project.
2. In the same project, register and install the GitLab agent.
3. Configure the agent configuration file so that the agent monitors the project for changes to the Kubernetes manifests. Use the GitOps configuration reference for guidance.

Any time you commit updates to your Kubernetes manifests, the agent updates the cluster.

GitOps configuration reference

The following snippet shows an example of the possible keys and values for the GitOps section of an agent configuration file.

gitops:
  manifest_projects:
  - id: gitlab-org/cluster-integration/gitlab-agent
    default_namespace: my-ns
    paths:
      # Read all YAML files from this directory. 
    - glob: '/team1/app1/*.yaml'
      # Read all .yaml files from team2/apps and all subdirectories.
    - glob: '/team2/apps/**/*.yaml'
      # If 'paths' is not specified or is an empty list, the configuration below is used.
    - glob: '/**/*.{yaml,yml,json}'
    reconcile_timeout: 3600s
    dry_run_strategy: none
    prune: true
    prune_timeout: 3600s
    prune_propagation_policy: foreground
    inventory_policy: must_match

GitOps annotations

The GitLab agent for Kubernetes has annotations you can use to:

• Sort resources: Apply or delete resources in a specific order.
• Use apply-time mutation: Dynamically substitute fields from one resource configuration to another.

The agent has default sorting, but with annotations, you can fine-tune the order and apply time-value injection.

To provide the GitOps functionality, the GitLab agent for Kubernetes uses the cli-utils library, a Kubernetes SIG project. You can read more about the available annotations in the cli-utils documentation.

• Learn more about apply sort ordering.
• Learn more about apply-time mutation.

Automatic drift remediation

Drift happens when the current configuration of an infrastructure resource differs from its expected configuration. Typically, this is caused by manually editing resources directly through the service that created the resource. Minimizing the risk of drift helps to ensure configuration consistency and successful operations.

In GitLab, the agent for Kubernetes regularly compares the expected state from the git repository with the known state from the cluster. Deviations from the git state are fixed at every check. These checks happen automatically every 5 minutes. They are not configurable.

The agent uses server-side applies. As a result, every field in a resource can have different managers. Only fields managed by git are checked for drift. This facilitates the use of in-cluster controllers to modify resources like Horizontal Pod Autoscalers.

Additional resources

The following documentation and examples can help you get started with a GitOps workflow.

• Managing Kubernetes secrets in a GitOps workflow
• Application and manifest repository example

Troubleshooting

Avoiding conflicts when you have multiple projects

The agent watches each glob pattern set under a project's paths section independently, and makes updates to the cluster concurrently. If changes are found at multiple paths, when the agent attempts to update the cluster, a conflict can occur.

To prevent this from happening, consider storing a logical group of manifests in a single place and reference them only once to avoid overlapping globs.

For example, both of these globs match *.yaml files in the root directory and could cause conflicts:

gitops:
  manifest_projects:
  - id: project1
    paths:
    - glob: '/**/*.yaml'
    - glob: '/*.yaml'

Instead, specify a single glob that matches all *.yaml files recursively:

gitops:
  manifest_projects:
  - id: project1
    paths:
    - glob: '/**/*.yaml'

Use multiple agents or projects

If you store your Kubernetes manifests in separate GitLab projects, update your agent configuration file with the location of these projects.

WARNING: The project with the agent's configuration file can be private or public. Other projects with Kubernetes manifests must be public. Support for private manifest projects is tracked in this issue.