GitLab

How GitLab runs

GitLab runs the official Enterprise Edition image gitlab/gitlab-ee as the gitlab Deployment in the gitlab namespace. Its license is under Admin > Subscription. Keep the gitlab-ee image on every upgrade: a gitlab-ce tag turns the instance into Community Edition and drops the licensed features.

Everything in the gitlab namespace is declared under manifests/ in nrp-nautilus/gitlab and synced by the Argo CD application gitlab (namespace argo) with auto-sync, prune and self-heal. Make every change there and merge it: kubectl edit, scale, set image and set env on those objects are reverted. kubectl rollout restart is kept, and so are one-off Jobs that are not in the repository. Secrets are committed as SealedSecrets; the repository’s README describes how to reseal one. To see which commit is live:

kubectl -n argo get application gitlab -o jsonpath='{.status.sync.revision} {.status.sync.status}{"\n"}'

• Settings: ConfigMap gitlab-config (manifests/gitlab/gitlab-config-configmap.yaml), key base.rb, loaded through GITLAB_OMNIBUS_CONFIG. Secrets reach it as environment variables from Kubernetes Secrets, read in base.rb with ENV[...]; never write a secret into the ConfigMap.
• Data: PVC gitlab, mounted at /home/git/data (repositories, shared, uploads, builds, backups), owned by uid/gid 1000 (user['uid']/user['gid'] in base.rb; the container command removes the image’s ubuntu user).
• State: PVC gitlab-state holds /etc/gitlab (including gitlab-secrets.json) and /var/opt/gitlab (Redis, Gitaly runtime). gitlab-secrets.json holds the keys for data GitLab encrypts; a copy is in Secret gitlab-secrets-json, sealed in the repository, to reseal whenever the file changes.
• SSH host keys: Secret gitlab-ssh-host-keys, copied into /etc/gitlab by the host-keys init container. Clients connect to gitlab-ssh.nrp-nautilus.io:30622 (Service gitlab-ssh).
• Database: the Postgres cluster gitlab-db (Zalando operator). Registry: the separate gitlab-reg Deployment; GitLab signs its tokens with the key in Secret ssl-key. Pages: port 8090 behind the *.pages.nrp-nautilus.io ingress.
• Placement: region us-west, like gitlab-db and gitlab-reg, preferring the SDSC nodes.
• Resources: every container in the namespace sets requests; limits are optional. GitLab requests 12 CPUs and 16Gi with a 32Gi memory limit, and runs 10 Puma workers capped at 2048 MB each.
• Readiness: the probe requests /-/health from inside the pod.

To change a setting, edit base.rb in the ConfigMap manifest and merge. Argo updates the ConfigMap but does not restart GitLab: once the new commit shows as synced, restart with kubectl -n gitlab rollout restart deploy/gitlab (about 5 minutes of downtime), outside the backup run. Check the result in the rendered /var/opt/gitlab/gitlab-rails/etc/gitlab.yml inside the pod.

Upgrading GitLab

1. Pick the next version on the upgrade path and read its upgrade notes. Never skip a required stop. Before moving past one, wait until Admin > Monitoring > Background migrations shows nothing pending.

2. If the upgrade could need a rollback, copy the database first, while GitLab is stopped. Self-heal undoes kubectl scale, so stop it from the repository: merge replicas: 0 in manifests/gitlab/gitlab-deployment.yaml, wait for the pod to go, then copy on the Postgres leader:

   leader=$(kubectl -n gitlab get pod -l cluster-name=gitlab-db,spilo-role=master -o name)
   kubectl -n gitlab exec "$leader" -c postgres -- psql -U postgres \
     -c "create database gitlab_before_upgrade template gitlab_production owner gitlab"

   Put replicas: 1 back in the same change as the new tag. Drop the copy once the upgrade is confirmed.

3. In manifests/gitlab/gitlab-deployment.yaml, set gitlab/gitlab-ee:<version>-ee.0 on both the gitlab container and the host-keys init container, and merge outside the backup run. Argo applies it and the Deployment recreates the pod (strategy Recreate), which runs gitlab-ctl reconfigure and the database migrations before starting:

   kubectl -n gitlab logs -f deploy/gitlab -c gitlab   # wait for "gitlab Reconfigured!"

4. Check the sign-in page, a git clone over SSH, a Pages site, a registry login and the agent endpoint (see below).

5. Upgrade the registry to the version that release ships: it is set in config/software/registry.rb of omnibus-gitlab at the tag <version>+ee.0. The registry stores images in S3 without a metadata database, so the update is a rolling image change: set the registry container in manifests/registry/gitlab-reg-deployment.yaml to registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v<registry version>-gitlab.

6. Move the AI Gateway to the newest self-hosted-v<major>.<minor>.*-ee tag of the new GitLab version that has a published image (a release can be tagged without one): set the ai-gateway container in manifests/ai-gateway/ai-gateway-deployment.yaml to registry.gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/model-gateway:self-hosted-v<version>-ee.

Use version tags in the manifests, not digests.

Backups

The container command starts busybox crond with one job: gitlab-backup create at 04:00 Pacific, skipping ci_secure_files. The tarball (about 250 GB) is uploaded in parts to the Google Cloud Storage bucket nautilus-gitlab through its S3-compatible API, which keeps it 95 days; local copies older than two days are deleted. The log is /var/opt/gitlab/backup.log. gitlab-secrets.json and the SSH host keys are not in the tarball; restoring needs them too.

A run takes 16 to 20 hours, and restarting GitLab kills it. gitlab-backup removes its staging files (db/, repositories/, *.tar.gz in /home/git/data/backups) when a run ends, not when the next one starts, so a killed run leaves them behind and the next run packs them into its tarball. While no backup is running, delete them:

kubectl -n gitlab exec deploy/gitlab -c gitlab -- sh -c \
  'cd /home/git/data/backups && rm -rf db repositories *.tar.gz backup_information.yml'

Kubernetes integration and agents

The GitLab agent server (KAS) runs in the GitLab pod (gitlab_kas['enable'] = true in base.rb). Agents connect to wss://gitlab.nrp-nautilus.io/-/kubernetes-agent/ and CI/CD jobs reach them through https://gitlab.nrp-nautilus.io/-/kubernetes-agent/k8s-proxy/; the pod’s nginx serves both paths, and the ingress carries the WebSockets. KAS upgrades with GitLab; agents should run the same version. Its keys are in gitlab-secrets.json.

Users install the agent in their own namespace with namespace-scoped rights (the chart’s default cluster-admin binding is turned off), as described in K8s GitLab Integration. The older certificate-based integration stays available through the certificate_based_clusters feature flag.

To check KAS: gitlab-ctl status gitlab-kas in the pod, and a plain request to the agent endpoint, which answers 426 Upgrade Required when the path works:

curl -s -o /dev/null -w '%{http_code}\n' https://gitlab.nrp-nautilus.io/-/kubernetes-agent/

GitLab Duo Self-Hosted

GitLab Duo runs on the NRP-managed LLMs. Three parts are involved:

• AI Gateway: Deployment and Service ai-gateway in the gitlab namespace, from GitLab’s model-gateway:self-hosted-v<version>-ee image. One container serves the AI Gateway on HTTP port 5052 and the GitLab Duo Agent Platform service on gRPC port 50052. Its JWT keys are in Secret ai-gateway-jwt. GitLab reaches it in-cluster, set under Admin > GitLab Duo > Change configuration:

  • Local AI Gateway URL: http://ai-gateway.gitlab.svc.cluster.local:5052
  • Local URL for the GitLab Duo Agent Platform service: ai-gateway.gitlab.svc.cluster.local:50052, with TLS off.

  The host is also in the outbound local-request allowlist (Admin > Settings > Network > Outbound requests).

• Model: the self-hosted model “Service” (Admin > GitLab Duo > Configure models for GitLab Duo) has endpoint https://ellm.nrp-nautilus.io/v1, identifier custom_openai/service and an LLM API token, and the Duo features are set to it. service answers from deepseek-v4-flash and falls back through the other models (see Managing models).

• Runner: foundational flows run on Deployment gitlab-runner-duo, an instance runner that takes only jobs tagged gitlab--duo and runs them with the Kubernetes executor in the gitlab namespace. Its token is key duo in Secret gitlab-runner-token.

The Education license includes only GitLab Duo Core. It covers Agentic Chat and flows, not the classic Duo Chat, which answers with error M3006; classic Chat on self-hosted models needs paid GitLab Duo Enterprise seats. GitLab meters Agentic Chat and flows in GitLab Credits even on the self-hosted model, and Education licenses get no included credits, so these features stop once GitLab enforces its limits. Never accept the usage billing terms or buy credits in the Customers Portal: that makes the usage billable and cannot be undone.

To check the setup, use the health check on Admin > GitLab Duo, or in the GitLab pod:

gitlab-rake "gitlab:duo:verify_self_hosted_setup[<admin username>]"