Hi! I would like to host a transparent proxy for cache.nixos.org on my local kubernetes cluster.
I took the following NGINX config https://nixos.wiki/wiki/FAQ/Private_Cache_Proxy and created all the folders on the mounted storage.
This is the kubernetes deployment:
apiVersion: v1
kind: PersistentVolume
metadata:
name: nix-cache-volume
spec:
capacity:
storage: 500Gi
storageClassName: manual
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/k8s/nix-cache" # Needs exists before PV is created!
persistentVolumeReclaimPolicy: Retain
***
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nix-cache-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: manual
resources:
requests:
storage: 500Gi
***
apiVersion: apps/v1
kind: Deployment
metadata:
name: nix-cache
spec:
replicas: 1
selector:
matchLabels:
app: nix-cache
template:
metadata:
labels:
app: nix-cache
name: nix-cache
spec:
volumes:
- name: nix-cache-storage
persistentVolumeClaim:
claimName: nix-cache-pvc
- name: nix-cache-config
configMap:
name: nix-cache-config
containers:
- name: nix-cache
image: nginx:1.27.0
ports:
- containerPort: 80
volumeMounts:
- name: nix-cache-storage
mountPath: /data
- name: nix-cache-config
mountPath: /etc/nginx/sites-available/default
resources:
limits:
memory: "512Mi"
cpu: "300m"
requests:
memory: "256Mi"
cpu: "200m"
***
apiVersion: v1
kind: Service
metadata:
name: nix-cache
spec:
selector:
app: nix-cache
ports:
- protocol: TCP
port: 80
targetPort: 80
***
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nix-cache-ingress
annotations:
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
rules:
- host: "nix-cache.raspi.home"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: nix-cache
port:
number: 80
tls:
- secretName: nix-cache-raspi-home-tls
hosts:
- "nix-cache.raspi.home"
***
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: nix-cache.raspi.home
spec:
commonName: nix-cache.raspi.home
dnsNames:
- "nix-cache.raspi.home"
secretName: nix-cache-raspi-home-tls
issuerRef:
name: ca-issuer
kind: ClusterIssuer
***
apiVersion: v1
kind: ConfigMap
metadata:
name: nix-cache-config
data:
nginx.conf: |
server {
listen 80;
server_name nix-cache.raspi.home;
location ~ ^/nix-cache-info {
proxy_store on;
proxy_store_access user:rw group:rw all:r;
proxy_temp_path /data/nginx/nix-cache-info/temp;
root /data/nginx/nix-cache-info/store;
proxy_set_header Host "cache.nixos.org";
proxy_pass https://cache.nixos.org/;
}
location ~^/nar/.+$ {
proxy_store on;
proxy_store_access user:rw group:rw all:r;
proxy_temp_path /data/nginx/nar/temp;
root /data/nginx/nar/store;
proxy_set_header Host "cache.nixos.org";
proxy_pass https://cache.nixos.org/;
}
}
To use the cache I added it to the substituters.
nix.settings.substituters = [
"https://nix-cache.raspi.home/"
];
But when I try to use it, get the error:
# Trigger a download
nix develop nixpkgs#just
# Error message
warning: 'https://nix-cache.raspi.home/' does not appear to be a binary cache
In the logs of the NGINX I see the following error:
2024/08/03 12:09:30 [error] 31#31: *3 open() "/usr/share/nginx/html/nix-cache-info" failed (2: No such file or directory), client: 10.42.2.7, server: localhost, request: "GET /nix-cache-info HTTP/1 │
│ 10.42.2.7 - - [03/Aug/2024:12:09:30 +0000] "GET /nix-cache-info HTTP/1.1" 404 153 "-" "curl/8.8.0 Nix/2.18.5" "10.42.2.1" │
│ 10.42.2.7 - - [03/Aug/2024:12:09:30 +0000] "PUT /nix-cache-info HTTP/1.1" 405 157 "-" "curl/8.8.0 Nix/2.18.5" "10.42.2.1"
Any ideas whats wrong? I'm neither an nix nor an nginx expert, so maybe it is something really simple but I cannot figure it out.
Many small companies use it for their internal crates, as kellnr builds and host the corresponding rustdocs, too. Another prevalent use-case is as a crates.io cache to speed a build times or lower the network traffic.