Install

Cortex currently relies on cloud provider specific functionality such as load balancers and storage. Kubernetes clusters in the following cloud providers are supported:

Cortex uses helm to install the Cortex operator and its dependencies on your Kubernetes cluster.

AWS

Prerequisites

  • kubectl

  • aws cli

  • helm 3

  • EKS cluster

    • at least 3 t3.medium (2 vCPU, 4 GB mem) instances

You may install Cortex in any namespace in your cluster. In the guide that follows, the "default" namespace is assumed; if you're using a different namespace, replace all occurrences of "default" with the name of your namespace.

Note that installing Cortex on your Kubernetes cluster will not provide some of the cluster-level features such as cluster autoscaling and spot instances with on-demand backup.

Download Cortex charts

wget https://s3-us-west-2.amazonaws.com/get-cortex/0.29.0/charts/cortex-0.29.0.tar.gz
tar -xzf cortex-0.29.0.tar.gz

Create a bucket in S3

The Cortex operator will use this bucket to store API state and dependencies.

aws s3api create-bucket --bucket <CORTEX_S3_BUCKET> --region <CORTEX_REGION>

Credentials

The credentials need to have at least these permissions.

export CORTEX_AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID>
export CORTEX_AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY>
kubectl --namespace default create secret generic 'aws-credentials' \
--from-literal='AWS_ACCESS_KEY_ID'=$CORTEX_AWS_ACCESS_KEY_ID \
--from-literal='AWS_SECRET_ACCESS_KEY'=$CORTEX_AWS_SECRET_ACCESS_KEY \
-o yaml --dry-run=client | kubectl apply -f - >/dev/null

Install Cortex

Define a values.yaml with the following information provided:

# values.yaml
cortex:
region: <CORTEX_REGION>
bucket: <CORTEX_S3_BUCKET> # e.g. "my-cortex-bucket" (without s3://)
cluster_name: <CORTEX_CLUSTER_NAME>
global:
provider: "aws"
helm install cortex charts/ --namespace default -f values.yaml

Configure Cortex client

Wait for the loadbalancers to be provisioned and connected to your cluster.

Get the Cortex operator endpoint:

kubectl get service --namespace default ingressgateway-operator -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

You can use the curl command below to verify that your load balancer is ready. It can take 5-10 minutes for the setup to complete. You can expect to encounter Could not resolve host or timeouts when running the verification request before the load balancer is initialized.

export CORTEX_OPERATOR_ENDPOINT=$(kubectl get service --namespace default ingressgateway-operator -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
curl http://$CORTEX_OPERATOR_ENDPOINT/verifycortex --max-time 5

A successful response looks like this:

{"provider":"aws"}

Once you receive a successful response, configure your Cortex client:

cortex env configure --operator-endpoint $CORTEX_OPERATOR_ENDPOINT

Using GPU/Inf resources on your cluster

The following tolerations are added to Deployments and Jobs orchestrated by Cortex.

- effect: NoSchedule
key: nvidia.com/gpu
operator: Exists
- effect: NoSchedule
key: aws.amazon.com/neuron
operator: Equal
value: "true"

GCP

Prerequisites

  • kubectl

  • gsutil

  • helm 3

  • GKE cluster

    • at least 2 n1-standard-2 (2 vCPU, 8 GB mem) (with monitoring and logging disabled)

You may install Cortex in any namespace in your cluster. In the guide that follows, the "default" namespace is assumed; if you're using a different namespace, replace all occurrences of "default" with the name of your namespace.

Note that installing Cortex on your Kubernetes cluster will not provide some of the cluster-level features such as cluster autoscaling and preemptible instances.

Download Cortex charts

wget https://s3-us-west-2.amazonaws.com/get-cortex/0.29.0/charts/cortex-0.29.0.tar.gz
tar -xzf cortex-0.29.0.tar.gz

Create a bucket in GCS

The Cortex operator will use this bucket to store API state and dependencies.

gsutil mb gs://CORTEX_GCS_BUCKET

Credentials

The credentials need to have at least these permissions.

export CORTEX_GOOGLE_APPLICATION_CREDENTIALS=<PATH_TO_CREDENTIALS>
kubectl create secret generic 'gcp-credentials' --namespace default --from-file=key.json=$CORTEX_GOOGLE_APPLICATION_CREDENTIALS

Install Cortex

Define a values.yaml with the following information provided:

# values.yaml
cortex:
project: <CORTEX_PROJECT>
zone: <CORTEX_ZONE>
bucket: <CORTEX_GCS_BUCKET> # e.g. "my-cortex-bucket" (without gs://)
cluster_name: <CORTEX_CLUSTER_NAME>
global:
provider: "gcp"
helm install cortex charts/ --namespace default -f values.yaml

Configure Cortex client

Wait for the loadbalancers to be provisioned and connected to your cluster.

Get the Cortex operator endpoint:

kubectl get service --namespace default ingressgateway-operator -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

You can use the curl command below to verify that your load balancer is ready. It can take 5-10 minutes for the setup to complete. You can expect to encounter Could not resolve host or timeouts when running the verification request before the load balancer is initialized.

export CORTEX_OPERATOR_ENDPOINT=$(kubectl get service --namespace default ingressgateway-operator -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl http://$CORTEX_OPERATOR_ENDPOINT/verifycortex --max-time 5

A successful response looks like this:

{"provider":"gcp"}

Once you receive a successful response, configure your Cortex client:

cortex env configure --operator-endpoint $CORTEX_OPERATOR_ENDPOINT

Using GPU resources on your cluster

The following tolerations are added to Deployments and Jobs orchestrated by Cortex.

- effect: NoSchedule
key: nvidia.com/gpu
operator: Exists