Kubernetes E2E tests

Kubernetes E2E tests guidance.

Prerequisite

  • An azure service principal

    Please follow this guide for creating an azure service principal The service principal should either have:

    • Contributor permission of a subscription
    • Contributor permission of a resource group. In this case, please create the resource group first
  • Docker daemon enabled

How to run Kubernetes e2e tests locally

  1. Prepare dependency project
  • aks-engine

    Binary downloads for the latest version of aks-engine for are available on Github. Download AKS Engine for your operating system, extract the binary and copy it to your $PATH.

    On macOS, you can install aks-engine with Homebrew. Run the command brew install Azure/aks-engine/aks-engine to do so. You can install Homebrew following the instructions.

    On Windows, you can install aks-engine via Chocolatey by executing the command choco install aks-engine. You can install Chocolatey following the instructions.

    On Linux, it could also be installed by following commands:

    $ curl -o get-akse.sh https://raw.githubusercontent.com/Azure/aks-engine/master/scripts/get-akse.sh
    $ chmod 700 get-akse.sh
    $ ./get-akse.sh
    
  • Kubernetes

    This serves as E2E tests case source, it should be located at $GOPATH/src/k8s.io/kubernetes.

    cd $GOPATH/src
    go get -d k8s.io/kubernetes
    
  • kubectl

    Kubectl allows you to run command against Kubernetes cluster, which is also used for deploying CSI plugins. You can follow here to install kubectl. e.g. on Linux

    curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
    chmod +x kubectl
    sudo mv kubectl /usr/local/bin/
    
  1. Build docker image azure-cloud-controller-manager and push it to your docker image repository.

    git clone https://github.com/kubernetes-sigs/cloud-provider-azure $GOPATH/src/sigs.k8s.io/cloud-provider-azure
    cd $GOPATH/src/sigs.k8s.io/cloud-provider-azure
    export IMAGE_REGISTRY=<username>
    export IMAGE_TAG=<tag>
    make image
    make push # or manually `docker push`
    
  2. Deploy a Kubernetes cluster with the above azure-cloud-controller-manager image.

    To deploy a cluster, export all the required environmental variables first and then invoke make deploy:

    export RESOURCE_GROUP_NAME=<resource group name>
    export AZURE_LOCATION=<location>
    export AZURE_SUBSCRIPTION_ID=<subscription ID>
    export AZURE_CLIENT_ID=<client id>
    export AZURE_CLIENT_SECRET=<client secret>
    export AZURE_TENANT_ID=<tenant id>
    export USE_CSI_DEFAULT_STORAGECLASS=<true/false>
    export K8S_RELEASE_VERSION=<k8s release version>
    export CCM_IMAGE=<image of the cloud controller manager>
    export CNM_IMAGE=<image of the cloud node manager>
    
    make deploy
    

    To connect the cluster:

    export KUBECONFIG=$GOPATH/src/sigs.k8s.io/cloud-provider-azure/_output/$(ls -t _output | head -n 1)/kubeconfig/kubeconfig.$LOCATION.json
    kubectl cluster-info
    

To check out more of the deployed cluster , replace kubectl cluster-info with other kubectl commands. To further debug and diagnose cluster problems, use kubectl cluster-info dump

  1. Get kubetest binary
go get -u k8s.io/test-infra/kubetest
  1. Run E2E tests

Please first ensure the kubernetes project locates at $GOPATH/src/k8s.io/kubernetes, the e2e tests will be built from that location.

cd $GOPATH/src/k8s.io/kubernetes

make WHAT='test/e2e/e2e.test'
make WHAT=cmd/kubectl
make ginkgo

export KUBERNETES_PROVIDER=azure
export KUBERNETES_CONFORMANCE_TEST=y
export KUBERNETES_CONFORMANCE_PROVIDER=azure
export CLOUD_CONFIG=$GOPATH/src/sigs.k8s.io/cloud-provider-azure/tests/k8s-azure/manifest/azure.json

# some test cases require ssh configurations
export KUBE_SSH_KEY_PATH=path/to/ssh/privatekey
export KUBE_SSH_USER={ssh_user}

# Replace the test_args with your own.
kubetest --test --provider=local --check-version-skew=false --test_args='--ginkgo.focus=Port\sforwarding'

Last modified January 11, 2022 : Polish node manager image building (19b0e4353)