AzureFile CSI Driver Known Issues

k8s versionstable version
v1.71.7.14+
v1.81.8.11+
v1.91.9.7+
v1.101.10.2+
v1.111.11.8+
v1.121.12.6+
v1.131.13.4+
v1.141.14.0+

1. azure file mountOptions setting

file/dir mode setting:

Issue details:

  securityContext:
    runAsUser: XXX
    fsGroup: XXX
versionfileMode, dirMode value
v1.6.x, v1.7.x0777
v1.8.0 ~ v1.8.5, v1.9.00700
v1.8.6 or later, v1.9.1 ~ v1.10.9, v1.11.0 ~ v1.11.3, v1.12.0 ~ v.12.10755
v1.10.10 or later0777
v1.11.4 or later0777
v1.12.2 or later0777
v1.13.x0777

other useful mountOptions setting:

  • mfsymlinks: make azure file(cifs) mount supports symbolic link
  • nobrl: Do not send byte range lock requests to the server. This is necessary for certain applications that break with cifs style mandatory byte range locks (and most cifs servers do not yet support requesting advisory byte range locks). Error message could be like following:
Error: SQLITE_BUSY: database is locked

Related issues

2. permission issue of azure file dynamic provision in acs-engine

Issue details:

From acs-engine v0.12.0, RBAC is enabled, azure file dynamic provision does not work from this version

error logs:

Events:
  Type     Reason              Age   From                         Message
  ----     ------              ----  ----                         -------
  Warning  ProvisioningFailed  8s    persistentvolume-controller  Failed to provision volume with StorageClass "azurefile": Couldn't create secret secrets is forbidden: User "system:serviceaccount:kube-syste
m:persistent-volume-binder" cannot create secrets in the namespace "default"
  Warning  ProvisioningFailed  8s    persistentvolume-controller  Failed to provision volume with StorageClass "azurefile": failed to find a matching storage account

Related issues

Workaround:

kubectl create -f https://raw.githubusercontent.com/andyzhangx/Demo/master/aks-engine/rbac/azure-cloud-provider-deployment.yaml
  • delete the original PVC and recreate PVC

Fix

3. Azure file support on Sovereign Cloud

Azure file on Sovereign Cloud is supported from v1.7.11, v1.8.0

4. azure file dynamic provision failed due to cluster name length issue

Issue details: k8s cluster name length must be less than 16 characters, otherwise following error will be received when creating dynamic privisioning azure file pvc, this bug exists in [v1.7.0, v1.7.10]:

Note: check cluster-name by running grep cluster-name /etc/kubernetes/manifests/kube-controller-manager.yaml on master node

persistentvolume-controller    Warning    ProvisioningFailed Failed to provision volume with StorageClass "azurefile": failed to find a matching storage account

Fix

k8s versionfixed version
v1.71.7.11
v1.81.8.0
v1.91.9.0

5. azure file dynamic provision failed due to no storage account in current resource group

Issue details:

When create an azure file PVC, there will be error if there is no storage account in current resource group, error info would be like following:

Events:
  Type     Reason              Age               From                         Message
  ----     ------              ----              ----                         -------
  Warning  ProvisioningFailed  10s (x5 over 1m)  persistentvolume-controller  Failed to provision volume with StorageClass "azurefile-premium": failed to find a matching storage account

Related issues

Workaround: specify a storage account in azure file dynamic provision, you should make sure the specified storage account is in the same resource group as your k8s cluster. In AKS, the specified storage account should be in shadow resource group(naming as MC_+{RESOUCE-GROUP-NAME}+{CLUSTER-NAME}+{REGION}) which contains all resources of your aks cluster.

Fix

k8s versionfixed version
v1.71.7.14
v1.81.8.9
v1.91.9.4
v1.101.10.0

6. azure file plugin on Windows does not work after node restart

Issue details: azure file plugin on Windows does not work after node restart, this is due to New-SmbGlobalMapping cmdlet has lost account name/key after reboot

Related issues

Workaround:

  • delete the original pod with azure file mount
  • create the pod again

Fix

k8s versionfixed version
v1.7not support in upstream
v1.81.8.10
v1.91.9.7
v1.101.10.0

7. file permission could not be changed using azure file, e.g. postgresql

error logs when running postgresql on azure file plugin:

initdb: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
fixing permissions on existing directory /var/lib/postgresql/data

Issue details: azure file plugin is using cifs/SMB protocol, file/dir permission could not be changed after mounting

Workaround:

Use mountOptions with dir_mode, file_mode set as 0777:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azurefile
provisioner: kubernetes.io/azure-file
mountOptions:
  - dir_mode=0777
  - file_mode=0777

follow detailed config here

Related issues Persistent Volume Claim permissions

8. Could not delete pod with AzureFile volume if storage account key changed

Issue details:

  • kubelet fails to umount azurefile volume when there is azure file connection, below is an easy repro:
    • create a pod with azure file mount
    • regenerate the account key of the storage account
    • delete the pod, and the pod will never be deleted due to UnmountVolume.TearDown error

error logs

nestedpendingoperations.go:263] Operation for "\"kubernetes.io/azure-file/cc5c86cd-422a-11e8-91d7-000d3a03ee84-myvolume\" (\"cc5c86cd-422a-11e8-91d7-000d3a03ee84\")" failed. No retries permitted until 2018-04-17 10:35:40.240272223 +0000 UTC m=+1185722.391925424 (durationBeforeRetry 500ms). Error: "UnmountVolume.TearDown failed for volume \"myvolume\" (UniqueName: \"kubernetes.io/azure-file/cc5c86cd-422a-11e8-91d7-000d3a03ee84-myvolume\") pod \"cc5c86cd-422a-11e8-91d7-000d3a03ee84\" (UID: \"cc5c86cd-422a-11e8-91d7-000d3a03ee84\") : Error checking if path exists: stat /var/lib/kubelet/pods/cc5c86cd-422a-11e8-91d7-000d3a03ee84/volumes/kubernetes.io~azure-file/myvolume: resource temporarily unavailable
...
kubelet_volumes.go:128] Orphaned pod "380b02f3-422b-11e8-91d7-000d3a03ee84" found, but volume paths are still present on disk

Workaround:

manually umount the azure file mount path on the agent node and then the pod will be deleted right after that

sudo umount /var/lib/kubelet/pods/cc5c86cd-422a-11e8-91d7-000d3a03ee84/volumes/kubernetes.io~azure-file/myvolume

Fix

k8s versionfixed version
v1.7no fix(no cherry-pick fix is allowed)
v1.81.8.8
v1.91.9.7
v1.101.10.0

Related issues

9. Long latency compared to disk when handling lots of small files

Related issues

10. allow access from selected network setting on storage account will break azure file dynamic provisioning

When set allow access from selected network on storage account and will get following error when creating a file share by k8s:

persistentvolume-controller (combined from similar events): Failed to provision volume with StorageClass "azurefile": failed to create share kubernetes-dynamic-pvc-xxx in account xxx: failed to create file share, err: storage: service returned error: StatusCode=403, ErrorCode=AuthorizationFailure, ErrorMessage=This request is not authorized to perform this operation.

That’s because k8s persistentvolume-controller is on master node which is not in the selected network, and that’s why it could not create file share on that storage account.

Workaround:

use azure file static provisioning instead

  • create azure file share in advance, and then provide storage account and file share name in k8s, here is an example

Related issues

11. azure file remount on Windows in same node would fail

Issue details:

If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since New-SmbGlobalMapping command would fail if file share is already mounted on the node.

error logs

Error logs would be like following:

E0118 08:15:52.041014    2112 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-file/42c0ea39-1af9-11e9-8941-000d3af95268-pvc-d7e1b5f9-1af3-11e9-8941-000d3af95268\" (\"42c0ea39-1af9-11e9-8941-000d3af95268\")" failed. No retries permitted until 2019-01-18 08:15:53.0410149 +0000 GMT m=+732.446642701 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"pvc-d7e1b5f9-1af3-11e9-8941-000d3af95268\" (UniqueName: \"kubernetes.io/azure-file/42c0ea39-1af9-11e9-8941-000d3af95268-pvc-d7e1b5f9-1af3-11e9-8941-000d3af95268\") pod \"deployment-azurefile-697f98d559-6zrlf\" (UID: \"42c0ea39-1af9-11e9-8941-000d3af95268\") : azureMount: SmbGlobalMapping failed: exit status 1, only SMB mount is supported now, output: \"New-SmbGlobalMapping : Generic failure \\r\\nAt line:1 char:190\\r\\n+ ... ser, $PWord;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Cred ...\\r\\n+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\\r\\n    + CategoryInfo          : NotSpecified: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMa \\r\\n   pping], CimException\\r\\n    + FullyQualifiedErrorId : HRESULT 0x80041001,New-SmbGlobalMapping\\r\\n \\r\\n\""

Fix

k8s versionfixed version
v1.10no fix
v1.111.11.8
v1.121.12.6
v1.131.13.4
v1.141.14.0

Related issues

12. update azure file secret if azure storage account key changed

Issue details: There would be azure file mount failure if azure storage account key changed

Workaround: User needs to update azurestorageaccountkey field manually in azure file secret(secret name format: azure-storage-account-{storage-account-name}-secret in default namespace):

kubectl delete secret azure-storage-account-{storage-account-name}-secret
kubectl create secret generic azure-storage-account-{storage-account-name}-secret --from-literal azurestorageaccountname=... --from-literal azurestorageaccountkey="..." --type=Opaque

make sure there is no \r in the account name and key, here is a failed case

  • delete original pod(may use --force --grace-period=0) and wait a few minutes for new pod retry azure file mount

13. Create Azure Files PV AuthorizationFailure when using advanced networking

Issue details:

When create an azure file PV using advanced networking, user may hit following error:

err: storage: service returned error: StatusCode=403, ErrorCode=AuthorizationFailure, ErrorMessage=This request is not authorized to perform this operation

Before api-version 2019-06-01, create file share action is considered as data-path operation, since 2019-06-01, it would be considered as control-path operation, not blocked by advanced networking any more.

Related issues

Fix

k8s versionfixed version
v1.18no fix
v1.191.19.0

Workaround:

Shut down the advanced networking when create azure file PV.

14. initial delay(5s) in mounting azure file

Issue details:

When starting pods with AFS volumes, there is an initial delay of five seconds until the pod is transitioning from the “Scheduled” state. The reason for this is that currently the volume mounting happens inside a wait.Poll which will initially wait a specified interval(currently 5 seconds) before execution. This issue is introduced by PR fix: azure file mount timeout issue with v1.15.11+, v1.16.8+, v1.17.4+, v1.18.0+

Fix

Fix

k8s versionfixed version
v1.15no fix
v1.161.16.14
v1.171.17.10
v1.181.18.7
v1.191.19.0

Last modified November 23, 2020 : Update azurefile.md (af5c2382f)