access issue while connecting to azure data lake gen 2 from databricks

Question

I am getting this below access issue while trying to connect from databricks to gen2 data lake using Service principal and OAuth 2.0

Steps performed: Reference article

1. created new service principal
2. provide necessary access to this service principal from azure storage account IAM with Contributor role access.
3. Firewalls and private end points connection has been enabled on databricks and storage account.

StatusCode=403
StatusDescription=This request is not authorized to perform this operation using this permission.
ErrorCode=AuthorizationPermissionMismatch
ErrorMessage=This request is not authorized to perform this operation using this permission.

However when I tried connecting via access keys it works well without any issue. Now I started suspecting if #3 from my steps is the reason for this access issue. If so, do I need to give any additional access to make it success? Any thoughts?

question from:https://stackoverflow.com/questions/65540805/access-issue-while-connecting-to-azure-data-lake-gen-2-from-databricks

Answer 1

When performing the steps in the Assign the application to a role, make sure to assign the Storage Blob Data Contributor role to the service principal.

Repro: I have provided owner permission to the service principal and tried to run the “dbutils.fs.ls("mnt/azure/")”, returned same error message as above.

Solution: Now assigned the Storage Blob Data Contributor role to the service principal.

Finally, able to get the output without any error message after assigning Storage Blob Data Contributor role to the service principal.

For more details, refer “Tutorial: Azure Data Lake Storage Gen2, Azure Databricks & Spark”.