[Bug] Runtime pip install broken on Linux Consumption Plan Python 3.11 - Previously working for 1+ year now fails with OSError permission errors

[Pruthvi-Ramesh]

Expected Behavior

We have been using python function app for deploying some scripts as HTTP trigger function. To deploy function app we use below ARM template.

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "appName": {
      "type": "String",
      "metadata": {
        "description": "The name of the function app that you wish to create."
      }
    },
    "storageAccountType": {
      "type": "string",
      "defaultValue": "Standard_LRS",
      "allowedValues": [
        "Standard_LRS",
		"Standard_GRS",
        "Standard_RAGRS"
      ]
    },
    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]",
      "metadata": {
        "description": "Location for all resources."
      }
    },
    "functionWorkerRuntime": {
      "type": "string",
      "defaultValue": "python"
    },
    "linuxFxVersion": {
      "type": "string",
      "defaultValue": "python|3.11"
    },
	"storageAccountName": {
      "type": "string",
      "defaultValue": "test",
      "metadata": `{`
        "description": "Storage Account Name"
      }
    }
  },
  "variables": {
    "functionAppName": "[parameters('appName')]",
    "hostingPlanName": "[parameters('appName')]",
    "storageAccountName": "[parameters('storageAccountName')]",
	"storageAccountid": "[concat(resourceGroup().id,'/providers/','Microsoft.Storage/storageAccounts/', variables('storageAccountName'))]"
  },
  "resources": [
    {
      "type": "Microsoft.Storage/storageAccounts",
      "apiVersion": "2022-05-01",
      "name": "[variables('storageAccountName')]",
      "location": "[parameters('location')]",
      "sku": {
        "name": "[parameters('storageAccountType')]"
      },
      "kind": "StorageV2",
	  "properties": {
        "supportsHttpsTrafficOnly": true,
        "defaultToOAuthAuthentication": true
      }
    },
    {
      "type": "Microsoft.Web/serverfarms",
      "apiVersion": "2022-03-01",
      "name": "[variables('hostingPlanName')]",
      "location": "[parameters('location')]",
      "sku": {
        "name": "Y1",
        "tier": "Dynamic",
        "size": "Y1",
        "family": "Y"
      },
      "properties": {
        "reserved": true
      }
    },
    {
      "type": "Microsoft.Web/sites",
      "apiVersion": "2022-03-01",
      "name": "[variables('functionAppName')]",
      "location": "[parameters('location')]",
	  "identity": {
        "type": "SystemAssigned"
      },
      "kind": "functionapp,linux",
      "properties": {
        "reserved": true,
        "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]",
        "siteConfig": {
          "linuxFxVersion": "[parameters('linuxFxVersion')]",
          "appSettings": [
            {
              "name": "AzureWebJobsStorage",
              "value": "[format('DefaultEndpointsProtocol=https;AccountName={0};EndpointSuffix={1};AccountKey={2}', variables('storageAccountName'), environment().suffixes.storage, listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2022-05-01').keys[0].value)]"
            },
            {
              "name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
              "value": "[format('DefaultEndpointsProtocol=https;AccountName={0};EndpointSuffix={1};AccountKey={2}', variables('storageAccountName'), environment().suffixes.storage, listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), '2022-05-01').keys[0].value)]"
            },
            {
              "name": "WEBSITE_CONTENTSHARE",
              "value": "[toLower(variables('functionAppName'))]"
            },
            {
              "name": "FUNCTIONS_EXTENSION_VERSION",
              "value": "~4"
            },
            {
              "name": "FUNCTIONS_WORKER_RUNTIME",
              "value": "[parameters('functionWorkerRuntime')]"
            }
		],
		  "cors": {
                        "allowedOrigins": [
                            "*"
                        ]
                    }
			},
         "publicNetworkAccess": "Enabled"
      },
      "dependsOn": [
        "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]",
        "[resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName'))]"
      ]
    }
  ],
  "outputs": {
    "appURL": {
      "type": "string",
      "value": "[concat('https://',parameters('appName'),'.azurewebsites.net')]"
    },
    "functionAppName": {
      "type": "string",
      "value": "[variables('functionAppName')]"
    },
    "WEBSITE_CONTENTSHARE": {
      "type": "string",
      "value": "[toLower(variables('functionAppName'))]"
    }
  }
}

Once function app is deployed, we create a requirements file at "site/wwwroot" in fileshare linked to that function app. Later we create a python modules installation function in the function app. Below is the script that we write in installation function.

import azure.functions as func
import subprocess
import logging

def install_requirements():
    requirements_path = '/home/site/wwwroot/requirements.txt'
    logging.info('Installing packages from requirements.txt...')
    return subprocess.run(['pip', 'install', '-r', requirements_path], capture_output=True, text=True)

def list_installed_packages():
    return subprocess.run(['pip', 'list'], capture_output=True, text=True)

def main(req: func.HttpRequest) -> func.HttpResponse:
    try:
        result = install_requirements()
        if result.returncode != 0:
            logging.error(f"pip install failed with error: {result.stderr}")
            return func.HttpResponse(f"An error occurred while installing packages: {result.stderr}", status_code=500)
        logging.info(f"pip install succeeded: {result.stdout}")

        installed_packages_result = list_installed_packages()
        if installed_packages_result.returncode != 0:
            logging.error(f"Failed to list installed packages: {installed_packages_result.stderr}")
            return func.HttpResponse(f"An error occurred while listing installed packages: {installed_packages_result.stderr}", status_code=500)

        logging.info("Function logic executed successfully.")

        return func.HttpResponse(f"Function executed successfully. Installed packages:\n{installed_packages_result.stdout}", status_code=200)

    except Exception as e:
        logging.error(f"Error occurred: {e}")
        return func.HttpResponse(f"An error occurred: {e}", status_code=500)

Once the installation function created in the function app, we call that function to install the packages that we have specified in requirements file (which will be needed to execute our scripts/ function that we upload once all the module installation is done). This feature involves dynamic package installation or removal. For around 1+ year it used to work fine (Was working fine 1 month back). All the above-mentioned deployments are done through code (C#). We are using V1 model. Below are the screenshots of the folders in file share for reference.

When we go to site/wwwroot we have below files, each function/script we upload has folder in which there will be some file including init.py file.

Actual Behavior

Now when we call python module installation function, it is breaking with below error

ERROR: Could not install packages due to an OSError: [Errno 1] Operation not permitted: '/home/.local/lib/python3.11/site-packages/azure_common-1.1.28.dist-info/INSTALLERh4gisrx2.tmp' [notice] A new release of pip is available: 24.0 -> 26.1.1 [notice] To update, run: pip install --upgrade pip

Steps to Reproduce

Mentioned in expected behavior.

Relevant code being tried

Mentioned in expected behavior.

Relevant log output

ERROR: Could not install packages due to an OSError: [Errno 1] Operation not permitted: '/home/.local/lib/python3.11/site-packages/azure_common-1.1.28.dist-info/INSTALLERh4gisrx2.tmp' [notice] A new release of pip is available: 24.0 -> 26.1.1 [notice] To update, run: pip install --upgrade pip

requirements.txt file

azure-identity
azure-mgmt-resource
azure-mgmt-compute
google-auth
google-auth-oauthlib
google-auth-httplib2
google-api-python-client
boto3
requests
azure-mgmt-storage

Where are you facing this problem?

Production Environment

Function app name

The function app name - cs19c034aef8664ac1.azurewebsites.net

Additional Information

No response