@4 version broken by a patch release

[githubcdr]

Version

Hi Team,

After upgrading our software to version 4.0.6594, we encountered severe issues where the application failed to start entirely. Despite extensive debugging efforts, we were unable to resolve the issue, and the only viable solution was to revert and pin the version back to 4.0.6280.

Given the significant impact and breaking changes introduced, we’re left wondering why this was released as a patch release, which typically implies non-breaking, backward-compatible changes. Could you provide insight into the reasoning behind this release versioning, and are there any known issues or changes in 4.0.6594 that could explain this behavior?

We appreciate any guidance or information you can offer to help us understand and address this problem.

Thank you.

Description

After upgrading our software to version 4.0.6594, we encountered severe issues where the application failed to start entirely. Despite extensive debugging efforts, we were unable to resolve the issue, and the only viable solution was to revert and pin the version back to 4.0.6280.

Steps to reproduce

Change version to 4.0.6594

[kshyju]

@githubcdr

1. What operating system are you using?
2. How/where did you update the core tools to the latest version from? Are you using NPM?
3. What is your function project type? dotnet-isolated or dotnet in-proc or some other language? If dotnet/dotnet-isolated, What version of .NET (TargetFramework value)?
4. Can you share the error message you are seeing in the terminal, if any?

[githubcdr]

1. We are using this docker container, see below.
2. NPM is used
3. TFM, in proc? What does that mean? We had to revert the runtime for most of our services.
4. There are no messages in stdout, the container just crashloops

FROM node:18-slim@sha256:b175cd7f3358c399f7bcee9b1032b86b71b1afa4cfb4dd0db55d66f871475a3e

# Disable ICU
# https://github.com/Azure/azure-functions-core-tools/issues/2554#issuecomment-820935686
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true

# install func CLI required to run Azure Functions
RUN apt update && \
  apt install  --no-install-recommends -y \
  ca-certificates \
  libssl1.1 \
  && \
  rm -rf /var/lib/apt/lists/* && \
  npm install -g [email protected] --unsafe-perm true

[Zemfm]

I'm also having issues when running a Function App from a docker container. In my case, I have a docker-compose.yml file that I was using in Visual Studio until recently without issues.

Recently, when launching from Visual Studio Pro v17.12.0 (which I believe uses the latest version of this tool), the container logs this instead of normal startup/application logs:

Select a number for worker runtime:
1. dotnet (isolated worker model)
2. dotnet (in-process model)
3. node
4. python
5. powershell
6. custom

This is happening to all our Function App projects and only in Visual Studio. This does not occur when running docker-compose up from the command line for the same projects.

Update: I was able to fix this issue by moving the setting "FUNCTIONS_WORKER_RUNTIME" from our docker-compose.yml environment variables to local.settings.json. This was not needed in the past.

[kshyju]

Azure Functions Core Tools are designed for local development and testing of function apps on your computer. The Docker images published by the Functions team already include the Function runtime, which executes your function app when the container is started. Could you share more about your use case? What’s the specific reason you need Core Tools inside the container?

[Zemfm]

I don't need Core Tools and am not adding it to the container. Here's my dockerfile:

#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.

FROM mcr.microsoft.com/azure-functions/dotnet-isolated:4-dotnet-isolated8.0 AS base
WORKDIR /home/site/wwwroot
EXPOSE 8080

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG BUILD_CONFIGURATION=Release
WORKDIR /src
COPY ["src/Job/our.func.app.Job.csproj", "src/Job/"]
COPY ["src/Application/our.func.app.Application.csproj", "src/Application/"]
COPY ["src/Domain/our.func.app.Domain.csproj", "src/Domain/"]
RUN dotnet restore "src/Job/our.func.app.Job.csproj"
COPY . .
WORKDIR "/src/src/Job"
RUN dotnet build "our.func.app.Job.csproj" -c $BUILD_CONFIGURATION -o /app/build

FROM build AS publish
RUN dotnet publish "our.func.app.Job.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /home/site/wwwroot
COPY --from=publish /app/publish .
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true

The issue was that previously, we were defining FUNCTIONS_WORKER_RUNTIME in our docker-compose.yml environment variables and did not need a local.settings.json file.
Currently, if you set FUNCTIONS_WORKER_RUNTIME only in docker-compose.yml, it seems Core Tools does not read this environment variable and asks the user to specify the runtime on startup.

Here's how to recreate:

1. Create a new function app project in Visual Studio with Docker support.
2. Right-click function app project -> Add.. -> Container Orchestrator Support -> Docker Compose.
3. Delete local.settings.json file.
4. Add this environment section (FUNCTIONS_WORKER_RUNTIME) and Azurite image to the generated yml file:

    environment:
      AzureWebJobsStorage: UseDevelopmentStorage=true;DevelopmentStorageProxyUri=http://azurite:10000/;
      FUNCTIONS_WORKER_RUNTIME: dotnet-isolated
    depends_on:
      - azurite
    
  azurite:
    image: mcr.microsoft.com/azure-storage/azurite:latest
    command: "azurite --loose --blobHost 0.0.0.0 --blobPort 10000 --queueHost 0.0.0.0 --queuePort 10001 --tableHost 0.0.0.0 --tablePort 10002 --location /workspace --debug /workspace/debug.log"
    ports:
       - "20000:10000"
       - "20001:10001"
       - "20002:10002"
    environment: 
      - AZURE_STORAGE_ACCOUNT=devstoreaccount1
      - AZURE_STORAGE_KEY=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

This used to work but if you try and check the Function App container logs you will see:

Select a number for worker runtime:
1. dotnet (isolated worker model)
2. dotnet (in-process model)
3. node
4. python
5. powershell
6. custom

And your function app will not be executed.

[kshyju]

@Zemfm I followed your repro steps, and I am able to run the app without any issues. The only additional thing I had to do was the port mapping for the azure function app from the container to the host.

https://github.com/kshyju/DockerMissingFWRRepro/blob/ee05571185be454390addaf5aa745ab9b890ed80/docker-compose.yml#L11-L12

I uploaded my working app to this repo with instructions on how to run it locally. You can see that my repro app does not have a local.settings.json file. The FUNCTIONS_WORKER_RUNTIME environment variable is defined in the docker-compose file.

Can you compare this with your local repro and see what is different? If you are still running into the issue, can you share a minimal repro (similar to how I shared the working repro)? Thanks!

[ksgopal4]

I'm also having issues when running a Function App from a docker container. In my case, I have a docker-compose.yml file that I was using in Visual Studio until recently without issues.

Recently, when launching from Visual Studio Pro v17.12.0 (which I believe uses the latest version of this tool), the container logs this instead of normal startup/application logs:

Select a number for worker runtime:
1. dotnet (isolated worker model)
2. dotnet (in-process model)
3. node
4. python
5. powershell
6. custom

This is happening to all our Function App projects and only in Visual Studio. This does not occur when running docker-compose up from the command line for the same projects.

Update: I was able to fix this issue by moving the setting "FUNCTIONS_WORKER_RUNTIME" from our docker-compose.yml environment variables to local.settings.json. This was not needed in the past.

I have exactly the same problem. I dont use docker, my test starts the function app. The test has been successfully starting Function app fine until now and with the new functions runtime, it no longer works. It now needs local.settings.json in the functions folder and for it to contain FUNCTIONS_WORKER_RUNTIME setting, before I can run func start. Although FUNCTIONS_WORKER_RUNTIME was being passed in as an environment variable, its no longer enough.

[Zemfm]

@Zemfm I followed your repro steps, and I am able to run the app without any issues. The only additional thing I had to do was the port mapping for the azure function app from the container to the host.

https://github.com/kshyju/DockerMissingFWRRepro/blob/ee05571185be454390addaf5aa745ab9b890ed80/docker-compose.yml#L11-L12

I uploaded my working app to this repo with instructions on how to run it locally. You can see that my repro app does not have a local.settings.json file. The FUNCTIONS_WORKER_RUNTIME environment variable is defined in the docker-compose file.

Can you compare this with your local repro and see what is different? If you are still running into the issue, can you share a minimal repro (similar to how I shared the working repro)? Thanks!

Hi @kshyju ,

I have the exact same repository as you and I still have the issue. Did you run this from Visual Studio's latest version? As I said, I have no issues when running docker-compose up from the command line. Only when running in Visual Studio with "docker-compose" as the startup project.

[Zemfm]

I may have found the cause for this issue. In this PR #4142 , the added method EnsureWorkerRuntimeIsSet() only checks local.settings.json through ISecretsManager.GetSecrets(). I believe it should also check environment variables and return immediately if the runtime is set there (or use it to set GlobalCoreToolsSettings.CurrentWorkerRuntime?).