**Production** Namespace is not set when dispatching a backend task at the frontend app

[miztaka]

This is happening on production Java17 EE8.

Weird thing starts happening for 3 days in production environment.
My app has frontend instances for web request and backend instances for tasks.
Frontend app may dispatch tasks to the backend like:

  TaskOptions opt =
        TaskOptions.Builder.withUrl("/_a/exportTask")
            .param("ids", StringUtils.join(request.getForumIds(), "-"))
            .param("status", status)
            .param("mindt", mindt != null ? mindt : "")
            .param("maxdt", maxdt != null ? maxdt : "")
            .param("to", user.getUserEmail())
            .header("Host", AppConst.backendHostname());
  QueueFactory.getQueue(AppConst.NORETRY_QUEUE).add(opt);

Note that this operation happens where Namespace is set.

It has worked fine for long times but the error "RuntimeException: namespace is not set. " starts happening at the task execution since 3 days ago. It seems to occur 100% so far.
It worked fine before 3 days in the same version.

Please let me know if you need more information.

Thanks
Mitsu

[miztaka]

Forgot to mention about environment. This is happening on production Java17 EE8.

[ludoch]

Thanks for the report,
It might be due to a pending experiment we are running.
To be sure, can you go to the appengine logs and send us the logs you see during the boot process of the jvm?
You should see something with AppEngineWebXmlInitialParse class logging, this is what we need to look at.

Also what is your app Id?

[ludoch]

Also, one more data would be which API jar version do you depend on?

[miztaka]

@ludoch Hi, my app-id is powerapps-bbs , cookai-o-bbs .
appengine api sdk version in pom is 2.0.12.

[miztaka]

This is stderr log where I can see AppEngineWebXmlInitialParse. Hope this works.
downloaded-logs-20260307-065500.json

[ludoch]

OK, we recently turned down the http connector experiment for unrelated maintainance, and we are slowing ramping it up.
It was at 100% before, so your app was implicitly using it.

I see your current app is not yet on it. Please see https://github.com/GoogleCloudPlatform/appengine-java-standard/blob/main/httpconnector.md and try to force using it before it reaches your app while the ramp up is happening.

ie add

 <system-properties>
        <property name="appengine.use.httpconnector" value="true"/>
    </system-properties>

in your appengine-web.xml and redeploy.

If you still see the error, let me know asap, I will try to see if other recent changes could cause this regression.
Thanks for your help!

[ludoch]

The reason why I am saying this is I see in the logs:

...
DEFAULT 2026-03-03T04:29:38.879295Z INFO: AppEngine runtime configuration: runtimeId=java17, with Jetty 12, with HTTP Connector, appengine.use.EE8=true
...
DEFAULT 2026-03-03T04:47:01.109223Z INFO: AppEngine runtime configuration: runtimeId=java17, with Jetty 12, appengine.use.EE8=true
...

Can you see if it was working on 2026-03-03T04:29:38.879295Z and start failing at 2026-03-03T04:47:01.109223Z when the experiment was off?

[miztaka]

@ludoch good news and bad news. Yes, I can see that it was working until 03-03 4:29 and start failing since then. And confirmed that HTTP Connector was used until that time.
Bad news is, even if I have put that configuration in my appengine-web.xml, the instance came up without HTTP Connector and the namespace problem remains. My test environment is cookai-o-bbs-dev and the version o365-stg-2-01-043 has the config with HTTP Connector. I wondered if java17 works with this config??

[ludoch]

Oh, it might be a typo with the case of the property name

Try appengine.use.HttpConnector

with capital H and C

Let me know

[miztaka]

@ludoch I have tried capitalized version but it doesn't work unfortunately..

{
  "textPayload": "INFO: AppEngine runtime configuration: runtimeId=java17, with Jetty 12, appengine.use.EE8=true",
  "insertId": "69ab9e610005fb7a313311aa",
  "resource": {
    "type": "gae_app",
    "labels": {
      "version_id": "o365-stg-2-01-043-rc1",
      "project_id": "cookai-o-bbs-dev",
      "zone": "us17",
      "module_id": "backend"
    }
  },
  "timestamp": "2026-03-07T03:41:21.392058Z",
  "labels": {
    "clone_id": "00da6cd2c46128c933232a0f1adff4ab4fae27b8b92b180380c83c86125f32af8b2f52bd5e92dd245a476ab02223e3b853775a7b7f19cb9e51b878e5c5865f1d2eafcc24838cf650152006656f3b0a"
  },
  "logName": "projects/cookai-o-bbs-dev/logs/stderr",
  "receiveTimestamp": "2026-03-07T03:41:21.700348117Z"
}

[ludoch]

Really sorry about that!!! Hard to understand. One last try is via env var:

    <env-variables>
       <env-var name="EXPERIMENT_ENABLE_HTTP_CONNECTOR_FOR_JAVA" value="true" />
   </env-variables> 

[miztaka]

@ludoch Interesting, the instance came up with HTTP Connector but the namespace issue is still happening..

{
  "textPayload": "INFO: AppEngine runtime configuration: runtimeId=java17, with Jetty 12, with HTTP Connector, appengine.use.EE8=true",
  "insertId": "69aba9c800004b34d855f6c2",
  "resource": {
    "type": "gae_app",
    "labels": {
      "version_id": "o365-stg-2-01-043-rc2",
      "zone": "us17",
      "module_id": "backend",
      "project_id": "cookai-o-bbs-dev"
    }
  },
  "timestamp": "2026-03-07T04:30:00.019252Z",
  "labels": {
    "clone_id": "00da6cd2c476559d8469c3829c820f7ed0af1a6ffbace290df38b162593563d94f29189be29a851d1d279a333cddc4a1f17736f504e4ad84e53d1ef44d0cb38d5e677031d38442d8bc5657accd"
  },
  "logName": "projects/cookai-o-bbs-dev/logs/stderr",
  "receiveTimestamp": "2026-03-07T04:30:00.068202282Z"
}

This log shows me that namespace is not set.

{
  "textPayload": "Namespaceがセットされていません。パラメータからセットしました:527610003",
  "insertId": "69aba9cd000f30fa81d37597",
  "resource": {
    "type": "gae_app",
    "labels": {
      "zone": "us17",
      "version_id": "o365-stg-2-01-043-rc2",
      "project_id": "cookai-o-bbs-dev",
      "module_id": "backend"
    }
  },
  "timestamp": "2026-03-07T04:30:05.995578Z",
  "severity": "WARNING",
  "labels": {
    "clone_id": "00da6cd2c476559d8469c3829c820f7ed0af1a6ffbace290df38b162593563d94f29189be29a851d1d279a333cddc4a1f17736f504e4ad84e53d1ef44d0cb38d5e677031d38442d8bc5657accd"
  },
  "logName": "projects/cookai-o-bbs-dev/logs/%2Fvar%2Flog%2Fapp",
  "trace": "projects/cookai-o-bbs-dev/traces/015dffc6204600ef90d18f461832825c",
  "receiveTimestamp": "2026-03-07T04:30:06.320858496Z",
  "spanId": "0521d7a55210f5d3"
}

[ludoch]

Thanks for trying....
Now, the runtime you are using did not change unless i:m mistaken, so no clue so far... Sorry
Will continue trying to understand tomorrow...
Is it relatevily new code on your side?

[miztaka]

@ludoch No, this app is running over 13 years. I guess this issue is related to the change 3 days ago but may not be related to the "HTTP Connector vs old one" thing.. Hopefully you could find the root cause soon.. Appreciate your involvement anyway. Please keep in touch.