Conformance tests

.github/workflows/conformance.yml

@@ -0,0 +1,62 @@
+name: Conformance Tests
+
+on:
+  pull_request: {}
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+jobs:
+  server:
+    name: Server Conformance
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up JDK 17
+        uses: actions/setup-java@v4
+        with:
+          java-version: '17'
+          distribution: 'temurin'
+          cache: 'maven'
+
+      - name: Build and start server
+        run: |
+          mvn clean install -DskipTests
+          mvn exec:java -pl conformance-tests/server-servlet -Dexec.mainClass="io.modelcontextprotocol.conformance.server.ConformanceServlet" &
+          timeout 30 bash -c 'until curl -s http://localhost:8080/mcp > /dev/null 2>&1; do sleep 0.5; done'
+
+      - name: Run conformance tests
+        uses: modelcontextprotocol/conformance@v0.1.11
+        with:
+          mode: server
+          url: http://localhost:8080/mcp
+          suite: active
+          expected-failures: ./conformance-tests/conformance-baseline.yml
+
+  client:
+    name: Client Conformance
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        scenario: [initialize, tools_call, elicitation-sep1034-client-defaults, sse-retry]
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up JDK 17
+        uses: actions/setup-java@v4
+        with:
+          java-version: '17'
+          distribution: 'temurin'
+          cache: 'maven'
+
+      - name: Build client
+        run: mvn clean install -DskipTests
+
+      - name: Run conformance test
+        uses: modelcontextprotocol/conformance@v0.1.11
+        with:
+          mode: client
+          command: 'java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar'
+          scenario: ${{ matrix.scenario }}
+          expected-failures: ./conformance-tests/conformance-baseline.yml

.gitignore

@@ -10,6 +10,7 @@ build/
 out
 /.gradletasknamecache
 **/*.flattened-pom.xml
+**/dependency-reduced-pom.xml
 
 ### IDE - Eclipse/STS ###
 .apt_generated

conformance-tests/VALIDATION_RESULTS.md

@@ -0,0 +1,82 @@
+# MCP Java SDK Conformance Test Validation Results
+
+## Summary
+
+**Server Tests:** 37/40 passed (92.5%)  
+**Client Tests:** 3/4 scenarios passed (9/10 checks passed)
+
+## Server Test Results
+
+### Passing (37/40)
+
+- **Lifecycle & Utilities (4/4):** initialize, ping, logging-set-level, completion-complete
+- **Tools (11/11):** All scenarios including progress notifications ✨
+- **Elicitation (10/10):** SEP-1034 defaults (5 checks), SEP-1330 enums (5 checks)
+- **Resources (4/6):** list, read-text, read-binary, templates-read
+- **Prompts (4/4):** list, simple, with-args, embedded-resource, with-image
+- **SSE Transport (2/2):** Multiple streams
+- **Security (1/2):** Localhost validation passes
+
+### Failing (3/40)
+
+1. **resources-subscribe** - Not implemented in SDK
+2. **resources-unsubscribe** - Not implemented in SDK  
+3. **dns-rebinding-protection** - Missing Host/Origin validation (1/2 checks)
+
+## Client Test Results
+
+### Passing (3/4 scenarios, 9/10 checks)
+
+- **initialize (1/1):** Protocol negotiation, clientInfo, capabilities
+- **tools_call (1/1):** Tool discovery and invocation
+- **elicitation-sep1034-client-defaults (5/5):** Default values for string, integer, number, enum, boolean
+
+### Partially Passing (1/4 scenarios, 1/2 checks)
+
+- **sse-retry (1/2 + 1 warning):** 
+  - ✅ Reconnects after stream closure
+  - ❌ Does not respect retry timing
+  - ⚠️ Does not send Last-Event-ID header (SHOULD requirement)
+
+**Issue:** Client treats `retry:` SSE field as invalid instead of parsing it for reconnection timing.
+
+## Known Limitations
+
+1. **Resource Subscriptions:** SDK doesn't implement `resources/subscribe` and `resources/unsubscribe` handlers
+2. **Client SSE Retry:** Client doesn't parse or respect the `retry:` field, reconnects immediately, and doesn't send Last-Event-ID header
+3. **DNS Rebinding Protection:** Missing Host/Origin header validation in server transport
+
+## Running Tests
+
+### Server
+```bash
+# Start server
+cd conformance-tests/server-servlet
+../../mvnw compile exec:java -Dexec.mainClass="io.modelcontextprotocol.conformance.server.ConformanceServlet"
+
+# Run tests (in another terminal)
+npx @modelcontextprotocol/conformance server --url http://localhost:8080/mcp --suite active
+```
+
+### Client
+```bash
+# Build
+cd conformance-tests/client-jdk-http-client
+../../mvnw clean package -DskipTests
+
+# Run all scenarios
+for scenario in initialize tools_call elicitation-sep1034-client-defaults sse-retry; do
+  npx @modelcontextprotocol/conformance client \
+    --command "java -jar target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
+    --scenario $scenario
+done
+```
+
+## Recommendations
+
+### High Priority
+1. Fix client SSE retry field handling in `HttpClientStreamableHttpTransport`
+2. Implement resource subscription handlers in `McpStatelessAsyncServer`
+
+### Medium Priority
+3. Add Host/Origin validation in `HttpServletStreamableServerTransportProvider` for DNS rebinding protection

conformance-tests/client-jdk-http-client/README.md

@@ -0,0 +1,135 @@
+# MCP Conformance Tests - JDK HTTP Client
+
+This module provides a conformance test client implementation for the Java MCP SDK using the JDK HTTP Client with Streamable HTTP transport.
+
+## Overview
+
+The conformance test client is designed to work with the [MCP Conformance Test Framework](https://github.com/modelcontextprotocol/conformance). It validates that the Java MCP SDK client properly implements the MCP specification.
+
+## Architecture
+
+The client reads test scenarios from environment variables and accepts the server URL as a command-line argument, following the conformance framework's conventions:
+
+- **MCP_CONFORMANCE_SCENARIO**: Environment variable specifying which test scenario to run
+- **Server URL**: Passed as the last command-line argument
+
+## Supported Scenarios
+
+Currently implemented scenarios:
+
+- **initialize**: Tests the MCP client initialization handshake only
+  - ✅ Validates protocol version negotiation
+  - ✅ Validates clientInfo (name and version)
+  - ✅ Validates proper handling of server capabilities
+  - Does NOT call any tools or perform additional operations
+
+- **tools_call**: Tests tool discovery and invocation
+  - ✅ Initializes the client
+  - ✅ Lists available tools from the server
+  - ✅ Calls the `add_numbers` tool with test arguments (a=5, b=3)
+  - ✅ Validates the tool result
+
+- **elicitation-sep1034-client-defaults**: Tests client applies default values for omitted elicitation fields (SEP-1034)
+  - ✅ Initializes the client
+  - ✅ Lists available tools from the server
+  - ✅ Calls the `test_client_elicitation_defaults` tool
+  - ✅ Validates that the client properly applies default values from JSON schema to elicitation responses (5/5 checks pass)
+
+- **sse-retry**: Tests client respects SSE retry field timing and reconnects properly (SEP-1699)
+  - ⚠️ Initializes the client
+  - ⚠️ Lists available tools from the server
+  - ⚠️ Calls the `test_reconnection` tool which triggers SSE stream closure
+  - ✅ Client reconnects after stream closure (PASSING)
+  - ❌ Client does not respect retry timing (FAILING)
+  - ⚠️ Client does not send Last-Event-ID header (WARNING - SHOULD requirement)
+
+## Building
+
+Build the executable JAR:
+
+```bash
+cd conformance-tests/client-jdk-http-client
+../../mvnw clean package -DskipTests
+```
+
+This creates an executable JAR at:
+```
+target/client-jdk-http-client-0.18.0-SNAPSHOT.jar
+```
+
+## Running Tests
+
+### Using the Conformance Framework
+
+Run a single scenario:
+
+```bash
+npx @modelcontextprotocol/conformance client \
+  --command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
+  --scenario initialize
+
+npx @modelcontextprotocol/conformance client \
+  --command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
+  --scenario tools_call
+
+npx @modelcontextprotocol/conformance client \
+  --command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
+  --scenario elicitation-sep1034-client-defaults
+
+npx @modelcontextprotocol/conformance client \
+  --command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
+  --scenario sse-retry
+```
+
+Run with verbose output:
+
+```bash
+npx @modelcontextprotocol/conformance client \
+  --command "java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar" \
+  --scenario initialize \
+  --verbose
+```
+
+### Manual Testing
+
+You can also run the client manually if you have a test server:
+
+```bash
+export MCP_CONFORMANCE_SCENARIO=initialize
+java -jar conformance-tests/client-jdk-http-client/target/client-jdk-http-client-0.18.0-SNAPSHOT.jar http://localhost:3000/mcp
+```
+
+## Test Results
+
+The conformance framework generates test results showing:
+
+**Current Status (3/4 scenarios passing):**
+- ✅ initialize: 1/1 checks passed
+- ✅ tools_call: 1/1 checks passed
+- ✅ elicitation-sep1034-client-defaults: 5/5 checks passed
+- ⚠️ sse-retry: 1/2 checks passed, 1 warning
+
+Test result files are generated in `results/<scenario>-<timestamp>/`:
+- `checks.json`: Array of conformance check results with pass/fail status
+- `stdout.txt`: Client stdout output
+- `stderr.txt`: Client stderr output
+
+### Known Issue: SSE Retry Handling
+
+The `sse-retry` scenario currently fails because:
+1. The client treats the SSE `retry:` field as invalid instead of parsing it
+2. The client does not implement retry timing (reconnects immediately)
+3. The client does not send the Last-Event-ID header on reconnection
+
+This is a known limitation in the `HttpClientStreamableHttpTransport` implementation.
+
+## Next Steps
+
+Future enhancements:
+
+- Fix SSE retry field handling (SEP-1699) to properly parse and respect retry timing
+- Implement Last-Event-ID header on reconnection for resumability
+- Add auth scenarios (currently excluded as per requirements)
+- Implement a comprehensive "everything-client" pattern
+- Add to CI/CD pipeline
+- Create expected-failures baseline for known issues

conformance-tests/client-jdk-http-client/pom.xml

@@ -0,0 +1,78 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+	<modelVersion>4.0.0</modelVersion>
+	<parent>
+		<groupId>io.modelcontextprotocol.sdk</groupId>
+		<artifactId>conformance-tests</artifactId>
+		<version>0.18.0-SNAPSHOT</version>
+	</parent>
+	<artifactId>client-jdk-http-client</artifactId>
+	<packaging>jar</packaging>
+	<name>MCP Conformance Tests - JDK HTTP Client</name>
+	<description>JDK HTTP Client conformance tests for the Java MCP SDK</description>
+	<url>https://github.com/modelcontextprotocol/java-sdk</url>
+
+	<scm>
+		<url>https://github.com/modelcontextprotocol/java-sdk</url>
+		<connection>git://github.com/modelcontextprotocol/java-sdk.git</connection>
+		<developerConnection>git@github.com/modelcontextprotocol/java-sdk.git</developerConnection>
+	</scm>
+
+	<dependencies>
+		<dependency>
+			<groupId>io.modelcontextprotocol.sdk</groupId>
+			<artifactId>mcp</artifactId>
+			<version>0.18.0-SNAPSHOT</version>
+		</dependency>
+
+		<!-- Logging -->
+		<dependency>
+			<groupId>ch.qos.logback</groupId>
+			<artifactId>logback-classic</artifactId>
+			<version>${logback.version}</version>
+			<scope>runtime</scope>
+		</dependency>
+	</dependencies>
+
+	<build>
+		<plugins>
+			<!-- Maven Shade Plugin for creating executable JAR with dependencies -->
+			<plugin>
+				<groupId>org.apache.maven.plugins</groupId>
+				<artifactId>maven-shade-plugin</artifactId>
+				<version>3.5.1</version>
+				<executions>
+					<execution>
+						<phase>package</phase>
+						<goals>
+							<goal>shade</goal>
+						</goals>
+						<configuration>
+							<transformers>
+								<transformer
+									implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+									<mainClass>io.modelcontextprotocol.conformance.client.ConformanceJdkClientMcpClient</mainClass>
+								</transformer>
+								<transformer
+									implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
+							</transformers>
+							<filters>
+								<filter>
+									<artifact>*:*</artifact>
+									<excludes>
+										<exclude>META-INF/*.SF</exclude>
+										<exclude>META-INF/*.DSA</exclude>
+										<exclude>META-INF/*.RSA</exclude>
+									</excludes>
+								</filter>
+							</filters>
+						</configuration>
+					</execution>
+				</executions>
+			</plugin>
+		</plugins>
+	</build>
+
+</project>