Track: Track1; Team name: howyadoin; Model: SheafAN

2026_tdl_challenge/run_evaluation.ipynb

@@ -1,174 +1,174 @@
 {
-  "cells": [
-    {
-      "cell_type": "markdown",
-      "id": "021b3a4c",
-      "metadata": {},
-      "source": [
-        "# 2026 TDL Challenge: GraphUniverse\n",
-        "\n",
-        "This notebook evaluates your model across a grid of GraphUniverse's synthetic graph distributions:\n",
-        "- **Homophily** (low, mid, high)\n",
-        "- **Average degree** (low, high)  \n",
-        "- **Power-law exponent** (low, high)\n",
-        "\n",
-        "Each configuration is trained with multiple random seeds. The best checkpoint from each run is then evaluated on all other grid settings (out-of-distribution evaluation).\n",
-        "\n",
-        "## Setup Requirements\n",
-        "\n",
-        "**Make sure Weights & Biases is configured:**\n",
-        "```bash\n",
-        "wandb login\n",
-        "```\n",
-        "\n",
-        "## How to Use\n",
-        "\n",
-        "1. Set your `MODEL_CONFIG` (path to your model yaml)\n",
-        "2. Run the evaluation\n",
-        "3. Results will be saved in:\n",
-        "   - `results.json` with detailed metrics\n",
-        "   - Heatmap visualizations showing performance across the grid\n",
-        "   - OOD performance delta plots"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "53c1d2fb",
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import sys\n",
-        "from pathlib import Path\n",
-        "\n",
-        "# Setup paths\n",
-        "_ROOT = Path.cwd().resolve()\n",
-        "_REPO = _ROOT if (_ROOT / \"configs\" / \"run.yaml\").exists() else _ROOT.parent\n",
-        "if str(_REPO) not in sys.path:\n",
-        "    sys.path.insert(0, str(_REPO))\n",
-        "\n",
-        "from utils import (\n",
-        "    resolve_project_root,\n",
-        "    run_challenge_grid,\n",
-        "    save_challenge_artifacts,\n",
-        ")\n",
-        "\n",
-        "PROJECT_ROOT = resolve_project_root(_REPO)"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "id": "config_section",
-      "metadata": {},
-      "source": [
-        "## Configuration\n",
-        "\n",
-        "**Set your model config path here:**"
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "config_cell",
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "# Your model configuration (e.g., \"graph/gcn\", \"graph/gin\", \"graph/gat\")\n",
-        "MODEL_CONFIG = \"graph/gin\""
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "id": "run_section",
-      "metadata": {},
-      "source": [
-        "## Run Evaluation\n",
-        "\n",
-        "This will train and evaluate your model across all grid configurations."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "93a4bb1f",
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "results, study_id = run_challenge_grid(\n",
-        "    project_root=PROJECT_ROOT,\n",
-        "    model_config=MODEL_CONFIG,\n",
-        ")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "id": "save_section",
-      "metadata": {},
-      "source": [
-        "## Save Results\n",
-        "\n",
-        "Generate JSON output and visualization plots."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "save_cell",
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "output_paths = save_challenge_artifacts(\n",
-        "    results,\n",
-        "    model_config=MODEL_CONFIG,\n",
-        "    study_id=study_id,\n",
-        ")\n",
-        "\n",
-        "print(f\"\\nResults saved to: {output_paths['dir']}\")\n",
-        "print(f\"JSON: {output_paths['json']}\")"
-      ]
-    },
-    {
-      "cell_type": "markdown",
-      "id": "inspect_section",
-      "metadata": {},
-      "source": [
-        "## Inspect Results\n",
-        "\n",
-        "View results as a DataFrame for quick inspection."
-      ]
-    },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "id": "inspect_cell",
-      "metadata": {},
-      "outputs": [],
-      "source": [
-        "import pandas as pd\n",
-        "\n",
-        "pd.DataFrame(results)"
-      ]
-    }
-  ],
-  "metadata": {
-    "kernelspec": {
-      "display_name": "tb",
-      "language": "python",
-      "name": "python3"
-    },
-    "language_info": {
-      "codemirror_mode": {
-        "name": "ipython",
-        "version": 3
-      },
-      "file_extension": ".py",
-      "mimetype": "text/x-python",
-      "name": "python",
-      "nbconvert_exporter": "python",
-      "pygments_lexer": "ipython3",
-      "version": "3.11.3"
-    }
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "021b3a4c",
+   "metadata": {},
+   "source": [
+    "# 2026 TDL Challenge: GraphUniverse\n",
+    "\n",
+    "This notebook evaluates your model across a grid of GraphUniverse's synthetic graph distributions:\n",
+    "- **Homophily** (low, mid, high)\n",
+    "- **Average degree** (low, high)  \n",
+    "- **Power-law exponent** (low, high)\n",
+    "\n",
+    "Each configuration is trained with multiple random seeds. The best checkpoint from each run is then evaluated on all other grid settings (out-of-distribution evaluation).\n",
+    "\n",
+    "## Setup Requirements\n",
+    "\n",
+    "**Make sure Weights & Biases is configured:**\n",
+    "```bash\n",
+    "wandb login\n",
+    "```\n",
+    "\n",
+    "## How to Use\n",
+    "\n",
+    "1. Set your `MODEL_CONFIG` (path to your model yaml)\n",
+    "2. Run the evaluation\n",
+    "3. Results will be saved in:\n",
+    "   - `results.json` with detailed metrics\n",
+    "   - Heatmap visualizations showing performance across the grid\n",
+    "   - OOD performance delta plots"
+   ]
   },
-  "nbformat": 4,
-  "nbformat_minor": 5
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "53c1d2fb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sys\n",
+    "from pathlib import Path\n",
+    "\n",
+    "# Setup paths\n",
+    "_ROOT = Path.cwd().resolve()\n",
+    "_REPO = _ROOT if (_ROOT / \"configs\" / \"run.yaml\").exists() else _ROOT.parent\n",
+    "if str(_REPO) not in sys.path:\n",
+    "    sys.path.insert(0, str(_REPO))\n",
+    "\n",
+    "from utils import (  # noqa: E402\n",
+    "    resolve_project_root,\n",
+    "    run_challenge_grid,\n",
+    "    save_challenge_artifacts,\n",
+    ")\n",
+    "\n",
+    "PROJECT_ROOT = resolve_project_root(_REPO)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "config_section",
+   "metadata": {},
+   "source": [
+    "## Configuration\n",
+    "\n",
+    "**Set your model config path here:**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "config_cell",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Your model configuration\n",
+    "MODEL_CONFIG = \"graph/san\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "run_section",
+   "metadata": {},
+   "source": [
+    "## Run Evaluation\n",
+    "\n",
+    "This will train and evaluate your model across all grid configurations."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "93a4bb1f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results, study_id = run_challenge_grid(\n",
+    "    project_root=PROJECT_ROOT,\n",
+    "    model_config=MODEL_CONFIG,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "save_section",
+   "metadata": {},
+   "source": [
+    "## Save Results\n",
+    "\n",
+    "Generate JSON output and visualization plots."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "save_cell",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "output_paths = save_challenge_artifacts(\n",
+    "    results,\n",
+    "    model_config=MODEL_CONFIG,\n",
+    "    study_id=study_id,\n",
+    ")\n",
+    "\n",
+    "print(f\"\\nResults saved to: {output_paths['dir']}\")\n",
+    "print(f\"JSON: {output_paths['json']}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "inspect_section",
+   "metadata": {},
+   "source": [
+    "## Inspect Results\n",
+    "\n",
+    "View results as a DataFrame for quick inspection."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "inspect_cell",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "\n",
+    "pd.DataFrame(results)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "tb",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
 }

configs/model/graph/san.yaml

@@ -0,0 +1,48 @@
+_target_: topobench.model.TBModel
+
+model_name: san
+model_domain: graph
+
+feature_encoder:
+  _target_: topobench.nn.encoders.${model.feature_encoder.encoder_name}
+  encoder_name: AllCellFeatureEncoder
+  in_channels: ${infer_in_channels:${dataset},${oc.select:transforms,null}}
+  out_channels: 64
+  proj_dropout: 0.0
+
+backbone:
+  _target_: topobench.nn.backbones.SANEncoder
+  input_dim: ${model.feature_encoder.out_channels}
+  hidden_dim: ${model.feature_encoder.out_channels}
+  num_layers: 2 # SWEEP: [2, 4, 8]
+  dropout: 0.4 # SWEEP: [0.0, 0.25, 0.5]
+  sheaf_type: bundle # SWEEP: [diag, bundle, general]
+  d: 3 # paper setting; hidden_channels = hidden_dim // d internally
+  input_dropout: 0.0
+  num_heads: 4 # SWEEP: [1, 4, 8]
+  residual: false # toggle for Res-SheafAN
+  orth: cayley # cayley or matrix_exp
+
+backbone_wrapper:
+  _target_: topobench.nn.wrappers.${model.backbone_wrapper.wrapper_name}
+  _partial_: true
+  wrapper_name: GNNWrapper
+  out_channels: ${model.feature_encoder.out_channels}
+  residual_connections: false
+  num_cell_dimensions: ${infer_num_cell_dimensions:${oc.select:model.feature_encoder.selected_dimensions,null},${model.feature_encoder.in_channels}}
+
+readout:
+  _target_: topobench.nn.readouts.${model.readout.readout_name}
+  readout_name: MLPReadout
+  num_cell_dimensions: ${infer_num_cell_dimensions:${oc.select:model.feature_encoder.selected_dimensions,null},${model.feature_encoder.in_channels}}
+  in_channels: ${model.feature_encoder.out_channels}
+  hidden_layers: [16]
+  out_channels: ${dataset.parameters.num_classes}
+  task_level: ${define_task_level:${dataset.parameters.task_level},${dataset.split_params.learning_setting}}
+  pooling_type: sum
+  dropout: 0.2
+  act: "relu"
+  norm: null
+  final_act: null
+
+compile: false