chore: skill helper for integrating firebaseui-web#1356
chore: skill helper for integrating firebaseui-web#1356russellwheatley wants to merge 7 commits intomainfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces the integration-helper agent skill and a comprehensive evaluation harness to assist and benchmark the integration of FirebaseUI Web v7 across various frameworks like React, Next.js, and Angular. The skill includes detailed reference documentation for framework setup, authentication flows, and migration strategies. The evaluation harness supports multiple runners (Codex, Cursor, OpenCode) and implements both deterministic and LLM-based grading. Feedback focuses on improving the robustness of the evaluation scripts, specifically by using unique temporary directory names to avoid collisions on shared machines and implementing more resilient JSON parsing for LLM-generated judge responses.
I am having trouble creating individual review comments. Click here to see my feedback.
evals/integration-helper/scripts/run.mjs (29)
Using a hardcoded directory name in os.tmpdir() can lead to collisions or permission issues if multiple users run the evaluation harness on the same shared machine. Appending the username or a unique identifier to the path is a safer practice.
const defaultScratchRoot = path.join(os.tmpdir(), "firebaseui-web-integration-helper-eval-" + (process.env.USER || process.env.USERNAME || "default"));evals/integration-helper/scripts/run.mjs (491-493)
The JSON.parse call is fragile because LLMs often wrap JSON responses in markdown code blocks (e.g., ```json ... ```), even when instructed otherwise. Extracting the JSON content by finding the first and last curly braces makes the parsing more robust.
try {
const rawText = judgeRun.finalAssistantText;
const start = rawText.indexOf("{");
const end = rawText.lastIndexOf("}");
const jsonContent = (start !== -1 && end !== -1) ? rawText.substring(start, end + 1) : rawText;
grading = JSON.parse(jsonContent.trim());
} catch {
No description provided.