README updated

Author: DzmitryPihulski

README.md

@@ -42,14 +42,14 @@ The package doesn't have the dataset, it is stored on our [HuggingFace page](htt
 
 ## Latest News 📣
 
+* [2026/03] Fully functional CLI commands for inference and evaluation. See this [guide](./llmsql/_cli/README.md).
+
 * [2026/03] Added support for API inference, for now only for OpenAI-compatable APIs, see [`inference_api()` function](./llmsql/inference/inference_api.py#inference_api)
 
 * [2026/03] The page now contains first version of [leaderboard](https://llmsql.github.io/llmsql-benchmark/#:~:text=%F0%9F%93%8A%20Leaderboard%20%E2%80%94%20Execution%20Accuracy%20%28EX)!
 
 * [2026/02] The new LLMSQL 2.0 version is out now! See the [dataset](https://huggingface.co/datasets/llmsql-bench/llmsql-2.0). The support is already added with the `version` parameter to each `inference` function.
 
-* [2025/12] Evaluation class converted to function see [new `evaluate(...)` function](./llmsql/evaluation/evaluate.py#evaluate)
-
 
 
 ## Usage Recommendations

examples/test_output.jsonl

@@ -0,0 +1,20 @@
+{"question_id": 15, "completion": "SELECT \"Launched\" FROM \"Table\" WHERE \"Laid down\" = \"September 1, 1964\";"}
+{"question_id": 48, "completion": "SELECT \"Tone (Latvian notation: /~/ - level, /^/ - broken)\" FROM \"Table\" WHERE \"Translation\" = \"Gen.Sing. plague\";"}
+{"question_id": 4, "completion": "SELECT COUNT(\"School/Club Team\") FROM \"Table\" WHERE \"Player\" = \"Jalen Rose\";"}
+{"question_id": 21, "completion": "SELECT \"US Air Force equivalent\" FROM \"Table\" WHERE \"Commonwealth equivalent\" = \"Pilot Officer\";"}
+{"question_id": 31, "completion": "SELECT COUNT(\"Total W–L\") FROM \"Table\" WHERE \"Doubles W–L\" = \"11–11\";"}
+{"question_id": 42, "completion": "SELECT \"BTs retail price (regulated)\" FROM \"Table\" WHERE \"Tariff code\" = \"ff0 PRS\";"}
+{"question_id": 7, "completion": "SELECT \"Date\" FROM \"Table\" WHERE \"Circuit\" = \"Misano\";"}
+{"question_id": 28, "completion": "SELECT \"Rank Each wrestlers total number of days as champion are ranked highest to lowest; wrestlers with the same number mean that they are tied for that certain rank.\" FROM \"Table\" WHERE \"Wrestler\" = \"Go Shiozaki\";"}
+{"question_id": 36, "completion": "SELECT \"Frequency\" FROM \"Table\" WHERE \"Market/Rank\" = \"Burlington - Plattsburgh , Vermont - New York /143\";"}
+{"question_id": 23, "completion": "SELECT \"Rank in Spanish\" FROM \"Table\" WHERE \"Rank in English\" = \"Major\";"}
+{"question_id": 11, "completion": "SELECT COUNT(DISTINCT \"Nationality\") FROM \"Table\" WHERE \"NHL team\" = \"New Jersey Devils\";"}
+{"question_id": 47, "completion": "SELECT \"BTs retail price (regulated)\" FROM \"Table\" WHERE \"Tariff code\" = \"g10\";"}
+{"question_id": 12, "completion": "SELECT \"Pick\" FROM \"Table\" WHERE \"Player\" = \"Dorain Anneck\";"}
+{"question_id": 16, "completion": "SELECT \"#\" FROM \"Table\" WHERE \"Commissioned\" = \"December 18, 1965\";"}
+{"question_id": 27, "completion": "SELECT \"Combined days\" FROM \"Table\" WHERE \"Wrestler\" = \"Go Shiozaki\";"}
+{"question_id": 32, "completion": "SELECT COUNT(\"Singles W–L\") FROM \"Table\" WHERE \"Doubles W–L\" = \"11–14\";"}
+{"question_id": 22, "completion": "SELECT \"Commonwealth equivalent\" FROM \"Table\" WHERE \"US Air Force equivalent\" = \"Major General\";"}
+{"question_id": 43, "completion": "SELECT \"Approx premium\" FROM \"Table\" WHERE \"Tariff code\" = \"g9\";"}
+{"question_id": 49, "completion": "SELECT MIN(\"Radius (R ☉ )\") FROM \"Table\";"}
+{"question_id": 34, "completion": "SELECT MAX(\"Ties played\") FROM \"Table\" WHERE \"Player\" = \"Josip Palada Category:Articles with hCards\";"}

llmsql/_cli/README.md

@@ -0,0 +1,73 @@
+# LLMSQL CLI Manual
+
+The `llmsql` CLI provides two main workflows:
+
+- `inference`: generate SQL predictions with a selected backend.
+- `evaluate`: score predictions on the LLMSQL benchmark.
+
+## Command Structure
+
+```bash
+llmsql <command> [options]
+```
+
+Available commands:
+
+- `llmsql inference transformers ...`
+- `llmsql inference vllm ...`
+- `llmsql inference api ...`
+- `llmsql evaluate ...`
+
+## Inference Commands
+
+### 1) Transformers backend
+
+```bash
+llmsql inference transformers \
+  --model-or-model-name-or-path Qwen/Qwen2.5-1.5B-Instruct \
+  --output-file outputs/preds_transformers.jsonl
+```
+
+This command calls [`inference_transformers()`](../inference/inference_transformers.py).
+
+### 2) vLLM backend
+
+```bash
+llmsql inference vllm \
+  --model-name Qwen/Qwen2.5-1.5B-Instruct \
+  --output-file outputs/preds_vllm.jsonl
+```
+
+This command calls [`inference_vllm()`](../inference/inference_vllm.py).
+
+### 3) OpenAI-compatible API backend
+
+```bash
+llmsql inference api \
+  --model-name gpt-5-mini \
+  --base-url https://api.openai.com/v1 \
+  --output-file outputs/preds_api.jsonl
+```
+
+This command calls [`inference_api()`](../inference/inference_api.py).
+
+## Evaluation Command
+
+```bash
+llmsql evaluate --outputs outputs/preds_transformers.jsonl
+```
+
+This command calls [`evaluate()`](../evaluation/evaluate.py).
+
+## Help
+
+Use built-in help to see all options:
+
+```bash
+llmsql --help
+llmsql inference --help
+llmsql inference transformers --help
+llmsql inference vllm --help
+llmsql inference api --help
+llmsql evaluate --help
+```

llmsql/evaluation/evaluate.py

@@ -14,7 +14,11 @@
 
 from rich.progress import track
 
-from llmsql.config.config import DEFAULT_WORKDIR_PATH, DEFAULT_LLMSQL_VERSION, get_repo_id
+from llmsql.config.config import (
+    DEFAULT_LLMSQL_VERSION,
+    DEFAULT_WORKDIR_PATH,
+    get_repo_id,
+)
 from llmsql.utils.evaluation_utils import (
     connect_sqlite,
     download_benchmark_file,