[docs/faq] Collect real user questions from GitHub issues and community channels

[mesutoezdil]

Context

This issue is part of the HAMi v2.9.0 Documentation Revamp Plan (WS6 - FAQ Revamp).

The current FAQ has been cleaned up for style and language but contains no new content. Before writing new FAQ entries, we need to know what questions users are actually asking. This issue covers the research phase only.

Before you start

Please reach out to the maintainers before picking this up:

• @rootsongjc (author, architecture decisions)
• @mesutoezdil (revamp reviewer, scope alignment)

Scope

Search the following sources and compile a list of recurring or unanswered questions:

• GitHub issues in Project-HAMi/HAMi and Project-HAMi/website (closed and open)
• GitHub Discussions (if any)
• Any public Slack or Discord channels referenced in the documentation

Pay particular attention to questions about:

• GPU virtualization model: how HAMi virtualizes GPU resources, what the limits are, how it differs from MIG
• Scheduling interaction: how HAMi integrates with the Kubernetes scheduler, how it works alongside Volcano
• Ecosystem integration: using HAMi with vLLM, monitoring tools, and cloud providers

For each question found, note:

• The source (link to issue or thread)
• A short summary of the question
• Whether it is already answered in the docs (and if so, where)
• Whether it would make a good FAQ entry

Acceptance criteria

• At least 10 candidate questions identified
• Each question has a source link
• Duplicates are merged into one entry
• Delivered as a comment on this issue or as a markdown file in a PR

Notes

• This is a research task, no writing of FAQ answers required at this stage
• The output will feed directly into the follow-up issue [docs/faq] Write new FAQ entries covering GPU virtualization, scheduling, and ecosystem integration #416 for writing new FAQ entries
• DCO sign-off required on every commit (git commit -s)

[mesutoezdil]

FAQ Candidate Questions - Research Output

Sources searched:

• Project-HAMi/HAMi issues (open and closed, ~300 reviewed)
• Project-HAMi/HAMi GitHub Discussions
• Project-HAMi/website issues
• Existing docs/faq/faq.md (used to exclude already-answered questions)

Each entry below is marked with whether it is already covered in the current FAQ.

---

Category 1: GPU Virtualization Model

Q1. How does HAMi virtualize GPU resources - does it use a kernel driver, hardware partition, or a library?

• Source: Understanding resource segmentation HAMi#962, Could we design a pluggable device virtualization framework for heterogeneous hardware, like k8s device plugin ? HAMi#1609
• Summary: Users consistently ask how HAMi actually enforces memory and compute limits - is it hardware MIG, kernel-level, or user-space?
• Already in FAQ: No (the FAQ mentions HAMi-core but does not explain the interception mechanism)
• Good FAQ entry: Yes - this is the single most-asked conceptual question

Q2. How does HAMi differ from NVIDIA MIG? When should I use HAMi vGPU and when should I use MIG?

• Source: different MIG mode on different device on the same node HAMi#1044, the division of mig-gpus is confusing when using dynamic-mig HAMi#1054, pod's resources.limits gpumem cannot be equal to memory defined in mig allowedGeometries HAMi#1074
• Summary: Users deploying on A100/H100 clusters ask whether to use HAMi vGPU or MIG, and whether they can use both simultaneously.
• Already in FAQ: No
• Good FAQ entry: Yes

Q3. Why does nvidia-smi inside my container show different memory from the host?

• Source: The GPU memory usage displayed by executing the nvidia-smi command in the container and on the host is different HAMi#1408, Different between memory usage reported by driver and inside container HAMi#1196, Inaccurate GPU Memory Usage Reporting HAMi#952, Mismatch vGPU Memory actual allocate for K8s Pods HAMi#1093
• Summary: Users expect container memory to match the host but HAMi deliberately shows only the allocated limit. This is intentional behavior that is never clearly explained.
• Already in FAQ: No
• Good FAQ entry: Yes - high volume, recurring confusion

Q4. Why is my nvidia.com/gpumem limit not enforced - the container can still use more memory than requested?

• Source: HAMi does not limit the size of video memory HAMi#1112, The Pod is sharing the GPU with other Pods when CUDA_DISABLE_CONTROL=true and nvidia.com/gpumem=XXX HAMi#1678, The limitations on gpumem and gpucores are not working correctly. HAMi#332, GPU Memory isolation not worked  HAMi#777
• Summary: Users set gpumem but find the limit is bypassed, often because CUDA_DISABLE_CONTROL=true is set or the container runtime is not configured correctly. Also bypassed in Docker-in-Docker (DinD) environments.
• Already in FAQ: No
• Good FAQ entry: Yes

Q5. What glibc version does HAMi-core require, and what happens on newer distributions (glibc 2.35+)?

• Source: GPU core isolation does not work properly in environments with higher versions of glibc, such as glibc2.35. When will support for higher versions of glibc be available? HAMi#1190
• Summary: Users on Ubuntu 22.04+ or Fedora 36+ hit glibc incompatibilities with libvgpu.so. This is a hard constraint not documented in the prerequisites.
• Already in FAQ: No
• Good FAQ entry: Yes

Q6. Why is there a ~1 minute startup delay when many containers initialize libvgpu.so at the same time?

• Source: [Performance] High latency (~1 minute) when hundreds of processes initialize libvgpu.so concurrently HAMi#1662
• Summary: At scale (hundreds of pods starting simultaneously), libvgpu.so initialization serializes. Users want to know if this is expected and what the workaround is.
• Already in FAQ: No
• Good FAQ entry: Yes - affects production deployments

---

Category 2: Scheduling Interaction

Q7. Does HAMi replace kube-scheduler or run alongside it?

• Source: 为什么需要另启动一个kube-scheduler HAMi#127, 调度器模块这里后续会考虑采用scheduling framework的方式来实现吗？ HAMi#96
• Summary: A very early and still recurring question. Users fear HAMi will break existing workloads by replacing the default scheduler.
• Already in FAQ: No (FAQ covers Volcano integration but not the basic scheduler architecture)
• Good FAQ entry: Yes

Q8. Can I use HAMi with NVIDIA TimeSlicing enabled on a node at the same time?

• Source: If I enable timeSlicing scheduling on the GPU in EKS, will it be affected when applying for resources using Hami scheduling policy HAMi#1245
• Summary: Users running EKS or other managed clusters where timeSlicing is already enabled ask whether HAMi conflicts.
• Already in FAQ: No
• Good FAQ entry: Yes

Q9. Does HAMi support pod preemption in the Kubernetes scheduler?

• Source: Can the Hami scheduler achieve K8s preemption ？ HAMi#1494, [Feature] Pod Preemption Support for HAMi Scheduler HAMi#1878
• Summary: Users want higher-priority jobs to preempt lower-priority ones waiting for vGPU allocation. The existing FAQ covers runtime priority (nvidia.com/priority) but not scheduler-level preemption.
• Already in FAQ: Partially (runtime priority is covered, but scheduler preemption is not)
• Good FAQ entry: Yes

Q10. Does HAMi work with the Kubernetes cluster autoscaler?

• Source: Node pool autoscaler not working HAMi#1099
• Summary: Users in auto-scaling environments ask whether HAMi scheduler state is correctly reflected when nodes are added or removed by the cluster autoscaler.
• Already in FAQ: No
• Good FAQ entry: Yes

Q11. Can I run HAMi scheduler as a plugin inside Volcano instead of as a separate extender?

• Source: Can HAMi scheduler be integrated in Volcano as extender plugin? HAMi#1281
• Summary: Users familiar with Volcano's plugin architecture ask whether HAMi can be embedded rather than running as an external extender. Distinct from the existing Volcano FAQ entry which only covers the volcano-vgpu-device-plugin integration.
• Already in FAQ: Partially (FAQ mentions Volcano but not this architectural question)
• Good FAQ entry: Yes

---

Category 3: Ecosystem Integration

Q12. How does HAMi work with vLLM? What are the known limitations for multi-GPU tensor parallelism?

• Source: vLLM crashes when booting 4 GPUs in volcano environment HAMi#1438, vLLM crashes under HAMi/libvgpu during NCCL initialization with Illegal device id and segfault in `multiprocess_memory_limit.c HAMi#1764, Multi-gpu processes such as vllm fail due to segfault error in nccl initialization HAMi#1853, vLLM 0.11.0 fails to initialize with NCCL/TCPStore errors under HAMi 2.7.0 (works on 2.6.1) HAMi#1479, vLLM 0.9.2 fails on HAMi vGPU when exposing only the “index-1” device: memory profiling assertion (Initial free == Current free) and HAMI-core ... host pid is error! HAMi#1381
• Summary: vLLM with tensor parallelism (tp>1) causes NCCL segfaults under older HAMi versions due to CUDA_DEVICE_MEMORY_SHARED_CACHE being set once per container. This is the highest-volume integration question in the repo and has been fixed in recent releases.
• Already in FAQ: No
• Good FAQ entry: Yes - very high volume

Q13. Is HAMi compatible with NVIDIA GPU Operator and DCGM metrics?

• Source: HAMI compatibility with GPU Operator monitoring (DCGM metrics) HAMi#1778
• Summary: Teams using GPU Operator for lifecycle management ask if HAMi's device plugin conflicts with the Operator's device plugin, and whether DCGM metrics still work.
• Already in FAQ: No
• Good FAQ entry: Yes

Q14. How do I set up Prometheus and Grafana monitoring with HAMi?

• Source: Cannot add prometheus following the doc HAMi#468, prometheus配置请教 HAMi#497, vgpu-monitor-metrics does not show in grafana HAMi#410, 监控的数据，有没有画好的grafana dashboard？ HAMi#60
• Summary: Users want to scrape HAMi's vGPU metrics with Prometheus and visualize them in Grafana. Recurring since early releases. A Grafana dashboard JSON is now included in the repo at static/grafana/gpu-dashboard.json but this is not documented in the FAQ.
• Already in FAQ: No
• Good FAQ entry: Yes

Q15. Is HAMi compatible with NVIDIA MPS (Multi-Process Service)?

• Source: MPS & HAMI compatibility HAMi#759
• Summary: MPS and HAMi both intercept CUDA at different layers. Users running high-throughput inference ask if they can use both.
• Already in FAQ: No
• Good FAQ entry: Yes

---

Summary

#	Question	Category	In FAQ?	Priority
Q1	How does HAMi virtualize GPU resources (library interception, not kernel/hardware)	Virtualization	No	High
Q2	HAMi vGPU vs NVIDIA MIG - when to use which	Virtualization	No	High
Q3	Why nvidia-smi shows different memory inside container vs host	Virtualization	No	High
Q4	Why is gpumem limit not enforced (CUDA_DISABLE_CONTROL, DinD)	Virtualization	No	High
Q5	glibc version requirements for HAMi-core / libvgpu.so	Virtualization	No	Medium
Q6	libvgpu.so startup latency at scale	Virtualization	No	Medium
Q7	Does HAMi replace kube-scheduler or extend it	Scheduling	No	High
Q8	HAMi with NVIDIA TimeSlicing enabled on the same node	Scheduling	No	Medium
Q9	Pod preemption support in HAMi scheduler	Scheduling	Partial	Medium
Q10	HAMi and cluster autoscaler compatibility	Scheduling	No	Medium
Q11	Running HAMi as a Volcano plugin vs external extender	Scheduling	Partial	Low
Q12	HAMi with vLLM multi-GPU tensor parallelism (tp>1)	Ecosystem	No	High
Q13	HAMi with NVIDIA GPU Operator and DCGM	Ecosystem	No	High
Q14	Prometheus and Grafana monitoring setup	Ecosystem	No	High
Q15	HAMi and NVIDIA MPS compatibility	Ecosystem	No	Medium

15 candidate questions identified, each with at least one source link. Duplicates have been merged (e.g., multiple vLLM NCCL issues merged into Q12, multiple Prometheus issues merged into Q14).

This list feeds directly into #416 for writing the FAQ entries.