Hi, I noticed loading of models with I64 mapping , like potion-code-16M, results in incorrect mapping because it assumes the mapping is I32 in the token_mapping code. Ignoring the mapping dtype in safetensors header.
let v: Vec<usize> = raw
.chunks_exact(4)
.map(|b| i32::from_le_bytes(b.try_into().unwrap()) as usize)
.collect();
Suggestion something like this to support loading both I32 and I64:
let v: Vec<usize> = match t.dtype() {
Dtype::I64 => raw
.chunks_exact(8)
.map(|b| i64::from_le_bytes(b.try_into().unwrap()) as usize)
.collect(),
Dtype::I32 => raw
.chunks_exact(4)
.map(|b| i32::from_le_bytes(b.try_into().unwrap()) as usize)
.collect(),
other => return Err(anyhow!("unsupported mapping dtype {:?}", other)),
};
Hi, I noticed loading of models with I64 mapping , like potion-code-16M, results in incorrect mapping because it assumes the mapping is I32 in the token_mapping code. Ignoring the mapping dtype in safetensors header.
Suggestion something like this to support loading both I32 and I64: