Struct candle_core::quantized::QTensor

pub struct QTensor { /* private fields */ }

Implementations

impl QTensor

pub fn new<S: Into<Shape>>(storage: QStorage, shape: S) -> Result<Self>

pub fn quantize(src: &Tensor, dtype: GgmlDType) -> Result<Self>

pub fn dtype(&self) -> GgmlDType

pub fn device(&self) -> Device

pub fn rank(&self) -> usize

pub fn shape(&self) -> &Shape

pub fn dequantize(&self, device: &Device) -> Result<Tensor>

pub fn dequantize_f16(&self, device: &Device) -> Result<Tensor>

pub fn storage_size_in_bytes(&self) -> usize

pub fn data(&self) -> Result<Cow<'_, [u8]>>

Trait Implementations

impl CustomOp1 for QTensor

fn name(&self) -> &'static str

fn cpu_fwd( &self, storage: &CpuStorage, layout: &Layout ) -> Result<(CpuStorage, Shape)>

The forward pass, as run on a cpu device. Note that the storage can use arbitrary strides, offsets etc so the associated layout should be used to access it.

fn metal_fwd( &self, storage: &MetalStorage, layout: &Layout ) -> Result<(MetalStorage, Shape)>

The forward pass, as run on a metal gpu device. Note that the storage can use arbitrary strides, offsets etc so the associated layout should be used to access it.

fn cuda_fwd( &self, storage: &CudaStorage, layout: &Layout ) -> Result<(CudaStorage, Shape)>

The forward pass, as run on a gpu device. Note that the storage can use arbitrary strides, offsets etc so the associated layout should be used to access it.

fn bwd( &self, _arg: &Tensor, _res: &Tensor, _grad_res: &Tensor ) -> Result<Option<Tensor>>

This function takes as argument the argument arg used in the forward pass, the result produced by the forward operation res and the gradient of the result grad_res. The function should return the gradient of the argument.

impl Debug for QTensor

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.