Skip to main content

TokenCounter

htmlp

Trait TokenCounter 

Source 
pub trait TokenCounter {
    // Required methods
    fn name(&self) -> &str;
    fn count(&self, text: &str) -> u64;
}
Expand description

Inject a tokenizer to keep the parser independent of large vocabulary tables. name must match the document’s tokenizer identifier. Implementations must count the supplied string as ordinary text, without interpreting special IDs.

Required Methods§

Source

fn name(&self) -> &str

Source

fn count(&self, text: &str) -> u64

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§