flant5-d 0.2.0

FLAN-T5 bindings for D via CTranslate2 + sentencepiece


To use this package, run the following command in your project's root directory:

Manual usage
Put the following dependency into your project's dependences section:

flant5-d

optimized inference of [FLAN-T5] models in D

  • can run any huggingface T5 family model (including FlanT5, T0, etc.)

powered by CTranslate2 and sentencepiece

⚠️ the static library build of CTranslate2 depends on CBLAS, MKL, and more, and it's typically better to let those be linked dynamically. so you will need to copy $PACKAGE_DIR/libctranslate2.so from this project when built to libctranslate2.so.3 next to your binary.

usage

models

obtain a model that is in the CTranslate2 (ct2) format. see the ctranslate2 model conversion documentation for specific instructions.

for cpu reference, int8 quantization is recommended for best performance and memory usage.

assuming you have a model in the ct2 format, you can use it like so:

auto gen = FlanT5Generator();

// load model
gen.load_model(model_path);

// generation params
auto gen_params = gen.default_gen_params;
gen_params.beam_size = 5;
gen_params.sampling_temperature = cli_options.temp;
gen_params.sampling_topk = 10;
gen_params.max_input_length = 1024;
gen_params.max_decoding_length = 150;
gen_params.repetition_penalty = 1.1;

// generate
auto test_output = gen.generate(test_input);
Authors:
  • redthing1
Dependencies:
none
Versions:
0.2.0 2023-May-25
0.1.4 2023-May-24
0.1.3 2023-May-24
0.1.2 2023-May-24
0.1.1 2023-May-24
Show all 7 versions
Download Stats:
  • 0 downloads today

  • 0 downloads this week

  • 34 downloads this month

  • 34 downloads total

Score:
1.3
Short URL:
flant5-d.dub.pm