add tokenizer files in OpenAI format that can be used in VSCode Copilot extension

Browse source

...

This commit is contained in:

Brendan Dolan-Gavitt 2022-08-08 15:41:57 -04:00

parent 6edeb54505

commit fc097e1f08

2 changed files with 50002 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

1

copilot_proxy/cgtok/openai_format/tokenizer.json Normal file

View file

File diff suppressed because one or more lines are too long

50001

copilot_proxy/cgtok/openai_format/vocab.bpe Normal file

View file

File diff suppressed because it is too large Load diff