github/fauxpilot
1
0
Fork 0
mirror of https://github.com/fauxpilot/fauxpilot.git synced 2025-08-14 02:28:06 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix: enabled !INT8 as a default CC type in the Python-backend

Browse source 
CC (Compute Capability) of Nvidia GPU supports INT8 from version 9.0.
Therefore, this patch is to change the default SM (Streaming Machine) type to INT8
so that stable operation is compatible with versions between CC 6.0 and CC 9.0.

* Prerequisites: An NVIDIA GPU with Compute Capability >= 6.0
  https://github.com/fauxpilot/fauxpilot#prerequisites

* Architecture of Compute Capability (CC)9.0
  https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#id100

Signed-off-by: Geunsik Lim <leemgs@gmail.com>
This commit is contained in:
Geunsik Lim 2023-01-03 14:51:16 +09:00
commit 1ecd68aeb1
1 changed files with 2 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

4
setup.sh
View file

@ -154,8 +154,8 @@ function python_backend(){
fi
# use int8? Allows larger models to fit in GPU but might be very marginally slower
read -rp "Do you want to use int8? y/n [y]: " USE_INT8
if [[ ${USE_INT8:-y} =~ ^[Nn]$ ]]; then
read -rp "Do you want to use int8? y/n [n]: " USE_INT8
if [[ ! $USE_INT8 =~ ^[Yy]$ ]]; then
USE_INT8="0"
else
USE_INT8="1"