00199 Hugging Face Llama Recipes - Local Inference

前言

简单的介绍如何Local Inference Llama。

src link: https://github.com/LuYF-Lemon-love/fork-huggingface-llama-recipes

Operating System: Ubuntu 22.04.4 LTS

参考文档

1. NLP Course - Local Inference

介绍

您想在本地运行Llama模型的推理吗？我们也是！内存需求取决于模型大小和权重的精度。下表显示了不同配置所需的大致内存：

Model Size	Llama Variant	BF16/FP16	FP8	INT4(AWQ/GPTQ/bnb)
1B	3.2	2.5 GB	1.25GB	0.75GB
3B	3.2	6.5 GB	3.2GB	1.75GB
8B	3.1	16 GB	8GB	4GB
70B	3.1	140 GB	70GB	35GB
405B	3.1	810 GB	405GB	204GB

这些是估计值，可能会根据特定的实现细节和优化而有所不同。

Llama-3.1-8B-Instruct in 4-bit bitsandbytes

src link: https://github.com/LuYF-Lemon-love/fork-huggingface-llama-recipes/blob/main/local_inference/4bit_bnb.ipynb

1. 首先，我们来安装所需的库：

pip install transformers[torch] bitsandbytes

2. 我们导入所需的库：

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

3. 让我们加载模型。为了即时量化模型，我们需要传递一个 quantization_config：

from modelscope import snapshot_download

model_dir = snapshot_download('LLM-Research/Meta-Llama-3.1-8B-Instruct')

quantization_config = BitsAndBytesConfig(load_in_4bit=True,
                                         bnb_4bit_compute_dtype=torch.bfloat16,
                                         bnb_4bit_use_double_quant=True,
                                         bnb_4bit_quant_type= "nf4"
                                         )

quantized_model = AutoModelForCausalLM.from_pretrained(
	model_dir, device_map="auto", torch_dtype=torch.bfloat16, quantization_config=quantization_config)

4. 然后，我们需要准备输入数据：

tokenizer = AutoTokenizer.from_pretrained(model_dir)
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

5. 最后，我们可以生成输出！

output = quantized_model.generate(**input_ids, max_new_tokens=256)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Llama-3.2-3B-Instruct in 8-bit bitsandbytes

src link: https://github.com/LuYF-Lemon-love/fork-huggingface-llama-recipes/blob/main/local_inference/8bit_bnb.ipynb

1. 首先，我们来安装所需的库：

pip install transformers[torch] bitsandbytes

2. 我们导入所需的库：

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

3. 让我们加载模型。为了实时量化模型，我们传递一个量化配置：

from modelscope import snapshot_download

model_dir = snapshot_download('LLM-Research/Llama-3.2-3B-Instruct')
quantization_config = BitsAndBytesConfig(load_in_8bit=True)

quantized_model = AutoModelForCausalLM.from_pretrained(
	model_dir, device_map="auto", torch_dtype=torch.bfloat16, quantization_config=quantization_config)

4. 然后，我们需要准备输入：

tokenizer = AutoTokenizer.from_pretrained(model_dir)
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

5. 最后，我们可以生成输出！

output = quantized_model.generate(**input_ids, max_new_tokens=10)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Llama-3.1-8B-Instruct with AWQ & fused ops

src link: https://github.com/LuYF-Lemon-love/fork-huggingface-llama-recipes/blob/main/local_inference/awq.ipynb

Model Checkpoint: https://huggingface.co/hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4

注意：虽然在这个例子中我们只使用了8B的Instruct检查点，但您可以使用相同的代码库来处理任何Llama 3.1模型的检查点，例如70B、405B（以及微调）！

1. 由于Llama 3.1带来了一些小的模型变化（主要是RoPE缩放），我们需要确保我们使用的是transformers的最新版本。

pip install -q --upgrade transformers autoawq accelerate

2. 加载分词器和模型检查点。

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, AwqConfig

model_id = "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4"

quantization_config = AwqConfig(
    bits=4,
    fuse_max_seq_len=512, # Note: Update this as per your use-case
    do_fuse=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
  model_id,
  torch_dtype=torch.float16,
  low_cpu_mem_usage=True,
  quantization_config=quantization_config
).to("cuda")

3. 定义提示并进行分词处理。

prompt = [
  {"role": "system", "content": "You are a helpful assistant, that responds as a pirate."},
  {"role": "user", "content": "What's Deep Learning?"},
]

inputs = tokenizer.apply_chat_template(
  prompt,
  tokenize=True,
  add_generation_prompt=True,
  return_tensors="pt",
  return_dict=True,
).to("cuda")

4. 生成。

outputs = model.generate(**inputs, do_sample=True, max_new_tokens=25)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

瞧！现在你拥有了一个智能且能干的助手！🦙

Llama-3.1-8B-Instruct in INT4 with GPTQ

src link: https://github.com/LuYF-Lemon-love/fork-huggingface-llama-recipes/blob/main/local_inference/gptq_generation.py

# INSTALLATION
# pip install -q --upgrade transformers accelerate optimum
# pip install -q --no-build-isolation auto-gptq
# REQUIREMENTS
# An instance with at least ~210 GiB of total GPU memory when using the 405B model.
# The INT4 versions of the 70B and 8B models require ~35 GiB and ~4 GiB, respectively.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "hugging-quants/Meta-Llama-3.1-8B-Instruct-GPTQ-INT4"
messages = [
    {"role": "system", "content": "You are a pirate"},
    {"role": "user", "content": "What's Deep Leaning?"},
]

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
    device_map="auto",
)

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to("cuda")

outputs = model.generate(**inputs, do_sample=True, max_new_tokens=256)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

结语

第一百九十九篇博文写完，开心！！！！

今天，也是充满希望的一天。