[request] add 4060 ti 16 gb support

[cerulliber]

[request] add 4060 ti 16 gb support

thanks!

[dreaming-panda]

we are working on this now!

[dreaming-panda]

{
    "model": "hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4",
    "draft_model": "hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4",
    "offload": true,
    "max_length": 8192,
    "num_cache_layers": 0,
    "generation_length": 256,
    "max_turns": 16,
    "topk": 32,
    "temperature": 0.6,
    "topp": 0.9,
    "repetition_penalty": 1.05,
    "width": 16,
    "num_beams": 24,
    "depth": 24,
    "engine": "dynamic",
    "template": "meta-llama3",
    "dtype": "float16"
}

How about this configuration? Or which model do you want to run? Now we support Llama, Qwen, QwQ and Mistral.

Looking for your feedback! @cerulliber