Antigravity Skills - Build and Discover AI Agent SkillsAntigravity Skills

Hugging Face Community Evals Skill · Data Ai

Local Hugging Face Model Evaluation Skill

Run local evaluations for Hugging Face Hub models using 3 specialized scripts.

Optimize GPU performance with vLLM and lighteval for faster local results.

  • Hugging Face
  • LLM Evals
  • vLLM
  • GPU Testing
  • inspect-ai

About This Skill

This skill provides 3 specialized scripts to run local evaluations for Hugging Face Hub models using 2 major frameworks: inspect-ai and lighteval. It streamlines GPU smoke tests and backend selection for developers.

Quick Start

  1. 1Install uv and set your HF_TOKEN environment variable.
  2. 2Choose between inspect-ai or lighteval scripts based on your task.
  3. 3Execute the evaluation locally using uv run.
Example Command
uv run scripts/inspect_vllm_uv.py --model meta-llama/Llama-3-8b

Core Capabilities

inspect-ai with local inference

Run evaluations using the inspect-ai framework against models on the Hugging Face Hub using local inference providers.

lighteval with local inference

Execute lighteval benchmarks on local hardware, supporting both vLLM and accelerate backends.

Backend Selection Strategy

Choose the optimal backend between vLLM, Hugging Face Transformers, and accelerate for your specific GPU setup.

Smoke Tests and Fallbacks

Perform GPU smoke tests and implement task selection with backend fallback strategies to ensure evaluation stability.

Usage Examples

Input

Run an inspect-ai evaluation on a Llama-3 model using local vLLM.

Output

Executing scripts/inspect_vllm_uv.py with vLLM backend for meta-llama/Llama-3-8b.

Before

Manual environment checking and script configuration.

After

Automated GPU verification and standardized script execution via uv.

Input

Evaluate a model using lighteval with the accelerate backend.

Output

Running scripts/lighteval_vllm_uv.py with accelerate configuration.

SKILL.md

---
source: "https://github.com/huggingface/skills/tree/main/skills/huggingface-community-evals"
name: hugging-face-community-evals
description: Run local evaluations for Hugging Face Hub models with inspect-ai or lighteval.
risk: unknown
---

# Overview

## When to Use
Use this skill for local model evaluation, backend selection, and GPU smoke tests outside the Hugging Face Jobs workflow.

This skill is for **running evaluations against models on the Hugging Face Hub on local hardware**.

It covers:
- `inspect-ai` with local inference
- `lighteval` with local inference
- choosing between `vllm`, Hugging Face Transformers, and `accelerate`
- smoke tests, task selection, and backend fallback strategy

It does **not** cover:
- Hugging Face Jobs orchestration
- model-card or `model-index` edits
- README table extraction
- Artificial Analysis imports
- `.eval_results` generation or publishing
- PR creation or community-evals automation

If the user wants to **run the same eval remotely on Hugging Face Jobs**, hand off to the `hugging-face-jobs` skill and pass it one of the local scripts in this skill.

If the user wants to **publish results into the community evals workflow**, stop after generating the evaluation run and hand off that publishing step to `~/code/community-evals`.

> All paths below are relative to the directory containing this `SKILL.md`.

# When To Use Which Script

| Use case | Script |
|---|---|
| Local `inspect-ai` eval on a Hub model via inference providers | `scripts/inspect_eval_uv.py` |
| Local GPU eval with `inspect-ai` using `vllm` or Transformers | `scripts/inspect_vllm_uv.py` |
| Local GPU eval with `lighteval` using `vllm` or `accelerate` | `scripts/lighteval_vllm_uv.py` |
| Extra command patterns | `examples/USAGE_EXAMPLES.md` |

# Prerequisites

- Prefer `uv run` for local execution.
- Set `HF_TOKEN` for gated/private models.
- For local GPU runs, verify GPU access before starting:

```bash
uv --version
printenv HF_TOKEN >/dev/null
nvidia-

Frequently Asked Questions

FAQ

Which evaluation tools are compatible with this skill?
This skill is specifically designed to work with inspect-ai and lighteval, supporting backends like vLLM, Transformers, and accelerate.
Who is the target audience for this skill?
It is built for AI Developers and Software Engineers who need to benchmark Hugging Face models on local hardware or private GPU clusters.
How does this differ from Hugging Face Jobs?
Hugging Face Jobs orchestrates remote evaluations; this skill focuses on local execution, smoke tests, and manual backend selection outside that workflow.
What programming languages are supported?
The skill provides Python-based scripts optimized for execution with the uv package manager.
What results can I expect from running these scripts?
You will get local evaluation logs, performance metrics, and model benchmarks directly on your machine, which can then be handed off for publishing.

Discussion

Discussion

0 comments
U

On This Page

Trigger Phrases

Use these phrases to activate this skill in your AI coding assistant:

run local hf evalinspect-ai local benchmarklighteval vllm setupgpu smoke test for hf modelslocal model evaluation script