HOWTO: Run Claude Code with local inference

Claude Code is a CLI tool for code assistance. 

Claude Code can run system commands - do not run any commands if you are unsure what they do!

Because Claude can run system commands on your behalf, users are encouraged to take proper security precautions including, but not limited to:

1. Only giving Claude access to files you are willing to lose. Consider backing up anything before allowing Claude access.

2. Enabling manual confirmation for commands run: Default Mode = Ask

3. Disabling additional directories access

4. Restricting risky commands by adding them to the Denylist (permissions.deny) - e.g. WebFetch, Bash(curl:*), Bash(wget:*), etc.

5. Disable all hooks and MCP servers by default, whitelisting only those you trust.

Users are responsible for complying with their department/institution's data and tool policies and are responsible for all commands run on OSC machines.

 

Run Claude Code by taking the following steps:

1. Install Claude Code (only required first time)
    a. curl -fsSL https://claude.ai/install.sh | bash

2. Get a GPU session (multiple options)
    a. sinteractive -g 1 -t 1:00:00
    b. Open Ondemand desktop session with GPU

3. Run vLLM with your target model, setting an alias like "qwen" (Claude can't handle HuggingFace-style slashes in model names)

    a. module load vllm/0.23.0

    b. vllm_start Qwen/Qwen3-8B --served-model-name qwen --enable-auto-tool-choice

    --tool-call-parser hermes > vllm.log 2&>1

You can substitute the model of your choice. See osc.edu/ai and the vLLM software page for more on available and custom models

4. Configure Claude to use Local LLM (we'll use vLLM)
    a. Create a claude_envvars.env file with values something like the following:

export ANTHROPIC_BASE_URL="http://localhost:${VLLM_API_PORT}"
export ANTHROPIC_API_KEY="empty"                         # Placeholder value
export ANTHROPIC_AUTH_TOKEN="empty"                # Placeholder value

# Map Anthropic's internal model tiers to your local Qwen alias
# Use the exact alias defined in your local server (e.g., 'qwen3.5', 'qwen3-coder', or the full HF ID)
export ANTHROPIC_MODEL="qwen"
export ANTHROPIC_DEFAULT_OPUS_MODEL="qwen"
export ANTHROPIC_DEFAULT_SONNET_MODEL="qwen"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="qwen"
export CLAUDE_CODE_SUBAGENT_MODEL="qwen"  

# Optimize for local inference
export CLAUDE_CODE_MAX_OUTPUT_TOKENS="4096"
export DISABLE_PROMPT_CACHING="1"
export DISABLE_AUTOUPDATER="1"
export DISABLE_TELEMETRY="1"
export DISABLE_ERROR_REPORTING="1"
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC="1"

 

5. Set the Claude environment variables you defined above:

    a. source claude_envvars.env

6. Run Claude (in your project's directory)

    a. claude

If configured correctly, you should not be prompted to login or add any API token or key.  If you are, quit and confirm your environment settings are correct and that the local inference server is running properly.