HOWTO: Run Claude Code with local inference

Claude Code is a CLI tool for code assistance. 

Because Claude can run system commands on your behalf, users are encouraged to take proper security precautions including, but not limited to:

• Only giving Claude access to files you are willing to lose. Consider backing up anything before allowing Claude access.
• Enabling manual confirmation for commands run: Default Mode = Ask
• Disabling additional directories access
• Restricting risky commands by adding them to the deny list.
• Disable all hooks and MCP servers by default, whitelisting only those you trust.

1. Create your Claude settings file.  Here is an example ~/.claude/settings.json that is a starting point, but you may have to tailor it to your needs.  Users are encouraged to familiarize themselves with the Claude settings including permissions in the official documentation.

{ "$schema": "https://json-schema.org/claude-code-settings.json", 
  "permissions": { "disableBypassPermissionsMode": "disable", 
                   "allowManagedPermissionRulesOnly": true, 
                   "allowManagedHooksOnly": true, 
                   "allowedManagedMcpServersOnly": true, 
                   "deny": [ "Bash(rm -rf *)", 
                             "Bash(rm -rf /)", 
                             "Bash(sudo *)", 
                             "Bash(curl *)", 
                             "Bash(wget *)", 
                             "Read(**/.env*)", 
                             "Read(**/.ssh/**)", 
                             "Read(**/.aws/**)", 
                             "Read(**/secrets/**)", 
                             "Read(**/credentials/**)", 
                             "Read(**/*.pem)", 
                             "Read(**/*.key)", 
                             "Write(**/.env*)", 
                             "Write(**/.ssh/**)" 
                            ], 
                   "ask": [ "Bash(git push *)", 
                            "Bash(docker *)", 
                            "Bash(kubectl *)", 
                            "Bash(pnpm install *)", 
                            "Bash(npm install *)", 
                            "Bash(yarn install *)" 
                          ], 
                   "allow": [ "Read", 
                              "Write", 
                              "Bash(ls *)", 
                              "Bash(cat *)", 
                              "Bash(grep *)", 
                              "Bash(git status *)", 
                              "Bash(git diff *)", 
                              "Bash(git add *)", 
                              "Bash(git commit *)" 
                            ] 
                }, 
  "sandbox": { "enabled": true, 
               "allowUnsandboxedCommands": false, 
               "network": { "allowLocalBinding": false, 
                            "allowedDomains": [] 
                          } 
             }, 
  "cleanupPeriodDays": 7, 
  "disableAllHooks": true 
}

Run Claude Code by taking the following steps:

1. Install Claude Code (only required first time)
    a. curl -fsSL https://claude.ai/install.sh | bash

    b. echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

    c. which claude

The which command above should return ~/.local/bin/claude if Claude has been installed and is available in your PATH

2. Get a GPU session (multiple alternatives)
    a. sinteractive -g 1 -c 16 -t 1:00:00 -A <PROJECT_CODE>
    b. Open Ondemand desktop session with GPU

The above are alternative options for GPU sessions - you can pick the one that is most comfortable to you.

3. Run vLLM with your target model, setting an alias like "qwen" (Claude can't handle HuggingFace-style slashes in model names)

    a. module load vllm/0.23.0

    b. vllm_start Qwen/Qwen3-8B --served-model-name qwen --enable-auto-tool-choice --tool-call-parser hermes > vllm.log 2>&1

    c. (optional) test vLLM - curl http://localhost:${VLLM_API_PORT}/v1/models | jq 

You will need to wait a few minutes until the vllm_start prompt returns your prompt.  Once you get your prompt back, you can test with the curl command above.

If successful, the curl test above will show a json with your model name and its alias.

If unsuccessful, you will see  curl: (7) Failed to connect to localhost port 80: Connection refused.  In this case, check that your vLLM setup was correct.

4. Configure Claude to use Local LLM (we'll use vLLM)
    a. Create a claude_envvars.env file with values something like the following:

export ANTHROPIC_BASE_URL="http://localhost:${VLLM_API_PORT}"
export ANTHROPIC_API_KEY="empty"                   # Placeholder value
export ANTHROPIC_AUTH_TOKEN="empty"                # Placeholder value

# Map Anthropic's internal model tiers to your local Qwen alias
# Use the exact alias defined in your local server (e.g., 'qwen3.5', 'qwen3-coder', or the full HF ID)
export ANTHROPIC_MODEL="qwen"
export ANTHROPIC_DEFAULT_OPUS_MODEL="qwen"
export ANTHROPIC_DEFAULT_SONNET_MODEL="qwen"
export ANTHROPIC_DEFAULT_HAIKU_MODEL="qwen"
export CLAUDE_CODE_SUBAGENT_MODEL="qwen"  

# Optimize for local inference
export CLAUDE_CODE_MAX_OUTPUT_TOKENS="4096"
export DISABLE_PROMPT_CACHING="1"
export DISABLE_AUTOUPDATER="1"
export DISABLE_TELEMETRY="1"
export DISABLE_ERROR_REPORTING="1"
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC="1"

5. Set the Claude environment variables you defined above:

    a. source claude_envvars.env

6. Run Claude (in your project's directory)

   a. cd <your_project_dir>    

   b. claude

Running Claude outside your project directory will give Claude access to files in the current directory.