Basic litellm proxy setup

What is litellm?

litellm is a proxy server you can add between your LLM app and the LLM service.

Why use litellm?

litellm can be used to configure access, load balancing, accounting and many other things you might be interested in once you’re getting into the LLM business.

Installation

Install litellm proxy as described here:

linux # python3 -m venv /opt/litellm-proxy
linux # source /opt/litellm-proxy/bin/activate
linux # pip install "litellm[proxy]"

Basic config file

linux # cat litellm_config.yaml
model_list:
  - model_name: gemma3:27b
    litellm_params:
      model: ollama/gemma3:27b
      api_base: "http://127.0.0.1:11435"

general_settings: 
  master_key: sk-supermasterkey
  database_url: postgresql://litellm:mypassword@127.0.0.1:5432/litellm

Install and create postgres database

Install postgres, create litellm user and database:

linux # apt install postgresql
linux # sudo -u postgres createuser --pwprompt litellm
Enter password for new role: 
Enter it again:
linux # sudo -u postgres createdb --encoding=UTF8 --locale=C --template=template0 --owner=litellm litellm

First run

linux # litellm --config litellm_config.yaml --detailed_debug

Problems on first run

The problems I encountered were related to the prisma python library:

The easy case was the module prisma missing. This will result in “ModuleNotFoundError: No module named 'prisma'” and can easily be resolved by installing the module:

linux # pip install prisma

Upon next start you might hit this bug, that’ll result in something like: “Exception: Unable to find Prisma binaries. Please run 'prisma generate' first.“. So let’s do what we’re told to:

linux # prisma generate --schema /opt/litellm/lib/python3.12/site-packages/litellm/proxy/schema.prisma

If that step succeeds, try restarting litellm and it should work now.

If you’re working with image input, you might also want to install “Pillow“, otherwise litellm cannot convert the imges:

linux # pip install Pillow

User interface

You should now be able to access the UI using port 4000. Login is “admin” and your master_key specified in the config file.

Creating API keys

Once a database is configured and working, creating API keys is as easy as:

linux # MASTER_KEY=sk-supermasterkey
linux # curl -L -X POST 'http://127.0.0.1:4000/key/generate' \
-H "Authorization: Bearer ${MASTER_KEY}" \
-H 'Content-Type: application/json' -d '{
  "user_id": "marcel"
}' | jq
{
  "key_alias": null,
  "duration": null,
  "models": [],
  "spend": 0.0,
  "max_budget": null,
  "user_id": "marcel",
  "team_id": null,
  "max_parallel_requests": null,
  "metadata": {},
  "tpm_limit": null,
  "rpm_limit": null,
  "budget_duration": null,
  "allowed_cache_controls": [],
  "config": {},
  "permissions": {},
  "model_max_budget": {},
  "model_rpm_limit": null,
  "model_tpm_limit": null,
  "guardrails": null,
  "prompts": null,
  "blocked": null,
  "aliases": {},
  "object_permission": null,
  "key": "sk-j-DhUashI7dasS3Cr3t",
  "budget_id": null,
  "tags": null,
  "enforced_params": null,
  "allowed_routes": [],
  "allowed_passthrough_routes": null,
  "allowed_vector_store_indexes": null,
  "rpm_limit_type": null,
  "tpm_limit_type": null,
  "key_name": "sk-...Cr3t",
  "expires": null,
  "token_id": "1234f18ef33d469bfec0f3c01c11379b9c0be6f85fa50b86fcd86e50dfaecdef",
  "organization_id": null,
  "litellm_budget_table": null,
  "token": "1234f18ef33d469bfec0f3c01c11379b9c0be6f85fa50b86fcd86e50dfaecdef",
  "created_by": "default_user_id",
  "updated_by": "default_user_id",
  "created_at": "2025-12-21T16:48:06.839000Z",
  "updated_at": "2025-12-21T16:48:06.839000Z"
}

In case you’re interested in load-balancing using litellm (and ollama) take a look here.