SAML discovery service – Linux – The Next Generation

Federated services

If you want to provide web bases services in a federated (SAML) environment, you’ll need a way for your users to select their home institution in order to log into the service using their local credentials.

That service is called discovery service.

What’s that discovery service doing?

So what happens if a user wants to access such a federated service:

Some kind if dialog to select/search your home institution will be shown
Once selected, the user will be redirected to the Identity Provider (IdP) of his institution
After authenticating (according to the home institutions rules, e.g. including MFA) the user will be directed back to the originating service
The Service Provider (SP) may do extra checks (using the transmitted SAML attributes) to check if the user is allowed to access the resource.
If everything’s fine, the user can now use the service.

So next question: Where do I get such a discovery service?

Possible (ways to) discovery services

From a web developer perspective the simple answer may be: Write your own! As I’m more a system guy (and I don’t like to waste my time fiddling around with some ugly javascript libraries or css classes) I was looking for a ready to use solution.

The search proofed harder than expected: First thing I tried was the “Embedded Discovery Service (EDS)” of the Shibboleth software package. However I didn’t like the looks and the usability of it, so I looked for something else.

After quite some time I found thiss.io and even if it took me some time to figure how this thing is supposed to work in the end it matched my needs (and I think it also looks much better).

So here’s how to set up the discovery service including the metadata server:

Environment

Fortunately both required thiss projects provide a docker configuration for their components, so I used docker-compose to set them up:

services:

  thiss-js:
    image: thiss-js:latest
    container_name: thiss-js
    environment:
      - BASE_URL=https://ds.mydomain.de
      - STORAGE_DOMAIN=mydomain.de
      - MDQ_URL=https://mdq.mydomain.de/entities
      - SEARCH_URL=https://mdq.mydomain.de/entities
      - DEFAULT_CONTEXT=default
      - LOGLEVEL=warn
      - WHITELIST=mydomain.de
      - MIN_SEARCH_LEN=3
    ports:
      - 127.0.0.1:8088:80
    restart: unless-stopped

  # s. https://github.com/TheIdentitySelector/thiss-mdq/blob/master/docs/install.rst
  thiss-mdq:
    image: thiss-mdq:latest
    container_name: thiss-mdq
    volumes:
      - ./metadata.json:/etc/metadata.json
      - ./trustinfo.json:/etc/trustinfo.json
    environment:
      - BASE_URL=https://mdq.mydomain.de
    ports:
      - 127.0.0.1:8089:3000
    restart: unless-stopped

As you may have noticed the exported hosts are only available on localhost. That’s because I use an apache reverse proxy for SSL termination and to make the services available to the outside world.

While it is possible to run both the discovery service and the metadata query service on the same host I decided to go for two separate hosts (cause it makes the reverse proxy config a little easier).

Make sure to build both docker images (s. next chapters) before starting the docker containers using:

linux # docker compose up

The snippets from my apache config look like this (make sure to set up two reverse proxies: one for the discovery service (https://ds.mydomain.de) and one for the mdq service (https://mdq.mydomain.de):

<...>
Define BACKEND_HOST 127.0.0.1
Define BACKEND_PORT 8088
<...>
ProxyPass         /  http://${BACKEND_HOST}:${BACKEND_PORT}/
ProxyPassReverse  / http://${BACKEND_HOST}:${BACKEND_PORT}/
<...>

Discovery service

Building docker image:

linux # git clone https://github.com/TheIdentitySelector/thiss-js.git
linux # cd thiss-js
linux # make docker
linux # docker build --no-cache=true -t thiss-js:latest .

Metadata query service

Building docker image:

linux # git clone https://github.com/TheIdentitySelector/thiss-mdq.git
linux # cd thiss-mdq
linux # docker build --no-cache=true -t thiss-mdq:latest .

Configuration files needed: metadata.json (IdP metadata) and trustinfo.json (SP metadata).

For examples look at test/edugain.json (IdP metadata) or test/edugain-trustinfo.json (SP metadata).

If you want to add your own IDPs/SPs select a (short) one of the entries and adapt that to your setup. One thing that might get you into trouble however is a SHA1 hash sum contained in each metadata entry. According to the specification that checksum is of the entityID string. Here’s an easy way to calculate one:

linux # echo -n "https://keycloak.mydomain/realms/MYDOMAIN" | sha1sum 
f32c43411947eb965ab5173382e359825ec42f05  -

linux # cat metadata.json
[
  <...>
  {
    "title": "My Domain",
    "descr": "Identity Provider of Linux NG (LNG)",
    "title_langs": {
      "de": "My Domain",
      "en": "My Domain"
    },
    "descr_langs": {
      "de": "Identity Provider von My Domain",
      "en": "Identity Provider of My Domain"
    },
    "auth": "saml",
    "entity_id": "https://keycloak.mydomain.de/realms/MYDOMAIN",
    "entityID": "https://keycloak.mydomain.de/realms/MYDOMAIN",
    "type": "idp",
    "hidden": "false",
    "scope": "mydomain.de",
    "entity_icon_url": {
      "url": "https://www.mydomain.de/favicon.ico",
      "width": "16",
      "height": "16"
    },
    "privacy_statement_url": "https://www.mydomain.de/privacy",
    "id": "{sha1}f32c43411947eb965ab5173382e359825ec42f05"
  }
]

By standard an MDQ service implementation supports the following endpoints

linux # curl https://mdq.mydomain.de/entities | jq
[
  <... listing of all metadata here ...>
]
linux # curl 
https://mdq.mydomain.de/entities/%7Bsha1%7Df32c43411947eb965ab5173382e359825ec42f05
< ... returns metadata of sha1 matching entry ...>

(‘%7Bsha1%7D‘ means ‘{sha1}‘ URL encoded)

The thiss implementation however does add some extras (described here), like system status, text based searches or a listing of available entries (or their sha1 counterpart):

linux # curl https://mdq.mydomain.de/status
OK<no newline>
linux # curl https://mdq.mydomain.de/entities/?q=MYDOMAIN | jq 
[
  {
    "title": "My Domain",
    "descr": "Identity Provider of Linux NG (LNG)",
    "title_langs": {
      "de": "My Domain",
      "en": "My Domain"
    },
    "descr_langs": {
      "de": "Identity Provider von My Domain",
      "en": "Identity Provider of My Domain"
    },
    "auth": "saml",
    "entity_id": "https://keycloak.mydomain.de/realms/MYDOMAIN",
    "entityID": "https://keycloak.mydomain.de/realms/MYDOMAIN",
    "type": "idp",
    "hidden": "false",
    "scope": "mydomain.de",
    "entity_icon_url": {
      "url": "https://www.mydomain.de/favicon.ico",
      "width": "16",
      "height": "16"
    },
    "privacy_statement_url": "https://www.mydomain.de/privacy",
    "id": "{sha1}f32c43411947eb965ab5173382e359825ec42f05"
  }
]
linux # curl https://mdq.mydomain.de/.well-known/webfinger | jq
{
  [
    <... long list here ...>
    {
      "rel": "disco-json",
      "href": "https://mdq.mydomain.de/entities/{sha1}f32c43411947eb965ab5173382e359825ec42f05"
    }
  ]
}

How to create/convert json metadata

As you may have noticed creating the metadata by hand is a tedious task. Most federations provide their metadata in XML format only. So there should be a way to automate the conversion … and guess what: Yes there is: pyFF.

It uses some kind of pipeline to allow all kinds of modifications of metadata files like merging or converting to json. A simple sample config for doing that looks like this:

- load:
  - "/data/input.xml"
- select
- stats
- discojson
- publish:
    output: "/data/output.json"
    raw: true
    update_store: false

Calling pyff with this pipeline configuration converts a sample XML input file (/data/input.xml) to JSON format (/data/output.json) and it also gives some information (s. -stats) about the number and type of entries processed:

linux # pyff /data/xml2json.fd
---
total size:     1800
selected:       1800
          idps: 635
           sps: 1163
---

But pyFF cannot only be used as a conversion tool – it comes with its own MDQ server implementation with more options – so stay tuned for another post where we’ll replace thiss-mdq with pyFF!