token认证过程

图解

1.docker client 尝试到registry中进行push/pull操作；
2.registry会返回401未认证信息给client(未认证的前提下)，同时返回的信息中还包含了到哪里去认证的信息；
3.client发送认证请求到认证服务器(authorization service)；
4.认证服务器(authorization service)返回token；
5.client携带这附有token的请求，尝试请求registry；
6.registry接受了认证的token并且使得client继续操作；

详细介绍6个步骤

Step 1，Client 向registry 发起连接

通常，Docker Client在进行pull/push操作时，会先尝试连接docker registry。
Note: 当你访问远程的registry时，会用到tls验证域名的有效性(证书)，否则会出现如下错误:

FATA[0000] Error response from daemon: v1 ping attempt failed with error:
Get https://registry.example.com/v1/_ping: tls: oversized record received with length 20527. 
If this private registry supports only HTTP or HTTPS with an unknown CA certificate,please add 
`--insecure-registry registry.example.com` to the daemon's arguments.
In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag;
simply place the CA certificate at /etc/docker/certs.d/registry.example.com/ca.crt

在docker的启动中加入下面的命令，来忽略对registry域名证书的审核:

--insecure-registry registry.example.com

Step 2，未认证响应(Unauthorized response)

Registry server会返回401并且会附带Authentication endpoint:

$ curl https://registry.example.com/v2 -k -IL
HTTP/1.1 301 Moved Permanently
Server: nginx/1.4.7
Date: Sun, 22 Nov 2015 09:01:42 GMT
Content-Type: text/plain; charset=utf-8
Connection: keep-alive
Docker-Distribution-Api-Version: registry/2.0
Location: /v2/

HTTP/1.1 401 Unauthorized
Server: nginx/1.4.7
Date: Sun, 22 Nov 2015 09:01:42 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 87
Connection: keep-alive
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://registry.example.com:5001/auth",service="Docker registry"
X-Content-Type-Options: nosniff

Authentication server返回的头信息中包含了如何去或许token，它有几个重要的查询参数:
realm: 描述了认证的enpoint。
realm=“https://registry.example.com:5001/auth”

service: 描述了持有资源服务器的名称。
service=“Docker registry”

scope: 描述了client端操作资源的方法(push/pull)
scope=“repository:shuyun/hello:push”

account: 描述了操作的账号
account=admin

Step 3&4，认证endpoint通讯

这2步描述了client与认证服务2者的验证过程，需要明确的是，你需要知道：client发送请求到认证服务器签署token，请求信息中包含的基本身份验证信息将于服务器中的用户列表做匹配，然后根据请求中的scope要操作的范围、方法进而进行匹配，最后服务器匹配成功后将token进行签名，并且将token返回给客户端。

Step 5&6，最后沟通

Client尝试与registry连接(这次带了token信息)，registry接到请求后验证token，继而开始pull/push操作。

方案

开源auth_server认证服务
认证服务后端存储采用mongodb
registry都采用nginx代理
四个组件都通过容器化部署

实践

目录层级说明

CA创建

CA部分是为了实现TLS，实现镜像层文件的安全传输，我们需要创建一份根CA和多份不同域名的CA证书。

mongodb存储

Mongodb作为DB，存储用户认证和访问权限相关信息。为什么用mongodb，而不用已有的mysql，因为我们使用了一个开源的token认证服务，该开源工具可以采用mongodb作为后端存储，但是不支持mysql。
后续，我们还会采用到最新发布的navicat premium 12，新增了mongodb客户端连接工具，使用navicat向mongodb存储一些测试数据。

mongodb容器启动配置

容器化启动compose配置如下：

  mongodb:
    image: mongo:3.2
    expose:
      - "27017"
    network_mode: "host"
    volumes:
      - /data/mongodb:/data/db
    environment:
      - MONGO_INITDB_ROOT_USERNAME=mongoadmin
      - MONGO_INITDB_ROOT_PASSWORD=mongoadmin
    container_name: mongodb
    restart: always

建库并写入数据

1. 新建集合
users
registry_acl
2. 添加用户
{
“username” : “wangyj”,
“password” : “$2y$05$05XkDeA5aSdSBF1uxQ6fPVAO5.D2Z9W/lb1BvGwFGXxLkFaRW57En0G”,
“labels” : {
“group” : [“gisquest”],
“project”: [“gisquest”,“dzzw”]
}
}

简要地对写入的内容做说明：
username和password，在测试阶段，可以用htpasswd命令生成，password必须要求是Bcrypt加密的，使用命令“htpasswd -Bbc admin.htpasswd wangyj wangy123”。
labels这个标签很贴合我们的需求，既可以实现按用户组区分用户的访问权限，又可以按项目名区分用户访问权限；我们的需求是根据项目名确定用户是否有权限访问该项目仓库。
3. 添加ACL

{"seq": 1,"match" :{"name" : "${labels:project}/*"}, "actions" : ["push", "pull"], "comment" :"Users can pull and push to to namespaces matching projects they are assigned to"}

简要说明：
在访问权限控制方面，通过用户所在的项目进行限制，当registry_acl集合中项目组访问权限已存在时，只需在users集合中录入用户信息；当新建项目的时候，需要更新users集合中用户组所有成员信息。
在访问控制粒度上，暂时没有计划控制到分用户和注册仓库，具体控制力度待讨论后决定。

auth_server认证服务

auth_server容器启动配置

容器化启动compose配置如下：

  dockerauth:
    image: cesanta/docker_auth
    expose:
      - "5001"
    network_mode: "host"
    volumes:
      - /data/auth_registry/config/auth_server/auth_config.yml:/config/auth_config.yml:ro
      - /var/log/docker_auth:/logs
      - /data/auth_registry/config/nginx/ssl:/ssl
    container_name: dockerauth
    command: /config/auth_config.yml
    restart: always
    depends_on:
      - mongodb

挂载卷说明：
/data/auth_registry/config/auth_server/auth_config.yml auth_server的配置文件
/var/log/docker_auth 日志
/data/auth_registry/config/nginx/ssl https需要，CA证书

auth_server配置说明

# This config lists all the possible config options.
#
# To configure Docker Registry to talk to this server, put the following in the registry config file:
# 把下面的auth配置写入registry的启动配置文件
#  auth:
#    token:
#      realm: "https://auth.gisquest.com:5001/auth"
#      service: "Docker registry"
#      issuer: "Auth server"
#      rootcertbundle: "/ssl/dockerauth/auth.gisquest.com.crt "

# Server settings.
server:
  # Address to listen on.
  addr: ":5001"

  # URL path prefix to use.
  path_prefix: ""

  # TLS options.
  #
  # Use specific certificate and key.
  certificate: "/ssl/dockerauth/auth.gisquest.com.crt"
  key: "/ssl/dockerauth/auth.gisquest.com.key"
  # Use LetsEncrypt (https://letsencrypt.org/) to automatically obtain and maintain a certificate.
  # Note that this only applies to server TLS certificate, this certificate will not be used for tokens
  letsencrypt:
    # Email is required. It will be used to register with LetsEncrypt.
    email: [email protected]
    # Cache directory, where certificates issued by LE will be stored. Must exist.
    # It is recommended to make it a volume mount so it persists across restarts.
    cache_dir: /data/sslcache
    # Normally LetsEncrypt will obtain a certificate for whichever host the client is connecting to.
    # With this option, you can limit it to a specific host name.
    # host: "docker.example.org"
  # If neither certificate+key or letsencrypt are configured, the listener does not use TLS.

  # Take client's address from the specified HTTP header instead of connection.
  # May be useful if the server is behind a proxy or load balancer.
  # If configured, this header must be present, requests without it will be rejected.
  # real_ip_header: "X-Forwarded-For"
  # Optional position of client ip in X-Forwarded-For, negative starts from end of addresses.
  # real_ip_pos: -2

token:  # Settings for the tokens.
  issuer: "Auth server"  # Must match issuer in the Registry config.
  expiration: 900
  # Token must be signed by a certificate that registry trusts, i.e. by a certificate to which a trust chain
  # can be constructed from one of the certificates in registry's auth.token.rootcertbundle.
  # If not specified, server's TLS certificate and key are used.
  # certificate: "..."
  # key: "..."

#mongo_auth是用户登录控制
mongo_auth:
  # Essentially all options are described here: https://godoc.org/gopkg.in/mgo.v2#DialInfo
  dial_info:
    # The MongoDB hostnames or IPs to connect to.
    addrs: ["127.0.0.1:27017"]
    # The time to wait for a server to respond when first connecting and on
    # follow up operations in the session. If timeout is zero, the call may
    # block forever waiting for a connection to be established.
    # (See https://golang.org/pkg/time/#ParseDuration for a format description.)
    timeout: "10s"
    # Database name that will be used on the MongoDB server.
    # mongodb创建的数据库
    database: "docker_auth"
    # The username with which to connect to the MongoDB server.
    username: "dockerauth"
    # Path to the text file with the password in it.
    # 这个文件还没搞明白怎么回事
    # password_file: "/config/passwd"
    password: "dockerauth"
    # Enable TLS connection to MongoDB (only enable this if your server supports it)
    enable_tls: false
  # Name of the collection in which ACLs will be stored in MongoDB.
  # mongodb里面的集合
  collection: "users"
  # Unlike acl_mongo we don't cache the full user set. We just query mongo for
  # an exact match for each authorization


# (optional) Define to query ACL from a MongoDB server.acl_mongo是用户访问控制
acl_mongo:
  # Essentially all options are described here: https://godoc.org/gopkg.in/mgo.v2#DialInfo
  dial_info:
    # The MongoDB hostnames or IPs to connect to.
    addrs: ["127.0.0.1:27017"]
    # The time to wait for a server to respond when first connecting and on
    # follow up operations in the session. If timeout is zero, the call may
    # block forever waiting for a connection to be established.
    # (See https://golang.org/pkg/time/#ParseDuration for a format description.)
    timeout: "10s"
    # Database name that will be used on the MongoDB server.
    database: "docker_auth"
    # The username with which to connect to the MongoDB server.
    username: "dockerauth"
    # Path to the text file with the password in it.
    # password_file: "/config/passwd"
    password: "dockerauth"
    # Enable TLS connection to MongoDB (only enable this if your server supports it)
    enable_tls: false
  # Name of the collection in which ACLs will be stored in MongoDB.
  collection: "registry_acl"
  # Specify how long an ACL remains valid before they will be fetched again from
  # the MongoDB server.
  # (See https://golang.org/pkg/time/#ParseDuration for a format description.)
  cache_ttl: "1m"

其他事项说明

一开始的时候，auth_server也想用nginx做代理，但是在CA认证这块过不去，提示有以下两种错误：

1. 客户端CA证书未匹配，说是客户端的CA证书不是auth.gisquest.com的证书
2. 客户端的请求时http协议，而服务端需要的https
   最后解决办法就是，不用nginx代理auth_server，直接让用户将获取token的请求发给https://auth.gisuqest.com:5001/auth

registry镜像注册服务

registry容器启动配置

  dev.registry:
    image: registry:2
    expose:
      - "5000"
    network_mode: "host"
    volumes:
      - /data/auth_registry/config/dev.registry/config.yml:/etc/docker/registry/config.yml:rw
      - /data/auth_registry/config/nginx/ssl:/ssl:rw
      - /data/dev.registry:/var/lib/registry:rw
    extra_hosts:
      - "auth.gisquest.com:192.168.20.103"
    container_name: dev.registry
    restart: always

registry配置说明

version: 0.1
log:
  level: info
  fields:
    service: registry
storage:
    cache:
        layerinfo: inmemory
    filesystem:
        rootdirectory: /var/lib/registry
    maintenance:
        uploadpurging:
            enabled: false
    delete:
        enabled: true
http:
    addr: :5000
    secret: placeholder
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
auth:
  token:
    realm: https://auth.gisquest.com:5001/auth
    service: Docker registry
    issuer: Auth server
    rootcertbundle: /ssl/dockerauth/auth.gisquest.com.crt

nginx代理服务

nginx容器启动配置

  nginx:
    image: nginx:1.11.5
    expose:
      - "443"
      - "80"
    network_mode: "host"
    container_name: nginx
    volumes:
      - /data/auth_registry/config/nginx:/etc/nginx:rw
    restart: always
    depends_on:
      - dev.registry

代理镜像注册服务

开发仓库

  upstream dev.registry {
    server 127.0.0.1:5000;
  }

  ## Set a variable to help us decide if we need to add the
  ## 'Docker-Distribution-Api-Version' header.
  ## The registry always sets this header.
  ## In the case of nginx performing auth, the header will be unset
  ## since nginx is auth-ing before proxying.
  map $upstream_http_docker_distribution_api_version $docker_distribution_api_version {
    '' 'registry/2.0';
  }

  server {
    listen 80;
    server_name dev.gisquest.com;

    location /v2/ {

      if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
        return 404;
      }
      proxy_pass                          http://dev.registry;
      proxy_set_header  Host              $http_host;   # required for docker client's sake
      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header  X-Forwarded-Proto $scheme;
      proxy_read_timeout                  900;

   }
 }

  server {
    listen 443 ssl;
    server_name dev.gisquest.com;

    # SSL
    ssl_certificate /etc/nginx/ssl/dev.registry/dev.gisquest.com.crt;
    ssl_certificate_key /etc/nginx/ssl/dev.registry/dev.gisquest.com.key;

    # Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
    ssl_protocols TLSv1.1 TLSv1.2;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;

    # disable any limits to avoid HTTP 413 for large image uploads
    client_max_body_size 0;

    # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
    chunked_transfer_encoding on;


    location /v2/ {

      # Do not allow connections from docker 1.5 and earlier
      # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
      if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
        return 404;
      }

      ## If $docker_distribution_api_version is empty, the header will not be added.
      ## See the map directive above where this variable is defined.
      add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;

      proxy_pass                          http://dev.registry;
      proxy_set_header  Host              $http_host;   # required for docker client's sake
      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header  X-Forwarded-Proto $scheme;
      proxy_read_timeout                  900;

    }

  }

测试

docker login

nginx日志

registry日志

当我login一个已经注册过的用户时，会登录成功；先nginx转发请求到registry，返回401需要先授权，然后根据返回的信息，转到dockerauth获取token认证，最后客户端拿着认证过的token再去请求registry，返回200状态。当login一个未注册的用户时，提示需要先授权。

docker pull/push

docker push一个gisquest/tomcat的镜像，登录的wangyj账号，在dockerauth上已经注册并且在gisquest项目中。

docker pull刚刚上传的gisquest/tomcat镜像

将刚刚的镜像重名为 dev.gisquest.com/bdcycpt/tomcat:8.0.35， 执行docker push，尝试将镜像上传，发现请求被拒绝，镜像无法上传。

registry api

获取token

# curl -X POST -uwangyj:123456 -v  --cacert /etc/docker/certs.d/auth.gisquest.com/CA.crt https://auth.gisquest.com:5001/auth?service=dev.registry\&scope=dev.gisquest.com:gisquest/tomcat:push\&account=wangyj

测试下来，post请求和get请求都可以获取到token，而且看过auth_docker的源码之后，如果不加account，会使用http basic认证里面的账号。
又经过测试，发现之前获取token的请求的scope参数是不对的，争取请求如下：

# curl -X POST -uwangyj:123456 -v  --cacert /etc/docker/certs.d/auth.gisquest.com/CA.crt https://auth.gisquest.com:5001/auth?service=dev.registry\&scope=repository:gisquest/tomcat:pull
# curl -X POST -uwangyj:123456 -v  --cacert /etc/docker/certs.d/auth.gisquest.com/CA.crt https://auth.gisquest.com:5001/auth?service=dev.registry\&scope=registry:catalog:*

利用token向registry发起请求

#curl -v -H 'Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IkM2SUY6RjM3QTpFNjZMOlJQQ086WUk0RzpFVFFBOlhENkw6RzJSMjpJSUZaOlVGV0k6Rjc1RjpaUVNNIn0.eyJpc3MiOiJBdXRoIHNlcnZlciIsInN1YiI6Indhbmd5aiIsImF1ZCI6ImRldi5yZWdpc3RyeSIsImV4cCI6MTU0MjYxODIzMCwibmJmIjoxNTQyNjE3MzIwLCJpYXQiOjE1NDI2MTczMzAsImp0aSI6IjU3MjA2MDIxNDk4MjUzNzY1OTkiLCJhY2Nlc3MiOlt7InR5cGUiOiJkZXYuZ2lzcXVlc3QuY29tIiwibmFtZSI6Imdpc3F1ZXN0L3RvbWNhdCIsImFjdGlvbnMiOlsicHVzaCJdfV19.kwgTlFGveIT0ColxH6bPUnNRHnhuoE2o35tlhdtxbFYR1MtX3ItkSluXooU8QOXsQPJr3YwmbFMIfkCM2Bi1ytaggS_MSjatlrIkS8wRnLPmx7lig0s_8D05GFhmJcbybpPQBmBrLkBDa-muTyfcPw2f-8aYjff6UOoB3ENLTWGDAAXQIZdoUP1kWpnT7GO3I9UbTo69u8ZNZ6qtH4wpZ_qJxPMBYmDz12E9aYVMTDXoOKnOgQowrN9QVZsfbzIj4pBvOkMOBsrGIn6Sv08xwXJ7soDsMbuHw8qXGgKzXss5UXVsltMi231Cgu99cHzlMPtV1Co5ycYuVJerW2SpXA' http://dev.gisquest.com/v2/

参考地址

http://dockone.io/article/845
https://github.com/cesanta/docker_auth/blob/master/docs/Backend_MongoDB.md
https://hub.docker.com/_/mongo/
https://docs.docker-cn.com/registry/spec/auth/token/
https://docs.docker-cn.com/registry/configuration/#token