This document is a deep dive into the reasoning and design for the API. The content of this document started from this API sketch.
Roles and personas
In the original design of Kubernetes, Ingress and Service resources were based on a self-service model of usage; developers who create Services and Ingresses control all aspects of defining and exposing their applications to their users.
We have found that the self-service model does not fully capture some of the more complex deployment and team structures that our users are seeing. The Gateway/Routes API will target the following personas:
- Infrastructure provider: The infrastructure provider (infra) is responsible for the overall environment that the cluster(s) are operating in. Examples include public cloud providers (AWS, Azure, GCP, ...), or PaaS providers within an organization.
- Cluster operator: The cluster operator (ops) is responsible for administration of entire clusters. They manage policies, network access, and application permissions.
- Application developer: The application developer (dev) is responsible for defining their application configuration (e.g. timeouts, request matching/filter) and Service composition (e.g. path routing to backends).
We expect that each persona will map approximately to a
Role in the Kubernetes
Role-Based Authentication (RBAC) system and will define resource model
responsibility and separation.
Depending on the environment, multiple roles can map to the same user. For example, giving the user all the above roles replicates the self-service model.
For more information on the roles and personas considered in the Service API design, refer to the Security Model.
Note: Resources will initially live in the
networking.x-k8s.ioAPI group as Custom Resource Definitions (CRDs). Unqualified resource names will implicitly be in this API group.
There are three main types of objects in our resource model:
GatewayClass defines a set of gateways with a common configuration and behavior.
Gateway requests a point where traffic can be translated to Services within the cluster.
Routes describe how traffic coming via the Gateway maps to the Services.
GatewayClass defines a set of Gateways that share a common configuration and behaviour. Each GatewayClass will be handled by a single controller, although controllers MAY handle more than one.
GatewayClass is a cluster-scoped resource. There MUST be at least one GatewayClass defined in order to be able to have functional Gateways. A controller that implements the Gateway API does so by providing an associated GatewayClass resource that the user can reference from their Gateway(s).
This is similar to IngressClass for Ingress and StorageClass for PersistentVolumes.
In Ingress v1beta1, the closest analog to GatewayClass is the
ingress-class annotation, and in IngressV1, the closest analog is the IngressClass object.
A Gateway describes how traffic can be translated to Services within the cluster. That is, it defines a request for a way to translate traffic from somewhere that does not know about Kubernetes to somewhere that does. For example, traffic sent to a Kubernetes Services by a cloud load balancer, an in-cluster proxy or external hardware load balancer. While many use cases have client traffic originating “outside” the cluster, this is not a requirement.
It defines a request for a specific load balancer config that implements the GatewayClass’ configuration and behaviour contract. The resource MAY be created by an operator directly, or MAY be created by a controller handling a GatewayClass.
As the Gateway spec captures user intent, it may not contain a complete specification for all attributes in the spec. For example, the user may omit fields such as addresses, ports, TLS settings. This allows the controller managing the GatewayClass to provide these settings for the user, resulting in a more portable spec. This behaviour will be made clear using the GatewayClass Status object.
A Gateway MAY contain one or more *Route references which serve to direct traffic for a subset of traffic to a specific service.
Route objects define protocol-specific rules for mapping requests from a Gateway to Kubernetes Services.
TCPRoute are currently the only defined Route objects. Additional protocol-specific Route
objects may be added in the future.
The combination of
define an implementable load-balancer. The diagram below illustrates the
relationships between the different resources:
A typical client/gateway API request flow for a gateway implemented using a reverse proxy is:
- A client makes a request to http://foo.example.com.
- DNS resolves the name to
- The reverse proxy receives the request on
gateway.spec.listeners[x].portand uses the Host header to match an
- Optionally, the reverse proxy can perform request header and/or path matching based
matchrules of the
- Optionally, the reverse proxy can manipulate the request, i.e. add/remove headers,
etc. based on
filterrules of the
- Lastly, the reverse proxy forwards the request to one or more objects in the cluster
actionrules of the
HTTPRoute. When specifying multiple
forwardToobjects, the request is split, i.e. forwarded to each object.
TLS configuration is tied to Gateway listeners. Although adding the option to configure TLS on other resources was considered, ultimately TLS configuration on Gateway listeners was deemed sufficient for the following reasons:
- In most cases, users that are configuring TLS will naturally also have access to Gateways.
- In other cases, TLS config could be implemented with a controller watching Routes and adding generated certs to corresponding Gateways.
- This does not solve the use case for users wanting to provide their own certs for Routes while not having access to a Gateway resource. This seems like it would be a rare edge case and is not worth supporting at this point. The security model outlined a potential approach to enable this in the future, but there does not seem to be a sufficient reason to work towards that now.
There are some general design guidelines used throughout this API.
Single resource consistency
The Kubernetes API guarantees consistency only on a single resource level. There are a couple of consequences for complex resource graphs as opposed to single resources:
- Error checking of properties spanning multiple resource will be asynchronous and eventually consistent. Simple syntax checks will be possible at the single resource level, but cross resource dependencies will need to be handled by the controller.
- Controllers will need to handle broken links between resources and/or mismatched configuration.
Separation and delegation of responsibility among independent actors (e.g between cluster ops and application developers) can result in conflicts in the configuration. For example, two application teams may inadvertently submit configuration for the same HTTP path. There are several different strategies for handling this:
As this API aims to cover a wide set of implementations and use cases, it will not be possible for all implementations to support all features at the present. However, we do expect the set of features supported to converge eventually. For a given feature, users will be guaranteed that features in the API will be portable between providers if the feature is supported.
To model this in the API, we are taking a similar approach as with sig-arch work on conformance profiles. Features as described in the API spec will be divided into three major categories:
- CORE features will be portable and we expect that there is a reasonable roadmap for ALL implementations towards support of APIs in this category.
- EXTENDED features are those that are portable but not universally supported across implementations. Those implementations that support the feature will have the same behavior and semantics. It is expected that some number of EXTENDED features will eventually migrate into the CORE. EXTENDED features will be part of the API types and schema.
- CUSTOM features are those that are not portable and are vendor-specific. CUSTOM features will not have API types and schema except via generic extension points.
Behavior and feature in the CORE and EXTENDED set will be defined and validated via behavior-driven conformance tests. CUSTOM features will not be covered by conformance tests.
By including and standardizing EXTENDED features in the API spec, we expect to be able to converge on portable subsets of the API among implementations without compromising overall API support. Lack of universal support will not be a blocker towards developing portable feature sets. Standardizing on spec will make it easier to eventually graduate to CORE when support is widespread.
We expect there will be varying levels of conformance among the different providers in the early days of this API. Users can use the results of the conformance tests to understand areas where there may be differences in behavior from the spec.
GatewayClass (source code) is cluster-scoped resource
defined by the infrastructure provider. This resource represents a class of
Gateways that can be instantiated.
Note: this serves the same function as the
kind: GatewayClass metadata: name: cluster-gateway spec: controller: "acme.io/gateway-controller"
We expect that one or more
GatewayClasses will be created by the
infrastructure provider for the user. It allows decoupling of which mechanism
(e.g. controller) implements the
Gateways from the user. For instance, an
infrastructure provider may create two
private to reflect
Gateways that define Internet-facing vs private, internal
kind: GatewayClass metadata: name: internet ... --- kind: GatewayClass metadata: name: private ...
The user of the classes will not need to know how
implemented. Instead, the user will only need to understand the resulting
properties of the class that the
Gateway was created with.
Providers of the
Gateway API may need to pass parameters to their controller
as part of the class definition. This is done using the
# GatewayClass for Gateways that define Internet-facing applications. kind: GatewayClass metadata: name: internet spec: controller: "acme.io/gateway-controller" parametersRef: name: internet-gateway --- kind: ConfigMap metadata: name: internet-gateway namespace: acme-system data: ip-address-pool: internet-vips ...
Note: parametersRef will expect a ConfigMap as a referenced object if
group are omitted.
The type of object referenced by
GatewayClass.spec.parametersRef will depend
on the provider itself. A
core.ConfigMap is used in the example above, but
controllers may opt to use a
CustomResource for better schema validation.
GatewayClasses MUST be validated by the provider to ensure that the configured
parameters are valid. The validity of the class will be signaled to the user via
kind: GatewayClass ... status: conditions: - type: InvalidParameters status: Unknown ...
GatewayClass will start with the
InvalidParameters condition set to
Unknown. At this point the controller has not seen the configuration. Once the
controller has processed the configuration, the condition will be set to
kind: GatewayClass ... status: conditions: - type: InvalidParameters status: False ...
If there is an error in the
GatewayClass.spec, the conditions will be
non-empty and contain information about the error.
kind: GatewayClass ... status: conditions: - type: InvalidParameters status: True Reason: BadFooBar Message: "foobar" is an FooBar.
Gateway is 1:1 with the life cycle of the configuration of
infrastructure. When a user creates a
Gateway, some load balancing
infrastructure is provisioned or configured
(see below for details) by the
Gateway is the
resource that triggers actions in this API. Other resources in this API are
configuration snippets until a Gateway has been created to link the resources
Gateway spec defines the following:
GatewayClassused to instantiate this Gateway.
- The Listener bindings, which define addresses and ports, protocol termination,
and TLS settings. The Listener configuration requested by a Gateway definition can
be incompatible with a given
GatewayClass(e.g. port/protocol combination is not supported).
- The Routes, which describe how traffic is processed and forwarded.
If the Listener configuration requested by a Gateway definition is incompatible with a given GatewayClass, the Gateway will be in an error state, signaled by the status field.
Depending on the
GatewayClass, the creation of the
Gateway could do any of
the following actions:
- Use cloud APIs to create an LB instance.
- Spawn a new instance of a software LB (in this or another cluster).
- Add a configuration stanza to an already instantiated LB to handle the new routes.
- Program the SDN to implement the configuration.
- Something else we haven’t thought of yet...
The API does not specify which one of these actions will be taken. Note that a GatewayClass controller that manages in-cluster proxy processes MAY restrict Gateway configuration scope, e.g. only be served in the same namespace.
Gateways track status for the
Gateway resource as a whole as well as each
Listener it contains. The status for a specific Route is reported in the
status of the
Route resource. Within
GatewayStatus, Listeners will have
status entries corresponding to their name. Both
ListenerStatus follow the conditions pattern used elsewhere in Kubernetes.
This is a list that includes a type of condition, the status of that condition,
and the last time this condition changed.