Gateway API for Layer 4 (NLB) Implementation¶

This section details the AWS Load Balancer Controller's (LBC) architecture and operational flow when processing Gateway API resources for Layer 4 traffic utilizing AWS NLB.

Gateway API Resources and Controller Architecture¶

The LBC implements Gateway API support through a bifurcated architecture, employing distinct controller instances for Layer 4 (L4) and Layer 7 (L7) routing. This design allows for specialized and optimized reconciliation aligned with the underlying AWS Load Balancer capabilities.

The LBC instances dedicated to L4 routing monitor the following Gateway API resources:

GatewayClass: For L4 routing, the LBC specifically manages GatewayClass resources with the controllerName set to gateway.k8s.aws/nlb.
Gateway: For every gateway which references a GatewayClass with the controllerName set to gateway.k8s.aws/nlb, The LBC provisions an AWS NLB.
TLSRoute: Defines TLS-specific routing rules, enabling secure Layer 4 communication. These routes are satisfied by an AWS NLB.
TCPRoute: Defines TCP-specific routing rules, facilitating direct TCP traffic management. These routes are satisfied by an AWS NLB.
UDPRoute: Defines UDP-specific routing rules, facilitating UDP traffic management. These routes are satisfied by an AWS NLB.
ReferenceGrant: Defines cross-namespace access. For more information see
LoadBalancerConfiguration (LBC CRD): A Custom Resource Definition utilized for fine-grained customization of the provisioned NLB. This CRD can be attached to a Gateway or its GatewayClass. For more info, please refer How customization works
TargetGroupConfiguration (LBC CRD): A Custom Resource Definition used for service-specific customizations of AWS Target Groups. This CRD is associated with a Kubernetes Service. For more info, please refer How customization works

The Reconciliation Loop¶

The LBC operates on a continuous reconciliation loop within your cluster to maintain the desired state of AWS Load Balancer resources:

Event Watching: The L4-specific controller instance constantly monitors the Kubernetes API for changes to the resources mentioned above to NLB provisioning.
Queueing: Upon detecting any modification, creation, or deletion of these resources, the respective object is added to an internal processing queue.
Processing:
- The controller retrieves the resource from the queue.
- It validates the resource's configuration and determines if it falls under its management (e.g., by checking the GatewayClass's controllerName). If it does, it enqueues a relevant gateway for processing
- The Kubernetes Gateway API definition is then translated into an equivalent desired state within AWS (e.g., specific NLB, Listeners, Target Groups etc).
- This desired state is meticulously compared against the actual state of AWS resources.
- Necessary AWS API calls are executed to reconcile any identified discrepancies, ensuring the cloud infrastructure matches the Kubernetes declaration.
Status Updates: Following reconciliation, the LBC updates the status field of the Gateway API resources in Kubernetes. This provides real-time feedback on the provisioned AWS resources, such as the NLB's DNS name and ARN or if the gateways are accepted or not.

Step-by-Step L4 Gateway API Resource Implementation with an Example¶

This section illustrates the step-by-step process of configuration of L4 Gateway API resources, demonstrating how the LBC provisions and configures an NLB.

Consider a scenario where an application requires direct TCP traffic routing:

# nlb-gatewayclass.yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: aws-nlb-gateway-class
spec:
  controllerName: gateway.k8s.aws/nlb
---
# my-nlb-gateway.yaml
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: my-tcp-gateway
  namespace: example-ns
spec:
  gatewayClassName: aws-nlb-gateway-class
  listeners:
  - name: tcp-app
    protocol: TCP
    port: 8080
    allowedRoutes:
      namespaces:
        from: Same
---
# my-tcproute.yaml
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: my-tcp-app-route
  namespace: example-ns
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: my-tcp-gateway
    sectionName: tcp-app # Refers to the specific listener on the Gateway
  rules:
  - backendRefs:
    - name: my-tcp-service # Kubernetes Service
      port: 9000

API Event Detection: The LBC's L4 controller continuously monitors the Kubernetes API server. Upon detecting the aws-nlb-gateway-class (with controllerName: gateway.k8s.aws/nlb), the my-tcp-gateway (referencing this GatewayClass), and my-tcp-app-route (referencing my-tcp-gateway's tcp-app listener) resources, it recognizes its responsibility to manage these objects and initiates the provisioning of AWS resources.
NLB Provisioning: An AWS Network Load Balancer (NLB) is provisioned in AWS for the my-tcp-gateway resource with default settings. At this stage, the NLB is active but does not yet have any configured listeners. As soon as the NLB becomes active, the status of the gateway is updated.
L4 Listener Materialization: The controller processes the my-tcp-app-route resource. Given that the TCPRoute validly references the my-tcp-gateway and its tcp-app listener, an NLB Listener is materialized on the provisioned NLB. This listener will be configured for TCP protocol on port 8080, as specified in the Gateway's listener definition. A default forward action is subsequently configured on the NLB Listener, directing all incoming traffic on port 8080 to the newly created Target Group for service my-tcp-service in backendRefs section of my-tcp-app-route.
Target Group Creation: An AWS Target Group is created for the Kubernetes Service my-tcp-service with default configuration. The cluster nodes are then registered as targets within this new Target Group.

Combined Protocols¶

AWS NLB supports combining TCP and UDP on the same listener; the protocol is called TCP_UDP. This powerful paradigm allows the load balancer to serve different protocols for different applications on the same listener port. The LBC implements this protocol merging capability.

Combined protocol quirks¶

AWS NLB assumes that in a combined protocol set up, all targets are able to serve both protocols. To prevent configuration duplication, we follow this same pattern for constructing the combined protocol listener. TCP_UDP listeners are able to attach routes of type TCP and UDP, each route attached generates a TCP_UDP target group.

Combined protocol examples¶

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-tcp-udp-gateway
  namespace: tcp-udp
spec:
  gatewayClassName: aws-nlb-gateway-class
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: tcp-app
    port: 80
    protocol: TCP
  - allowedRoutes:
      namespaces:
        from: Same
    name: udp-app
    port: 80
    protocol: UDP
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
  name: my-udp-app-route
  namespace: tcp-udp
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: my-tcp-udp-gateway
    sectionName: udp-app
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: udpechoserver
      port: 8080
      weight: 1

To customize the target group created, it's no different from a single protocol

apiVersion: gateway.k8s.aws/v1beta1
kind: TargetGroupConfiguration
metadata:
  name: example-tg-config
  namespace: tcp-udp
spec:
  defaultConfiguration:
    targetType: ip
  targetReference:
    group: ""
    kind: Service
    name: udpechoserver

To customize the listener:

apiVersion: gateway.k8s.aws/v1beta1
kind: LoadBalancerConfiguration
metadata:
  name: nlb-lb-config
  namespace: tcp-udp
spec:
  listenerConfigurations:
  - protocolPort: TCP_UDP:80
    listenerAttributes:
    - key: tcp.idle_timeout.seconds
      value: "60"