Amazon SageMaker HyperPod has introduced the Global Inference Gateway, a SageMaker-managed routing tier that places inference requests across every registered HyperPod cluster, whether in one AWS Region or several, according to the AWS Machine Learning Blog. The announcement follows the earlier per-cluster Inference Gateway, which routes requests to pods inside a single cluster using signals such as key-value (KV) cache utilization and queue depth. The Global Inference Gateway is described as the second tier of that architecture.

Why it matters

As models grow, teams spread GPU capacity across more clusters and Regions to obtain enough of it. The blog illustrates the operational cost with a team serving four models across five HyperPod clusters in two Regions. Without the Inference Gateway, clients must track 20 endpoints, one per model per cluster. The per-cluster Inference Gateway reduces that to five, but each client still chooses a cluster itself. That choice brings uneven load, with requests queuing in one Region while GPUs in another sit idle; manual failover, where an engineer reconfigures traffic when a cluster fails; and custom plumbing for load balancers, DNS, and cross-Region networking. The Global Inference Gateway is positioned to remove those burdens by giving clients one endpoint that selects the best-suited cluster and moves traffic off a failed one automatically.

Technical details

Setup involves two decisions. First, an operator publishes a cluster once by enabling the spec.globalRouting block in the cluster's existing InferenceGatewayConfig. The HyperPod Inference controller then publishes the cluster privately from the customer's own account, creating a Network Load Balancer in front of the cluster's Inference Gateway, publishing a VPC endpoint service on it, and uploading the listener's certificate to the customer's TLS bucket. Second, an operator creates the gateway once by calling CreateGlobalInferenceGateway with the clusters to route across; SageMaker returns one endpoint. A cluster is reachable only after it is published, and listing an unpublished cluster in a gateway fails with a message describing what to fix.

In the request path, the client signs with AWS Signature Version 4 (SigV4) and sends to the gateway's regional endpoint. The gateway authenticates the caller and picks the best cluster, AWS PrivateLink carries the request to that cluster in the same or another Region, and the cluster's Inference Gateway routes it to a model pod. Each cluster publishes its load back to the gateway. AWS operates the routing fleet that terminates TLS, authenticates callers, scores clusters, and forwards requests; it also creates one private endpoint per cluster, reads each cluster's VPC endpoint service name, certificate, and Region through a customer-created execution role, health-checks every cluster, and handles authentication and rate limits. The blog states that every request is authenticated with SigV4 and IAM and reaches clusters only over AWS PrivateLink.

Bagelcode, which runs its own LLM inference service on AWS, reported beta results. Jinhyung Park, VP (AI Platform) at Bagelcode, said the gateway routed requests across two AWS Regions through a single endpoint and kept serving them when one cluster could no longer serve the model, giving a small platform team cross-Region failover without building or maintaining its own routing layer.

What to watch

The blog frames the gateway as a way to use already-paid-for GPU capacity during peaks by overflowing requests to clusters with available capacity instead of queuing on a saturated one. Teams planning adoption should note the two-step model: publish each cluster, then create the gateway over the published set. Because the Network Load Balancer, VPC endpoint service, and certificate resources live in the customer's account, operators retain ownership of those components while SageMaker operates the routing fleet and failover.

About the author

AWS Machine Learning Blog

Thoai Dispatch Curated — curated source featured on Thoai Dispatch.

More by AWS Machine Learning Blog