AI clusters have gotten a shared infrastructure. Neoclouds, enterprise AI platform groups, monetary providers organizations, life sciences groups, and analysis teams have to share GPU capability. This shared infrastructure can undergo from decrease monetization, elevated operational complexity, and restricted management and visibility throughout tenants, workloads, hosts, and the community cloth.
EVPN/VXLAN is the sensible community basis. It supplies tenant-scoped overlay segmentation utilizing VRFs, VNIs, route distinguishers, and route targets. Nonetheless, tenant-aware segmentation is just not job-aware segmentation. The scheduler understands jobs; the community sometimes understands routes, interfaces, queues, drops, and flows.
Why AI clusters want multitenancy
Devoted GPU clusters are easy to isolate, however they’re inefficient to function at scale. As GPU estates develop, organizations need a shared useful resource pool that may serve a number of groups, clients, and workload lessons with out forcing each group into its personal bodily cluster. In any other case, one group can have stranded GPUs in a devoted island whereas one other waits in queue.
The requirement seems in a number of patterns:
- A GPU-as-a-Service supplier maps every tenant to an exterior buyer with its personal tackle and coverage area (per-customer isolation whereas maintaining the GPU pool shareable).
- An enterprise platform crew maps tenants to growth, testing, manufacturing fine-tuning, mannequin analysis, or regulated analytics (constant surroundings boundaries with out constructing separate clusters).
- A monetary service division separates fraud analytics, danger modeling, and analysis workloads on one coaching cluster (stronger management boundaries and auditability with out duplicating GPU islands).
- A analysis group assigns shared GPU capability to impartial analysis teams (clearer quota, utilization, and troubleshooting accountability throughout competing tasks).
This is the reason multitenancy can’t cease at compute allocation. Distributed coaching will depend on east-west GPU communication, sometimes over Ethernet materials, so the community turns into an integral a part of the isolation and efficiency boundary.
How business solves it at this time
Present AI multitenancy is often carried out throughout three layers:
- Orchestration and scheduler layer. Kubernetes-based platforms, GPU cloud orchestration programs, and Slurm schedulers outline the logical possession mannequin for the cluster. They monitor tenants or tasks, customers, queues or namespaces, job requests, node placement, and GPU allocation. For instance, Tenant A would possibly submit Job 100 requesting eight GPUs throughout two servers, whereas Tenant B submits Job 200 requesting 4 GPUs on a distinct set of nodes. As an example, in an orchestration platform like Rafay, the platform can personal tenant onboarding and infrastructure intent, whereas the precise job scheduling might occur in Kubernetes, Slurm, or a tenant-operated scheduler.
- Host isolation layer. The host enforces the native machine boundary for every workload. If a tenant receives entire servers, isolation is easier as a result of the server, GPU set, and NIC set will be handled as one tenant-owned unit. If a number of tenants or jobs share GPUs inside the similar server, the runtime should expose solely the assigned GPU units and bind the workload’s communication libraries, akin to NCCL or UCX, to the meant NICs. This host-side mapping issues as a result of a GPU server might have a number of NICs related to completely different switches or tenant-facing community segments. Material segmentation can isolate visitors as soon as it enters the community, but it surely can’t appropriate an incorrect native task the place the workload is allowed to make use of the improper GPU or NIC.
- Community segmentation layer. EVPN/VXLAN supplies scalable tenant segmentation throughout the material. VXLAN encapsulates tenant visitors and makes use of VNIs to determine the overlay phase or routing area. EVPN makes use of BGP to promote endpoint and prefix reachability and to manage which VTEPs import a tenant’s routes by route targets. In a routed AI cloth, a tenant generally maps to a VRF and a number of VNIs, with route distinguishers maintaining tenant routes distinctive and route targets controlling import-export coverage. This permits overlapping tenant tackle house and scoped reachability throughout a shared underlay.
ACLs or safety group ACLs can add supply and vacation spot coverage, particularly in brownfield L3 designs or the place the material can’t but eat richer workload identification. Their limitation is operational scale: static or manually up to date ACL and VRF insurance policies don’t naturally comply with fast-changing AI job placement.
Collectively, these layers present a workable tenant-level mannequin. The remaining hole is job context: the community can often see tenant context, interfaces, routes, queues, and counters, however not the particular scheduler job operating inside a tenant. Tenant segmentation itself doesn’t mechanically isolate Job 100 from Job 101 inside the identical tenant until job identification can also be carried, derived, or programmed into community coverage.
Cisco Nexus One integration with AI iorchestration platforms
Cisco Nexus One is effectively positioned because the broader basis for making tenant-aware AI materials extra deterministic. On this structure, Nexus One is the entire cloth automation, integration, and visibility floor for all the cloth.
Nexus One can present cloth topology context to an AI infrastructure orchestration platform akin to Rafay by integration workflows or APIs. That lets groups map tenant VRFs, VLANs, and port assignments on to a tenant, slightly than managing them solely as an summary tenant label.
As well as, Nexus One extends the mannequin past provisioning. Tenant-level visibility can present the tenant’s cloth path and related well being alerts akin to congestion, drops, and so forth. This enhances AI job observability: job-aware views can correlate scheduler, topology, optics, GPU telemetry, analytics, and anomalies, whereas tenant-specific Job-ID enforcement stays a separate future-facing coverage functionality.
Tenant-aware is just not job-aware
Tenant segmentation solutions the query, “Which buyer or group owns this visitors?” AI operations typically want, “Which coaching job is creating or experiencing this visitors inside a tenant?”
This distinction issues for segmentation in addition to throughout troubleshooting. A scheduler can determine the job, allotted nodes, GPUs, and runtime state. The community can determine interfaces, routes, queues, drops, ECN marks, PFC occasions, optics well being, and paths. With out correlation, operators should manually join these two views.
The result’s a standard operational drawback: the material reveals a scorching uplink or lossy interface, whereas the platform crew sees a sluggish coaching job. The lacking hyperlink is the workload identification within the community working mannequin.
Future route: AI Job-ID-aware segmentation
Job-ID-aware segmentation route—patent-pending know-how from Cisco—is the logical subsequent step. (Be aware that this describes our architectural route, not a delivery function.) The aim is for infrastructure orchestrator (akin to Rafay) and scheduler (akin to Slurm) intent to hold each tenant identification and job identification into the material management and data-plane mannequin.
In that mannequin, the material controller interprets job intent into coverage. The change information airplane carries or derives a job ID, for instance by VXLAN GPO bits, and enforces that solely endpoints in the identical licensed tenant and job can trade RoCEv2 visitors.
The anticipated advantages are operationally essential:
- Easier operations, as a result of groups can cause in tenants and jobs as an alternative of translating each grow to be static community objects—essential for NetOps and cloth operations groups.
- Deeper visibility, as a result of drops, congestion, paths, and optics will be correlated to workload context slightly than solely to interfaces or tenant VRFs—useful for platform engineering and SRE groups.
- Extra granular segmentation, as a result of coverage can comply with the lifecycle of a job slightly than stopping on the tenant boundary—essential for safety, compliance, and tenant governance groups.
This strategy is constructed on open requirements—not a proprietary overlay. EVPN/VXLAN is IETF-defined, and the Group Coverage Choice (GPO) follows the identical path: an IETF-defined mechanism that encodes a bunch/coverage identifier within the VXLAN header alongside the VNI, which Cisco NX-OS implements in alignment with the open specification. Tenant scope (VNI) and workload/job scope (GPO) are subsequently expressed in constructs a standards-compliant cloth can interpret—letting operators evolve from tenant-aware to job-aware enforcement with no cloth forklift.
Technical instance: tenant and job boundaries
Take into account a GPU-as-a-Service surroundings with two clients, Tenant A and Tenant B. Every tenant is mapped to its personal VRF/VNI within the EVPN/VXLAN cloth. Tenant-level segmentation prevents Tenant B from reaching Tenant A.
Now assume Tenant A runs two concurrent coaching jobs. Job 100 makes use of GPUs on servers 1 and a couple of. Job 101 makes use of completely different GPUs on the identical shared cloth. Tenant-level EVPN/VXLAN nonetheless treats each jobs as Tenant A visitors. Job-ID-aware segmentation would add one other enforcement dimension: Job 100 endpoints may trade RoCEv2 visitors with different Job 100 endpoints, however not with Job 101 endpoints, even inside the identical tenant.
That’s the architectural shift: EVPN/VXLAN stays the tenant basis, whereas Job ID turns into the long run workload-level coverage and observability attribute.
Advancing safety from tenant-level to job-level segmentation
AI information heart multitenancy begins with EVPN/VXLAN tenant segmentation, but it surely doesn’t finish there. The stronger working mannequin combines topology-aware provisioning, tenant-level enforcement, and end-to-end visibility at this time, then evolves towards Job-ID-aware segmentation as scheduler and orchestrator integration matures.
In case you are designing a shared AI cluster at this time, tenant-aware EVPN/VXLAN is the muse. Job-aware enforcement and observability are the subsequent frontier.
*Particular because of Ramesh Ponnapalli and his crew, whose engineering management has been instrumental in bringing this know-how to life.
Extra sources: