NVIDIA CUDA or AMD ROCm for local AI in 2026?

For most teams deploying local AI in 2026, NVIDIA CUDA remains the less risky choice. AMD ROCm should not be selected out of affection for an open stack or because an accelerator has an attractive price. Choose it when the exact models, library versions, and workload modes have already been tested on the exact AMD GPU. Without that test, engineering hours quickly consume the price difference.

This does not mean ROCm is only suitable for experiments. PyTorch has worked with HIP for years, vLLM publishes ROCm builds, and llama.cpp supports HIP alongside CUDA. The gap is no longer about whether a basic function exists. It is about the breadth of tested combinations, how quickly new kernels arrive, and how many unpleasant exceptions remain. Start with the workload and memory, then calculate energy and maintenance, and only then compare server prices.

The workload determines the choice

CUDA is the sensible starting point if the team trains models, changes architectures often, uses recent quantization methods, or wants to run third-party repositories without a separate porting project. It has more ready-made binary packages, examples, and containers. When a repository says only pip install, its author usually tested CUDA, even if pure PyTorch code can run through HIP.

ROCm makes sense when the workload is stable and fits the official compatibility matrix. A good candidate looks like this: a pinned Linux release, a supported AMD GPU, one or two models, a known weight format, a measurable request profile, and a team prepared to maintain its own container image. In that environment, a better memory capacity or server configuration can outweigh the software stack's inconvenience.

Do not merge three separate decisions. The first concerns development and training in PyTorch. The second concerns high-volume serving through vLLM. The third concerns compact local execution of quantized GGUF models through llama.cpp. One GPU vendor can win in the first scenario and lose in the third, so "we need an AI platform" is much too vague for procurement.

I use a short decision path:

1. For a research team with changing code, I choose CUDA unless supply or memory imposes a hard constraint.
2. For a stable vLLM service, I consider ROCm only after testing the required model, quantization, and parallelism.
3. For a self-contained assistant on llama.cpp, I compare specific GPUs by memory, speed, and power because the platform gap is smaller here.
4. For a mixed fleet, I calculate the cost of two sets of images, monitoring, and skills. That cost often exceeds the benefit of procurement flexibility.

This order removes the brand argument. The team gets a list of checks, and procurement gets requirements that can be included in acceptance.

Do not carry cloud test results over to a local server without verification. A cloud image may contain a selected driver, kernel patches, and communication settings that are absent from your organization's standard distribution. The same model's performance also changes with the power limit, PCIe topology, and CPU speed. A cloud instance can show how software behaves on the chosen GPU, but it does not answer the question about your complete system.

PyTorch hides differences only at the Python layer

Ordinary PyTorch code often needs no edits when moving from CUDA to ROCm. PyTorch documentation explicitly says that a HIP build retains the torch.cuda interfaces: the device is still specified as cuda, and torch.cuda.is_available() works for both stacks. That is a sound engineering choice, but people stretch it into an unjustified claim of full interchangeability.

The trouble starts below the Python API. A custom CUDA extension, a new Triton operator, a fused kernel from a model repository, or an uncommon torch.compile mode may have a separate HIP implementation, lag behind by a version, or build only from source. The word cuda in the code does not prove that it is tied to NVIDIA, while a shared API does not prove equal support for every kernel. The team must understand this distinction before buying hardware.

Backend detection should be part of diagnostics, not a guess based on the device name:

import torch

if not torch.cuda.is_available():
    raise RuntimeError("GPU backend is unavailable")

backend = "rocm" if torch.version.hip else "cuda"
print({
    "backend": backend,
    "torch": torch.__version__,
    "cuda_runtime": torch.version.cuda,
    "hip_runtime": torch.version.hip,
    "device": torch.cuda.get_device_name(0),
})

On CUDA, the result contains a version in cuda_runtime and None in hip_runtime. On ROCm, the result is reversed. Store this dictionary beside every test result. Without it, "everything worked on the test machine" is useless after a driver update.

For training, test more than the first iteration. Check that the loss curve matches, mixed precision remains stable, checkpoints save and load, distributed execution works, and the job resumes after failure. If the project depends on custom extensions, build them in a clean container without a cache. An error that appears only during a repeat build on a new node still counts as a platform error.

CUDA wins where researchers constantly bring in new code. ROCm can deliver a proper result on a known set of operators, but the team must treat the GPU, OS, and ROCm version matrix as a contract. AMD publishes separate matrices for compute accelerators and Radeon products. A GPU outside the list may run the code, but an unsupported experiment cannot become a production SLA.

vLLM requires the entire version chain to be pinned

Both CUDA and ROCm are available for vLLM in 2026, but installation convenience and the breadth of combinations differ. Current vLLM documentation specifies Linux as the primary OS, CUDA GPUs with suitable compute capability, and a limited list of AMD architectures. For ROCm, the project publishes prepared wheels only for specific ROCm and Python versions, while a mismatched PyTorch build may force the team to build vLLM from source.

This is more than a minor installer annoyance. vLLM compiles many specialized kernels, so binary compatibility depends on the PyTorch version, driver, runtime, and build parameters. CUDA can also produce a conflict when someone installs the package over an arbitrary environment. The difference is that the ready-made route is usually wider for common CUDA configurations, while ROCm makes ownership of the full build chain necessary sooner.

I do not accept a vLLM server after one API response. The acceptance run must cover:

• the exact model and weight revision;
• the selected data type or quantization format;
• the minimum, normal, and maximum context length;
• the required number of concurrent requests;
• one GPU and the actual parallelism scheme if the server uses several.

Check the specific quantization method in the vLLM matrix. AMD support in the project's heading does not mean AWQ, GPTQ, FP8, and every new scheme work equally on every architecture. A purchase based on "they will add it in the next release" is particularly dangerous. Acceptance can count only a version that the team can install and reproduce now.

Pin the container by an immutable identifier, not the latest tag. Store the host driver, firmware version, launch arguments, and a small set of control prompts beside it. After an update, compare not only tokens per second but also time to first token, tail latency, peak memory, and the share of failed requests.

The practical conclusion is direct: if vLLM is the foundation of an internal API and the team wants to adopt new models quickly, CUDA needs fewer qualifications. ROCm can be selected for a fixed service when a test proves the required performance and the image already builds without manual patches. Buying an AMD GPU and then discovering which Triton version builds on it today transfers an architecture decision to the engineer on call.

llama.cpp narrows the gap considerably

llama.cpp is better suited to comparing the GPUs themselves because the project controls more of the execution path and supports several backends. CUDA builds through GGML_CUDA, while ROCm builds through GGML_HIP. Quantized GGUF files can move between machines, although optimal parameters and speed will differ.

The minimum check is equally transparent:

# NVIDIA
cmake -S . -B build-cuda -G Ninja -DGGML_CUDA=ON
ninja -C build-cuda

# AMD
cmake -S . -B build-hip -G Ninja -DGGML_HIP=ON
ninja -C build-hip

After building, run the same GGUF file with identical -c, -b, CPU thread counts, and -ngl values. Record the built-in benchmark output separately for prompt processing and generation. One combined figure hides an important difference: a GPU may process a long prompt quickly but deliver modest sequential generation speed, or the reverse.

Do not treat multiple GPU support as a free doubling of speed. The llama.cpp feature matrix explains that backends can use several devices, while the CUDA code translated for ROCm through HIP can split rows between GPUs. This mode helps when the connection is fast enough, not with every pair of cards. Over ordinary PCIe lanes, transfer latency and uneven utilization can spoil the appealing arithmetic of total memory.

llama.cpp also offers Vulkan, but it should be evaluated as a separate backend. It is useful for portability and some desktop configurations, but the presence of Vulkan does not turn an unsupported ROCm GPU into the equivalent of a supported compute accelerator. For a maintainable server, select a path the team can rebuild and diagnose.

AMD can be entirely reasonable for a small number of users, a fixed GGUF model, and one GPU. Memory capacity is often more important here than access to the newest fused kernel. If the same server must move to vLLM tomorrow, train adapters, and accept experimental models, the llama.cpp advantage cannot be generalized to the whole stack.

Calculate memory before choosing a GPU model

For local AI, video memory sets the limit earlier than peak teraflops. If the model does not fit alongside the KV cache and working buffers, the team will offload layers to system memory, shorten the context, or split the workload between GPUs. Each escape changes latency, energy use, and complexity.

Start with model weights. A rough lower estimate is the parameter count multiplied by the bytes per parameter: about two bytes for FP16 or BF16, one for 8 bit, and half a byte for 4 bit. The actual file and allocation use more because of metadata, quantization scales, alignment, and temporary buffers. A 70-billion-parameter model at 4 bit therefore has about 35 GB of raw weights alone, and the practical budget must be higher.

The second memory consumer, the KV cache, grows with context and the number of concurrent sequences. For ordinary multi-head attention, the order of magnitude can be checked with this formula:

KV bytes ≈ 2 × layers × hidden_size × bytes_per_value × tokens × sequences

The factor of two accounts for keys and values. In models with grouped-query attention, size depends on the number of KV heads and is smaller, so use the model's own configuration rather than a universal calculator from someone else's note. vLLM also reserves execution memory and manages cache blocks, which means that matching the arithmetic sum to the GPU's rated capacity still does not guarantee a successful start.

Leave headroom for a peak, not only the state immediately after loading. For a service, measure memory at the longest permitted request and maximum concurrency. Training adds gradients, optimizer states, and activations to the weights, so its need can be many times the weight file size. A quantized inference configuration that fits 24 GB says nothing about whether the same model can be fine-tuned.

Compare CUDA and ROCm memory at the level of available server configurations. One GPU with sufficient capacity is almost always simpler than two smaller GPUs: there is less communication, fewer failure points, and more predictable latency. If an AMD configuration provides the required capacity on one supported accelerator, that is a strong argument. If saving money requires an unsupported desktop card and CPU offload to hide its limits, the saving exists only in the procurement spreadsheet.

Energy use is measured by work, not TDP

A GPU's rated power is not the consumption of the whole server. The bill includes the CPU, memory, storage, fans, power supplies, and conversion losses. TDP or a power limit helps estimate cooling and an upper bound, but it does not say how much energy one million tokens or an overnight fine-tuning job will consume.

Measure power at the server input and connect it to completed work. For each test, record average power in watts, duration, the number of processed input tokens, and the number of generated output tokens. Energy is simple to calculate:

energy_kWh = average_power_W × duration_hours / 1000

Take two readings: one under sustained load and one at idle with the model loaded. An internal assistant may wait for requests most of the day, so low power at 100 percent utilization does not compensate for a poor idle state. Batch processing, by contrast, more often benefits from maximum throughput and finishing the job quickly.

The nvidia-smi and amd-smi utilities provide device telemetry, but server comparison needs an external meter or data from a managed power distribution unit. A GPU software sensor cannot see the rest of the system. Reconcile both sources: the first helps find throttling and uneven utilization, while the second shows what the organization pays for.

A power cap often improves the result per watt, but it must be tested at the required latency. Lowering a limit that saves 15 percent power and extends a response by 40 percent increases energy per request. I do not use these percentages as a forecast for another GPU. They demonstrate why the product of power and time has to be calculated.

In Kazakhstan, the calculation should also include actual rack density, available cooling, the tariff, and the site's operating mode. A server that fits the electricity budget may fail the heat or backup power constraint. These limits should appear in the technical specification before the accelerator model number.

Desktop GPUs and server accelerators solve different jobs

An inexpensive card with plenty of memory looks tempting, but server operation imposes requirements that do not exist on a home test bench. Check physical dimensions, power delivery, directed airflow, permitted temperature, remote diagnostics, and whether the selected chassis officially supports the operating mode. A card that cools properly in an open case may throttle between neighboring accelerators in a rack.

For ROCm, the line between "it runs" and "it is officially supported" matters in particular. AMD's matrix lists combinations of GPUs, operating systems, and ROCm versions for compute workloads, while Radeon has separate conditions. An environment variable or an unofficial patch can sometimes make software recognize a different architecture, but that technique does not add vendor testing, spare parts, or a predictable update path. It is an acceptable laboratory experiment. For a service with a mandatory recovery time, it creates technical debt on day one.

NVIDIA gaming cards should not be treated as smaller server cards either. Direct communication between GPUs, virtualization, memory error correction, telemetry, and cooling behavior depend on the exact product and platform. CUDA guarantees a software backend, not identical operating properties across the full catalog.

Windows can work for workstations and llama.cpp when that exact route passes testing. vLLM's main documentation targets Linux and does not present native Windows as the primary production path. WSL and third-party builds are useful to a developer, but they add a layer that someone must update and diagnose. I do not include them in a server design without a named owner and a separate acceptance test.

Finally, check the availability of a like-for-like replacement. If a failed GPU cannot be replaced with the same architecture a year later, a new accelerator may require another driver and repeated image qualification. The price of a spare node and its delivery time belong in the platform calculation just as much as generation speed.

Maintenance hours can cancel the saving

ROCm's main hidden cost is not a license but engineering time spent on rare incompatibilities. CUDA's main hidden cost appears when an organization mistakes familiarity for permission to update everything without testing. Both platforms demand discipline, but ROCm usually has a narrower corridor of supported combinations.

A typical failure develops predictably. The team tests PyTorch on a workstation and sees an available GPU. It then installs vLLM in an existing environment where the PyTorch version does not match the kernels' build version. The package either fails during import or requires a build. During that build, the team discovers that the ROCm release does not match the image, while updating ROCm requires another supported OS release. One mismatched package turns into a host change.

A similar CUDA chain starts with an old driver or an arbitrary mixture of libraries. NVIDIA documentation describes backward compatibility of newer drivers with older CUDA applications and limited compatibility within a major toolkit branch, but that is not permission to mix any versions. PTX JIT, new driver features, and dynamic libraries create exceptions. A container does not replace the host's kernel driver.

Calculate monthly labor in hours for four kinds of work:

• building and testing base images;
• qualifying new models and quantization formats;
• updating drivers, the OS, and firmware;
• diagnosing failures and training the on-call shift.

Add recovery time. If the image is reproducible, the previous release is stored, and the control run is automated, rollback takes a knowable amount of time. If an engineer manually built a package inside a running container, a spare GPU from the same vendor will not save the service.

CUDA is usually cheaper to maintain for small teams, even when the hardware costs more. For a team that already operates Linux clusters on AMD, knows how to build kernels, and maintains a narrow model catalog, ROCm does not create the same overhead. Skills change the economics, so another organization's total cost calculation cannot simply be inserted into your purchase.

Put maintenance cost in the same table as hardware and electricity. Multiply planned update hours by the internal cost of an engineer, add the expected validation time for each new model, and reserve time for an unplanned failure. Precision down to the tenge is unnecessary. A range with honest assumptions is more useful than a zero that appears because salaried labor was mistakenly treated as free.

A pilot must end with an acceptance record

A pilot is not for an impressive chat demonstration. It must establish a reproducible configuration. Before ordering hardware, choose one real case of each relevant type: an interactive request with long context, a batch task, and training or fine-tuning if required. Run them on the CUDA and ROCm candidates with the same data and quality criteria.

The record must contain the GPU model, memory capacity, OS, driver, runtime, PyTorch, vLLM or llama.cpp commit, weight format, and every launch argument. Attach raw results: time to first token, sustained speed, peak memory, average wall power, and errors. An average without the worst observed result hides exactly the failure a user will see.

Repeat deployment on a clean node with an engineer who did not prepare the test bench. This simple exercise exposes undocumented packages, manual edits, and files from the author's home directory. If a second engineer cannot obtain the same result from the instructions, you have a demonstration but not yet a platform.

Set pass conditions in advance. For example: the model loads after a clean deployment; control responses contain no corrupted tokens; the service handles required concurrency without exhausting memory; a restart restores operation; the previous image rolls back without reinstalling the host. The numbers depend on your service, but each statement must permit an unambiguous "yes" or "no."

Do not change two variables at once. If the CUDA candidate runs FP8 while the ROCm candidate runs a 4-bit GGUF, you are comparing different serving models. First obtain the same format and engine. Then examine the optimal configuration for each platform separately and record honestly which differences change response quality.

For a server project, GSE.kz can build and integrate a locally manufactured platform with suitable accelerators and support across Kazakhstan. The acceptance certificate should still refer to your image and your test, not the logo of the server or GPU manufacturer.

CUDA remains the default, while ROCm needs a reason

Choose NVIDIA CUDA when the priorities are maximum compatibility with the PyTorch ecosystem, fast access to new vLLM capabilities, frequent model changes, or the smallest operations burden for a small team. It is a conservative choice in the useful sense: fewer unknowns stand between a developer's repository and the production server.

Choose AMD ROCm when a specific supported GPU offers a meaningful memory or configuration advantage, the workload is fixed, the Linux stack appears in the official matrix, and a pilot confirms the required functions. ROCm is particularly appropriate when the organization already owns the build chain and wants to reduce dependence on one supplier. Independence requires a second set of skills, or it remains a line in a presentation.

For llama.cpp and one quantized model, the decision can lean toward AMD more often. For vLLM with new quantization methods and for research PyTorch, CUDA is still safer. For training across several GPUs, test communication and recovery separately because the sum of memory says nothing about transfer speed.

If pilot results are close, select the platform that the on-call team can recover at night by following a written procedure. A few percent on a favorable test does not pay for a day of downtime after an update. If ROCm wins after the full cost of energy and engineering hours, choose ROCm without apology. If it wins only on GPU purchase price, the calculation is not finished.