Real load determines how many GPUs you need

You do not need 200 GPUs for 200 employees. In most office scenarios, a sensible starting point is two server GPUs with 48 GB each, provided that a 14B-class model verified on your tasks runs in a 4-bit or 8-bit format. One such card is enough for a pilot, while two provide separate replicas for load and maintenance. If acceptable quality requires a 32B-class model in BF16, the calculation quickly grows to two 80 GB cards for one replica and four for a fault-tolerant pair.

This is not a universal purchase specification. The employee count says almost nothing about load without four figures: how many requests arrive at the same time, how many tokens the model reads and writes, how much memory the exact build occupies, and how long users may wait. I have seen projects where an expensive accelerator sat idle because document retrieval was weak, and projects where the model fit in memory but the queue turned an ordinary question into a minute-long wait. Neither rated memory nor parameter count alone tells you how many cards to buy.

Two hundred accounts do not create two hundred concurrent requests

You need to count busy sequences and token flow during the peak interval, not issued accounts. An employee may open the assistant twice a day, or may hold a long conversation while preparing a contract. Both modes have the same user count and completely different server load.

Start with the request log from an existing pilot. For every request, record arrival time, input tokens, output tokens, time to first token, total generation time, and completion code. You do not need to retain request text for this calculation. If there is no pilot yet, run a working session with 15-20 people representing the relevant roles and record their scenarios: internal document search, email drafting, spreadsheet analysis, meeting-minutes summarization, and support responses. Synthetic ten-word phrases make an attractive test and a poor purchase decision.

Little's law gives a useful first approximation:

среднее число занятых запросов = запросов в секунду × среднее время обслуживания

Suppose 30 of 200 employees are active during the busiest five-minute interval. Each sends one request every 90 seconds on average, and the server generates an answer for 12 seconds. Average occupancy is then 30 / 90 × 12 = 4 sequences. An average does not cover bursts, so I would plan to test eight concurrent sequences and separately reproduce the moment when several people paste long documents.

There is another awkward detail: the interface may create extra calls. Generating a conversation title, classifying the request, rewriting it for retrieval, and producing the main answer can become four calls to one model. The user sees one click, while the server sees four jobs. Before sizing, draw the chain for one action and count every call, including retries after errors.

Fix the request profile before choosing hardware

The context-window size in a model description is not the normal request length. A model may support 32 thousand tokens, but that does not mean every conversation should reserve or send that amount. For an office assistant, it is more useful to define four working profiles and the share of each at peak.

The first profile is a short question with retrieval: about 1,500 input tokens after adding the system instruction and retrieved passages, with up to 250 output tokens. The second is document summarization: 6,000-10,000 input tokens and 500 output tokens. The third is a continuing long conversation whose history gradually grows. The fourth is a rare heavy request, such as comparing several policies. These figures are a measurement template, not a standard for your organization.

For each profile, record:

• median and 95th-percentile input tokens;
• median and 95th-percentile output tokens;
• requests per minute in the busiest window;
• acceptable time to first token;
• minimum output speed after the answer starts.

Do not mix up time to first token and total response time. A long answer can type for 20 seconds and still feel fine if its first words appeared after two seconds. A short answer that stays silent for ten seconds and then appears instantly feels worse. I normally set separate objectives, such as p95 time to first token of no more than three seconds and a p95 inter-token latency of no more than 80 milliseconds. This is an example service objective, not a promise about any particular accelerator.

Trimming conversation history often saves more than moving to the next card. The assistant does not always need the entire dialogue. Summarizing older turns, limiting retrieved passages, and preventing uncontrolled file attachment reduce input size, KV cache, and prefill time. But you cannot simply cut everything to 2,000 tokens: legal and technical answers may lose their basis. Choose the limit through quality testing on real tasks.

Weights, KV cache, and headroom all consume memory

A model that barely loads into accelerator memory is not ready to serve users. GPU memory is shared by weights, the KV cache for active sequences, temporary tensors, execution graphs, and the inference server itself. Headroom is not superstition: a peak operation or a longer batch can terminate the process with an out-of-memory error.

A rough weight estimate is simple:

память весов ≈ число параметров × бит на вес / 8

For a 14B model, the theoretical weight minimum is about 28 GB in BF16, 14 GB at 8 bits, and 7 GB at 4 bits. The actual file and occupied memory will be larger because of quantization coefficients, metadata, selected layers kept at higher precision, and runtime buffers. For 32B, the figures are about 64, 32, and 16 GB before overhead. The claim that a 32B INT4 model fits in 24 GB may therefore be true for one format and false for another.

The KV cache stores attention keys and values for tokens already processed in every active sequence. For a conventional transformer model, estimate it as follows:

KV на токен = 2 × слои × KV-головы × размер головы × байт на элемент
общий KV = KV на токен × активные токены всех последовательностей

The factor of 2 represents a key and a value. Take the parameters from config.json for the exact model, not a similar member of the family. For example, the Qwen2.5-14B-Instruct configuration specifies 48 layers, 8 KV heads, and a hidden size of 5120 across 40 attention heads. One head is therefore 128 units wide. With BF16, the cache per token is 2 × 48 × 8 × 128 × 2 = 196,608 bytes, or about 0.1875 MiB. Eight sequences with 4,000 occupied tokens each require about 5.9 GiB of KV cache. Qwen2.5-32B-Instruct has 64 layers with the other stated values unchanged, giving about 0.25 MiB per token, or about 7.8 GiB for the same batch.

This check exposes a distinction that often disappears in discussions about quantization. Quantized weights do not guarantee a correspondingly compact KV cache. The server may keep its cache in FP16, BF16, or supported FP8 regardless of the weight format. Check kv_cache_dtype and support on the chosen accelerator before calculating. FP8 can shrink the cache, but you should test its effect on quality and compatibility instead of assuming it.

The vLLM documentation explicitly defines gpu_memory_utilization as the share of memory used for weights, activations, and KV cache. When request preemption occurs, it recommends reducing the number of sequences or batched tokens, or adding memory through greater tensor parallelism. That is more useful than a marketing compatibility table: the server itself reports how many cache blocks it created and whether preemptions started. A clean startup only proves that the process launched.

Model size changes both quality and topology

Choose among 7B, 14B, and 32B by measuring quality on work examples, not by trying to fill purchased memory. For factual retrieval from a carefully prepared knowledge base, a small model can answer better than a large model fed poor passages. For difficult editing, multilingual documents, and long-instruction adherence, the difference between classes can be noticeable. Measure it through blind answer comparisons using criteria written by process owners.

In practical terms, the classes look like this:

The table does not replace testing. NVIDIA's NIM support matrix lists 80 GB for H100 and A100, 48 GB for L40S, and 24 GB for A10G, and labels untested combinations as estimates rather than guarantees. I agree with that qualification. Equal memory capacity does not make cards equal in memory bandwidth, low-precision compute, cooling, or continuous-duty operation.

Splitting one model across two cards is called tensor parallelism. It helps fit the weights and expands available cache, but it does not turn two cards into two independent service copies. Failure of either card stops the entire replica. Two separate replicas of a smaller model provide aggregate performance and allow one node to be serviced, while two cards under one large model provide only one service point. Put this distinction in the architecture, or the word two will mislead the procurement team.

Quantization is not a free win either. A 4-bit model saves memory and can sometimes generate faster, but a given method may reduce accuracy with numbers, formatting, or instruction following. Test the same weight file, chat template, and runtime that will go into production. A test of the original BF16 model proves nothing about a random 4-bit build.

Acceptable response time becomes token throughput

After checking memory, you must prove that the configuration can process peak flow. Generation has two useful load categories: input processing, commonly called prefill, and sequential production of new tokens, or decode. A long document puts heavy load on prefill, while many answers being typed at once load decode.

Estimate average output demand as follows:

выходных токенов в секунду =
запросов в секунду × среднее число выходных токенов

If 20 peak requests per minute return 250 tokens each, average demand is about 83 output tokens per second. You should not buy a configuration that produced 85 tokens per second in a lab. Request arrival is uneven, long inputs compete for compute, and utility calls add work. For an initial test, I multiply the observed peak by at least 1.5 and run a separate sudden burst. In this example, the test objective becomes at least 125 tokens per second while meeting latency limits, not just a maximum token count with an unrestricted queue.

Total throughput and single-answer speed are in tension. A server with continuous batching adds new requests to ongoing work and raises aggregate throughput. An oversized batch makes an individual user wait longer. TensorRT-LLM and vLLM use paged KV caches and in-flight request batching for denser serving, but you still have to tune the maximum batch against your latency objective.

Do not copy a tokens/s result from an accelerator card into the calculation without its conditions. You need to know the model, precision, input length, output length, concurrent sequence count, server version, and card-sharding method. A result from one short request shows interactive speed but not service capacity. A result from hundreds of batched requests shows a throughput ceiling but may hide unacceptable time to first token.

Voice and autocomplete have stricter requirements than document chat. A pause in voice dialogue is immediately obvious. For a summary, an employee can accept several seconds before output starts if the answer then arrives steadily. One cluster does not have to serve both modes identically: it can be cheaper to give a small fast model a separate replica for routing and short responses.

A calculation for 200 employees gives a range, not one number

Consider an organization with 200 employees where the assistant searches internal documents and prepares drafts. Thirty people are active in the busiest five minutes. The system receives 20 visible requests per minute, and each visible request creates 1.2 model calls on average because some retrieval queries are rewritten. That totals 24 calls per minute, or 0.4 calls per second.

Suppose the measured profile has 2,500 input tokens per call on average, 6,000 at p95, and 250 output tokens. Average active generation time is 12 seconds. Little's law gives 0.4 × 12 = 4.8 concurrently busy sequences. The test environment must handle eight for a burst, and an overload test should use twelve. With eight sequences and an average occupied context of 3,000 tokens, the 14B model from the previous example requires about 4.4 GiB of BF16 KV cache.

Assume quality evaluation found that the 14B model in a verified 4-bit format passes 92 of 100 control tasks, with the remaining eight routed to a person. That is an example acceptance threshold, not a market statistic. Its weights might occupy roughly 8-10 GB once the specific format's metadata is included. A 48 GB card then has enough room for cache, temporary buffers, and moderate context growth. Memory is not the main risk in this case. Compute throughput is.

Required average output is 0.4 × 250 = 100 tokens per second. With a factor of 1.5, the test target becomes 150 tokens per second while p95 time to first token stays within the adopted objective. If one card achieves that on the mixed profile, deploy two independent cards as two replicas. A load balancer divides requests, and either replica can carry a degraded service while the other is maintained. The result is two cards, but not because there are 200 employees.

If one card achieves only 95 tokens per second at the required latency, two replicas may provide enough total capacity, but test uneven distribution of long requests. If two still fail, examine routing, context length, and batching first. Buy a third card after measuring the bottleneck, not as a cure for every red graph.

Now replace the model with 32B BF16. Weights alone need about 64 GB before overhead. One 80 GB card may load the build, but the remaining space for a 6-8 GiB cache, buffers, and spikes will be tight. Two 80 GB cards in tensor parallelism provide a working replica with headroom, and a production pair of such replicas requires four cards. That is why the model-quality decision must come before the server specification.

A sensible starting point depends on the cost of an error

For a general internal assistant, I would start with two 48 GB cards and a 14B model in a verified quantized format. This is not the smallest functioning setup, but it lets you compare one and two replicas, survive maintenance, and gather honest peak data. If the pilot budget is limited, one 48 GB card is acceptable, but do not describe it as a production-ready architecture.

For a reference assistant with short answers, two 24 GB cards, one per replica, may be enough if a 7B-8B model passes quality evaluation and each card handles half the peak. For difficult document analysis where 32B has a confirmed advantage, consider two different starts: two 48 GB cards with a verified 4-bit model as independent replicas, or four 80 GB cards for two BF16 replicas using two cards each. Compare the cost of the complete service, not the price of one card.

A popular recommendation is to buy one maximum-power accelerator with spare capacity. It makes a purchasing table simple and creates an operational problem. You will still have to stop that server to update a driver, model, or firmware. When all spare capacity sits in one replica, scheduled maintenance means downtime. Two smaller independent replicas are often more useful than one large unit, as long as the selected model fits and meets the quality objective.

The opposite extreme is to assign one accelerator to every active user. Modern inference servers batch sequences, so one card serves several conversations. A dedicated card per user may be justified for isolation or an unusually heavy workload, but not for ordinary text chat. Queues, limits, and separate pools usually isolate departments more sensibly than a physical card for every account.

When the cost of a wrong answer is high, compute headroom does not replace controls. An assistant used in medicine, finance, or human resources should display sources, restrict actions, and refer uncertain cases to a specialist. A larger model can make a more convincing mistake. Include the load from embedding models, rerankers, document recognition, and control components in the hardware calculation if they share the same accelerators.

The test environment must reproduce a queue, not one attractive prompt

Run the test environment with the same driver version, inference server, model file, and chat template planned for production. Replacing any one of these can change the result. A workstation run helps with familiarization but does not validate a server with different cooling, power limits, and card interconnects.

A minimum test plan has five runs:

1. One short request measures the best interactive speed and finds template errors.
2. Constant calculated load for 30 minutes shows sustained throughput and heat behavior.
3. A mixed set of short and long inputs reproduces the office profile.
4. A twofold burst tests the queue, timeouts, and recovery after the peak.
5. Disabling one replica proves that the remaining system actually accepts requests.

For every run, collect p50, p95, and p99 for time to first token, inter-token latency, and total time. Also record queue length, error share, KV-cache preemptions, occupied GPU memory, power, and throttling. Average time hides occasional stalls that employees remember better than fast responses.

The load generator must reproduce a length distribution instead of sending one repeated text. Prepare an anonymized set of tokenized input sizes and expected answer lengths. You may replace the text with safe examples of the same size when confidentiality prevents requests from leaving their environment. Save the random seed and run configuration so the supplier and your team can repeat the same test.

Write the acceptance criterion before the run. For example: at 24 calls per minute with the mixed profile, p95 time to first token does not exceed three seconds, p95 output speed is at least 12.5 tokens per second, errors stay below an agreed threshold, and the service remains available with a documented degradation after one replica is disabled. Without a criterion, anyone can call any graph good.

Do not optimize only for maximum throughput. If a server sustains 250 tokens per second but one long request blocks short ones for 15 seconds, employees will click again and increase the load. A batched-token limit, priority for short requests, or a separate document queue can provide better service even with a lower maximum.

A production configuration includes more than a GPU count

Accelerators cannot fix slow model storage, insufficient system memory, a weak network, or power without redundancy. The server has to load weights, survive a restart, and remove heat under continuous load. Check RAM capacity, local drive speed, PCIe lanes, card and chassis compatibility, power-supply capacity, and the rack's actual thermal conditions.

Two replicas need a load balancer with a readiness check, not merely a check that the process is running. A replica may answer on a network port while its model is not loaded. During an update, bring up the new replica, warm it with a control request, add it to the pool, and only then remove the old one. This sequence requires temporary spare capacity.

Separate user limits from compute limits. A per-user limit protects the service from a faulty script that sends hundreds of requests. The server scheduler limits active sequences and tokens per batch. The queue needs a finite size and a clear overload response. An infinite queue does not preserve work. It moves the failure one minute later.

Tie observability to the user's request. One identifier should pass through retrieval, reranking, generation, and filters so an engineer can see where time was spent. Content can be hidden or hashed according to policy, but sizes, durations, and statuses are necessary for planning. After a month, real distributions will replace initial assumptions and make the card calculation more accurate.

For infrastructure purchased in Kazakhstan under requirements for local manufacturing, support, and supply transparency, GSE.kz can assemble the server and integration components for the measured profile, including AI and data-center infrastructure. A reproducible test must still confirm the exact accelerator model and card count: a system integrator cannot repeal queueing physics.

The purchase decision fits in one calculation sheet

The final sheet should link business load to every specification line. Include employee count, peak active users, model calls per action, input and output token percentiles, selected model and precision, weight size, KV per token, target concurrency, required throughput, single-card results, and the fault-tolerance design.

Use this calculation order:

1. вызовы/с = активные пользователи × действий/с × вызовов на действие
2. занятые последовательности = вызовы/с × длительность генерации
3. требуемый decode = вызовы/с × выходные токены
4. память = веса + KV активных токенов + измеренные буферы + запас
5. карты на реплику = максимум требования по памяти и теста производительности
6. всего карт = карты на реплику × число независимых реплик

Line 5 does not come from theoretical peak performance. Fill it with the test result. Line 6 depends on acceptable downtime: the multiplier may be one for a pilot, while a working service usually needs at least two independent replicas. If one model spans two cards, a pair of replicas makes the total four.

For the profile in this example, the answer is simple: one 48 GB card proves that the 14B model works, and two 48 GB cards form a sensible production starting point if each replica passes the mixed test. For 32B BF16, a sensible start may be four 80 GB cards because each replica occupies two. Between those points are options using 4-bit 32B, cards from other classes, and routing between two models.

Do not approve the specification until you have three artifacts: a quality evaluation set, a workload-profile log, and a repeatable test report. Without them, the GPU count remains an opinion. With them, one table explains the purchase, and you can recalculate it from facts after a month of operation.