The cost of storing event logs for more than a year

A year of log retention does not mean 365 daily volumes on expensive disks. The price depends on how much data arrives, what it becomes after indexing, how many copies the platform creates, and how much of the history must actually answer a query in seconds.

I have seen estimates that counted only the raw feed, leaving the cluster to run out of disk a few months later. I have also seen the opposite: an organization bought fast storage for events it accessed once a year. A sound calculation separates availability, preservation, and search speed. Each part of the history then gets a suitable storage medium, and the budget gets a formula that anyone can verify.

Measure the daily feed before looking at the current folder

The calculation starts with the event volume that collectors actually accept over a full day. The size of a directory on one server is almost always misleading: rotation may have already deleted some files, agents may have filtered records, and the index may have added or removed bytes. Measure at one point in the pipeline and use one set of units.

Take at least two ordinary weeks, plus a month-end close, infrastructure update, or other high-volume operation. Record accepted bytes and event count for each source. The average determines baseline capacity, while the highest sustained day shows whether the system can absorb a surge without losing data. A one-off emergency spike is better handled by an ingestion buffer than by multiplying the entire annual estimate by that spike.

For a file gateway, this reproducible command avoids guessing:

du -cb /var/log/export/2026-06-15/* | tail -1

The output looks like 318472901224 total: these are the bytes in the files for the selected day. If the data is in object storage, total the objects under the date prefix. If the platform bills for ingested gigabytes, export the daily value from its metrics and compare it with at least one independent measurement at the collector.

Do not mix GB and GiB. Disk manufacturers use decimal units: 1 TB is 1,000,000,000,000 bytes. An operating system may display binary TiB, where 1 TiB is 1,099,511,627,776 bytes. A 100 TB disk pool provides about 90.95 TiB before formatting. A difference of almost ten percent already consumes a substantial part of the reserve.

Finally, split the sources. Firewall feeds, detailed database audits, application logs, and system events grow for different reasons. One combined number is enough to buy disks, but it cannot guide retention by data type.

A usable formula counts copies, growth, and free space

One formula is enough for a first estimate when every coefficient comes from your own system:

Требуемая емкость = D × R × K × C × G / U

Here, D is raw volume per day, R is the number of days, K is the ratio of on-disk size to raw volume after parsing, indexing, and compression, C is the number of complete data copies, G is the growth allowance, and U is the permitted disk utilization. A 20% growth allowance is G = 1.20, while an 80% utilization ceiling is U = 0.80.

Teams often count C incorrectly. One primary and one replica mean two copies, so C = 2, not one. RAID does not replace a search-cluster replica, and neither replaces a backup. RAID helps the node survive a media failure, a replica keeps service running after a node loss, and a backup protects against logical deletion, corruption, or an administrator's mistake. They address different failures and belong on different budget lines.

Free space is not a luxury. The Elasticsearch documentation defines standard disk thresholds: at 85% the cluster restricts new shard allocation, at 90% it starts moving shards, and at 95% it blocks writes to affected indexes. Another platform will have different mechanics, but it still needs space for merges, temporary files, recovery, and rebalancing. Planning normal operation at 95% utilization means accepting emergency behavior by design.

Add the items that do not scale directly with data volume: system indexes, metadata, the platform's own logs, spare disks, room to restore a snapshot, and temporary coexistence of old and new data during a migration. These items matter more in a small cluster than in a large one. Treat the formula as a hypothesis to check with a pilot load, not as a purchase order.

Measure compression on your own events

The statement that logs compress fivefold is not fit for a budget. Repetitive text lines may compress well, but UUIDs, hashes, encrypted fields, traces, and precompressed payloads behave differently. An index adds dictionaries, search structures, and field values. An indexed data set can be smaller than the raw JSON, or it can be larger.

Take a representative three-to-seven-day sample, load it with the production field template, and wait for background merges to finish. Divide the actual size of the primary shards by the raw input size. Repeat the test for each large event class. The weighted average becomes K, while the range describes the risk.

For Elasticsearch, check the size of primary shards with this request:

GET _cat/indices/logs-*?h=index,pri,rep,docs.count,pri.store.size,store.size&s=index

If the raw input for the same period is 1.00 TB, pri.store.size is 0.62 TB, and store.size with one replica is 1.24 TB, then K = 0.62 and C = 2. Do not put 1.24 into the formula as the coefficient and then multiply by two copies again. That counts the replica twice.

Elastic's documentation for logsdb mode describes best_compression, based on ZSTD for stored fields, plus specialized codecs for numeric values. That is a useful optimization, not a promise of a specific percentage. Denser compression can also increase CPU use during writes and reads. Compare options with the same queries, the same hardware, and segments that have already merged.

The archive needs a separate test. Compress daily files in the intended format, record the ratio of archive size to raw size, and measure decompression time. Storing millions of tiny objects creates overhead: catalogs and APIs work harder, and some archive classes charge for a minimum object size or per-object metadata. Amazon S3 documentation, for example, specifies 40 KB of extra metadata for every object in Glacier Flexible Retrieval and Deep Archive. An hourly or daily package is usually more practical than one file per event.

A full year of fast search is rarely necessary

A requirement to retain logs for a year usually describes preservation, not the response time for an arbitrary query. If an auditor can wait several hours for an export, keeping the full year in a search cluster makes little sense. If an analyst must investigate a nine-month-old incident and build correlations immediately, a cold archive without a search layer will not meet the requirement.

Split retention into three promises. The hot tier accepts writes and answers frequent queries. The warm tier keeps older indexes on higher-capacity media but remains searchable. The archive preserves raw or normalized events, which must be restored, attached, or processed by another engine before analysis.

Set the boundary using access data, not a tidy number of days. Over a quarter, collect the age of the time range used in every query and investigation. If 96% of interactive queries cover the latest 30 days, that is strong evidence for keeping one month hot. The remaining four percent do not prove that all older history belongs in an archive: some tables may need a 90-day warm tier.

Microsoft's Azure Monitor documentation explicitly separates analytics retention from long-term retention. Long-term data does not have all the capabilities of a regular table, and retrieval requires a search job. The principle matters even in a different system: a lower storage price comes with another access method. Write that limitation in the SLA in terms users can understand.

Do not move the only copy to the archive until you have tested the entire recovery chain. An archive without a catalog, checksums, field schema, and working import tool exists on paper, but it cannot support an investigation.

Cold storage changes the price but does not remove costs

An archive lowers media cost and often removes search-tier licensing from old data. It adds object write operations, minimum storage durations, read charges, API requests, network egress, temporary recovery space, and administrator labor. Looking only at the monthly price of one GB is dangerous.

Archive classes have a practical consequence: data does not return instantly. Amazon S3 documentation gives retrieval times ranging from minutes to 12 hours for Glacier Flexible Retrieval, and 9 to 48 hours for Deep Archive. It also sets minimum storage durations of 90 and 180 days. Deleting or transitioning an object early may not stop charges immediately. Regional prices change, so the model should keep them as parameters instead of hard-coding them into the design.

Calculate annual cloud archive cost this way:

Хранение = сумма(средний GB за месяц × цена GB-месяца)
Извлечение = восстановленный GB × цена чтения
Операции = число запросов по типам × цена запроса
Передача = выведенный GB × цена передачи

During the first year, volume rises gradually. If the archive starts empty and receives the same amount every day, its average occupied volume for that year is close to half the year-end volume. In the second full year, with a policy that keeps the latest 365 days, the archive stays near its limit. An estimate that multiplies the final 30 TB by twelve months overstates first-year storage. An estimate that uses half the final volume in every later year understates it.

For an on-premises archive, count usable capacity after RAID or erasure coding, spare media, servers, shelves, network ports, power, cooling, warranty replacements, and labor. Add a second site if losing one building must not destroy the logs. A cheap disk on a price list does not mean cheap preservation.

Set retention for an event class, not the entire feed

One retention period for everything almost always wastes money. An application's debug log, an administrator's permission change, a network flow, and a successful health-check result have different operational and evidentiary value. The data owner, security team, operations team, and legal counsel first agree why each class is kept. The engineer then selects its storage tier.

A working classification may look like this:

• Account and permission changes: fast search for 90 days, total retention for 1-3 years under policy; keep the complete normalized event and the original after the fast tier.
• Security-control events: fast search for 30-90 days, total retention for one year; keep investigation fields and the source record in the archive.
• Info-level application logs: fast search for 14-30 days, total retention for 90 days; retain only required services or aggregates.
• Detailed debug logs: fast search for 3-7 days, total retention for 7-30 days; an archive is usually unnecessary after the defect is fixed.
• Health metrics: fast search for 30 days, total retention for one year; hourly or daily aggregates are enough afterward.

This is a design worksheet, not a ready-made policy. A law, contract, sector rule, or internal investigation may demand another period and immutability. NIST SP 800-92 advises organizations to define log-retention requirements in policy instead of leaving them as a side effect of server settings. I would add three details to that policy: when the retention clock starts, the permitted recovery time, and who authorizes deletion.

Filtering before storage produces the largest savings, but it needs care. Remove noise that has been proved useless, such as frequent successful health checks, only when their absence does not break availability controls. Do not discard fields merely because nobody queries them today. Subject identifier, exact time, source, action, result, and correlation identifier often become the connecting evidence in a later investigation.

Aggregation suits metrics and repetitive technical events, but it does not replace an audit trail. A daily count of failed logins cannot identify the account, source address, or exact time involved. Keep the aggregate for trend analysis and retain the source events for the investigation period.

A worked calculation for 300 GB per day

Assume collectors accept 300 GB of raw events per day. A pilot measured K = 0.65 for the search index and K = 0.22 for archive files. The search tier has one replica, so C = 2. The plan includes 20% growth and keeps disk utilization at no more than 80%.

Keeping the full year in one search tier gives this calculation:

300 × 365 × 0,65 × 2 × 1,20 / 0,80 = 213 525 GB

The project must provide about 213.5 TB of usable provisioned capacity before system data, temporary recovery space, and backups. On disks sold in TB, this is already a large array. Calculate cluster snapshots separately: they may compress and deduplicate differently, but they are not free.

Now keep 30 days in the fast tier and move the previous 335 days to the archive. Search requires:

300 × 30 × 0,65 × 2 × 1,20 / 0,80 = 17 550 GB

Archive volume at year end, without another application-level copy, is:

300 × 335 × 0,22 × 1,20 = 26 532 GB

The resulting 17.55 TB of fast storage and 26.53 TB of archive objects cannot be added as if they were equivalent disks. Their price, resilience, and access speed differ. The comparison still shows why tiering changes the design. The search cluster needs less than one tenth of the original 213.5 TB, while older history remains in a denser format.

A design with 90 days of fast search requires 52.65 TB of provisioned capacity. The archive for the remaining 275 days occupies about 21.78 TB with the same coefficients. The extra 60 days of fast access cost about 35.1 TB in the search tier. The investigation owner can now decide whether real queries justify that difference, and procurement gets the price of a specific SLA instead of a vague request for more disks.

Test two more cases: a 50% increase in the feed and a worse K at the upper bound of the pilot result. If one new log source breaks the design, the reserve is too optimistic. If the unfavorable case still leaves half the array empty, capacity can probably be purchased in stages.

Disk price is only one part of TCO

Total storage cost includes hardware or cloud charges, licenses based on volume or nodes, compute for indexing and queries, network, racks, power, cooling, backups, and staff time. Longer retention may require more than disks. Shard and object counts rise, the catalog grows, and checks and restores take longer.

Check how the search platform is licensed. If the product charges per ingested GB, archiving after ingestion may not reduce the ingestion fee. If the license depends on nodes or resources, moving older data to a cheaper tier may reduce the number of costly nodes. If scanned query volume is billable, occasional broad searches across the archive become another variable.

Compare on-premises options over the same horizon, usually three to five years. Include disk replacement, shelf expansion, support, and capital cost. Do not treat a purchased array as free after year one: it consumes resources and occupies space throughout its service life. A cloud option avoids the initial purchase, but payments continue and vary with growth, reads, and transfer.

In an infrastructure project, GSE.kz can connect the retention calculation to server, data-center, and software-layer configurations without binding the architecture to one vendor. Before contacting any integrator, collect D, K, copy requirements, and query-age statistics. Without them, every supplier must either add an expensive safety margin or guess.

Estimate verification cost separately. An archive that nobody has restored creates false savings. A quarterly test restore of a selected day should produce a verifiable file set, confirm checksums, load the schema, and run several recorded queries. The estimate needs temporary capacity and labor hours for that test.

Immutability needs its own calculation

Long retention often supports proof that a record has not changed as well as search. Replication and a regular backup do not prove that by themselves. An administrator with enough access can delete both copies, and an automated policy can faithfully repeat the deletion on every node. When logs support an audit, the design needs an immutable tier with a defined retention period and separate access management.

Immutability affects capacity. A mistakenly exported package cannot simply be overwritten with a more compact version, while an overly long retention period can prevent deletion of unneeded data. Test the policy on a separate container and a short interval first. Record who can enable retention, who can extend it, whether a legal hold is possible, and what happens when the period ends. Those answers matter more than the vendor's feature name.

Microsoft's Azure Monitor export documentation notes that records in Log Analytics cannot be modified after ingestion, but they can be removed through a purge. It recommends export to a storage account with an immutability policy when storage must resist modification. This exposes the distinction between protecting record content inside the search platform and protecting the entire record from deletion. Test both operations in your own system.

Each archive package needs a small manifest containing the source, interval start and end, record count, size, schema version, and checksum. Create the checksum after final compression and before archive transfer. A basic check looks like this:

sha256sum events-2026-06-15.json.zst > events-2026-06-15.sha256
sha256sum -c events-2026-06-15.sha256

Successful output looks like events-2026-06-15.json.zst: OK. Keep the checksum file beside the package and include it in the protected catalog. Otherwise, an attacker can replace the data and its checksum together. A strict evidentiary process may also require a digital signature, trusted timestamp, and access log, but the control owner must specify those requirements, not the storage-array administrator.

Archive encryption adds another dependency. The data, key, and recovery instructions must survive for the same period, but they should not sit in one place under identical permissions. During key rotation, verify that old objects remain readable and preserve the format version. I have encountered archives that remained physically intact for years but became useless after a service account was deleted or the key-management system changed.

The format is part of retention as well. Open line-delimited JSON is easier to restore with different tools, but it takes more space and reads more slowly. Parquet stores typed columns more densely and supports selective analysis, but it requires the schema and a compatible reader. Choose according to the expected future search method. A good test asks another team to read a package in a clean environment using only the manifest and instructions, without verbal help from the archive's author.

Do not enable immutability for every technical log merely as a precaution. It raises cost and makes classification errors harder to correct, while personal or secret data may have a conflicting deletion period. Select the classes that truly require protection from modification and calculate them separately. The immutable archive then remains a control instead of becoming an expensive dump that nobody has permission to clear.

Recovery time sets the sensible boundary

A practical boundary appears when the organization answers four questions in writing: what period needs search within seconds, how many hours users may wait for older data, how much data a typical restore covers, and what happens if the archive is unavailable. Hot and warm retention then follow from the work rather than habit.

For many feeds, test a baseline of 30 days of fast search, 90 days in the warm tier for the most useful tables, and the rest of the mandatory period in the archive. This is not a universal rule. Privileged-action audits may deserve a longer interactive window, while detailed debugging may need a much shorter one.

Make the decision an acceptance test. Pick a random date older than the fast tier, restore one event class into an isolated environment, verify record counts and checksums, run an investigation query, and then remove the temporary copy safely. Measure elapsed hours and labor. If the result misses the SLA, move the boundary or change the archive method. Buying another year of fast disks remains the most expensive way to avoid this test.

Recalculate after adding a major source, changing the schema, changing the replica count, or revising policy. Keep daily volume and K as monthly time series. Procurement can then add capacity before a utilization threshold, instead of reacting after writes are blocked. The sensible boundary sits where recovery still fits the job, not where the budget row ends.