Nginx Logs

This system is concerned with understanding what actually happened at the edge of a web service. It operates on the premise that every completed request leaves behind a factual record, and that those records are the most reliable source of truth available once a request has passed.

The focus here is on individual events, not summaries. Each request is treated as an observation, not a statistic. The system is designed to preserve those observations long enough to be questioned later, even when the questions were not known at the time the data was collected.

This scope explicitly values completeness over convenience. Data is not sampled, pre-aggregated, or reshaped to fit predefined dashboards. Any loss of detail is considered a tradeoff that must be consciously chosen, not an accidental side effect of optimization.

Just as important are the boundaries. This system does not attempt to explain application intent, user motivation, or business meaning. It does not infer success beyond what can be observed at the request boundary. It does not replace metrics, tracing, or application-level logging.

In practice, this means the system answers questions like what was requested, when it was requested, how it was handled, and what response was produced. Questions about why a user behaved a certain way or whether a request achieved a business goal are intentionally out of scope.

The goal is clarity, not completeness of narrative. By drawing a hard line around what is observed and refusing to speculate beyond it, the system remains dependable even when other sources of truth disagree.

At its core, this system begins with a single artifact: one line of text written by nginx after a request has completed. Each line is a factual record of an interaction between a client and the server.

Below are three real examples representing common outcomes. Each line corresponds to a single HTTP request and response cycle.

baremetalbridge.com 54.85.239.163 - - [08/Feb/2026:23:22:22 +0000] "GET /the-silent-rebellion-fast-food-wars-and-the-power-of-the-dollar/ HTTP/1.1" 404 659 "-" "Grammarly/1.0 (http://www.grammarly.com)"
baremetalbridge.com 114.119.156.9 - - [08/Feb/2026:23:22:34 +0000] "GET /monday-boot-sequence HTTP/1.1" 301 5 "-" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"
baremetalbridge.com 114.119.156.9 - - [08/Feb/2026:23:22:35 +0000] "GET /monday-boot-sequence/ HTTP/1.1" 200 4984 "-" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"

The first request resulted in a 404, indicating that the server successfully handled the request but could not find a matching resource.

The second request returned a 301, signaling a permanent redirect. The server is instructing the client to request a different location.

The third request completed with a 200, meaning the requested content was located and returned successfully.

These records describe outcomes, not intent. They tell us what was requested, how it was handled, and when it occurred. Everything else must be inferred carefully.

Nginx access logs do not have an inherent structure. They only become meaningful when a format is declared and consistently enforced. The log format defines both the order of fields and the implicit contract for how each request will be recorded.

Below is the access log format used as the canonical example throughout this document. Every log line shown earlier was produced using this definition.

log_format dm_access '$host $remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"';

Each field in this format exists for a specific reason. Together, they describe what was requested, when it happened, who made the request, and how the server responded. Importantly, this format records outcomes without embedding interpretation.

• $host identifies the virtual host that handled the request. This is critical when multiple sites share the same edge.
• $remote_addr records the client IP address as seen by nginx at the time of the request.
• $remote_user represents authenticated user identity, when applicable. A dash indicates no authenticated user.
• $time_local captures when the request completed, using the server’s local time format. This is later normalized but preserved here as emitted.
• $request contains the HTTP method, requested path, and protocol version as a single atomic string.
• $status records the HTTP response status code returned to the client.
• $body_bytes_sent indicates how many bytes were sent in the response body, excluding headers.
• $http_referer captures the referring page, when provided by the client.
• $http_user_agent records the client’s declared software identity.

None of these fields attempt to explain intent. They simply record what nginx observed. Meaning is derived later, not embedded here.

This format is applied uniformly across all hosts using a shared access log directive. This ensures that every log line, regardless of origin, adheres to the same structural contract.

access_log /var/log/nginx/dangerousmetrics.com/access.log dm_access;

Enforcing a single log format across all hosts is not a convenience. It is a prerequisite for reliable analysis. Without a consistent structure, parsing becomes guesswork and comparisons between systems become fragile.

By fixing the format at the edge, all downstream systems can assume the same field order, the same semantics, and the same guarantees. This is what allows logs from different machines to be treated as a single, coherent data surface.

Parsing is the act of assigning structure to text. Normalization is the act of assigning consistency to that structure. Together, they form the boundary between raw observation and usable data.

At this stage, the goal is not insight. The goal is fidelity. Parsing must extract what was recorded without adding meaning, and normalization must make values comparable without changing what they represent.

The parsing process begins with position and agreement. Each field in the log line is interpreted according to its declared order and expected format. If a field cannot be confidently identified, that failure is itself meaningful and must not be silently corrected.

Normalization follows parsing and exists to remove superficial variation. Timestamps are aligned to a common time base. Numeric values are converted from strings to numbers. Empty or missing fields are made explicit rather than implied.

There are strict limits on what normalization is allowed to do. It may standardize representation, but it must not collapse distinct values, infer intent, or rewrite outcomes. A request that returned a redirect remains a redirect. A missing field remains missing.

Parsing and normalization are intentionally conservative operations. They are designed to preserve the full expressive range of the original log line while making it possible to ask precise questions later.

Any transformation that introduces interpretation, correlation, or summarization belongs to a later stage. At this boundary, the only acceptable loss is superficial inconsistency. Loss of meaning is not permitted.

Time is the axis that makes logs useful, and it is also the axis most likely to betray you. Every request recorded in an access log is bound to a moment, and nearly every meaningful question you will ask later depends on that moment being represented correctly.

Consider the timestamp emitted in the example log entries:

[08/Feb/2026:23:22:22 +0000]

This value represents the time at which the request completed, as recorded by the nginx worker handling the connection. It includes a date, a time, and an explicit offset from UTC. At the moment it is written, it is still just text. Its meaning exists, but it is implicit.

For analysis systems like OpenSearch, that implicit meaning must be made explicit. Timestamps are not treated as strings because strings do not understand time. They sort lexically, not temporally. They cannot be ranged accurately. They cannot be aggregated into intervals without reinterpretation.

When a timestamp is stored as a true temporal type, the system gains the ability to order events correctly, even across hosts. It can answer questions about sequences, gaps, bursts, and correlations. Without this, time-based analysis quietly degrades into approximation.

Normalizing timestamps to UTC is a necessary step in this process. Different machines may run in different locales. Clock offsets may vary. Daylight saving changes introduce ambiguity. By converting all observed times to a single, unambiguous reference, events from different sources can be placed onto the same timeline.

Even with normalization, time remains imperfect. Clocks drift. Virtual machines pause. Network delays reorder arrival. Logs reflect when an event was observed, not when it was intended. This is not a flaw of logging, it is a property of distributed systems.

Because of this, time must be handled carefully and conservatively. It should be preserved, normalized, and indexed accurately, but never overinterpreted. Ordering is a tool for reasoning, not a guarantee of causality.

Treating timestamps as first-class temporal data is what allows later systems to ask precise questions without lying to themselves. Treating them as strings turns time into decoration.

Enrichment is the process of adding context to an event so it can be understood more easily later. In many logging systems, this step becomes a catch-all where missing information is inferred, joined, or reconstructed after the fact.

In this particular case, enrichment plays a much smaller role than it often does. That is intentional. The nginx access logs used here are already emitted with the full set of contextual fields required for meaningful analysis.

Host identity, request details, response status, byte counts, referrer, and user agent are all captured at the edge, at the moment the request completes. This means the most important enrichment has already occurred before the log line ever exists.

Because of this, there is no need to perform heavy downstream enrichment for these logs. No additional lookups are required to understand what happened. No external systems need to be consulted to add meaning that was missing at write time.

This section exists primarily as a reference point, not as an active transformation stage. It establishes where enrichment would occur if it were needed, and more importantly, where it should not be forced when it is unnecessary.

If you need a refresher on where and how this context was established, refer back to the Log format and fields section. That is where the enrichment boundary is intentionally drawn for these logs.

The guiding principle is simple. Context is most trustworthy when it is recorded at the point of observation. The further enrichment is pushed downstream, the more likely it is to introduce assumptions.

Later chapters will revisit enrichment in cases where raw signals do not carry sufficient meaning on their own. For nginx access logs, that work has already been done at the edge, and repeating it would add complexity without improving truth.

Once log events have been parsed and normalized, they must be placed somewhere that preserves their structure, ordering, and queryability over time. The storage layer is not just a destination. It is part of the contract that determines what questions can be asked later and how reliably they can be answered.

The strategy used here is built around treating access logs as immutable events. Each request is written once, indexed once, and never rewritten to fit a new interpretation. Corrections and derived insights are handled downstream rather than by mutating the original record.

This approach places several non-negotiable requirements on storage. Events must be stored in a system that understands time as a first class concept, supports structured fields with explicit types, and allows efficient querying across large volumes without requiring pre-aggregation.

The system used to satisfy these requirements is OpenSearch. It is chosen not because it is fashionable or convenient, but because it aligns well with the needs of event-oriented data. It provides indexing over time-based documents, flexible schemas that can be controlled explicitly, and powerful query capabilities without collapsing raw events into summaries prematurely.

Feeding data into this storage layer requires a reliable ingestion mechanism that can accept raw log lines, apply parsing and normalization consistently, and deliver structured events without loss. That role is handled by Vector, which acts as the boundary between raw text and indexed documents.

Importantly, this section describes intent, not implementation. The choice of Vector and OpenSearch reflects the constraints outlined here, but the reasoning matters more than the tools themselves. A different stack could be substituted if it honored the same rules.

The following two sections describe how these decisions are applied in practice. The first covers how log data is ingested and prepared for storage. The second covers how that data is indexed, mapped, and stored in OpenSearch to preserve its usefulness over time.

OpenSearch is used here as the system of record for parsed nginx access events. Its role is not to interpret or summarize data, but to store immutable events in a way that preserves structure, ordering, and queryability at scale.

The key design decision is that the shape of the data is defined explicitly before any events are indexed. This is done through an index template that declares field names, types, and constraints up front. Documents are expected to conform to this contract.

Below is the index template used for these access logs. It defines how each field is stored and indexed, and just as importantly, what is not allowed.

{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  },
  "mappings": {
    "dynamic": false,
    "properties": {
      "timestamp": {
        "type": "date",
        "format": "dd/MMM/yyyy:HH:mm:ss Z"
      },
      "host": {
        "type": "keyword"
      },
      "client_ip": {
        "type": "ip"
      },
      "user": {
        "type": "keyword"
      },
      "request": {
        "type": "text"
      },
      "method": {
        "type": "keyword"
      },
      "path": {
        "type": "keyword",
        "ignore_above": 2048
      },
      "protocol": {
        "type": "keyword"
      },
      "status": {
        "type": "integer"
      },
      "bytes": {
        "type": "long"
      },
      "referrer": {
        "type": "keyword",
        "ignore_above": 2048
      },
      "user_agent": {
        "type": "text"
      },
      "env": {
        "type": "keyword"
      },
      "log_type": {
        "type": "keyword"
      },
      "pipeline": {
        "type": "keyword"
      },
      "parse_ok": {
        "type": "boolean"
      },
      "source": {
        "type": "keyword"
      },
      "message": {
        "type": "text",
        "index": false
      }
    }
  }
}

Setting dynamic to false is a deliberate choice. It prevents OpenSearch from inventing fields on the fly based on incoming documents. Any field that is not declared here is rejected rather than silently indexed with an inferred type.

This protects the index from schema drift. Without this constraint, a single malformed event can introduce new fields, conflicting types, or unexpected mappings that permanently alter query behavior.

Field types are chosen to reflect how the data will be used. Keywords are used where exact matching or aggregation matters. Numeric types are used where ordering and range queries are required. The timestamp is stored as a true date so time-based queries and sorting behave correctly.

The template also establishes limits. Long strings are capped. Raw messages are stored but not indexed. This keeps storage predictable and avoids paying indexing costs for data that will never be queried directly.

Importantly, ingestion tools do not need to understand this mapping. Vector does not need to know field types or index settings. It only needs to emit documents with the correct field names and values.

Index templates are applied by OpenSearch at index creation time. When a matching index is created, the mapping is enforced automatically. Any incoming document is validated against it.

This separation of responsibility is intentional. OpenSearch owns the schema. Ingestion tools remain simple and focused on delivery. The result is a system where structure is centralized, enforced, and resistant to accidental change.

The next section describes how Vector prepares and emits events so they conform to this mapping without embedding OpenSearch-specific logic into the ingestion pipeline.

Vector is the ingestion boundary between raw log files and structured events. Its role is to accept log input from multiple sources, apply parsing and normalization deterministically, and emit documents that conform to the storage contract defined earlier.

Until recently, Logstash was the log parser of choice in this role. While powerful, its overhead and breadth of features were not justified for this use case. The pipeline here does not require the majority of Logstash filters, codecs, or dynamic behavior.

Filebeat was also considered, but it is tightly coupled to the Elastic ecosystem and assumes conventions that do not align cleanly with this deployment. Vector fills the gap by remaining lightweight, explicit, and agnostic about downstream storage.

The configuration below defines the complete ingestion pipeline for nginx access logs. It shows how log lines enter the system, how they are parsed and normalized, and how they are delivered to OpenSearch without embedding OpenSearch-specific logic into the parsing stage.

sources:
  nginx_dev_file:
    type: file
    include:
      - /var/log/nginx/dev.dangerousmetrics.com/access.log
    read_from: beginning

  nginx_www_file:
    type: file
    include:
      - /var/log/nginx/dangerousmetrics.com/access.log
    read_from: beginning

  nginx_syslog:
    type: syslog
    address: 0.0.0.0:1514
    mode: udp

transforms:
  parse_nginx:
    type: remap
    inputs:
      - nginx_dev_file
      - nginx_www_file
      - nginx_syslog
    source: |
      parsed, err = parse_regex(
        .message,
        r'^(?P<host>\S+) (?P<client_ip>\S+) - (?P<user>\S+) \[(?P<timestamp>[^\]]+)\] "(?P<request>[^"]*)" (?P<status>\d+|-) (?P<bytes>\d+|-) "(?P<referrer>[^"]*)" "(?P<user_agent>[^"]*)"'
      )

      if err != null {
        abort
      }

      . |= parsed

  normalize:
    type: remap
    inputs:
      - parse_nginx
    source: |
      if exists(.source) {
        if contains!(.source, "/var/log/nginx/dev") {
          .env = "dev"
        } else if contains!(.source, "/var/log/nginx/") {
          .env = "www"
        }
      }

      if !exists(.env) {
        .env = "remote"
      }

      .status = to_int(.status) ?? null
      .bytes  = to_int(.bytes)  ?? null

      .parse_ok = false

      if exists(.request) {
        req, err = parse_regex(
          .request,
          r'^(?P<method>\S+)\s+(?P<path>\S+)(?:\s+(?P<protocol>\S+))?$'
        )

        if err == null {
          .method   = req.method
          .path     = req.path
          .protocol = req.protocol
          .parse_ok = true
        }
      }

      .log_type = "nginx_access"
      .pipeline = "vector_dm_access"

sinks:
  opensearch_out:
    type: elasticsearch
    inputs:
      - normalize
    endpoints:
      - http://localhost:9200
    suppress_type_name: true
    mode: bulk
    bulk:
      index: nginx-access-%Y.%m

The pipeline begins with three sources. Two read nginx access logs directly from disk, representing different virtual hosts. The third listens for syslog input over UDP, allowing remote or forwarded logs to enter the same processing path.

All three sources feed into a single parsing transform. This transform applies a regular expression that mirrors the nginx log format described earlier. If a line does not match the expected structure, it is aborted rather than partially parsed.

This behavior is intentional. Failed parsing is treated as a signal, not something to be silently repaired. Only events that fully conform to the expected format are promoted to structured data.

The normalization stage applies controlled cleanup and decomposition. Environment tags are derived from the source path, allowing events to be grouped by origin without external lookup. Status codes and byte counts are converted to numeric types to match the OpenSearch mapping.

The request line is further decomposed into method, path, and protocol. A boolean flag records whether this secondary parsing succeeded, preserving visibility into partial failures without discarding the event.

Finally, static metadata fields are attached to identify the log type and the pipeline that produced the event. These fields provide trace ability and future-proofing without affecting core semantics.

The sink delivers events to OpenSearch using bulk indexing. The index name is time-based, allowing retention and lifecycle policies to be applied without rewriting data. Vector does not define the schema. OpenSearch enforces it using the index template described earlier.

This separation of concerns is the central design choice. Vector is responsible for correctness and consistency of events. OpenSearch is responsible for validation, indexing, and query behavior. Neither needs to understand the internals of the other.