Raw Logs export

Raw Logs is the account-level hourly export of CDN request logs to Amazon Simple Storage Service (S3), File Transfer Protocol (FTP), or SSH File Transfer Protocol (SFTP) storage. Logs include requests to edge cache servers. Pre-cache server requests are included only when Origin Shielding is enabled and Add logs from Origin Shielding is selected. Raw Logs remains available for existing hourly S3, FTP, or SFTP exports. For new log-export setups, use Logs Uploader. Logs Uploader is the current workflow: it supports 5–60-minute delivery, selectable fields, HTTP(S) and Alibaba Cloud Simple Log Service (SLS) destinations, and per-resource policies. To move an existing Raw Logs configuration to Logs Uploader, contact Gcore Support.

This is a paid feature. Contact Gcore Support to activate Raw Logs. After activation, enable Receive Raw Logs in the Gcore Customer Portal and configure export to S3, FTP, or SFTP storage.

Raw Logs settings

The CDN > Logs > Raw Logs page stores the export destination, empty-file preference, and Origin Shielding inclusion. Status, the hourly interval, and traffic totals come from that configuration.

Origin Shielding log inclusion

When Origin Shielding is enabled for the account, the Add logs from Origin Shielding checkbox appears in Raw Logs settings. Select it so the export includes requests to both edge servers and the pre-cache server.

If Add logs from Origin Shielding is not visible, Origin Shielding is not activated for the account. See the Origin Shielding guide.

Raw Logs statuses

The Raw Logs status in the Customer Portal can be:

Pending: interval between connecting to storage and the first export
OK: logs are exported from at least one CDN server
Failed: a storage connection error occurred, or no export completed within 24 hours
Pause: Raw Logs is paused

Export interval

Logs are exported at the end of each hour. If Raw Logs is activated at 00:30, the first export runs between 00:45 and 01:00, and the next between 01:45 and 02:00. If the covered CDN resources receive no requests during the export interval and Do not send empty logs is not selected, an empty log file (about 20 bytes) is sent to storage.

Log traffic calculation

Raw Logs traffic totals depend on whether Add logs from Origin Shielding is selected. For CDN report metrics, see View statistics on CDN resources.

Origin Shielding disabled

total_bytes = upstream_bytes + sent_bytes

upstream_bytes is $upstream_response_length: response length from the origin, in bytes
sent_bytes is $bytes_sent: bytes sent to the client from edge cache servers

Example: $upstream_response_length 10485760 and $bytes_sent 1514848 produce 12000608 bytes.

Origin Shielding enabled

total_bytes = upstream_bytes + sent_bytes + shield_bytes

upstream_bytes is $upstream_response_length: response length from the origin, in bytes
sent_bytes is $bytes_sent: bytes sent to the client from edge cache servers
shield_bytes is $bytes_sent on the pre-cache server: bytes sent from the shielding node

Log totals can differ from billing statistics when some files are missing. Common causes:

Origin Shielding is enabled, but Add logs from Origin Shielding is not selected
The storage imposes a rate limit, so some files are not delivered to storage

Log path example

s3://log-bucket-name/2019/08/20/15/nodename_cdn.example.com_access.log.gz

Log example

"203.0.113.45" "-" "-" "[26/Apr/2019:09:47:40 +0000]" "GET /ContentCommon/images/image.png HTTP/1.1"
"200" "1514283" "https://example.com/videos/10" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36"
"1514848" "[dh-up-gc18]" "https" "origin.example.com" "1.500" "0.445" "157" "bytes=0-1901653" "[dh]"
"MISS" "10485760" "192.0.2.1:80" "2510" "7399" "-" "-" "KZ" "-" "shield_no" "198.51.100.1" "80" "206" "-" "0.000"
"0.200" "198.51.100.2" "asnumber" "106980391" "1" "00-d5fe1dc9035165ce36952daf29686b6c-14330be33197dd1a-01" "-" "-"
"ECDHE-RSA-AES256-GCM-SHA384" "28a4184139cb43cdc79006cf2d1a4ac93bdc****" "r"
"application/json" "21" "PL" "45" "10" "100700"

Log format

"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status"
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent"
"$edgename" "$scheme" "$host" "$request_time"
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]"
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr"
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set"
"$geoip_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time"
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests"
"$http_traceparent" "$http_x_forwarded_proto" "$gcdn_internal_status_code" "$ssl_cipher"
"$ssl_session_id" "$ssl_session_reused" "$sent_http_content_type" "$tcpinfo_rtt"
"$server_country_code" "$gcdn_tcpinfo_snd_cwnd" "$gcdn_tcpinfo_total_retrans" "$gcdn_rule_id"
"$request_completion" "$sent_http_content_length" "$gcdn_finalize_reason"

New fields can appear at the end of the line. When fields are added, Gcore sends an email notification.

Log fields

The following table lists the available log fields. Fields marked as internal system variables belong to the CDN control plane and can be ignored for most analytics.

Log fields

Field	Log value example	Description
`$remote_addr`	`203.0.113.45`	Client IP address
`$remote_user` (internal system variable)	`-`	Username used in Basic authentication
`[$time_local]`	`[26/Apr/2019:09:47:40 +0000]`	Local time in Common Log Format
`$request`	`GET /ContentCommon/images/image.png HTTP/1.1`	HTTP method, requested file path, and HTTP version
`$status`	`200`	Response status code from a CDN server
`$body_bytes_sent`	`1514283`	Number of bytes sent to a user, excluding the response header size
`$http_referer`	`https://example.com/videos/10`	Value of the `Referer` request header. The header usually identifies the page that initiated the request.
`$http_user_agent`	`Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36`	User agent that was used to send a request (browser or other application)
`$bytes_sent`	`1514848`	Number of bytes sent to a user
`$edgename`	`[dh-up-gc18]`	CDN server that forwarded a requested file
`$scheme`	`https`	Protocol (HTTP or HTTPS) of a request
`$host`	`cdn.example.com`	Requested hostname of a CDN resource
`$request_time`	`1.500`	Request processing time in seconds (accurate to milliseconds); time elapsed between the first bytes of a request being processed and logging after the last bytes were sent to a user
`$upstream_response_time`	`0.445`	Number of seconds (accurate to milliseconds) it took to receive a response from the upstream server. When Origin Shielding is active, the upstream server may be an intermediate CDN cache rather than the origin. In case of multiple responses, commas and colons are used
`$request_length`	`157`	Request length (including request line, header, and request body)
`$http_range`	`bytes=0-1901653`	File fragment size in a Range request
`[$responding_node]`	`dh`	Responding data center
`$upstream_cache_status`	`MISS`	Status of a requested file in CDN cache: `HIT`: response served from the CDN cache. `STALE`: outdated response that failed to update (origin not responding or responding incorrectly). `UPDATING`: outdated response still updating from a previous request. `REVALIDATED`: response matching one on an origin (based on the `proxy_cache_revalidate` directive). `EXPIRED`: response that has expired in cache but still matches one on an origin; a request was sent to re-cache it. `MISS`: response served from the origin rather than from cache, including the first request after a purge. `BYPASS`: response served from the upstream server because configured cache-bypass conditions were met — for example a matching cookie, query parameter, or authorization header. Bypass does not invalidate the cached copy. A request for content that is absent from cache after a purge results in `MISS`, not `BYPASS`.
`$upstream_response_length`	`10485760`	Response length from an origin in bytes. In case of multiple responses, commas and colons are used
`$upstream_addr`	`192.0.2.1:80`	IP address and port of the upstream server
`$gcdn_api_client_id` (internal system variable)	`123`	Account ID in the Gcore system
`$gcdn_api_resource_id` (internal system variable)	`01`	CDN resource ID in the Gcore system
`$uid_got` (internal system variable)	`-`	Cookie name and received user ID
`$uid_set` (internal system variable)	`-`	Cookie name and provided user ID
`$geoip_country_code`	`KZ`	Client ISO 3166-1 alpha-2 country code
`$geoip_city`	`-`	Client city code
`$shield_type` (internal system variable)	`shield_no`	Indicates whether Origin Shielding is enabled: `shield_old` – enabled `shield_no` – disabled
`$server_addr` (internal system variable)	`198.51.100.1`	IP address of an Anycast zone or CDN server
`$server_port` (internal system variable)	`80`	Requested port
`$upstream_status`	`206`	Origin response code
`$upstream_connect_time`	`0.000`	Number of seconds (accurate to milliseconds) it took to access an origin server
`$upstream_header_time`	`0.200`	Number of seconds (accurate to milliseconds) it took to receive a response header from an origin server
`$shard_addr` (internal system variable)	`198.51.100.2`	IP address of a CDN server that was first to accept a request if the Cache Sharding feature is enabled
`$geoip2_data_asnumber`	`asnumber`	Number of an autonomous system that sent a request
`$connection` (internal system variable)	`2897494295`	Connection serial number
`$connection_requests` (internal system variable)	`1`	Current number of requests made through a connection
`$http_traceparent`	`00-d5fe1dc9035165ce36952daf29686b6c-14330be33197dd1a-01`	Incoming client `traceparent` request header. If the client omits the header, the field contains `-`; if the value is invalid, the field preserves that value. The CDN-generated replacement is not stored here. See Traceparent header for troubleshooting.
`$http_x_forwarded_proto`	`-`	Initial protocol of an incoming request (HTTP or HTTPS)
`$gcdn_internal_status_code` (internal system variable)	`-`	Internal CDN status code. `-` means no internal code was recorded. Other values identify the rule or feature that generated the response. See internal status codes.
`$ssl_cipher` (internal system variable)	`ECDHE-RSA-AES256-GCM-SHA384`	Cipher name used for an established SSL connection
`$ssl_session_id` (internal system variable)	`28a4184139cb43cdc79006cf2d1a4ac93bdc****`	Session ID of an established SSL connection
`$ssl_session_reused` (internal system variable)	`r`	Shows whether a session was reused (`r`) or not (`.`)
`$sent_http_content_type`	`application/json`	Value of the `Content-Type` HTTP header, indicating the MIME type of a transmitted file
`$tcpinfo_rtt`	`21`	Average time (latency) it takes to transfer a packet to/from a server. The unit of time is microseconds.
`$server_country_code`	`PL`	Server ISO 3166-1 alpha-2 country code
`$gcdn_tcpinfo_snd_cwnd`	`45`	Size of the TCP congestion window: the maximum number of TCP segments the connection can send before an acknowledgment is required
`$gcdn_tcpinfo_total_retrans`	`10`	Total number of retransmitted packets over the life of the connection
`$gcdn_rule_id`	`100700`	Initial rule ID (beta). Possible values are `-` or `100700`
`$request_completion`	`OK`	Whether the CDN finished writing the response body. `OK` means the request completed. An empty value means it did not. Use with `$sent_http_content_length` and `$body_bytes_sent` to detect truncated `200` or `206` responses.
`$sent_http_content_length`	`53214`	`Content-Length` sent to the client: the number of body bytes the CDN declared. Compare with `$body_bytes_sent` to measure truncation. `0` means the header was absent (chunked transfer, no `Content-Length`, or a row from before the field was added), not that zero bytes were sent.
`$gcdn_finalize_reason`	`ok`	Why the request stopped producing a body. `ok` means the body finished. Empty or `unset` means no reason was recorded. Other values identify the cause: client (`client_closed`, `client_timeout`, `client_error`); upstream (`upstream_connect_failed`, `upstream_connect_timeout`, `upstream_headers_null`, `upstream_header_timeout`, `upstream_body_null`, `upstream_body_partial`, `upstream_body_timeout`, `upstream_ssl_error`, `upstream_error`, `upstream_send_timeout`); this proxy (`internal_error`, `shutdown`); or slice (`slice_bad_status`, `slice_missing`, `slice_etag`, `slice_range`).

Internal status codes

$gcdn_internal_status_code identifies the CDN rule or feature that generated the response.

Reason	HTTP code	Internal code	Comment
Country ACL	403	1001
Referer ACL	403	1002
IP ACL	403	1003
User-Agent ACL	403	1004
Secure Token	403	1005	Returned when Secure Token validation fails and `$secure_link` is empty
Secure Token	410	1006	Returned when the link lifetime has expired and `$secure_link` is `"0"`
WAF	403	1007	Request blocked by WAF
Bot challenge / Testcookie	307	1008	Redirection sent by bot challenge (testcookie)
Blocklist	403	1009	Request blocked by bot protection blocklist
HTTP method	405	1200	Returned when `AllowedHttpMethods` is enabled and the request method is not in the allowed list
Streaming disabled (`.ts` or `.m3u8` files)	402	1201	Streaming feature disabled for the client
LE validation /.well-known/acme-challenge/	404	1202	Returned when `http_user_agent` is not `cert-manager-challenges`, `acme.zerossl.com`, `Cpanel-HTTP-Client`, `Buypass validation client`, `Google-Certificates-Bridge`, `vercel-fetch`, `win-acme`, `Typhoeus`, or `Go-http-client`
Shield requestor header	403	1203	Unauthorized request to the shield
Force Return	ANY	1204	Response generated by the Force Return status code option

When a Secure Token link passes the authenticity check, $secure_link is set to the link extracted from the request URI. That successful validation does not produce a 403 or 410.

S3 export

Amazon S3
Other S3-compatible storage

Open Raw Logs

In the Gcore Customer Portal, navigate to CDN > Logs > Raw Logs.

Enable the export

Turn on Receive Raw Logs. Keep Do not send empty logs selected to skip empty files, or clear it to receive empty files.

Include Origin Shielding logs

If Origin Shielding is enabled, select Add logs from Origin Shielding to include edge and pre-cache logs.

Select Amazon S3

Under Receive logs using, select S3. Under Choose S3 storage, select Amazon.

Amazon S3 export form in Raw Logs settings

Enter credentials

Enter the AWS access key ID and AWS secret access key. See the Amazon access key guide.

Set region and buckets

Optionally choose an AWS region. Then choose one bucket for all CDN resources or a separate bucket per resource, and enter an existing bucket name. Add a folder name to write files inside that bucket.

Save the configuration

Click Save changes.

Open Raw Logs

In the Gcore Customer Portal, navigate to CDN > Logs > Raw Logs.

Enable the export

Turn on Receive Raw Logs. Keep Do not send empty logs selected to skip empty files, or clear it to receive empty files.

Include Origin Shielding logs

If Origin Shielding is enabled, select Add logs from Origin Shielding to include edge and pre-cache logs.

Select other S3 storage

Under Receive logs using, select S3. Under Choose S3 storage, select Other.

Other S3 export form in Raw Logs settings

Enter the hostname

Enter the storage hostname. For Gcore Object Storage, copy the hostname from Details.

Enter credentials

Enter the access key ID and secret access key. For Gcore Object Storage, create the key under access keys and copy the secret when the key is created.

Enter the bucket hostname

Enter the bucket hostname in {bucket_name}.{hostname} format, using the hostname from the hostname step. For Gcore Object Storage the value looks like examplename.s-ed1.cloud.gcore.lu.

Enter the region

Optionally enter the region (location ID). For Gcore Object Storage this field is required. The location ID is the hostname prefix before the first dot. See storage Details.

Set buckets

Choose one bucket for all CDN resources or a separate bucket per resource, and enter an existing bucket name. Add a folder name to write files inside that bucket.

Save the configuration

Click Save changes.

FTP and SFTP export

SFTP
FTP

Open Raw Logs

In the Gcore Customer Portal, navigate to CDN > Logs > Raw Logs.

Enable the export

Turn on Receive Raw Logs. Keep Do not send empty logs selected to skip empty files, or clear it to receive empty files.

Include Origin Shielding logs

If Origin Shielding is enabled, select Add logs from Origin Shielding to include edge and pre-cache logs.

Select SFTP

Under Receive logs using, select SFTP.

Enter the hostname and port

Enter hostname:port. For Gcore SFTP Storage, copy the hostname from Details and append port 2200, for example ams.origin.gcdn.co:2200. For another provider, use that host and its SFTP port.

Omitting the port returns a wrong connection settings error.

Enter credentials

Enter the Login and Password. For Gcore SFTP Storage, the login is Storage/User name in Details.

Enter the destination folder

Optionally enter a Prepend folder. Then enter the Folder. For Gcore SFTP Storage, use the root folder at the end of the Upload path in Details. The home folder is receive-only and cannot be written to directly. The only writable default folder is htdocs. For another provider, enter the folder that should receive files, or leave the field empty if the account starts in that folder.

Organize folders

Choose one folder for all CDN resources or a separate folder per resource. A missing folder name sends files to the root folder.

Save the configuration

Click Save changes.

FTP sends credentials and log data without transport encryption. Use SFTP when logs contain IP addresses, cookies, or request URLs.

Open Raw Logs

In the Gcore Customer Portal, navigate to CDN > Logs > Raw Logs.

Enable the export

Turn on Receive Raw Logs. Keep Do not send empty logs selected to skip empty files, or clear it to receive empty files.

Include Origin Shielding logs

If Origin Shielding is enabled, select Add logs from Origin Shielding to include edge and pre-cache logs.

Select FTP

Under Receive logs using, select FTP.

Enter the hostname and port

Enter hostname:port. Use the FTP host and port from the storage provider, typically port 21, for example ftp.example.com:21.

Omitting the port returns a wrong connection settings error.

Enter credentials

Enter the Login and Password.

Enter the destination folder

Optionally enter a Prepend folder. Then enter the Folder, or leave it empty if the FTP account starts in that folder.

Organize folders

Choose one folder for all CDN resources or a separate folder per resource. A missing folder name sends files to the root folder.

Save the configuration

Click Save changes.

Raw Logs deactivation

Turn off the export

In the Gcore Customer Portal, navigate to CDN > Logs > Raw Logs and turn off Receive Raw Logs.

Save the change

Click Save changes. Delivery stops only after this save. The stored S3, FTP, or SFTP settings remain and apply again when Receive Raw Logs is turned on and saved.

Raw Logs page with Receive Raw Logs turned off

Discard changes restores the last saved configuration. If Receive Raw Logs was on when the page was last saved, Discard changes turns the toggle back on and does not deactivate the export.

Account settings

Developer Tools

CDN

FastEdge

Edge Cloud

AI

Gclaw

Managed DNS

Public DNS

Hosting

Colocation

Object Storage

Video Streaming

DDoS protection

Edge Proxy

WAAP

Raw Logs settings

Origin Shielding log inclusion

Raw Logs statuses

Export interval

Log traffic calculation

Log path example

Log example

Log format

Log fields

Internal status codes

S3 export

FTP and SFTP export

Raw Logs deactivation

​Raw Logs settings

​Origin Shielding log inclusion

​Raw Logs statuses

​Export interval

​Log traffic calculation

​Log path example

​Log example

​Log format

​Log fields

​Internal status codes

​S3 export

​FTP and SFTP export

​Raw Logs deactivation

Raw Logs settings

Origin Shielding log inclusion

Raw Logs statuses

Export interval

Log traffic calculation

Log path example

Log example

Log format

Log fields

Internal status codes

S3 export

FTP and SFTP export

Raw Logs deactivation