Fastly
Fastly's Amazon S3 logging endpoint writes log files to an S3 bucket managed by EINBLIQ.IO, as with the other CDNs.
Updated
Values provided by EINBLIQ.IO
| Value | Used for |
|---|---|
<BUCKET_NAME> |
S3 bucket |
<S3_ENDPOINT> |
Host name of the S3 storage endpoint |
<BUCKET_PREFIX> |
Path for your files inside the bucket, ending with / |
<ACCESS_KEY_ID> |
Access key ID for the bucket |
<SECRET_ACCESS_KEY> |
Secret access key |
The bucket can be hosted on Amazon S3 or on other S3-compatible storage. Fastly writes to both through its Amazon S3 endpoint type; the domain provided by EINBLIQ.IO points it to the right storage.
Prerequisites
- Permission to edit and activate the configuration of each Fastly service you want to include.
Create the logging endpoint
- Open the service, select Edit configuration and clone the active version.
- Go to Logging and select Amazon S3 from the Add endpoint menu.
- Fill in the form with the values below and paste the log format.
- Open Advanced options and set the remaining values.
- Select Create, then activate the new version with Activate on Production.
| Setting | Value |
|---|---|
| Name | EINBLIQ.IO |
| Placement | Format Version Default |
| Log format | The format below |
| Timestamp format | Leave the default |
| Bucket name | <BUCKET_NAME> |
| Domain | <S3_ENDPOINT> |
| Access method | User credentials |
| Access key | <ACCESS_KEY_ID> |
| Secret key | <SECRET_ACCESS_KEY> |
| Period | 30 seconds |
Under Advanced options:
| Setting | Value |
|---|---|
| Path | <BUCKET_PREFIX> |
| Select a log line format | Blank |
| Compression | Gzip |
Leave Redundancy level, PGP public key, ACL and Server side encryption at their defaults.
The period sets how often Fastly finalises a file and starts a new one. Fastly’s default is 3,600 seconds; EINBLIQ.IO uses 30 seconds in its own Fastly setup, so new data becomes available sooner. Blank writes each log record without a syslog prefix, so every record stays valid JSON.
Log format
Paste this format into the Log format field:
{
"timestamp": "%{strftime(\{"%Y-%m-%dT%H:%M:%S%z"\}, time.start)}V",
"timestamp_ms": %{time.start.msec}V,
"client_ip": "%{req.http.Fastly-Client-IP}V",
"geo_country": "%{client.geo.country_name}V",
"geo_city": "%{client.geo.city}V",
"host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V",
"url": "%{json.escape(req.url)}V",
"url_path": "%{json.escape(req.url.path)}V",
"query_string": "%{json.escape(req.url.qs)}V",
"request_method": "%{json.escape(req.method)}V",
"request_protocol": "%{json.escape(req.proto)}V",
"request_referer": "%{json.escape(req.http.referer)}V",
"request_user_agent": "%{json.escape(req.http.User-Agent)}V",
"cmcd_object": "%{json.escape(req.http.CMCD-Object)}V",
"cmcd_request": "%{json.escape(req.http.CMCD-Request)}V",
"cmcd_session": "%{json.escape(req.http.CMCD-Session)}V",
"cmcd_status": "%{json.escape(req.http.CMCD-Status)}V",
"response_state": "%{json.escape(fastly_info.state)}V",
"response_status": %{resp.status}V,
"response_reason": %{if(resp.response, "%22"+json.escape(resp.response)+"%22", "null")}V,
"response_body_size": %{resp.body_bytes_written}V,
"response_completed": %{if(resp.completed, "true", "false")}V,
"response_locally_generated": %{if(resp.is_locally_generated, "true", "false")}V,
"req_restarts": %{req.restarts}V,
"duration_usec": %{time.elapsed.usec}V,
"server_ip": "%{server.ip}V",
"fastly_server": "%{json.escape(server.identity)}V",
"fastly_is_edge": %{if(fastly.ff.visits_this_service == 0, "true", "false")}V
}The format contains all required log fields. It records CMCD from the query string and from the four CMCD headers, so header-based players such as Apple AVPlayer are covered without further changes. fastly_is_edge separates edge requests from shield requests when you use shielding.
Keep CMCD out of the cache key
Fastly’s default cache key is the URL, including its query string, and the host. A CMCD parameter therefore creates a separate cache entry for almost every request. Set the cache key in a request setting rather than rewriting the URL: CMCD then stays in the logged query string and still reaches your origin.
- In the service configuration, open Settings and select Create request setting under Request settings.
- Open Advanced options and enter one of the cache keys below.
- Create the setting and activate the version.
| Cache keys | Effect |
|---|---|
querystring.filter(req.url, "CMCD"), req.http.host |
Removes only CMCD from the cache key |
req.url.path, req.http.host |
Removes the whole query string from the cache key |
Check the delivery
Fastly streams log lines continuously, but a file appears in the bucket only once its upload is complete, at the end of the first period. When the new version is active, inform EINBLIQ.IO. The delivery check then confirms that the files arrive and contain CMCD for your players.
Fastly documentation
- Log streaming: Amazon S3 describes every field of the endpoint form.
- Manipulating the cache key explains cache keys in request settings.
- Changing log line formats explains the Blank, Classic, Loggly and Logplex formats.
- Changing log compression options lists the compression codecs.