• Stars
    star
    140
  • Rank 261,473 (Top 6 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 6 years ago
  • Updated 12 days ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Tool to improve AMP URLs via Signed Exchanges

AMP Packager

AMP Packager is a tool to improve AMP URLs by serving AMP using Signed Exchanges. By running it in a proper configuration, web publishers enable origin URLs to appear in AMP search results.

The AMP Packager works by creating Signed HTTP Exchanges (SXGs) containing AMP documents, signed with a certificate associated with the origin, with a maximum lifetime of 7 days. The Google AMP Cache will fetch, cache, and serve them, similar to what it does for normal AMP HTML documents. When a user loads such an SXG, Chrome validates the signature and then displays the certificate's domain in the URL bar instead of google.com, and treats the web page as though it were on that domain.

The packager is an HTTP server that sits behind a frontend server; it fetches and signs AMP documents as requested by the AMP Cache.

As an alternative to running the packager, you can sign up for one of the SXG service providers.

Packager/Signer

How to use

In all the instructions below, replace amppackageexample.com with a domain you own and can obtain certificates for.

Development server

Manual installation
  1. Install Go version 1.13 or higher.

  2. Get amppackager.

    git clone https://github.com/ampproject/amppackager.git my-amp-directory
    cd my-amp-directory
    make build
    
  3. Optionally, move the built amppkg wherever you like.

  4. Prepare a temporary certificate and private key pair to use for signing the exchange when testing your config. Follow WICG instructions to ensure compliance with the WICG certificate requirements.

  5. Create a file amppkg.toml. A minimal config looks like this:

    LocalOnly = true
    CertFile = 'path/to/fullchain.pem'
    KeyFile = 'path/to/privkey.pem'
    OCSPCache = '/tmp/amppkg-ocsp'
    
    [[URLSet]]
      [URLSet.Sign]
        Domain = "amppackageexample.com"
    

    More details can be found in amppkg.example.toml.

  6. amppkg -development

    If amppkg.toml is not in the current working directory, pass -config=/path/to/amppkg.toml.

Docker

Follow the instructions here on how to deploy a local Docker container.

Test your config

  1. Run Chrome with the following command line flags:
    alias chrome = [FULL PATH TO CHROME BINARY]
    PATH_TO_FULLCHAIN_PEM = [FULL PATH TO fullchain.pem]
    chrome --user-data-dir=/tmp/udd\
        --ignore-certificate-errors-spki-list=$(\
           openssl x509 -pubkey -noout -in $PATH_TO_FULLCHAIN_PEM |\
           openssl pkey -pubin -outform der |\
           openssl dgst -sha256 -binary | base64)\
        --enable-features=SignedHTTPExchange\
           'data:text/html,<a href="https://localhost:8080/priv/doc/https://amppackageexample.com/">click me'
    
  2. Open DevTools. Check 'Preserve log'.
  3. Click the click me link.
  4. Watch the URL transmogrify! Verify it came from an SXG by switching DevTools to the Network tab and looking in the Size column for (from signed-exchange) and in the Type column for signed-exchange. Click on that row and then click on the Preview tab, to see if there are any errors.

Demonstrate privacy-preserving prefetch

This step is optional; just to show how privacy-preserving prefetch works with SXGs.

  1. go get -u github.com/ampproject/amppackager/cmd/amppkg_dl_sxg.
  2. amppkg_dl_sxg https://localhost:8080/priv/doc/https://amppackageexample.com/
  3. Stop amppkg with Ctrl-C.
  4. go get -u github.com/ampproject/amppackager/cmd/amppkg_test_cache.
  5. amppkg_test_cache
  6. Open Chrome and DevTools, as above.
  7. Visit https://localhost:8000/. Observe the prefetch of /test.sxg.
  8. Click the link. Observe that the cached SXG is used.

Productionizing

For now, productionizing is a bit manual. The minimum steps are:

  1. Don't pass -development flag to amppkg. This causes it to serve HTTP rather than HTTPS, among other changes.

  2. Don't expose amppkg to the outside world; keep it on your internal network.

  3. Configure your TLS-serving frontend server to conditionally proxy to amppkg:

    1. If the URL starts with /amppkg/, forward the request unmodified.

    2. If the URL points to an AMP page and the AMP-Cache-Transform request header is present, rewrite the URL by prepending /priv/doc and forward the request.

      NOTE: If using nginx, prefer using proxy_pass with $request_uri, rather than using rewrite, as in this PR, to avoid percent-encoding issues.

    3. If at all possible, don't send URLs of non-AMP pages to amppkg; its transforms may break non-AMP HTML.

    4. DO NOT forward /priv/doc requests; these URLs are meant to be generated by the frontend server only.

  4. For HTTP compliance, ensure the Vary header set to AMP-Cache-Transform, Accept for all URLs that point to an AMP page, irrespective of whether the response is HTML or SXG. (SXG responses that come from amppkg will have the appropriate Vary header set, so it may only be necessary to explicitly set the Vary header for HTML responses.)

  5. Get an SXG cert from your CA. It must use an EC key with the prime256v1 algorithm, and it must have a CanSignHttpExchanges extension. One provider of SXG certs is DigiCert. You MUST use this in amppkg.toml, and MUST NOT use it in your frontend.

  6. Every 90 days or sooner, renew your SXG cert (per WICG/webpackage#383) and restart amppkg (per #93).

  7. Keep amppkg updated from releases (the default branch, so go get works) about every ~2 months. The wg-caching team will release a new version approximately this often. Soon after each release, Googlebot will increment the version it requests with AMP-Cache-Transform. Googlebot will only allow the latest 2-3 versions (details are still TBD), so an update is necessary but not immediately. If amppkg doesn't support the requested version range, it will fall back to serving unsigned AMP.

    To keep subscribed to releases, you can select "Releases only" from the "Watch" dropdown in GitHub, or use various tools to subscribe to the releases branch.

You may also want to:

  1. Launch amppkg as a restricted user.
  2. Save its stdout to a rotated log somewhere.
  3. Use the provided tools to verify that your published AMP documents are valid, for instance just before publication, or with a regular audit of a sample of documents. The transforms are designed to work on valid AMP pages, and may break invalid AMP in small ways.
  4. Setup monitoring of amppackager and related requests to AMP document server.

Once you've done the above, you should be able to test by launching Chrome without any command line flags. To test by visiting the packager URL directly, first add a Chrome extension to send an AMP-Cache-Transform: any request header. Otherwise, follow the above "Demonstrate privacy-preserving prefetch" instructions.

Security Considerations

Signed exchanges come with some security considerations that publishers should consider. A starting list of recommendations based on that:

  • Use different keys for the signed exchange cert and the TLS cert.
  • Only sign public content that's OK to be shared with crawlers.
  • Don't sign personalized content. (It's OK to sign content that includes static JS that adds personalization at runtime.)
  • Be careful when signing inline JS; if it includes a vulnerability, it may be possible for attackers to exploit it without intercepting the network path, for up to 7 days.

Testing productionization without a valid certificate

It is possible to test an otherwise fully production configuration without obtaining a certificate with the CanSignHttpExchanges extension. amppkg still needs to perform OCSP verification, so the Issuer CA must be valid (i.e. no self-signed certificates). e.g. You can use a certificate from Let's Encrypt.

Running amppkg with the -invalidcert flag will skip the check for CanSignHttpExchanges. This flag is not necessary when using the -development flag.

Chrome can be configured to allow these invalid certificates with the --ignore-certificate-errors-spki-list command line flag:

google-chrome --ignore-certificate-errors-spki-list=<hashes> --user-data-dir=<dir>

where <hashes> is a comma separated list of Base64-encoded SHA-256 SPKI Fingerprints and it is necessary to specify --user-data-dir with a valid or creatable directory <dir> when --ignore-certificate-errors-spki-list is used.

As an example, the hash for a PEM certificate can be obtained with OpenSSL:

openssl x509 -pubkey -noout -in mycert.crt | openssl pkey -pubin -outform der | openssl sha256 -binary | openssl base64

Redundancy

If you need to load balance across multiple instances of amppkg, you'll want your OCSPCache to be backed by a shared storage device (e.g. NFS). It doesn't need to be shared among all instances globally, but perhaps among all instances per datacenter. The reason for this is to reduce the number of OCSP requests amppkg needs to make, per OCSP stapling recommendations.

How will these web packages be discovered by Google?

Googlebot makes requests with an AMP-Cache-Transform header. Responses that are acceptable AMP SXGs will be eligible for display to SXG-supporting browsers, and the HTML payload will be extracted and eligible for use in the AMP viewer in other browsers.

Limitations

Currently, the packager will refuse to sign any AMP documents that hit the size limit of 4MB. You can monitor the size of your documents that have been signed, to see how close you are to the limit.

The packager refuses to sign any URL that results in a redirect. This is by design, as neither the original URL nor the final URL makes sense as the signed URL.

To account for possible clock skew in user agents, the packager back-dates packages by 24h, which means they effectively last only 6 days for most users.

This tool only packages AMP documents. To sign non-AMP documents, look at the commandline tools on which this was based, at https://github.com/WICG/webpackage/tree/master/go/signedexchange.

<amp-install-serviceworker> will fail inside of a signed exchange, due to a Chrome limitation. The recommendation is to ignore the console error, for now. This is because amp-install-serviceworker will still succeed in the unsigned AMP viewer case, and crawlers may reuse the contents of the signed exchange when displaying an AMP viewer to browser versions that don't support SXG.

<amp-script>

If you have any inline <amp-script>s (those with a script attribute), then the expiration of the SXG will be set based on the minimum max-age of those <amp-script>s, minus one day (due to backdating). If possible, prefer external <amp-script>s (those with a src attribute), which do not have this limitation.

If inline is necessary, you will need to weigh the security risks against the AMP Cache requirement for a minimum max-age of 345600 (4 days). For SXGs shorter than that, the Google AMP Cache will treat them as if unsigned (by showing an AMP Viewer).

How does amppackager process a document it cannot sign?

Packager will respond to every request with either a signed document, an unsigned document, or an error.

It will sign every document it can. It may, however, decide not to, for a number of reasons: the certificate may be invalid, the page may not be a valid AMP page, the page may not be an AMP page at all, the page may be 4MB or larger, etc.

If packager cannot sign the document but can fetch it, it will proxy the document unsigned.

If there was a problem with the gateway fetch request, or with the original request, packager will respond with an HTTP error, and log the problem to stdout.

You can monitor the packager's error rates, as well as the rates of signed vs unsigned documents, via the tools discussed in the next section.

Specifically, you can monitor the requests that resulted in a signed or an unsigned document via amppackager_signer_documents_total metric, and the ones that resulted in an error - via amppackager_http_duration_seconds_count metric.

Monitoring amppackager in production via its Prometheus endpoints

Once you've run the amppackager server in production, you may want to monitor its health and performance. You may also monitor the performance of the underlying requests to the AMP document server. You can monitor both servers via the Prometheus endpoints provided by amppackager. A few examples of questions you can answer:

  • Is amppackager up and running?
  • How many requests has it processed since it's been up?
  • What was the 0.9 percentile latency of handling those request?
  • How many of those requests have triggered a gateway request to the AMP document server?
  • For those gateway requests, what was the 0.9 percentile latency of the AMP document server?

You can perform one-off manual health inspections, visualize the real-time stats, set up alerts, and more. To learn what are all the things you can monitor, and how to do it, check the monitoring manual.

Local Transformer

The local transformer is a library within the AMP Packager that transforms AMP HTML for security and performance improvements. Ports of or alternatives to the AMP Packager will need to include these transforms.

More info here.

More Repositories

1

amphtml

The AMP web component framework.
JavaScript
14,887
star
2

worker-dom

The same DOM API and Frameworks you know, but in a Web Worker.
TypeScript
3,206
star
3

amp-wp

Enable AMP on your WordPress site, the WordPress way.
PHP
1,790
star
4

amp-by-example

DEPRECATED: AMP by Example has been merged into amp.dev
HTML
752
star
5

amp.dev

The AMP Project Website.
HTML
584
star
6

amp-toolbox

A collection of AMP tools making it easier to publish and host AMP pages.
HTML
450
star
7

samples

HTML
444
star
8

ampstart

AMP Start source code and templates .
HTML
419
star
9

rollup-plugin-closure-compiler

Leverage Closure Compiler to minify and optimize JavaScript with Rollup.
TypeScript
292
star
10

remapping

Remap sequential sourcemaps through transformations to point at the original source code
TypeScript
103
star
11

meta

Information about the AMP open source project.
78
star
12

filesize

Monitor the size of files in your project specified within package.json.
TypeScript
75
star
13

amp-toolbox-php

AMP Optimizer PHP library
PHP
73
star
14

amp-sw

A drop in service worker library to help your AMP pages gain network resiliency in 1 line
JavaScript
70
star
15

ampbench

AMPBench: AMP URL validation and troubleshooting tools (DEPRECATED)
JavaScript
66
star
16

wg-amp4email

Responsible for the AMP4Email project. Facilitator: @nainar
56
star
17

bentojs.dev

Bento Website
SCSS
49
star
18

eleventy-plugin-amp

Quickly build interactive websites with Eleventy & AMP.
JavaScript
46
star
19

bento

An easy to use component library that helps you achieve a great page experience
39
star
20

amp-viewer

Objective-C
38
star
21

amp-react-prototype

A scratch pad to experiment with React rendered AMP Components
JavaScript
36
star
22

amp-github-apps

GitHub Apps for the AMP Project
TypeScript
34
star
23

wg-stories

Responsible for implementing and improving AMP's story format (amp-story). Facilitator: @newmuis
28
star
24

meta-ac

The AMP Advisory Committee
25
star
25

animations

TypeScript
23
star
26

cloudflare-amp-optimizer

Implementation of AMP Optimizer for Cloudflare Workers
JavaScript
22
star
27

design-docs

Design docs contributed to AMP.
20
star
28

wg-bento

JavaScript
17
star
29

amp-react

JavaScript
17
star
30

meta-tsc

The AMP Technical Steering Committee
16
star
31

amp-email-viewer

TypeScript
16
star
32

wg-outreach

Responsible for helping developers who use AMP remain productive and keeping the AMP community healthy. This includes maintaining and enhancing AMP's developer-facing sites and documentation, maintaining communication channels, organizing community events, etc. Facilitator: @sebastianbenz
JavaScript
15
star
33

amp-readiness

AMP Readiness is a chrome extension allows you to quickly see what technologies used on the page are AMP compatible.
JavaScript
14
star
34

error-tracker

AMP Project's error logging server
JavaScript
13
star
35

wg-components

Responsible for AMP components, the overall health of AMP Pages, analytics features, and integrations with partner technologies. Facilitator: @alanorozco
11
star
36

wg-access-subscriptions

Responsible for user specific controlled access to AMP content (amp-subscriptions, amp-access and related extensions) Facilitator: @jpettitt Slack: #wg-access-subs
11
star
37

cdn-worker

CDN worker source code
TypeScript
10
star
38

amphtml-build-artifacts

This repository contains build artifacts related to the ampproject/amphtml project
10
star
39

bento-compiler

Server-render AMP Components with worker-dom
HTML
9
star
40

validator-java

A validator for the AMP HTML format implemented in Java
Java
9
star
41

wg-monetization

Responsible for monetization features and integrations in AMP. Facilitator: @powerivq
8
star
42

wg-caching

Responsible for AMP's validator and features related to AMP caches. Facilitator: @Gregable
8
star
43

px-toolbox-php

PHP
8
star
44

storybook-addon-amp

The storybook AMP addon
TypeScript
8
star
45

cdn-configuration

Configuration settings for AMP Project CDNs
TypeScript
7
star
46

wg-performance

Monitoring and improving AMP's load and runtime performance for compliant documents. Facilitator: @erwinmombay
7
star
47

amp-status

HTML
7
star
48

amp-wp-qa-tester

Easily test pre-release versions of the AMP Plugin for WordPress.
PHP
6
star
49

wg-analytics

Deprecated Working Group, previously responsible for analytics features.
6
star
50

wg-viewers

Responsible for ensuring support for AMP viewers and for the amp-viewer project. Facilitator: @gmajoulet
5
star
51

amp-closure-compiler

JavaScript
5
star
52

wg-runtime

Responsible for AMP's core runtime, such as layout/rendering and data binding. Facilitator: @jridgewell
4
star
53

wg-infra

Responsible for AMP's infrastructure, including building, testing and release. Facilitator: @danielrozenberg
4
star
54

wg-codeofconduct

Responsible for enforcing AMP's Code of Conduct. (The Technical Steering Committee delegates its responsibility for enforcement of the Code of Conduct to this Working Group.) Facilitator: @nainar
3
star
55

npw

A workspace-aware npm wrapper to aid with developing in monorepos
JavaScript
3
star
56

wg-approvers

Responsible for approving changes that have a significant impact on AMP's behavior or significant new features as described in the feature and bug fix process. Facilitator: @dvoytenko
3
star
57

wg-foundation-onboarding

Responsible for work/coordination related to AMP completing the OpenJS Foundation onboarding
3
star
58

error-reporting

Contains production error tracking issues.
2
star
59

wg-security-privacy

2
star