Issue Description
On iOS and MacOS Safari fails to load a site over HTTPS served by NGINX acting as a reverse proxy in front of Apache. Safari can’t open the page displaying the error: “The operation couldn’t be completed. Protocol error” (NSPOSIXErrorDomain:100)
Explanation
Here’s whats happening:
sequenceDiagram
participant safari
participant nginx
safari->>nginx: I speak h1, h1.1, h2
nginx-->>safari: Great, let's speak h2
safari->>nginx: GET index.html
nginx->>apache: I speak h1, h1.1, h2
apache-->>nginx: Great, let's speak h2c
Note left of apache: Upgrade: h2c
nginx-->>safari: here's index.html but I was told to upgrade the connection
safari->>safari: Error: cannot upgrade bc we speak h2 already
Note left of safari: Safari: Abort!
Nginx when installed as a reverse proxy with Apache as a back-end fetches resources from Apache using HTTP/1.1, which the back-end server tries to upgrade to HTTP/2 by sending the “Upgrade: h2c” header:
Upgrade: h2, h2c
By default Apache allows the following HTTP protocol versions:
Protocols h2 h2c http/1.1
Nginx is transmitting the header Upgrade from Apache to a client, i.e. browser. And browsers on iOS (on iPhone) and on macOS High Sierra from Apple might fail here and drop a connection to such a site.
And that is simply the result of following the HTTP/2 specification. It seems other browsers are more lenient with this issue and silently drop the forbidden header fields:
An endpoint MUST NOT generate an HTTP/2 message containing connection-specific header fields; any message containing connection-specific header fields MUST be treated as malformed (Section 8.1.2.6)…. connection- specific header fields, such as Keep-Alive, Proxy-Connection, Transfer-Encoding, and Upgrade
Source: https://httpwg.org/specs/rfc7540.html#rfc.section.8.1.2.2
Given that all major browsers do not support HTTP/2 without TLS anyway and that no Upgrade header is allowed for HTTP/2 over TLS the solution here is to remove the header from the nginx response to the client, so implement one of the following options:
Solution 1) nginx
proxy_hide_header Upgrade;
Solution 2) Apache
Header unset Upgrade
Troubleshooting steps
Update curl
When I initially started troubleshooting this issue I quickly realized that my curl version was too old and did not support HTTP/2. Browsing the web I found the following suggestion:
brew reinstall curl --with-openssl --with-nghttp2
This command fails because these parameters have since been removed. The correct command is:
brew install curl-openssl
Check nginx
Next I queried the affected site and curl was also issuing an error just like Safari:
curl -v --http2 --head https://dev.example.com
* Trying 4.122.230.187:443...
* TCP_NODELAY set
* Connected to dev.example.com (4.122.230.187) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /usr/local/etc/openssl/cert.pem
CApath: /usr/local/etc/openssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: OU=Domain Control Validated; OU=PositiveSSL Wildcard; CN=*.example.com
* start date: Mar 25 00:00:00 2019 GMT
* expire date: May 23 23:59:59 2020 GMT
* subjectAltName: host "dev.example.com" matched cert's "*.example.com"
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fa515005600)
> HEAD / HTTP/2
> Host: dev.example.com
> User-Agent: curl/7.65.3
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [upgrade], value: [h2,h2c]
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host dev.example.com left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
Check Apache
My next step was to open an SSH tunnel to my back-end Apache and test its response headers:
ssh -L 9000:localhost:8080 dev-web1b
curl -v --http2 http://localhost:9000
* Trying ::1:9000...
* TCP_NODELAY set
* Connected to localhost (::1) port 9000 (#0)
> GET / HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.65.3
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Thu, 22 Aug 2019 09:35:03 GMT
< Server: Apache/2.4.39 ()
< Last-Modified: Thu, 25 Jul 2019 12:45:22 GMT
< ETag: "910-58e80cc8a0dee"
< Accept-Ranges: bytes
< Content-Length: 2320
< Vary: Accept-Encoding,User-Agent
< Cache-Control: public
< Content-Type: text/html; charset=UTF-8
And indeed the Apache was sending the upgrade header that nginx is forwarding to the clients!
Validate the solution
I applied the header fix and tested again:
curl -v --http2 --head https://dev.example.com
* Trying 4.122.230.187:443...
* TCP_NODELAY set
* Connected to dev.example.com (4.122.230.187) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /usr/local/etc/openssl/cert.pem
CApath: /usr/local/etc/openssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: OU=Domain Control Validated; OU=PositiveSSL Wildcard; CN=*.example.com
* start date: Mar 25 00:00:00 2019 GMT
* expire date: May 23 23:59:59 2020 GMT
* subjectAltName: host "dev.example.com" matched cert's "*.example.com"
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fcc6f80b200)
> HEAD / HTTP/2
> Host: dev.example.com
> User-Agent: curl/7.65.3
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
HTTP/2 200
< server: nginx/1.17.3
server: nginx/1.17.3
< date: Sat, 24 Aug 2019 07:38:19 GMT
date: Sat, 24 Aug 2019 07:38:19 GMT
< content-type: text/html; charset=UTF-8
content-type: text/html; charset=UTF-8
< content-length: 2320
content-length: 2320
< last-modified: Thu, 25 Jul 2019 12:45:22 GMT
last-modified: Thu, 25 Jul 2019 12:45:22 GMT
< etag: "910-58e80cc8a0dee"
etag: "910-58e80cc8a0dee"
< accept-ranges: bytes
accept-ranges: bytes
< vary: Accept-Encoding,User-Agent
vary: Accept-Encoding,User-Agent
< cache-control: public
cache-control: public
<
* Connection #0 to host dev.example.com left intact
It’s fixed, everything works as expected. This confirmed my suspicion that the Upgrade header was indeed causing the issues.
Analysing and documenting the issue cost me quite some time because Safari masked the actual error and my old curl version only used HTTP/1.1 which completely bypassed the conditions that cause problems for clients in the first place.