Using the Google Cloud SDK through an SSL Interceptor on MacOS¶
Source¶
Medium: Using the Google Cloud SDK through an SSL Interceptor on MacOS
What is an SSL Interceptor?¶
In today’s world, companies can be very careful to secure their networks. Instead of just using firewalls that prevent your workstation from communicating to the web except through HTTP/HTTPS, instead of implementing a web proxy, they may install a firewall/interceptor such as Zscaler or FireEye SSL Intercept.
These are specialized firewalls that do more than just permit/deny traffic. For HTTP/HTTPS traffic, they are Layer 5 aware — they look at the request and response for security issues such as Cross-Site Scripting (XSS) attacks. In order to do that with HTTPS, the specialized firewall decrypts the SSL traffic (to examine it), and then re-encrypts it, often with a company specific, self-signed Certificate Authority (CA). This certificate authorities public key and certificate are then distributed to your system’s SSL certificate chain, which your browser will check. Because of this, you may notice that the URL has been rewritten, but within the browser, as long as everything works, you do not care much.
The Command-Line Issue¶
Many popular packages include certificate chains that may not be able to validate the traffic, however, leading command-line tools to fail. It is often hard to diagnose this — if you search on GCP SSL certificates, you will learn a lot about configuring Load Balancers, maybe using Certificate Manager within GCP, etc. You will not learn however learn how to change the certificate chain used by the client software to validate certificates unless you look for that specifically, maybe including the name of the software package involved.
For AWS Client 2, the client software includes the Python packages certifi used by urllib3. Google Cloud SDK uses a different python package, httplib2, within the command line gcloud.
I am lucky in that my employer provides me a MacBook, and so I can use Homebrew to install versions of the software packages necessary. However, when I use these, my workstation is behind our SSL interceptor, and the software cannot validate the certificates of the server. In gcloud, this presents as a time-out followed by a failure, because the command-line retries by default:
ERROR: gcloud crashed (SSLError): HTTPSConnectionPool(host='cloudresourcemanager.googleapis.com', port=443): Max retries exceeded with url: /v1/projects?alt=json&filter=lifecycleState%3AACTIVE&pageSize=500 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
gcloud's default CA certificates failed to verify your connection, which can happen if you are behind a proxy or firewall.
To use a custom CA certificates file, please run the following command:
gcloud config set core/custom_ca_certs_file /path/to/ca_certs
If you would like to report this issue, please run the following command:
gcloud feedback
To check gcloud for common problems, please run the following command:
gcloud info --run-diagnostics
I also have some Python code that uses the Google Calendar API to update my Google Calendar to include my corporate calendar meetings. That code raises a Python exception right away, without any waiting:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
How to fix the issue¶
To fix this issue, the general strategy is:
Export the system certificates into a file in PEM format. Configure the client software to use that file for validation. On MacOS, the certificates are held in 3 KeyChain files — the system root certificates, the system store, and the login chain (which is user specific). Rather than guess where my admins have placed the certificates, I will export all of them in PEM format as follows:
security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain >cacerts.txt
security find-certificate -a -p /Library/Keychains/System.keychain >>cacerts.txt
security find-certificate -a -p ~/Library/Keychains/login.keychain-db >>cacerts.txt
For GCP, I have only to set the configuration to use this file:
However, this is not enough for my code that uses the Google Calendar API, for that I must also export an environment variable that tells the Python library httplib2 about the custom certificate authority file:
With GCP, if you are using your own workstation rather than the Google Cloud Shell, you may also be accessing multiple projects through multiple configurations — remember that this setting is per configuration. Remember to set it, maybe from an environment variable, when you create a new configuration:
gcloud configurations config create my-new-project
gcloud config set project my-new-project-id
gcloud config set core/custom_ca_certs_file $HTTPLIB2_CA_CERTS
What about AWS and Boto3?¶
For some reason, this has not been an issue with boto3, but the boto3 configuration documentation clearly suggests you define the following environment variable: