Bugzilla – Bug 9686
network operations should use accept-encoding when appropriate
Last modified: 2009-07-01 16:40:13 UTC
You need to log in before you can comment on or make changes to this bug.
To decrease the size of transfer by pkg(5) clients, network operations should use accept-encoding with gzip, deflate where appropriate (such as 'manifest', 'catalog', etc. but not where data is already compressed such as 'file'). This change could result in significant bandwidth savings for clients. For example, the packagemanager gui when pointed at en empty user image using a copy of a build 101 repository will transfer this many bytes at startup (during the background as it is retrieving all manifests): 65,367,147 A simple addition to the new transport engine prototype code in src/modules/client/transport/engine.py:__setup_handle: hdl.setopt(pycurl.ENCODING, "") ...and a small change in src/depot.py: "tools.gzip.on": True (after tools.encode.on) brings this number down to: 18,399,927 That's about a 72% reduction in the amount of bytes transferred! This RFE will only be for the transport change. A separate RFE will be opened for the depot server.
I'll leave it to j to decide if this should be dependent on bug 9687, as that will have to be done for clients to benefit, although clients with the change described in this RFE should experience no issues connecting to an older depot server.
Thanks for filing this, Shawn. I don't think that we need to depend upon 9687, since the accept-encoding that libcurl sends to the server always allows for it to fallback to identity encoding. In the e-mail you sent me off-list, this was the header libcurl was purported to be sending "Accept-Encoding: gzip, deflate". According to RFC2616, this implicitly includes identity and should be fine for the cases where we want to get compressed data. http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html (see 14.3)
The latest transport webrev contains support for this option: http://cr.opensolaris.org/~johansen/webrev-xport-2/
Integrated 1Jul2009 as change set a48bee2a4b2e9c8345c29acea63116acf77dddb3