Use the Accept Header in Your XHRs

An example of a common scenario: You are building a browser-based app for a grocery store. The app shows a list of all the valid coupons this week. Your browser-based app needs to load all of the store’s coupons by requesting the coupon data from the server. The server is capable of returning the coupons in JSON or XML format. Your browser app only knows how to deal with a JSON response and you need to indicate that in your request. You consider several options to indicate the desired response format.

Your first option, and probably the most popular option in use today, is to use a format extension in the request URL.

var xhr = new XMLHttpRequest();
xhr.open('GET', '/coupons.json', true);
xhr.onreadystatechange = function () {
    if (xhr.readyState == 4) {
        if (xhr.status == 200) {
            var data = JSON.parse(xhr.responseText);
            // do something with data
        } else {
            //handle errors
        }
    }
};
xhr.send();

The resulting HTTP request

GET /coupons.json HTTP/1.1
Accept: */*
Host: api.nestorsmarket.com

(Note that the above Accept header says the browser can handle any response format. That’s a lie as you know your app can only handle a JSON response.)

Your second option, which is very similar to the first option, is to use a format parameter in the query string of the request URL.

var xhr = new XMLHttpRequest();
xhr.open('GET', '/coupons?format=json', true);
xhr.onreadystatechange = function () {
    if (xhr.readyState == 4) {
        if (xhr.status == 200) {
            var data = JSON.parse(xhr.responseText);
            // do something with data
        } else {
            //handle errors
        }
    }
};
xhr.send();

The resulting HTTP request

GET /coupons?format=json HTTP/1.1
Accept: */*
Host: api.nestorsmarket.com

(More lying.)

Your third option, and probably the least commonly used option, is to use the Accept header.

var xhr = new XMLHttpRequest();
xhr.open('GET', '/coupons', true);
xhr.setRequestHeader('Accept', 'application/json');
xhr.onreadystatechange = function () {
    if (xhr.readyState == 4) {
        if (xhr.status == 200) {
            var data = JSON.parse(xhr.responseText);
            // do something with data
        } else {
            //handle errors
        }
    }
};
xhr.send();

The resulting HTTP request

GET /coupons HTTP/1.1
Accept: application/json
Host: api.nestorsmarket.com

The first and second options are ugly hacks. These hacks were only ever used because, in the past, the third option didn’t work or server-side programmers did not know how to deal with it. You may be asking, “what is so terribly wrong with the first and second options?”

Intermediary Cache

Imagine you have two types of client apps that want to access the same data. The first client type is a browser-based app that can only handle JSON responses. The second client type is a native iOS app that can handle JSON or XML responses. Also imagine there is an intermediary caching server somewhere between the clients and your web server. A browser-based client makes an XHR request for the JSON format using the format extension technique.

GET /coupons.json HTTP/1.1
Accept: */*
Host: api.nestorsmarket.com

The cache on the intermediary server is empty so the request is passed on to the web server. The web server responds with

HTTP/1.1 200 OK
Cache-Control: max-age=86400
Content-Type: application/json

[
    {
        "upc": 827454737,
        "description": "20% OFF Rib Eye Steak!"
    },
    {
        "upc": 546373635,
        "description": "Save $3.18 on Rocket Fuel french roast coffee!"
    }
]

The caching server sees from the Cache-Control header that this response is cacheable for one more day so it caches it to speed up responses for the remainder of the day. It passes the response back to the client.

Soon after, the iOS app makes a request. It uses the Accept header technique indicating it can handle both data formats.

GET /coupons HTTP/1.1
Accept: application/json, application/xml
Host: api.nestorsmarket.com

When the request arrives to the caching server, the server checks to see if it has a response cached for /coupons. There is nothing matching in the cache. What a pity. There is a match for /coupons.json but that does not match /coupons. The caching server has no idea how to interpret these strings. It checks for exact matches only.

Because the caching server cannot make a match, the caching server must again pass the request to the web server. The web server must again generate the response. The caching server will again cache the response. This wastes end user time, bandwidth, CPU time on the web server, and storage space on the caching server.

Suppose that instead, the browser-based app had originally made this request indicating only one acceptable response format.

GET /coupons HTTP/1.1
Accept: application/json
Host: api.nestorsmarket.com

Then when the iOS app made its request, the caching server would see it has a match in the cache and could respond quickly to the client without bothering the web server.

The intermediary cache is only one compelling example demonstrating that if we use HTTP as it was designed, many things are improved.

Opera Bug

My exploration into this topic started way back in 2008 with the discovery described in the email to the authors of the XMLHttpResponse specification.

From: "Peter Michaux" <petermichaux@gmail.com>
To: public-webapi@w3.org
Date: Wed, 16 Apr 2008 16:49:44 -0700
Subject: XHR setting headers

The XMLHttpRequest spec says "The setRequestHeader() method appends a
value if the HTTP header given as argument is already part of the list
of request headers."

This is fine but what is a problem is whether or not a new
XHMHttpRequest object has any default headers. I was trying to use the
Accept header a few days ago and I wanted to have only

Accept: application/json

but Opera has a default header

Accept: text/html, text/xhtml, etc

so my application/json was appended to the front of that list which
makes my Accept header useless as part of the client-server
communication. The server thinks that the client knows what to do with
text/html. My JavaScript certainly does NOT know what to do with
text/html. My JavaScript only knows how to handle application/json.

I think all XMLHttpRequest headers should be specified as blank when
the object is created. Then the JavaScript can add any headers it
needs to add. If, when the call to send() occurs, some essential
header(s) is missing the XHMLHttpRequest object should add these
automatically but only according to specified behavior.

Peter

To make the problem described in my email above more clear, in Opera 9.27, I would do this.

var xhr = new XMLHttpRequest();
xhr.open('GET', '/coupons', true);
xhr.setRequestHeader('Accept', 'application/json');
xhr.onreadystatechange = function () {
    if (xhr.readyState == 4) {
        if (xhr.status == 200) {
            var data = JSON.parse(xhr.responseText);
            // do something with data
        } else {
            //handle errors
        }
    }
};
xhr.send();

The resulting HTTP request would be this.

GET /coupons HTTP/1.1
Accept: application/json, text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg,
Host: api.nestorsmarket.com

You can download old versions of Opera to try for yourself.

The final email in the thread of discussion was from the author of the XMLHttpRequest specification (who’s email address happened to end in opera.com.)

From: "Anne van Kesteren" <annevk@opera.com>
To: "Julian Reschke" <xxxx.xxxx@xxx.de>
Cc: "Peter Michaux" <petermichaux@gmail.com>, public-webapi@w3.org
Date: Tue, 27 May 2008 14:14:02 +0200
Subject: Re: XHR setting headers

On Tue, 13 May 2008 09:24:42 +0200, Julian Reschke <xxxx.xxxx@xxx.de> wrote:
> Anne van Kesteren wrote:
> > On Mon, 12 May 2008 17:24:16 +0200, Julian Reschke <xxxx.xxxx@xxx.de> wrote:
> > > Well, we just heard from people complaining about XHR implementations  
> > > pre-filling request headers, and thus causing clients to create broken  
> > > content-type headers (because of the append functionality).
> >
> > Actually, that was specifically about Accept and it seems to be simply  
> > an Opera bug per the specification.
>
> So does the specification say which headers may be prefilled by the  
> implementation, and when?

It is now made clear that user agents can only control the headers that  
authors can not set and can control a limited set of headers if the author  
has not set them (e.g. the specification allows the author to control  
caching himself, but if he does not do that the user agent can do it).

You can read the whole email thread in the mailing list archives.

The related parts of the XMLHttpRequest specification are in The setRequestHeader() method and The send() method sections where the "author request headers" are discussed.

I was happy with the outcome. The specification was improved. Of course, at that time, it was still 2008 and the buggy versions of Opera were still out there in the wild being used. I thought, "Great! In 10 years I can start using the Accept header!!" Well, it is almost 8 years now and I think it is time to start.

The first version of Opera to ship with this bug fixed was Opera 10.50 beta 1 which was released on February 11, 2010. That is only 6 years ago but Opera has released new versions and has fallen in popularity. For one site I monitor, in the year prior to the release of Opera 10.50 beta 1, 1 out of 125 sessions was from the buggy versions of Opera. Now, only 1 out of 17241 sessions comes from the old buggy versions of Opera. Low enough for me.

I’m not aware of any other browser that has had this bug or any other bug preventing the proper use of the Accept header...and I’ve been looking. All of the following browsers worked.

Operating SystemBrowser
Windows XP SP2Internet Explorer 6.0.2900.2180.xpsp_sp2_rtm.040803-2158
Internet Explorer 7.0.5720.13
Firefox 42.0
Chrome 48.0.2564.11 m
Opera 10.50 Beta 1
Windows 7Internet Explorer 8.0.7600.16385
Firefox 14.0.1
Chrome 47.0.2526.111 m
Internet Explorer 9.0.8112.16421
Firefox 35.0.1
Windows 10Internet Explorer 11.63.10586.0
OS X 10.4.11Safari 4.0.5 (4531.22.7)
Firefox 3.6.28
OS X 10.11.3Firefox 44.0.2
Chrome 48.0.2564.116
Safari 9.0.3 (11601.4.4)
Opera 10.50 Beta 1
Opera 11.11
Opera 35.0.2066.68
Debian 7.4Iceweasel 17.0.10
Ubuntu 15.10Firefox 44.0.2
iOS 6.1.6Safari 6.0
iOS 9.2.1Safari 9.0
Chrome 48.0.2564.87
Android 4.4.2Internet 4.4.2.I747MVLUFFOB3
Chrome 47.0.2526.83
Windows Phone 8.1Internet Explorer Mobile 11.0

Even good-old Internet Explorer 6 allowed successful control and setting of the Accept header in an XHR!

Conclusion

In the browser world, it takes a long time for old bugs to go away. I think it is now time to ditch the format hacks and take advantage of the proper use of HTTP Accept header.

Do you already use the Accept for your XHR requests from browser-based apps?

Have you tried to use Accept for your XHR requests from browser-based apps and had problems?

I’d like to know about your experiences.

Comments

Have something to write? Comment on this article.