Web Services + JSON = Dump Your Proxy

In my post on "How to build a Maps Mash-up" I mentioned that there are different ways to overcome the browser security restrictions to retrieve data from another domain (cross-domain restriction). The previous sample used the software proxy method to make the Web service requests and this post talks about a way to make a request without a proxy. It’s the dynamic script tag method.

Today Yahoo! added a new output option for part of their Web services called JSON. This makes it possible to make the JavaScript WS request without using the XMLHTTPRequest object. It is a great way to pull data from another domain because you can dump your proxy and all the data will not route through your server anymore. I will talk about the pros and cons of both these approaches later, but first I want to give an overview of what JSON is, how it works and show some sample code.

What is JSON?

On Doug Crockford’s page it reads like that: "JSON (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate." And that’s how it look like:

{"ResultSet":{"Result":[{"precision":"city","Latitude":"34.05217","Longitude":"-118.243469","Address":"","City":"LOS ANGELES","State":"CA","Zip":"","Country":"US"}]}}}

The string above is returned by Y! Geocoder for the query “LA”. JSON is a serialized JavaScript object, which JavaScript can turn back into an object. For Yahoo! WS the structure of the JSON string is similar to the XML result but the difference between and attribute and element can’t be made. The following is a comparison of the XML result for the same call.

<ResultSet ... >
  <Result precision="city">
    <Latitude>34.05217</Latitude>
    <Longitude>-118.243469</Longitude>
    <Address></Address>
    <City>LOS ANGELES</City>
    <State>CA</State>
    <Zip></Zip>
    <Country>US</Country>
  </Result>
</ResultSet>

One way to get from JSON to a JavaScript object is to call eval(), with the string as argument. The following sample uses the Geocoder result to display LA’s Latitude and Longitude in an alert box. This is just static.

eval.html
<HTML>
  <BODY>
    <script language"javascript">
      var location = eval({"ResultSet":{"Result":[{"precision":"city","Latitude":"34.05217","Longitude":"-118.243469","Address":"","City":"LOS ANGELES","State":"CA","Zip":"","Country":"US"}]}});
      alert("Lat:" + location.ResultSet.Result[0].Latitude + " Lon: " + location.ResultSet.Result[0].Longitude );
    </script>
  </Body>
</HTML>

This is nice but doesn’t do too much in the real world. The problem was to get the data from a Web service that is located on another domain imported without using a proxy.

The secret sauce

Adding the <Script> tag dynamically in the DOM tree of the browser is the answer and the JSON response helps to get the data in a format that is easy to digest for JavaScript. When a Script tags gets dynamically added to the DOM tree the code (script URL) gets executed on the fly. The trick is that instead pointing to a JavaScript library, we include a Web service request in the tag that returns data in the above mentioned format. The Yahoo! Web services that offer the JSON output option also supports a parameter called ‘callback’ and all it does is wrap the return data in a function with the name of the callback value. http://api.local.yahoo.com/MapsService/V1/geocode?appid=dantheurer&location=la&output=json&callback=getLocation would result in something like this getLocation({"ResultSet":{"Result":[{"precision":"city",....) which tries to call the getLocation function (callback) that needs to be implemented to deal with the data.

Below is a sample that takes a location as an input parameter, then calls the Y! Geocoder WS and displays Long / Lat in the page.

geocodeJson.html
<script type="text/javascript" src="jsr_class.js"></script>
<script type="text/javascript">
  //<![CDATA[
  var appid = "dantheurer";

  //That is the callback function that is specified in the request url and gets executed after the data is returned
  function getLocation(jData) {
    if (jData == null) {
      alert("There was a problem parsing search results.");
      return;
    }
    //get the values out of the object
    var lat = jData.ResultSet.Result[0].Latitude;
    var lon = jData.ResultSet.Result[0].Longitude;
    //build some html
    var smart = "Long: " + lon + "<br />" + "Lat: " + lat;
    //add it in the page DOM
    document.getElementById(‘result’).innerHTML = smart;
  }

  function addGeocode() {
    var location = document.getElementById("geoquery").value;
    // Build the Yahoo! web services call
    request = 'http://api.local.yahoo.com/MapsService/V1/geocode?appid=' + appid + '&location=' + location + '&output=json&callback=getLocation';
    // Create a new script object
    aObj = new JSONscriptRequest(request);
    // Build the script tag
    aObj.buildScriptTag();
    // Execute (add) the script tag
    aObj.addScriptTag();
  }
  //]]>
</script>

Jason, my cube neighbor, wrote a really nice class that deals with the dirty work of adding, removing and making sure the tags are unique. If you open up the file, it even has a security warning from the inventor of JSON in there. Below is the code snippet:

jsr_class.js

function JSONscriptRequest(fullUrl) {
  // REST request path
  this.fullUrl = fullUrl;
  // Keep IE from caching requests
  this.noCacheIE = '&noCacheIE=' + (new Date()).getTime();
  // Get the DOM location to put the script tag
  this.headLoc = document.getElementsByTagName("head").item(0);
  // Generate a unique script tag id
  this.scriptId = 'YJscriptId' + JSONscriptRequest.scriptCounter++;
}

// Static script ID counter
JSONscriptRequest.scriptCounter = 1;

// buildScriptTag method
JSONscriptRequest.prototype.buildScriptTag = function () {
  // Create the script tag
  this.scriptObj = document.createElement("script");

  // Add script object attributes
  this.scriptObj.setAttribute("type", "text/javascript");
  this.scriptObj.setAttribute("src", this.fullUrl + this.noCacheIE);
  this.scriptObj.setAttribute("id", this.scriptId);
}

// removeScriptTag method
JSONscriptRequest.prototype.removeScriptTag = function () {
  // Destroy the script tag
  this.headLoc.removeChild(this.scriptObj);
}

// addScriptTag method
JSONscriptRequest.prototype.addScriptTag = function () {
  // Create the script tag
  this.headLoc.appendChild(this.scriptObj);
}

Here is what the script does in some bullet points

  • Build request URL with input parameter and callback function
  • Build the script tag that contains the request URL
  • Add the new tag to the DOM tree
  • As soon as the tag gets addes, the WS request gets executed and what gets returned is the JSON response wrapped in a function call. The name of the function got specified in the callback parameter.
  • The response, which is a function call now calls the matching function with the JSON data as parameter. This is where the data can get extracted.
  • The script tag gets removed again

The sample I wrote for the Maps launch depends on a PHP proxy. I took that sample and wrote a version that uses JSON instead. Maybe it's because it's new, but for some reason I like the new version better.

jsonSample

All this is not just a hack that might not work again tomorrow because of a browser upgrade. Most of the dynamic ads use the dynamic script tag to display themselfes depending on the context. There are of course ups and downs for both technologies and not everyone agrees, but here are some points to think about.

Proxy method

  • More robust, error handling is easier
  • More control on the server side
  • It has some security implications on the server side as the proxy could be abused.
  • The server side can have additional functionality implemented that is hidden to the caller e.g. login, exchange secrets...

Dynamic script tag

  • No XML parsing necessary
  • Performance win
  • No traffic gets routed (and counted) on your end.
  • JSON converters don't know that they should define an array if they is only one nested element in a tag, even if the Schema allows 1..n
  • More cross-browser issues
  • Positive impact on rate limiting if it's done per IP
  • No need to set up a proxy

For more information about JSON, JavaScript and Web services have a look at our brand new Yahoo! Developer Network JavaScript Developer Center.

48 Responses to “Web Services + JSON = Dump Your Proxy”

  1. Nicely written, Dan. In the last couple of days I’ve pulled together SpiffY!Search, a proxy-free prototype that works exactly as you say it will.

    I’m very proud of Yahoo! for stepping up and offering JSON-wrapped search results; I think it’s going to be a major turning point for Web services.

  2. [...] (More at 1:40: Dan Theurer’s Web Services + JSON = Dump Your Proxy.) [...]

  3. Dan says:

    Kent, this is really cool! That is one of the things I had in mind for a perfect JSON use case.

  4. It would be really nice if the web service took an extra parameter
    which it would pass unaltered as second argument to a callback function.

    Otherwize it is rather cumbersome to build generic class that
    will track which requests got a response and which were
    timed out, for example.

  5. Dan says:

    Artem,

    Yes, I agree that this would be useful addition. I will check with our engineering team if they can get that in.

    Thanks

  6. Shawn Brown says:

    Last month I started using a comparable technique at Independence Ave (IndependenceAve.org). But I went a slightly different route: instead of wrapping the data in a function call, I’ve been appending a block statement that recreates IE’s native callback for other browsers (used with JSONRequest()). Both of these setups have the same effect but wrapping it in a function looks simpler.

    One thing not mentioned is that current KHTML browsers (Konqueror & Safari) don’t run dynamically added <script> tags. But this isn’t a showstopper because setups like this can be made to support KHTML too.

    If a KHTML browser is detected, it’s easy enough to fall back to a traditional page-refresh method — by including the request as a URL variable (or saving it in a cookie) and refreshing the page. When the page reloads, an init-loader object fetches the off-site data which can then be included without problems. The “permalink” behaviour on Independence Ave’s Lookup application manually triggers its KHTML-friendly fallback technique.

  7. [...] and, probably most importantly, you can use it in a cross-domain environment. This exists due to the fact that you can execute remote Javascript (aka a JSON object), no matter what domain you’re coming from. You can now completely skip the previously necessary (for XML) proxies. [...]

  8. Thanks, Dan. I wish I could get a bit more press for it; nobody but you seems to get how much of a change proxy-free on-page interactions are going to cause. It’s a completely new ballgame; providers brave enough to serve the data up will see it all over the Web, but with no guarantee that they’ll be credited as its source.

    Shawn: SpiffY!Search runs fine on Safari 1.3 or higher. (Not sure about 1.2; if anybody out there wants to test and report back, please do.) The secret seemed to be setting the script node’s attribute and src before appending it to the head.

  9. Dan says:

    Hey Kent,

    That is a perfect candidate for the Y! Search Application Gallery. I will get it in there.

    You are right in that there is no guarantee as a source to get credited, but some API’s have it in their Terms of Service that you need to mention the source, e.g. Y! Shopping.

  10. Shawn Brown says:

    Kent wrote:
    > SpiffY!Search runs fine on Safari 1.3 or
    > higher. (Not sure about 1.2; if anybody
    > out there wants to test and report back,
    > please do.) The secret seemed to be
    > setting the script node’s attribute and
    > src before appending it to the head.

    I have been setting the properties before attaching the nodes (again, as per this function). The issue seems to be that I did most of my testing on Konqueror 3.4.0 — I don’t have the luxury of testing in Safari myself.

    I just tried SpiffY!Search with Konqueror 3.4.0 and 3.4.3 (it didn’t work with either). As I experienced with my own project, Konqueror isn’t running the appended script tags.

    I’ve been treating all versions of Safari the same but it looks like I should stop explicitly triggering that fallback for newer versions. Thanks for the heads up!

  11. Tom Stewart says:

    If jsr_class.js were used on a closed Intranet for XMLHttp activities would the JSON security warning still apply?

  12. Well there is no guarantee as a source to get credited.

  13. Happy New Year to Everyone!

  14. Alan Brown says:

    Nice tutorial. I compared the two versions of your maps mash-up example, and the JSON version plotted the results twice as fast; the payoff of not having to use a proxy or parse XML is pretty clear.

  15. Dan says:

    It is correct that you can’t make sure that you get credited as a source, but it doesn’t make a difference if you provide a REST, SOAP or JSON Web service. Besides that, if you provide the information on a website, it could be scraped as well… (yes that brakes every once in a while)

  16. Dan says:

    Alan,

    Thanks for posting that. I didn’t know the difference was that much. I used an external parser for the XML mash-up, I am sure it’s faster if the sample uses the one that the brower provides.

  17. Dan says:

    Hi Tom,

    JSON is save to use when you get your data from a source that you can trust. There is a potential security issue if you call a service that might return malicious code. To answer your question, JSON save if you know that the souce only returns what you expect.

  18. Alex B says:

    Great tutorial Dan Have you ever tried creating a weather mash-up using Yahoo! weather datas?

  19. Dan says:

    Hi Alex,

    I didn’t work with the weather data yet.

  20. David says:

    Dan,

    My university does not allow the scripting required to build a proxy for personal pages. The json code you provided helped me use the Yahoo Geocoder in concert with the Google Maps API without having to worry about using a proxy to get xml information. It would have been virtually impossible for me to impliment a geocoder otherwise. Thanks for the tutorial and code samples.

    David Padgett

  21. Dan says:

    Hi David,

    Great that it worked our for you. I really like JSON, it make it easier to pull in 3rd party data. Let me know if you have any comments on the tutorial or maybe something that you wish to have covered.

    –dan

  22. andy says:

    Dan –

    I’ll ask you the same question I just posed over on Mindsack:

    Do you know if there’s a way to “execute” (read eval) the JSON returned from a server without the provider having a built-in callback capability?

    I know it could be done server-side by sucking in the page and then eval-ing whatever you get back, but is the client-side “script tag hack” an option without a callback capability? That is, if I just wanted to pick up “dumb” JSON from a server that no one’s tacked a callback feature onto, is there a way to do anything with it?

  23. Dan says:

    Hi Andy,

    The only way this would work is if you use the XMLHTTPRequest method (which requires a proxy) instead of the script tag. JSON by itself is not a valid JavaScript expression and that’s why the script tag hack fails.

  24. andy says:

    Hmm… yeah, that’s what I figured. Thanks for the response. Love what you’re doing with JSON, btw.

  25. Dan says:

    Thanks. JSON is a great tool and I am sure we will see a fairly high adoption for data driven “AJAX” applications. Let me know if you have any other questions.

  26. This is a great guide. JSON and on demand JavaScript are going to be a powerful set of tools.

    Example: I have created a batch geocoding service. It is simple to your example except it can do many hundreds of geocodes from a tab-delimited source file (copy/paste straight from Excel.) It can take the results and draw a Map, or just return them again in tab delimited format to be pasted back into Excel. Yahoo does all the heavy lifting and my web server is only needed to serve up about 20k worth of HTML and JavaScript.

    Since Yahoo’s 50,000 query per day limit applies to IP address, when you’re using JSON without a proxy then the 50,000 a day limit applies to the END USER IP of course, not your server. Thanks Dan and thanks Yahoo!

    http://k9.dv8.org/~dax/batchGeocoder/

    -Phillip

  27. Mark Chipman says:

    In regard to this comment: “Do you know if there’s a way to “execute” (read eval) the JSON returned from a server without the provider having a built-in callback capability?”

    I think that this still might be do-able… I just read an interesting idea concerning this very thing at http://24ways.org/advent/.

    In a very brief summary, the idea is that you append onto the last bit of the JSON datapackage (after the last curly brace) a reference to a image and use the image’s onLoad capabilities to call your function on the fly… (i.e. .

    I havn’t tried this yet… it’s still conceptual, but I think it will work.

    Just an idea, but maybe worth a shot.

    -Mark

  28. Donald says:

    Maybe I am missing something, based on thsi article I know how to get the data from the http://api.local.yahoo.com/MapsService … web service but I have a .net 1.1 web service that I want to be able to access from romate domains with similar Javascript but I do not know how to make it return JSON in the manner that the Yahoo ws does. Thanks in advance for any advice.

  29. Dan says:

    Hi Donald,

    What you need to do is make the WS accept two additional parameters: output and callback. Of cause you can call them whatever you want.

    Output specifies if your WS should return regular XML or JSON. The callback parameter is to define what function call the JSON response should be wrapped in, e.g myCallback({“ResultSet”:{“Result”:[...]}}}).

    To get the .Net server to return data in JSON encoding, you have 2 options. Either download a JSON serialzer at http://json.org or create the the output format “by hand”. If you work with C#, this should do it for you http://www.json.org/cs.zip.

    –dan

  30. scottandrew says:

    Just wanted to chime in to reaffirm that this x-domain approach still doesn’t work in Safari 2.0 (at least for me). This is a real PITA considering that Safari 1.3.2 had no problem executing dynamically inserted SCRIPT elements.

    Both the maps mashup and Spiffy Search (linked above) do nothing in Safari 2.0 AFAICT. Is anyone else seeing different results?

  31. Dallas says:

    Using your expample I am wondering if there is a way to display all the result of the in the json file beside jData.ResultSet.Result[0] like jData.ResultSet.Result[1] and jData.ResultSet.Result[2] and so on. I have tried this and get an error when result 1 or 2 or so on is null. any direction you could give me would be great. the example work great for what I am doing except i need to build in some error checking for user who can’t type in an address correctly. Thanks

  32. Rui Dias says:

    I’m trying to get muliple results from JSON like if I type an address where there is Nort and South I wanted to see both addreses

    The correct address is: 631 N 20th or 631 S 20th but I type 631 20th I wanted to get a return of but addresses

    here is what I get
    precision Latitude Longitude Address City State Zip Country
    undefined 39.76889 -94.846741 ST JOSEPH MO US

    Below is my code:

    “>
    “>
    “>

    // Callback function, will be called from JSON output
    function results(rs) {
    var tbMatches = document.getElementById(‘matches’);

    if (typeof(rs.ResultSet.Result[0].warning) != ‘undefined’) {
    document.writeln(” + rs.ResultSet.Result[0].warning + ” );
    document.writeln(”);
    document.writeln(”);
    document.writeln(”);
    document.writeln(”);

    }

    document.writeln(”);
    document.writeln(”);
    document.writeln(”);
    document.writeln(‘precision ‘);
    document.writeln(‘Latitude ‘);
    document.writeln(‘Longitude ‘);
    document.writeln(‘Address ‘);
    document.writeln(‘City ‘);
    document.writeln(‘State ‘);
    document.writeln(‘Zip ‘);
    document.writeln(‘Country ‘);
    document.writeln(”);
    for (i = 0; i ‘);
    document.writeln(”);
    document.writeln(” + rs.ResultSet.Result[0].Precision + ”);
    document.writeln(” + rs.ResultSet.Result[0].Latitude + ”);
    document.writeln(” + rs.ResultSet.Result[0].Longitude + ”);
    document.writeln(” + rs.ResultSet.Result[0].Address + ”);
    document.writeln(” + rs.ResultSet.Result[0].City + ”);
    document.writeln(” + rs.ResultSet.Result[0].State + ”);
    document.writeln(” + rs.ResultSet.Result[0].Zip + ”);
    document.writeln(” + rs.ResultSet.Result[0].Country + ”);
    document.writeln(”);
    }
    }

    &city=&state=&results=10&output=json&callback=results”>

    Back

  33. Vince says:

    Dan,

    Today the addLocal search stopped working. I thought I’d screwed it up but soon discovered that it no longer works.

    Do you have any idea what has teken place?

  34. Rui Dias -
    Guys, the JSON api for the yahoo geocoder is no more:

    http://groups.yahoo.com/group/yws-maps/message/1760

    I guess it was too good to be true (for long anyway.)

    -Phillip

  35. [...] For dynamically adding script tags, I use this library found over at Dan Theurer’s Blog. Its pretty simple and prevents duplicate script tags (which is a necessity). [...]

  36. Bob says:

    Now that’s a shame that this ended up being “too good to be true”. Got me all excited for awhile and just left me hanging? I hope there’s an alternative out there soon.

  37. Dan says:

    I agree! I really liked that feature as well. The alternative is the the proxy approach.

  38. [...] Here’s the step by step: (I use a JavaScript Object I found over at Dan Theurer’s Blog for the dynamic adding of script elements…it helps avoid duplicate script tags). [...]

  39. layouts says:

    that’s too bad I was looking forward to testing it out.

    ~Scotty

  40. Sindy says:

    2 Dan – i think, that proxys is not anonimous way for internet security.

  41. Tim says:

    Great tutorial. Very useful information. It helped me in my project.
    Thanks!

  42. Rowell says:

    Oh no, Sindy what are you talking about?
    So what is really security on internet?
    Tell me please.

  43. Patrick says:

    So,

    is this solution available *ONLY* if the server supports JSON output ?

    If the server doesn’t support JSON output, using JSON it doesn’t make sense, and I can use PHP proxy, or server-side proxy. Is it right ?

    sorry for english! :)

  44. Dan says:

    Hi Patrick,

    Yes that’s correct. The server needs to support JSON with callback. If it doesn’t, you can use the PHP proxy!

    –dan

  45. Has anybody given any thought on how to secure a json-based api to an application?

    For instance, let’s say I have an application running on a server which serves out functionality through a json api. I’ll call this server the parent.

    I then have a customer who wants to build a public website (which I’ll call child1) where all the data processing is handled by the parent through the API. I then have customer number 2 who wants to do the same thing with an app they accessed through child2. Repeat repeat repeat.

    With the assumption that the customer data contains sensitive information, is building this type of application feasible? If I take the approach detailed here, I would dynamically include script tags which would interact with the api on the parent. Is there a way to lock down the api to the approved children where other parties could not interact with the API themselves?

    The first thought that comes to mind is creating some type of token based system where the token would be included in the src/uri of the script tag. I would also assume that the token would be time-limited because it would be fairly trivial to read it from the html and included on another website. If this were the case, I’m left with having to fetch a token from the parent by the child servers which would take some of the shine off not having to write a proxy. Although writing a token fetching procedure would not be that difficult, you would have to write a version for the various server technologies out there.

    The second thought I have is to restrict access to the api based on http_referer on request of the document in the script tag, but I believe that http_referer is a) unreliable and b) easily spoofed.

    I do not wish the end user to have to authenticate themselves (implying that they must create an account) prior to using a working child.

    Is it possible to secure a json api in this scenario or am I just dreaming?

  46. [...] – PHP – JavaScript – XTemplate – RemoteJSOutput: a simple script by Matthew Batchelder (me) and Zach Tirrell – jsr_class via Theurer.cc (This won’t be mentioned in this article until part II is published) [...]

  47. Sell says:

    Thanks for posting that. I didn’t know the difference was that much. I used an external parser for the XML mash-up, I am sure it’s faster if the sample uses the one that the brower provides.

  48. Hi Dan,

    Our company is in the process of implementing JSON apis for our services, and we’ve run into a serious limitation in JSONscriptRequest: it causes Internet Explorer to leak memory. See this for details — even after you remove the script tag from the document head, IE decides to keep all of the js text around for, oh, some reason.

    Given that we’re rolling this out on a serious scale (involving sending sometimes hundreds of kB of json across the communications channel every five minutes or so) this was causing serious problems.

    We’ve worked around this problem by rewriting JSONscriptRequest; instead of attaching a script tag to the document head, we add an iframe element to the page instead, which points to a cgi which loads the json script…….. This is obviously not something we wanted, but since IE stubbornly refuses to garbage collect as well as the other browsers, we don’t see any choice.

    If you email me I can send you a test case which illustrates the problem.

    –Greg

Leave a Reply