The first thing to notice about Example 4.1 is that it does not use a JSON serializer or parser: a JSON-encoded MQL query is constructed with string concatenation and the desired results are extracted with regular expressions. These shortcuts keep the example simple and allow us to focus on how the mqlread URL is built and its content fetched. More sophisticated applications, however, use a JSON encoder to serialize the query and a JSON decoder to parse the result. Building queries with string manipulation can be reasonable in the simplest applications (though caution is required to avoid MQL injection attacks), but attempting to extract results with regular expressions is brittle and not a technique to emulate in your own code!

Example 4.2 is a higher-level version of Example 4.1. It uses a JSON serializer and parser and also a higher-level API for URL manipulation. To use it, you must have the JSON.pm module (which you can find at http://search.cpan.org) installed. This version of the program also uses a somewhat more sophisticated query to sort albums by their release date, and also does error checking and error reporting in case anything goes wrong with the query. Finally, this version of the program demonstrates how to log in to Metaweb to obtain authentication credentials. Instead of hardcoding your cookie data into the script, you must instead hardcode your Freebase username and password. Although this example uses the Metaweb login service, that service is not formally documented until Chapter 6.

Note that Example 4.2 places the query in the inner and outer envelope objects before JSON serialization. Note also that the mqlread service returns the query results in its own two-layer response envelope object. The outer object includes a property with the same name as the outer object of the query envelope. The inner object of the response has a property named result. The value of the result property is the result of the MQL query.

#!/usr/bin/perl -w
use strict;           # Don't allow sloppy syntax
use JSON;             # JSON encoding and decoding
use URI::Escape;      # URI encoding
use LWP::UserAgent;   # High-level HTTP API

# Some constants for this script
my $SERVER = 'http://www.freebase.com';            # The Metaweb server
my $QUERYURL = $SERVER . '/api/service/mqlread';   # Path to mqlread service
my $LOGINURL = $SERVER . '/api/account/login';     # Path to login service
my $USERNAME = 'user';     # Enter your Freebase username name
my $PASSWORD = 'pass';     # Enter your Freebase password here

# Create the HTTP "user agent" we'll use to send the query
my $ua = LWP::UserAgent->new;

# Login to Metaweb to get authentication credentials for our UA object.
# This will add an authentication cookie to subsequent HTTP requests.
# The login() subroutine is defined below.
&login($ua, $USERNAME, $PASSWORD);

# What band did the user ask about?
my $band = $ARGV[0];

# Construct a Metaweb query as a Perl data structure
my $query =  {
    type => "/music/artist",    # We're looking for a band
    name => $band,              # This is the name of the band
    album => [{                 # Return some albums
        name => undef,          # undef is Perl's null
        sort => "release_date", # sort by release date
        release_date => undef   # return release date, too
    }]
};

# Put the query in an envelope object
my $envelope = {        # This is the outer envelope object
    albumquery => {     # "albumquery" is an arbitrary name for inner envelope
        query => $query # The "query" property of inner envelope holds query
    }                   # End of inner envelope
};                      # End of outer envelope

# Convert the envelope object from Perl hash to JSON string, and URI encode it
my $json = JSON->new();                     # Create JSON parser/serializer
my $encoded = $json->objToJson($envelope);  # Serialize object to string
my $escaped = uri_escape($encoded);         # URI encode the string

# Build the complete query url
my $url = $QUERYURL . "?queries=" . $escaped;

# Send request to the server and get the response
my $response = $ua->get($url);

if ($response->is_success) {                     # If we get HTTP 200 OK
    my $responsetext = $response->content;       # Get result as JSON text
    my $outer=$json->jsonToObj($responsetext);   # Parse text to a Perl hash
    my $inner = $outer->{albumquery};            # Open outer envelope

    if ($inner->{code} !~ m|^/api/status/ok|) {  # If the query was not okay
        my $err = $inner->{messages}[0];         # Get the error message obj
        die $err->{code}.': '.$err->{message};   # and exit with error message
    }

    my $result = $inner->{result};               # Open inner envelope
    my $albums = $result->{album};               # Get albums from result
    for my $album (@$albums) {                   # Loop through albums
        print "$album->{name}";                  # Print the name of each
        if ($album->{release_date}) {            # Print release date
            print " [" . substr($album->{release_date},0,4) . "]"; 
        }
        print "\n";                              # Add a newline
    }
}
else {                                           # If query failed 
    die "Server returned error code " . $response->code . "\n";
}

# This subroutine calls the Metaweb login service to obtain authentication
# credentials. It asks the UA to send those credentials as cookies in
# all future requests.
sub login {
    my($ua, $username, $password);
    ($ua, $username, $password) = @_;   # Get subroutine arguments

    # Post username and password to the login service
    my $res = $ua->post($LOGINURL, {username=>$username,password=>$password});

    my $raw = $res->header('Set-Cookie');  # Get raw cookies from the response
    die "Login failed" if !$raw;           # If none, then login failed
    my @cookies = split(', ',$raw);        # Break cookies at commas
    
    # Each cookie is broken into fields with semicolons. 
    # We want the only first field of each cookie
    my $credentials = ''; # We'll accumulate login credentials here
    for my $cookie (@cookies) {                        # Loop through cookies
        my @parts = split(";", $cookie);               # Split each one on ;
        $credentials = $credentials . $parts[0] . ';'; # Remember first part
    }
    chop($credentials);   # Remove trailing semicolon

    # Tell the UA to send our credentials on every request
    $ua->default_header('Cookie' => $credentials);
}

Now that we've seen some working code, this section explains more formally how mqlread works. As you've seen in the preceding examples, the URL for the mqlread service on freebase.com is:

http://www.freebase.com/api/service/mqlread

The sub-sections that follow document mqlread input and output, and also specify the format of the query and response envelopes.

{               # This is the outer envelope object
  "q0": {       # This is the first inner envelope.  The name "q0" is arbitrary
    "query": {  # The first MQL query goes here
    }
  },
  "q1": {       # This is the second inner envelope
    "query": [{ # Second MQL query goes here.  Note that this one is in []
    }]
  }
}

{               # This is the outer envelope object
  "q0": {       # This is the first inner envelope.  The name "q0" is arbitrary
    "code":"/api/status/ok"  # The query was successful
    "result": {                # The result of the first MQL query goes here
    }
  },
  "q1": {       # This is the second inner envelope
    "code":"/api/status/ok"  # The query was successful
    "result":[{                # The result of the second MQL query goes here.
    }]
  }
}

Now that we've seen how mqlread works in more formal detail, let's return to example code, and re-write our album listing script in Python. Example 4.3 is a Python module that defines the utility function metaweb.read(). This function:

This code relies on the simplejson module for JSON encoding and parsing. You can find the simplejson code at http://cheeseshop.python.org/pypi/simplejson.

import urllib        # URL encoding
import urllib2       # Higher-level URL content fetching
import simplejson    # JSON serialization and parsing

host = 'www.freebase.com'              # The Metaweb host
readservice = '/api/service/mqlread'   # Path to mqlread service

# Submit the MQL query q and return the result as a Python object.
# If authentication credentials are supplied, use them in a cookie.
# Raises MQLError if the query was invalid. Raises urllib2.HTTPError if
# mqlread returns an HTTP status code other than 200 (which should not happen).
def read(q, credentials=None):
    # Put the query in an envelope
    env = {'qname':{'query':q}}
    # JSON serialize and URL encode the envelope and the query parameter
    args = urllib.urlencode({'queries':simplejson.dumps(env)})
    # Build the URL and create a Request object for it
    url = 'http://%s%s?%s' % (host, readservice, args)
    req = urllib2.Request(url)

    # Send our authentication credentials, if any, as a cookie.
    # The need for mqlread authentication is a temporary restriction.
    if credentials: 
        req.add_header('Cookie', credentials)

    # Now upen the URL and and parse its JSON content
    f = urllib2.urlopen(req)        # Open the URL
    response = simplejson.load(f)   # Parse JSON response to an object
    inner = response['qname']       # Open outer envelope; get inner envelope

    # If anything was wrong with the invocation, mqlread will return an HTTP
    # error, and the code above with raise urllib2.HTTPError.
    # If anything was wrong with the query, we won't get an HTTP error, but
    # will get an error status code in the response envelope.  In this case
    # we raise our own MQLError exception.
    if not inner['code'].startswith('/api/status/ok'):
        error = inner['messages'][0]
        raise MQLError('%s: %s' % (error['code'], error['message']))

    # If there was no error, then just return the result from the envelope
    return inner['result'];
                 
# If anything goes wrong when talking to a Metaweb service, we raise MQLError.
class MQLError(Exception):
    def __init__(self, value):     # This is the exception constructor method
        self.value = value
    def __str__(self):             # Convert error object to a string
        return repr(self.value)

With the metaweb.read() function defined, we can now write our album listing code in Python. Example 4.4 shows how we do this. Note that this example uses the metaweb.login() function for authentication. The implementation of this function is in Chapter 6.

import sys
import metaweb  # Defines the metaweb.read() and login() functions

# Compose our MQL query using a Python data structure
band = sys.argv[1]                               # The band we want
query = { 'type': '/music/artist',               # Our MQL query in Python
          'name': band,
          'album': [{ 'name': None,              # None is Python's null
                      'release_date': None,
                      'sort': 'release_date' }]}

# Login to get authentication credentials
# Insert your Freebase username and password here.
credentials = metaweb.login("username", "password");

# Submit the query using metaweb.read() and check for valid results
result = metaweb.read(query, credentials)
if not result or not result['album']: sys.exit('Unknown band')

# A utility function to get year from a MQL datetime value
def getYear(date): 
    if not date: return ''
    return "[%s]" % date[0:4]

# Now output the results
for album in result['album']:
    print "%s %s" % (album['name'], getYear(album['release_date']))

In this section, we'll demonstrate how to create an online version of our album-lister application. We'll use the the server-side scripting language PHP to create the web application that was shown in Figure 1.2 of Chapter 1. Example 4.5 is a PHP file that defines a class named Metaweb. This class has a single method, named read that behaves just like the metaweb.read() function defined in Example 4.3.

The code in Example 4.5 is commented and you should be able to follow it even if you are not familiar with PHP. One point to note is that in PHP the data structure known as an array works as both a sequential array and as an associative array. That is, JSON objects and JSON arrays are both arrays in PHP. Example 4.5 depends on an external module for JSON serialization and parsing. The module used here is from http://pear.php.net.

json = new Services_JSON(SERVICES_JSON_LOOSE_TYPE);
  }
  
  // This method submits a query and synchronously returns its result.
  // If authentication credentials are passed, it uses them as an HTTP Cookie.
  function read($queryobj, $credentials) {
    // Put the query into an envelope object
    $envelope = array("qname" => array("query" => $queryobj));

    // Serialize the envelope object to JSON text
    $querytext = $this->json->encode($envelope);

    // Then URL encode the serialized text
    $encoded = urlencode($querytext);

    // Now build the URL that represents the query
    // Note that we use an HTTP GET request for read queries 
    $url = $this->URL . "?queries=" . $encoded;

    // Use the curl library to send the query and get response text
    $request = curl_init($url);

    // Return the result instead of printing it out.
    curl_setopt($request, CURLOPT_RETURNTRANSFER, TRUE);

    // If we have credentials, send them with the request as a cookie
    if ($credentials) curl_setopt($request, CURLOPT_COOKIE, $credentials);

    // Now fetch the URL
    $responsetext = curl_exec($request);
    curl_close($request);

    // Parse the server's response from JSON text into a PHP array
    $response = $this->json->decode($responsetext);

    // Return null if the query was not successful
    if (strpos($response["qname"]["code"], "/api/status/ok") !== 0) return null;

    // Otherwise, open the envelope and just return the actual result object
    return $response["qname"]["result"];
  }
}
?>

With the PHP utility function defined in Example 4.5, it becomes easy to write simple Metaweb-enabled web applications in PHP. Example 4.6 demonstrates. It displays an HTML form in which the user can enter the name of a band. When the form is submitted, it lists the albums by that band.

Like Example 4.1, this example requires you to hard-code the value of your freebase.com authentication cookie into the script. See the instructions earlier in this chapter for finding the value of the metaweb-user cookie.

Band: 

 "/music/artist", // We want an musical artist
                 "name" => $band,           // This is its name
                 "album" => array());       // Fill in this empty albums array!
  
  // Insert your own freebase.com cookie data into the string below
  $credentials = 'metaweb-user=### Put your cookie data here ### ';

  // Submit the query using the utility function defined earlier
  $result = $metaweb->read($query, $credentials);
  
  // This is the array of albums we want
  $albums = $result["album"];
  
  // Display the albums on the web page 
  echo "
Albums by " . $band . "
";
  foreach ($albums as $album) echo $album . "
";
}
?>

Since the MQL syntax is based on JSON, Metaweb queries are most gracefully expressed in JavaScript. We haven't seen a JavaScript-based Metaweb application so far for one important reason: the same-origin policy. The same-origin policy is a sweeping (but necessary) security restriction in JavaScript that says that code embedded in a document that was served by server A can only interact with content that is also served by server A. This restriction applies to the XMLHttpRequest object which is what is typically used to fetch the contents of a URL. A web application hosted at www.freebase.com can use XMLHttpRequest to submit MQL queries to the mqlread service on that same server, but this is not allowed for web applications hosted on any other server.

There are two workarounds to this restriction. The first, and most obvious, is to run a proxy script on your own site that behaves like the mqlread service but simply forwards your query to freebase.com.

The second workaround relies on the fact that a query result, in JSON format, is valid JavaScript code. This means that a mqlread URL can be used as the value of the src attribute of a <script> tag. When the server returns its result, the <script> tag evaluates the JSON text as JavaScript code. Evaluating the JSON text creates the JavaScript object we want, but to make this scheme work, the script then has to be able to do something with that object. The solution is to add another URL parameter to the mqlread service. If the URL for your query includes a callback= parameter, then mqlread will take the value of that parameter to be the name of a JavaScript function. Then, instead of simply returning a JSON text, it will return the specified function name, an open parenthesis, the JSON text and a close parenthesis. When used this way with a <script> tag, the JSON text is evaluated, and the object that results is passed to the specified function (which you have defined previously).

We use the <script> technique in this chapter: it is simple, elegant and in common use across the internet. If you prefer a proxy and XMLHttpRequest-based approach, you can find sample proxy code in Appendix A.

One thing you'll notice about our JavaScript examples here (and in in Appendix A) is that they are asynchronous: when you submit a query you do not get the result immediately. Instead, the callback function you specify is invoked when the result is available. This asynchronous programming model is common in client-side JavaScript, but is substantially different from the synchronous model demonstrated in Example 4.5 and other examples.

4.5.1. Listing Albums and Tracks with JavaScript

Let's jump right in. Example 4.7 is a JavaScript-based album and track lister application, pictured in Figure 4.1. Notice that this code depends on two external modules. json.js is a file that defines JavaScript functions for parsing and serializing JSON. The code for this module is not shown here; you can find it as Example A.1 in Appendix A. The second module of external code is metaweb.js. This module, whose code listed in Example 4.8, defines the utility function Metaweb.read() that submits a MQL query, through a <script> tag, to the mqlread service.

Figure 4.1. Listing albums and tracks

Example 4.7 lists the albums by a specified band, and also displays the tracks on an album when the user clicks on the name of the album. There are several features worth noting in this example. First, notice that this code uses the Metaweb.read() function to send queries to the mqlread service. We'll see how Metaweb.read() is implemented in the next section. Second, note that the code displays a "Loading..." message to the user while the queries are pending and also displays an appropriate message if a query fails. Finally, Example 4.7 demonstrates two different ways to insert Metaweb query results into an HTML document. When the album query returns, the album list is populated using DOM methods to build each text node and

tag. When the track query returns, on the other hand, the list of tracks is built as a string of HTML text and is inserted into the document by setting the innerHTML property of the container element.

In addition to querying album and track names, the queries in this example also ask for album release date and track length. The example includes utility functions to massage this data, extracting a year from a /type/datetime string and converting a track length in seconds into the more familiar mm:ss format.

Example 4.7. albumlist.html: a JavaScript album and track lister

      
   
<script>                                

/* Display albums by the specified band */
function listalbums(band) {
    // Find the document elements we need to insert content into
    var title = document.getElementById("title");
    var albumlist = document.getElementById("albumlist");
    var tracklist = document.getElementById("tracklist");

    title.innerHTML = "Albums by " + band;           // Set the page title 
    albumlist.innerHTML = "Loading..." // Album list is coming...
    tracklist.style.visibility = "hidden";           // Hide any old tracks
    
    var query = {                      // This is our MQL query 
        type: "/music/artist",         // Find a band
        name: band,                    // With the specified name
        album: [{                      // We want to know about albums         
            name:null,                 // Return album names
            release_date:null,         // And release dates
            sort: "release_date"       // Order by release date
        }]
    };

    // Issue the query and invoke the function below when it is done
    Metaweb.read(query, displayAlbums);

    // This function is invoked when we get the result of our MQL query
    function displayAlbums(result) {  
        // If no result, the band was unknown.
        if (!result || !result.album) {
            albumlist.innerHTML = "Unknown band: " + band + "";
            return;
        }
        
        // Otherwise, the result object matches our query object, 
        // but has album data filled in.  
        var albums = result.album;  // the array of album data
        // Erase the "Loading..." message we displayed earlier
        albumlist.innerHTML = "";
        // Loop through the albums
        for(var i = 0; i < albums.length; i++) {
            var name = albums[i].name;                   // album name 
            var year = getYear(albums[i].release_date);  // album release year
            var text = name + (year?(" ["+year+"]"):""); // name+year

            // Create HTML elements to display the album name and year.
            var div = document.createElement("div");
            div.className = "album";
            div.appendChild(document.createTextNode(text));
            albumlist.appendChild(div);

            // Add an event handler to display tracks when an album is clicked
            div.onclick = makeHandler(band, albums[i].name);
        }

        // This function returns a function.  We do it this way to create
        // a closure that captures the band and album names.
        function makeHandler(band, album) {
            return function(e) { listtracks(band, album); }
        }
    }

    // A utility function to return the year portion Metaweb /type/datetime
    function getYear(date) {
        if (!date) return null;
        if (date.length == 4) return date;
        if (date.match(/^\d{4}-/)) return date.substring(0,4);
        return null;
    }
}

/* Display the tracks on the specified album by the specified band */
function listtracks(band, albumname) {
    // Begin by displaying a Loading... message
    var tracklist = document.getElementById("tracklist");
    tracklist.innerHTML = "
" + albumname + "
Loading...";
    tracklist.style.visibility = "visible";

    // This is the MQL query we will issue
    var query = {
        type: "/music/album",
        name: albumname,
        artist: band,
        // Get track names and lengths, sorted by index
        track: [{name:null, length:null, index:null, sort:"index"}]
    };

    // Issue the query, invoke the nested function when the response arrives
    Metaweb.read(query, function(result) {
                     if (result && result.track) { // If result is defined
                         var tracks = result.track;  // array of tracks
                         // Build an array of track names + lengths
                         var listitems = []
                             for(var i = 0; i < tracks.length; i++) {
                                 var n = tracks[i].name + " (" +
                                   toMinutesAndSeconds(tracks[i].length)+")";
                                 listitems.push(n);
                             }
                         // Display the track list by setting innerHTML
                         tracklist.innerHTML = "
" + albumname + "
" +
                             "
" + listitems.join("
") + "
";
                     }
                     else {
                         // If empty result display error message
                         tracklist.innerHTML = "
" + albumname + "
" +
                             "
No track list is available.";
                     }
                 });

    // Convert track length in seconds to minutes:seconds format
    function toMinutesAndSeconds(seconds) {
        var minutes = Math.floor(seconds/60);
        var seconds = Math.floor(seconds-(minutes*60));
        if (seconds <= 9) seconds = "0" + seconds;
        return minutes + ":" + seconds;
    }
};



 
#albumlist { width:50%; padding: 5px; }
#tracklist {
  width: 45%; float:right; visibility:hidden;
  padding: 5px; border: solid black 2px; margin-right: 10px;
  background-color: #8a8;
}
#tracklist h2 { font: bold 16pt sans-serif;  text-align: center;}
#tracklist p { text-align: center; font: italic bold 12pt sans-serif; }
#tracklist li { font-style: italic;}
div.album {  font: bold 12pt sans-serif; margin: 2px;}
div.album:hover {text-decoration: underline;}







Enter the name of a band: 





        

  
  

&callback=Metaweb._3

Metaweb._3(/* JSON object goes here */)

/**
 * metaweb.js: 
 *
 * This file implements a Metaweb.read() utility function using a <script>
 * tag to generate the HTTP request and the URL callback parameter to
 * route the response to a specified JavaScript function.
 **/
var Metaweb = {};                               // Define our namespace
Metaweb.HOST = "http://www.freebase.com";       // The Metaweb server
Metaweb.QUERY_SERVICE = "/api/service/mqlread"; // The service on that server
Metaweb.counter = 0;                            // For unique function names

// Send query q to Metaweb, and pass the result asynchronously to function f
Metaweb.read = function(q, f) {
    // Define a unique function name
    var callbackName = "_" + Metaweb.counter++

    // Create a function by that name in the Metaweb namespace.
    // This function expects to be passed the outer query envelope.
    // If the query fails, this function throws an exception.  Since it
    // is invoked asynchronously, we can't catch the exception, but it serves
    // to report the error to the JavaScript console.
    Metaweb[callbackName] = function(outerEnvelope) {
        var innerEnvelope = outerEnvelope.qname;         // Open outer envelope
        // Make sure the query was successful.
        if (innerEnvelope.code.indexOf("/api/status/ok") != 0) {  // Check for errors
          var error = innerEnvelope.messages[0]          // Get error message
          throw error.code + ": " + error.message      // And throw it!
        }
        var result = innerEnvelope.result;   // Get result from inner envelope
        document.body.removeChild(script);   // Clean up <script> tag
        delete Metaweb[callbackName];        // Delete this function
        f(result);                           // Pass result to user function
    };

    // Put the query in inner and outer envelopes
    envelope = {qname: {query: q}}

    // Serialize and encode the query object
    var querytext = encodeURIComponent(JSON.serialize(envelope));

    // Build the URL using encoded query text and the callback name
    var url = Metaweb.HOST + Metaweb.QUERY_SERVICE +  
        "?queries=" + querytext + "&callback=Metaweb." + callbackName

    // Create a script tag, set its src attribute and add it to the document
    // This triggers the HTTP request and submits the query
    var script = document.createElement("script");
    script.src = url
    document.body.appendChild(script);
};

A number of things can go wrong when using mqlread. If you invoke it with incorrect URL parameters or supply a query envelope that is not valid JSON, mqlread will response with an HTTP "400 Bad Request" error. The body of the response will be a JSON object that provides details about the error. It might look like this:

{
  "code": "/api/status/error",
  "messages": [
    {
      "message": "JSON parse error: Expecting property name at line 1 column 1", 
      "code": "/api/status/error/input/invalid", 
      "info": {
        "field": "queries", 
        "value": "{<}\r\n"
      }
    }
  ]
}

This kind of error can occur if you are cutting-and-pasting raw mqlread URLs or if you are entering MQL queries as JSON text into a query editor application. When you write scripts that use JSON serializers and invoke mqlread using tested code, this kind of error should not occur. Errors are still possible, however: a MQL query can be invalid, even if it is expressed using well-formed JSON and passed to mqlread using the correct URL parameters.

If you submit an invalid MQL query, mqlread returns an HTTP error code of "200 OK", but the code property of the inner response envelope begins with "/api/status/error". The inner response envelope also includes a property named messages instead of a property named result. The value of the messages property is an array (usually of length 1) of message objects each of which has the following properties:

When a MQL query is to be submitted to mqlread, it is placed inside an inner query envelope object, as the value of a property named query. Often, this is the only property of the inner query envelope. But a property named cursor is also allowed.

Use a cursor when you want to retrieve results in batches from a large result set. Start by including this property in your inner query envelope:

cursor: true

When you do this, mqlread will include a cursor property in the inner envelope of its response. If the value of the response cursor property is false, then mqlread has returned the complete set of query results to you. If the cursor property is not false, then it will be a string containing opaque data. Take the value of this cursor property, insert it back into your inner query envelope, and send the query back to mqlread. mqlread will send you the next batch of results and will again include a cursor property in the inner response envelope. Repeat this process until the cursor property of the response is false.

Example 4.9 is a metaweb.readall() function written in Python. It works like the metaweb.read() function of Example 4.3, but uses a cursor to iterate through a large result set, making multiple queries and concatenating the results into a single array before returning them. (Note that this function doesn't allow any kind of parallelism: it does not allow the first batch of results to be processed while the second batch is being fetched, for example. So if you're using a limit directive and cursors to improve response time, this readall() methods is not appropriate.)

import urllib        # URL encoding
import urllib2       # Higher-level URL content fetching
import simplejson    # JSON serialization and parsing

host = 'www.freebase.com'               # The Metaweb host
readservice = '/api/service/mqlread'    # Path to mqlread service

# Submit the MQL query q and return the result as a Python object
# This function behaves like read() above, but uses cursors so that
# it works even for very large result sets
def readall(q, credentials=None):
    # This is the start of the mqlread URL.
    # We just need to append the envelope to it
    urlprefix = 'http://%s%s?queries=' % (host, readservice)

    # The query and most of the envelope are constant. We just need to append
    # the encoded cursor value and some closing braces to this prefix string
    jsonq = simplejson.dumps(q);
    envelopeprefix = urllib.quote_plus('{"q0":{"query":'+jsonq+',"cursor":')
    
    cursor = 'true'   # This is the initial value of the cursor
    results = []      # We accumulate results in this array

    # Loop until mqlread tells us there are no more results
    while cursor:
        # append the cursor and the closing braces to the envelope
        envelope = envelopeprefix + urllib.quote_plus(cursor + '}}')
        # append the envelope to the URL
        url = urlprefix + envelope

        # Begin an HTTP request for the URL
        req = urllib2.Request(url)

        # Send our authentication credentials, if any, as a cookie.
        # The need for mqlread authentication is a temporary restriction.
        if credentials: 
            req.add_header('Cookie', credentials)

        # Read and parse the URL contents
        f = urllib2.urlopen(req)          # Open URL
        response = simplejson.load(f)     # Parse JSON response
        inner = response['q0']            # Get inner envelope from outer

        # Raise a MQLError if there were errors
        if not inner['code'].startswith('/api/status/ok'):
            error = inner['messages'][0]
            raise MQLError('%s: %s' % (error['code'], error['message']))

        # Append this batch of results to the main array of results.
        results.extend(inner['result']);

        # Finally, get the new value of the cursor for the next iteration
        cursor = inner['cursor']
        if cursor:                        # If it is not false, put it
            cursor = '"' + cursor + '"'   #  in quotes as a JSON string

    # Now that we're done with the loop, return the results array
    return results

It is important to understand that cursors only work when multiple results are expected at the top-level of the query. The cursor property is part of the mqlread envelope syntax, not part of the MQL query language, and it cannot be applied to sub-queries of a query. Another way to say this is that it only makes sense to include "cursor":true in an envelope if the first character following "query": in the envelope is [. The query must be expressed as an array in order for a cursor to be meaningful.

Consider the code in Example 4.4. It contains this query:

query = { 'type': '/music/artist',               # Our MQL query in Python
          'name': band,
          'album': [{ 'name': None,              # None is Python's null
                      'release_date': None,
                      'sort': 'release_date' }]}

This is a perfectly valid query, and works just fine in Example 4.4. But suppose we wanted to port that script to use the metaweb.readall() function defined above. To do this, we'd also have to alter the query so that the array of albums was at the top level of the query:

query = [{'type': '/music/album',
          'artist': band,
          'name': None,
          'release_date': None,
          'sort': 'release_date',
          'limit': 10}]

Note that we've added an explicit limit directive to this modified query. In general, it makes sense to specify an explicit limit when using cursors.

As explained in Chapter 2, Metaweb is really two databases in one. One database is the graph of nodes and relationships. The second is the content store that holds chunks of data such as HTML documents and graphical images. We use mqlread service to retrieve data from the graph, and we use the trans service to retrieve content from the content store.

The trans service is so named because in addition to fetching the requested data, it can also translate it for you. For example, it can "translate" an image to thumbnail size.

The trans service is HTTP based, just as mqlread is. Content is retrieved by specifying the desired translation and the content id, with a URL of this form:

http://www.freebase.com/api/trans/translation/guid

Here, for example, is an actual trans URL at freebase.com:

http://www.freebase.com/api/trans/raw/%239202a8c04000641f8000000003c1978c

The translation portion of a trans URL must be one of the following:

The path component that follows the translation is the URL-encoded version of a Metaweb guid. %23 is the encoding of the # character, and the letters and digits that follow are the hexadecimal digits of the guid. The guid passed to trans must identify an object of type /type/content, /common/image or /common/document. These three types are closely related:

The trans service does not support a callback parameter as the mqlread service does, so you cannot use it with <script> tags. If you implement a proxy on your own web server, then you can invoke the trans service indirectly to retrieve content with XMLHttpRequest, however.

It is usually easier, however, to use the trans service with and tags. To retrieve and display an image, simply use a trans URL as the src attribute of an tag. And to retrieve and display the HTML content of a document, use a trans URL as the src attribute of an .

Like mqlread, the trans service requires cookie-based authentication during the freebase.com roll-out period. The examples in this chapter assume that you are using the trans service in a web browser that has visited and logged on to www.freebase.com.

<script>

// These are a few important constants
var HOST = "http://www.freebase.com";
var READ = "/api/service/mqlread";
var RAW = "/api/trans/raw/";
var THUMB = "/api/trans/image_thumb/";
var BLURB = "/api/trans/blurb/";

/**
 * Send the queries named in the outer envelope object to Metaweb,
 * and pass the outer response envelope to the function f.  This is a
 * variant of the Metaweb.read() function that runs multiple queries.
 */ 
function sendQueries(queryEnvelope, f) {
    // Define a unique function name
    var callbackName = "_" + sendQueries.counter++

    // Create a function by that name, using sendQueries as a namespace.
    // This function expects to be passed the response to the query
    sendQueries[callbackName] = function(responseEnvelope) {
        document.body.removeChild(script);  // Remove <script> tag
        delete sendQueries[callbackName];   // Delete this function
        f(responseEnvelope);                // Pass response to user function
    };

    // Serialize and encode the query object
    var queries = encodeURIComponent(JSON.serialize(queryEnvelope));

    // Build the URL using encoded query text and the callback name
    var url = HOST + READ + "?queries=" + queries +
        "&callback=sendQueries." + callbackName

    // Create a script tag, set its src attribute and add it to the document
    // This triggers the HTTP request and submits the query
    var script = document.createElement("script");
    script.src = url
    document.body.appendChild(script);
};
sendQueries.counter = 0;  // Initialize the counter

// How many images and how many documents do we display?
var N = 10;                                          // This is the default
if (window.location.search.substring(0,3) == "?n=")  // URL argument overrides
    N = parseInt(window.location.search.substring(3));

// These are the queries we issue to find the n newest images and documents
var queries = {
  images: {
    query: [{
      type:"/common/image", id:null,         // Return image ids
      timestamp:null, sort:"-timestamp",     // Most recent first
      limit:N,                               // Only N of them
      "/type/content/media_type":null,       // Check image type, too
      "/type/content/media_type|=":[         // We only want images that are:
        "/media_type/image/gif",             // GIF or
        "/media_type/image/png",             // PNG or 
        "/media_type/image/jpeg"             // JPEG
      ]
    }]
  },
  docs: {
    query: [{
      type:"/common/document", id:null,      // Return document ids
      timestamp:null, sort:"-timestamp",     // Most recent first
      limit:N                                // Only N of them
    }]
  }
};

// When the document has loaded, send the queries above to freebase.com.
// Then call the function below with the results
window.onload = function() { sendQueries(queries, displayResults) }

// This function gets called with our query results
function displayResults(response) {
    // First, display image thumbnails
    var images = response.images.result;                // Array of images
    var container=document.getElementById("newimages"); // Where they go
    for(var i = 0; i < images.length; i++) {            // Loop through them
        var id = encodeURIComponent(images[i].id);      // Image id in URL form

        var thumbnail = document.createElement("img");  // Create  tag
        thumbnail.src = HOST + THUMB + id;      // url for image thumbnail
        thumbnail.title = images[i].timestamp;  // timestamp as tooltip
        
        var link = document.createElement("a"); // Hyperlink for image
        link.href = HOST + RAW + id;            // to a full-size image
        link.target = "_new";                   // displayed in a new window

        link.appendChild(thumbnail);            // Put thumbnail inside link
        container.appendChild(link);            // Put link inside container
    }

    // Next display document blurbs
    var docs = response.docs.result;                // Array of documents
    container = document.getElementById("newdocs"); // Where they go
    for(var i = 0; i < docs.length; i++) {          // Loop through them
        var id = encodeURIComponent(docs[i].id);      // Doc id in URL form
        var blurb = document.createElement("iframe"); // Create an iframe
        blurb.src = HOST + BLURB + id;                // to hold doc blurb
        var link = document.createElement("a");       // Hyperlink
        link.href = HOST + RAW + id;                  // To full document
        link.target = "_new";                         // In a new window
        link.innerHTML = docs[i].timestamp;           // Timestamp as link text
        var listitem = document.createElement("li");  // Create list item
        listitem.appendChild(blurb);                  // Put blurb in item
        listitem.appendChild(link);                   // Put link in item
        container.appendChild(listitem);              // Put item in container
    }
}

 /* Make it all look good with a stylesheet */
img { margin: 5px;}
iframe { width: 70%; height: 75px; vertical-align: top;}
li a { vertical-align: bottom; }
h2 { margin-bottom: 5px; }




The Newest Images
Click thumbnail for full-size image


The Newest Documents
Click timestamp for full document