Monday, August 29, 2011

easy cross-platform inter process communication (IPC) between your program and swi-prolog

Image available under creative commons license from the Flickr photostream of DailyPic.


The problem

In this tutorial I will show how to set up communication between your program (can be written in any language) and swi-prolog. If you, like me, are using python, you might be tempted to use the python/swi-prolog bridge pySWIP, but it has a major disadvantage: it just doesn't work on 64-bit platforms (yet?). If you use any other language, your mileage may vary: e.g. in JAVA you will often be advised to use a prolog that runs on a JVM.

What is presented here is a slightly different approach, which works beautifully across platforms, operating systems and network boundaries. Moreover, it ensures that your prolog code remains very well separated from your main program's code, which cannot be a bad thing.

This tutorial explains matters from the point of view of talking from a python program to a prolog backend.

Nothing prevents you from talking from a prolog frontend to a python backend. The prolog predicates required to do so are not explained in this tutorial (you can read about it in the manual of swi-prolog, in the part about http/http_client)

Approach


Swi-prolog makes it quite easy to write a web service. Swi-prolog also has built-in support for JSON-RPC. By tying together those two functionalities we can interrogate the prolog backend using JSON-RPC requests.

Details


We want to keep it simple for the sake of clarity so we set out to achieve the following goal: a python program will communicate with prolog to calculate the sum of two numbers.

In order to do this we need two things:

  • a prolog program that implements some functionality in prolog and that also responds to JSON-RPC requests (i.e. a JSON-RPC server)
  • a client program that can talk JSON-RPC with our prolog server, and understands its replies (we will create one in python for the sake of simplicity).


Create a python client


After installing jsonrpclib, communication with a JSON-RPC web service is incredibly easy.


First an object is created that represents the server. Here it is assumed that the prolog JSON-RPC server will be running on localhost, port 5000, and it will respond to JSON-RPC requests at url http://localhost:5000/handle.

Once this server proxy is created, one can call methods on it. Here we ask the server to give back the result of "add(5,6)". The jsonrpclib library will automagically convert this method call into a JSON-RPC request, send it to the web-server, read the reply, unparse it from JSON back to python, and return the result. How's that for cool? ;)

If you use another programming language, you will most likely be able to find some specialized library to work with JSON-RPC.

Creating a prolog JSON-RPC server




The prolog JSON-RPC web service isn't that much harder to understand too.
First we import a bunch of libraries that provide much of the required functionality in creating web services out of the box.

Then we get

This line tells prolog that the predicate handle_rpc will be executed (it implicitly assumes that handle_rpc will take a Request as parameter) whenever someone visits the url handle (as indicated by the root(handle) part of the line). Note: nothing prevents you from handling url handle by a predicate with the same name. I gave them different names here (handle and handle_rpc) to make it more explicit in which ways the parameters of http_handler are used.

The next line
http_json:json_type('application/json-rpc').
tells prolog that 'application/json-rpc' is a valid mime-type for a JSON-RPC request. As it turns out, the built-in JSON-RPC support by default only recognizes mime-types of 'application/jsonrequest' and 'application/json'. 'application/json' is the suggested mime-type adopted by RFC4627. The python client, on the other hand uses 'application/json-rpc'. Luckily the makers of swi-prolog decided to make the http_json:json_type predicate a multifile predicate, meaning that it can be extended outside its original file (in other words: by us, the user of the library). If you don't allow 'application/json-rpc' as a valid mime type on the server, the python client will keep receiving an "500 Internal server error" error.

The lines

configure a web server on port Port. After starting prolog, and consulting your prolog server file, you can start the web server on port 5000 by evaluating the following goal:

This will spawn some threads and immediately return to the console prompt, which opens up possibilities for debugging your web service (read more about these possibilities in the HOWTO part of the swi-prolog documentation).

Then comes the handling of the request:


First the json is extracted from the Request, and read into a variable JSONIn.
The http_read_json predicate is provided by the http/http_json library included with swi-prolog. Next, the JSONIn variable (which is basically a string; a datatype described in a subset of javascript) is parsed into a prolog data type. The json_to_prolog predicate is provided by the http/json_convert library included with swi-prolog. After that we evaluate the contents of PrologIn, and from that create PrologOut. Using the current sample code json_to_prolog won't do anything. It can be used to automatically convert the prolog representation of the json reply into a more "prologic" form. To make this magic possible, one needs to declare json_objects. Using prolog_to_json, PrologOut is converted to a more "JSONic" prolog term again, and then sent back to the client with reply_json. This conversion relies on the same json_object declarations as json_to_prolog. For now I don't fully see the advantages of this process. Please chime in, if you have more insights. prolog_to_json is provided by the http/json_convert library.

The evaluate predicate is one we have to write ourselves. It's where the computations take place that are needed to handle the request (in our case: the addition of two numbers). Here's how the evaluate predicate works:
A json request consists of predefined fields:

  • jsonrpc: A String specifying the version of the JSON-RPC protocol. MUST be exactly "2.0".
  • method: A String containing the name of the method to be invoked. Method names that begin with the word rpc followed by a period character (U+002E or ASCII 46) are reserved for rpc-internal methods and extensions and MUST NOT be used for anything else.
  • params : A Structured value that holds the parameter values to be used during the invocation of the method. This member MAY be omitted.
  • id : An identifier established by the Client that MUST contain a String, Number, or NULL value if included. If it is not included it is assumed to be a notification. The value SHOULD normally not be Null and Numbers SHOULD NOT contain fractional parts. The id is sent by the client, and should be sent back with the response so the client knows what query is being answered.


More specifically, if we use our python client, after translation to prolog, we get the following term for PrologIn:

We unify it to extract the parameter values, id, and method:


Note that in its current form, the prolog web service is rather inflexible: it will just know how to do an "add" of two numbers. I'll leave it up to your imagination to see how this can be generalized or abstracted.

A JSON-RPC response also has a predefined format. It contains the following fields:

  • jsonrpc: A String specifying the version of the JSON-RPC protocol. MUST be exactly "2.0".
  • result: This member is REQUIRED on success. This member MUST NOT exist if there was an error invoking the method. The value of this member is determined by the method invoked on the Server.
  • error: This member is REQUIRED on error. This member MUST NOT exist if there was no error triggered during invocation. The value for this member MUST follow some predefined rules (see JSON-RPC specification for details).
  • id : This member is REQUIRED. It MUST be the same as the value of the id member in the Request Object. If there was an error in detecting the id in the Request object (e.g. Parse error/Invalid Request), it MUST be Null.


So we are ready to calculate and formulate our reply:


The MethodName = add just verifies that we really were asked to do an add on two numbers.
The Sum = Param1 + Param2 performs the calculation, and
the PrologOut contains the reply that will be sent back to the client.

Running the programs


The hard work is done. Now we can enjoy the fruits of our labour:
Start a console (dos prompt) and start the prolog server on port 5000:

swipl -f server.pl -g "server(5000)."

Start a second console and run the python program:
python rpc.py

If all went well, running the python program should print:
The result is: 11

To kill the prolog server, you can manually type
halt.
on its console prompt, but you could of course also design an RPC request that tells the interpreter to shut itself down.

Your imagination is the only limiting factor.



7 comments:

  1. According to Wikipedia, the correct MIME type for JSON is "application/json"
    http://en.wikipedia.org/wiki/JSON#MIME_type

    So this line:
    http_json:json_type('application/json-rpc').
    Should be:
    http_json:json_type('application/json').

    ReplyDelete
  2. Please ignore my last comment. I now understand that the python client was the problem.

    ReplyDelete
  3. Hi, does there exist a more general Prolog RPC server? I'd like to use one, but I don't really have the time to write one myself. I want to be able to define rules and issue arbitrary queries, etc...

    ReplyDelete
    Replies
    1. I have no idea to be honest. I did make something slightly more general at some point, but then got worried about security implications of being able to submit random prolog queries to a web server. Perhaps try asking on the swi-prolog newsgroup, or on comp.lang.prolog ? I believe in the mean time (i.e. since writing this post) someone has been working on some kind of safe call predicate, which could be used to make a more generic rpc server that is reasonably safe.

      Delete
    2. I was thinking, if it's possible to host the server on 127.0.0.1 only, then that would suit me, and I wouldn't have to worry about security. I only want to be able to make calls to it from the same machine. Unfortunately, I didn't see anything in swi-prolog's documentation that mentioned specifying the local loopback as the host IP.

      I'll check out the swi-prolog newsgroup, but in the meantime, I figured I'd try something: I put "call(MethodName, Param)" in my "evaluate" clause on the server.

      Then I tried to make an assertion from the client: "server.assertz(father(john,sandy))"

      This makes Python complain that 'father' is not defined. The way that one calls a remote function from the jsonrpclib doesn't make a whole lot of sense when trying to call Prolog from Python because a statement like "assertz(father(john,sandy))" that is valid in Prolog isn't valid in Python.

      It would be better to issue remote procedure calls as strings. Something like "server.call('assertz(father(john,sandy))')"





      Delete
    3. And only now I realize that your problem may be much more mundane as follows:

      You can only assert/asserta/assertz a fact into a prolog database that is declared as
      dynamic.

      Try adding this to your prolog program (near the top):

      :- dynamic
      father/2.

      Delete
  4. For what it's worth: here's what I did.
    Please be aware that it presents a HUGE security risk if you 'd decide to run this on a (semi-)public site,
    as it allows anyone to run any code on the server...

    server.pl:
    **********

    :- use_module(library(http/thread_httpd)).
    :- use_module(library(http/http_dispatch)).
    :- use_module(library(http/html_write)).
    :- use_module(library(http/json)).
    :- use_module(library(http/json_convert)).
    :- use_module(library(http/http_json)).

    :- http_handler(root(handle), handle_rpc, []).

    http_json:json_type('application/json-rpc').

    % sample prolog program
    f1(1).
    f1(2).
    f1(4).
    f1(5).
    f2(3).
    f2(4).
    f2(6).

    server(Port) :-
    http_server(http_dispatch, [port(Port)]).

    handle_rpc(Request) :-
    http_read_json(Request, JSONIn),
    json_to_prolog(JSONIn, PrologIn),
    evaluate(PrologIn, PrologOut), % application body
    PrologOut = JSONOut,
    reply_json(JSONOut).

    evaluate(PrologIn, PrologOut) :-
    PrologIn = json([jsonrpc=Version, params=[Query], id=Id, method=MethodName]),
    MethodName = eval,
    atom_to_term(Query, Term, Bindings),
    Goal =.. [findall, Bindings, Term, IR],
    call(Goal),
    sort(IR, Result),
    format(atom(StringResult), "~q", [Result]),
    PrologOut = json([jsonrpc=Version, result=StringResult, id=Id]).

    client: rpc.py
    **************
    import jsonrpclib
    server = jsonrpclib.Server('http://localhost:5000/handle')
    print "The result is: ", server.eval("f1(R), f2(S)")


    In terminal 1, I run:

    swipl -f server.pl -g "server(5000)"

    In terminal 2, I run:

    python rpc.py

    and this prints:

    The result is: [['R'=1,'S'=3],['R'=1,'S'=4],['R'=1,'S'=6],['R'=2,'S'=3],['R'=2,'S'=4],['R'=2,'S'=6],['R'=4,'S'=3],['R'=4,'S'=4],['R'=4,'S'=6],['R'=5,'S'=3],['R'=5,'S'=4],['R'=5,'S'=6]]


    ReplyDelete