Polyglot

by jruizwp

From its beginning the objective of the durable_rules project has been to provide a library, which can be leveraged in the most popular languages for building internet applications. Even though durable_rules borrows a healthy number of concepts from the Production Business Rules and Expert Systems domains, I consider it critical to tap into the infrastructure and knowledge base, which has been built over the last few years with languages such as JavaScript, Python and Ruby.

The initial durable_rules prototypes were built using JavaScript and the node.js environment. As I considered targeting a larger audience, the implementation evolved into a core engine written in C, using JSON as the lingua-franca for defining both rulesets and the data to be processed. A set of thin adapters would allow using the engine in JavaScript, Python and Ruby. I borrowed the MongoDB JSON query syntax as the linguistic abstraction for defining rulesets. For optimal performance the data in Redis would be stored using the message pack format (not JSON).

arch

Because scripting languages have great support for the JSON data model (hash sets, arrays and primitives) I was optimistic (perhaps naive) this approach would be a one-size fits all solution. However, as the project progressed, I found a couple of problems with this idea. 

First problem

In durable_rules the logical (and, or) and sequence operator (when_all, when_any) precedence matters. In MongoDB an expression like: ‘qty > 100 and price < 9.95’ is represented as ‘{{ qty: { $gt: 100 } }, { price: { $lt: 9.95 } }}’. This is nice and simple, but the JSON data model doesn’t imply hash-set ordering. What if I don’t want the engine to evaluate price when qty < 100? …And then came other features such as correlations, statecharts, flowcharts, parallel, salience, tumbling windows, count windows… I departed from MongoDB queries. The example below shows how a typical rule is defined in the latest durable_rules implementation. This rule waits for three events: The first of type ‘purchase’. The second with the same ‘ip’ address as the first but different ‘cc’ number. And the third with same ‘ip’ address as the first, but ‘cc’ number different than that of the second and the first.

JSON.stringify({
    suspect: {
        all: [
            {first: {t: 'purchase'}},
            {second: { 
                $and: [
                    {ip: {first: 'ip'}}, 
                    {$neq: {cc: {first: 'cc'}}} 
                ]
            }},
            {third: { 
                $and: [
                    {ip: {first: 'ip'}}, 
                    {$neq: {cc: {first: 'cc'}}}, 
                    {ip: {second: 'ip' }}, 
                    {$neq: {cc: {second: 'cc'}}}
                ]
            }}
        ],
    }
})

Second problem

The rule above expresses precisely what I want. But I just cannot imagine manually defining tens, hundreds or thousands of rules using the format above. It is not sustainable. Ruby, with its great DSL support, gave me the creative inspiration for making significant improvements over the user model. By leveraging language extensibility I was able to define and follow the set of principles for the three libraries I have implemented so far JavaScript, Python, Ruby:

  1. Leverage the language
    • Use existing expression language
    • Use existing scoping constructs
    • Use blocks, lambdas and decorators
  2. Enable compact and maintainable code
    • Avoid nesting\scoping when possible
    • Encourage the user to define short phrases, avoid cascading
    • Facilitate troubleshooting

To the extent possible I tried driving uniformity across library implementations, however I did not sacrifice usability for uniformity. Below is a summary of the features I used for implementing the libraries. 

arch

Notes on Ruby

Ruby has the most comprehensive language extensibility: Operator overloading for logical and arithmetic expressions. Property accessor interception to leverage ‘.’ notation. Block support to seamlessly integrate actions as part of the rule definition. Contexts applied to block execution to avoid unnecessary variable qualification in expression rvalues and action implementations. All these features lead to a compact and fairly well integrated ruleset definitions.  

Durable.ruleset :suspect do
  when_all c.first = m.t == "purchase",
           c.second = (m.ip == first.ip) & (m.cc != first.cc),
           c.third = (m.ip == second.ip) & (m.cc != first.cc) & (m.cc != second.cc) do
    puts "detected " + first.cc + " " + second.cc + " " + third.cc
  end
end

‘durable_rules’ exposes a REST API to post events and assert facts. I used the Sinatra framework to implement such an API. It was very easy to integrate, except for the threading model: for each request a synchronous response is expected and multiple concurrent requests are dispatched in different threads in a single process.

One caveat, which really bothers me: In Ruby it is not possible to overload logical ‘and’ and ‘or’. I had to use bitwise operators instead. The bitwise operator precedence is different from that of their logical counterparts. So, it is required to enclose logical expressions with parens ‘(m.ip == first.ip) & (m.cc != first.cc)’ as opposed to ‘m.ip == first.ip && m.cc != first.cc’.

Notes on Python

Python has great language extensibility. Similar to Ruby: operator overloading for logical and arithmetic expressions and property accessor interception allows using ‘.’ notation for tuple names. Function decorators for rule action integration. Contexts for leveraging scoping constructs when defining rulesets, statecharts and flowcharts. This results in compact and well integrated ruleset definitions.

with ruleset(’suspect'):
    @when_all(c.first << m.t == 'purchase',
              c.second << (m.ip == first.ip) & (m.cc != first.cc),
              c.third << (m.ip == second.ip) & (m.cc != first.cc) & (m.cc != second.cc))
    def detected(c):
        print ('detected {0} {1} {2}'.format(c.first.cc, c.second.cc, c.third.cc))

I used the Werkzeug framework to implement the REST API. I considered using Flask, but I really didn’t need all its features. Similar to Sinatra and Ruby: for each request a synchronous response is expected and multiple concurrent requests are dispatched in different threads in a single process.

Python has the same caveat as Ruby regarding logical ‘and’ and ‘or’ overloading. Lambda support in Python is fairly limited, so I resorted to function decorators, still with a very reasonable outcome. In python ‘=‘ is a statement, so I had to use the ‘<<‘ operator for naming events and facts in correlated sequences.  

Notes on JavaScript

JavaScript does not have operator overloading, so a set of functions for describing logical and arithmetic expressions needs to be provided. Property accessor overloading is not readily available, I played a nifty trick to overcome this obstacle (see note below). Lambda support enables rule action integration. Contexts allow leveraging scoping constructs when defining rulesets, statecharts and flowcharts. While the result seems acceptable, the question still remains: has this crossed the usability tipping point?

with (d.ruleset('suspect')) {
    whenAll(c.first = m.amount.gt(100),
            c.second = m.ip.eq(c.first.ip).and(m.cc.neq(c.first.cc)), 
            c.third = m.ip.eq(c.second.ip).and(m.cc.neq(c.first.cc), m.cc.neq(c.second.cc)),
        function(c) {
            console.log('detected ' + c.first.cc + ' ' + c.second.cc + ' ' + c.third.cc);
        }
    );
}

Node.js provides great support for implementing the REST API. In addition the single threaded continuation based model is a beautiful invention (in my opinion), it helps with JavaScript ‘global’ variable management, enables non-blocking IO calls and provides nice concurrency control using multiple processes.

‘m.amount.gt(100)’: It is easy to take it for granted. In order to enable such an expression without requiring the user to define object hierarchies, I leveraged V8 extensibility in C++ (given that the rulesets are run in node.js). In V8 it is possible to provide a proxy object that intercepts Property Set\Get and call back a JavaScript lambda:

C++ class definition:

class Proxy {
public:

  explicit Proxy(Handle<Function> gvalue, Handle<Function> svalue) { 
    Isolate* isolate = Isolate::GetCurrent();
    _gfunc.Reset(isolate, gvalue); 
    _sfunc.Reset(isolate, svalue); 
  }
  Handle<Object> Wrap();

private:

  static Proxy* Unwrap(Local<Object> obj);
  static void Get(Local<String> name, const PropertyCallbackInfo<Value>& info);
  static void Set(Local<String> name, Local<Value> value, const PropertyCallbackInfo<Value>& info);
 
  Persistent<Function> _gfunc;
  Persistent<Function> _sfunc;
};

From JavaScript

var s = r.createProxy(
    function(name) {
      ….
    },
    function(name, value) {
    }
 );

Please find the full implementation in durable_rules code base.

A note on multi-platform support

Unfortunately a Polyglot framework does not imply multi-platform support. In the Unix ecosystem, the ‘durable_rules’ libraries (including the C engine implementations) have a high degree of portability. That is to say: having implemented the framework in the Mac OS X platform, very few changes were needed to make it work in Linux. Integrating the Windows platform has been a lot more difficult: VisualStudio, which is the premier toolset for Windows, doesn’t fully support C99 (variable length arrays, asprintf, snprintf…), luckily all these limitations can be overcome without needing fundamental changes nor creating a different code fork. Hiredis is not officially supported by Redis for Windows, the MS Open Tech team has done a nice job porting and maintaining the Redis code base for Windows. I extracted the ‘Hiredis’ portion for the ‘durable_rules’ C engine as it heavily depends on it.