Rete_D

by jruizwp

The past few months have been a time of intense study and creative software implementation. Which led to the full development of the durable_rules forward inference algorithm. Before I release what I consider the durable_rules beta version, I would like to present my thoughts on what I have done with the Rete algorithm. It is difficult to talk about an algorithm without naming it. That is why, in this note, I refer to durable_rules forward inference implementation as Rete_D (D for distributed).

As I read blogs and white papers on rules engines implementations, at least in principle, I found interesting similarities between Rete_D and Drools (ReteOO as well as PHREAK). In particular it is worth mentioning the features:

  • Alpha node indexing: Rete_D accomplishes by using a trie written in C (see my previous blog entry).
  • Beta node indexing: in Rete_D `join` and `not` nodes are indexed using a lua table.
  • Set oriented propagation:  Inserts and deletes are queued. During rule evaluation, inserts and deletes are evaluated against a set of frames which facts and events stored in redis.
  • Heap based agendas: The agenda for rule execution is managed with a redis sorted set and lists ordered by salience.

Rete_D has a number of important features, which make it unique. To demonstrate them I use a simple fraud detection rule written in Ruby.

Durable.ruleset :fraud do
  when_all c.first = m.t == "purchase",
                 c.second = (m.t == "purchase”) & (m.location != first.location) do
    puts "fraud detected " + first.location + " " + second.location
  end
end

Scripting

In the example above, you might have noticed there is no types nor models definition. Events and facts are JSON objects and rules are written for the JSON type system (sets of name value pairs). A good analogy is the MongoDB query language, in which the existence of JSON objects is assumed and an explicit structured schema definition is not needed. In fact in my earliest Rete_D implementation I used the MongoDB syntax for defining rules. I departed from MongoDB query because expression precedence when using operators such as And, Or, All and Any is ambiguous. In today’s durable_rules implementation our example will be translated to:

fraud: {
    all: [
        {first: {t: 'purchase'}},
        {second: {$and: [{t: 'purchase'}, {$neq: {location: {first: 'location'}}}]}}
     ],
}

Events and facts are fully propagated down the network, that is, there is no property extraction constructs in either the library nor the Rete_D structure. In a rule definition, events and facts have to be named and referenced by such a name when expressing correlations. Any property name can be used. The results:

  1. Simpler rule definitions. 
  2. Integration with scripting languages, that provide great support for JSON type system.

Distribution

With Rete_D, the tree evaluation can distributed among different compute units (call them processes or machines). There are three important reasons for doing so:

  1. Availability: avoid single points of failure.   
  2. Scalability: distribute the work utilizing resources available.
  3. Performance: consolidate event ingestion with real-time rule evaluation.

To make this viable, pushing evaluation as close to the data a possible is critical. Alpha nodes are evaluated by the Rete_D C library and can be distributed symmetrically across compute units (such as Heroku web dynos). The Beta nodes are evaluated by a lua script (created during rule registration) in the Redis servers where the state is kept. Events and facts can be evaluated in different Redis servers depending on the context they belong to. Finally actions are to be executed by the compute units responsible for alpha node evaluation.

trie

Events

Events are a first class citizen in the Rete_D data model. The difference between events and facts is events can only be seen once by any action. This principle implies that events can be deleted as soon as a rule is resolved and before the corresponding action is scheduled for dispatch. This can dramatically reduce the combinatorics of join evaluation and the amount of unnecessary computations for some scenarios.  

To illustrate this point, let’s add the following rule to the example above:


  when_start do
    assert :fraud1, {:id => 1, :sid => 1, :t => "purchase", :location => "US"}
    assert :fraud1, {:id => 2, :sid => 1, :t => "purchase", :location => "CA"}
    assert :fraud1, {:id => 3, :sid => 1, :t => "purchase", :location => "UK"}
    assert :fraud1, {:id => 4, :sid => 1, :t => "purchase", :location => "GE"}
    assert :fraud1, {:id => 5, :sid => 1, :t => "purchase", :location => "AU"}
    assert :fraud1, {:id => 6, :sid => 1, :t => "purchase", :location => "MX"}
    assert :fraud1, {:id => 7, :sid => 1, :t => "purchase", :location => "FR"}
    assert :fraud1, {:id => 8, :sid => 1, :t => "purchase", :location => "ES"}
    assert :fraud1, {:id => 9, :sid => 1, :t => "purchase", :location => "BR"}
    assert :fraud1, {:id => 10, :sid => 1, :t => "purchase", :location => "IT"}
  end

Because each fact matches the first and the second rule expression, there are 10 x 10 comparisons, which produce 10 x (10 -1) = 90 results. In contrast, let’s replace ‘when_start’ with the following rule:


  when_start do
    post :fraud1, {:id => 1, :sid => 1, :t => "purchase", :location => "US"}
    post :fraud1, {:id => 2, :sid => 1, :t => "purchase", :location => "CA"}
    post :fraud1, {:id => 3, :sid => 1, :t => "purchase", :location => "UK"}
    post :fraud1, {:id => 4, :sid => 1, :t => "purchase", :location => "GE"}
    post :fraud1, {:id => 5, :sid => 1, :t => "purchase", :location => "AU"}
    post :fraud1, {:id => 6, :sid => 1, :t => "purchase", :location => "MX"}
    post :fraud1, {:id => 7, :sid => 1, :t => "purchase", :location => "FR"}
    post :fraud1, {:id => 8, :sid => 1, :t => "purchase", :location => "ES"}
    post :fraud1, {:id => 9, :sid => 1, :t => "purchase", :location => "BR"}
    post :fraud1, {:id => 10, :sid => 1, :t => "purchase", :location => "IT"}
  end

Remember each event can only be see once, therefore there are 10 comparisons, which produce only 5 results. As you can see events can dramatically improve performance for some scenarios. 

Context References

It is common to provide rule configuration information by asserting facts. In some cases this might lead to increasing join combinatorics. Rete_D implements an eventually consistent state cache, which can be referenced during alpha node evaluation, thus reducing the beta evaluation load. Consider the rule implemented in Ruby:

Durable.ruleset :a8 do
  when_all m.amount > s.id(:global).min_amount do
    puts "a8 approved " + m.amount.to_s
  end
  when_start do
    patch_state :a8, {:sid => :global, :min_amount => 100}
    post :a8, {:id => 2, :sid => 1, :amount => 200}
  end
end

An event is compared against the ‘min_amount’ property of the context state object called ‘global’. This evaluation is done by an alpha node using the state cached in the compute unit and refreshed every two minutes (by default). Again this can lead to reduction in join combinatorics and significant performance improvements for some scenarios.

Conclusion

Rete_D provides a number of unique features, which I believe make it relevant for solving Streaming Analytics problems where rules are expected to handle a large amount of data in real time. The following features are not yet implemented in Rete_D. They represent exciting areas for future research.

  • Beta node sharing
  • Lazy rule evaluation
  • Rule based partitioning
  • Backward inference