Discover Top Posts Tagged with #state_machine

state_machine

Well, it has come to this. You wrote this beautiful complex thing of a state_machine. You have 14 states and handle a ton of state changes in various combinations. You also wrote zero docs. Thank you.

#rails #state_machine #documentation

A secure and reliable event tracking system for online betting

Gambling development has its own rules and if you have chance to face them, you may know how hard it can be to meet all the gambling regulatory authority requirements. In one of our recent projects (horses racing betting website) we had to follow a lot of rules from [the ARJEL](http://www.arjel.fr) - regulatory Authority for online gambling in France. The ARJEL provides french gambling operators with license, but only if they prove that they can protect players, prevent fraud, and, the most important, prove integrity of gambling activity. Which means for developers that each bet, and most of the users' actions, must be tracked in secured locations in append-only mode. In practice, we need to follow this workflow: ![ARJEL typical workflow](http://media.tumblr.com/tumblr_m7yy6gQpt11qa44ov.png) 1. User sends a secure (https) request to the Collector, the request is intercepted before hitting the Gaming Platform (aka "GP"); 2. The Collector will hit the GP, but all params are saved from the initial request (so data can't be altered by the GP); 3. The GP is answering to the Collector: if status is OK, then the process continues, if there's error - the Collector respond to the user with the corresponding message and the process is stopped); 4. The Collector is sending data to the secure Vault; the Vault is answering to the Collector, if status is OK, then the process continues, otherwise GP (which is important) must rollback the action; 5. the Collector sends the answer back from the GP to the User (if 1-5 went with OK status). Such process is quite simple on the paper. But really hard to put in place in the real life. This article explains how we resolved this using only a few lines of Ruby. The technical specs can be found (in French only) [here](http://www.arjel.fr/-Obligations-des-operateurs-.html) and especially in this [doc](http://www.arjel.fr/IMG/pdf/annexe.pdf) ## Saving and protecting parameters Parameters sent by the player must be saved locally in the Collector, to ensure their consistency when events are sent to the vault. They can't be altered in any way by the Gaming Platform, so the best solution is to never trust it. That was solved by using ruby [closures](http://en.wikipedia.org/wiki/Closure_(computer_science)), that make up most of collector's logging. Closures will allow us to write code that can be passed like an object (to be called later) with all the variables in context saved. A very simplified version of this code could be: class Handler attr_reader :queue def initialize @queue = [] end def enqueue(trace_type, options={}, &block) @queue << block end end which can be called like this: collector = Collector.new.enqueue do Bet.create(params[:bet]) end ## Performances Next major problem was the performance - we need to take into account that Collector->Vault requests had to be as fast as possible because : * while vaulting, the user is still waiting for an answer from the server (see schema above) * for horse racing, most of the bets are placed in the last 2 minutes before the start and cannot be placed after the race start, so if the vaulting is slow, the number of the accepted bets might be low. Unfortunately, by default, only one request can be sent to the Vault at once. And here comes [Typhoeus](https://github.com/typhoeus/typhoeus), a Curl wrapper that can send send multiple requests in parallel, just what we need to improve performance (the vault has a simple HTTP API). Typhoeus gathers all the requests to send in a queue. This queue is called "hydra" and once fired, it sends all the requests with a nice concurrency. That way, we could hit the vault with dozens of requests (ie: bets) at once, in parallel. ## Atomicity As one of the ARJEL requirements, on the step 4 of the workflow it's necessary to track all the events in the requests, so the Collector creates XML document with all the info about the request (some kind of a log entry) and sends it to the Vault in the request. The main issue here is that only one XML document can be sent per request. Why is it an issue? Just imagine a typical signup request, which contains 2 XML documents (or they are also called "traces") for 2 events: one straight for signup and one for accepting Terms of Services, or "TOS" (in France you must accept TOS to register). Those 2 traces must be sent together, but as mentioned above the Vault allows only one trace per request. So, if the signup trace was not be successfully recorded (vaulted), the "Accept TOS" trace must not be sent to the Vault. That's why we introduced a [DFS](http://en.wikipedia.org/wiki/Depth-first_search) algorythm (Deep first Search) in our CollectorTransaction model, where traces are stacked and can even be in a tree within one request: canceling a node will also cancel the following sub-nodes. def cancel_dependent_transactions @stack = [] dfs(self.on_complete) @stack.each(&:cancel) true end def dfs(ctransactions) ctransactions.each do |ct| @stack << ct dfs(ct.on_complete) end end (Surprisingly, it's pretty rare to see recursive functions in Ruby.) Remember that events are sent in parallel to the vault (using Typhoeus). If we send the 2 events in parallel, there's a (small) chance that one of them fails: that's why we force some events to be sent after others. This DFS implementation will help to cancel and clean dependent traces waiting to be sent. ## Some Glue In addition, it's really important to be aware of requests' state (e.g. in case of server down). That's the role of the ```CollectorTransaction``` model, where state is maintained with [State Machine](https://github.com/pluginaweek/state_machine/), giving 2 advantages : * in case of crash, we know exactly what was the state of a request; * we could match our data with the vault data (vault wasn't directly accessible to us). The state machine was quite simple: state_machine :status, :initial => :pending do event :send_to_vault do transition :pending => :sent end event :cancel do transition :pending => :cancelled end event :rollback do transition :pending => :rollbacked end before_transition :on => :rollback, :do => :restore_model after_transition :on => :rollback, :do => :cancel_dependent_transactions before_transition :on => :send_to_vault, :do => :save_model_id after_transition :on => :send_to_vault, :do => :fire_model_callbacks state :sent do validates :response_code, :inclusion => {:in => [200]} end state :rollbacked do validates :response_code, :exclusion => {:in => [200]} # 0 = could not connect, 409 = no IDE header returned by vault end end ## Enqueuing events to be vaulted So, here's typical example of enqueued collector's request : bet = Bet.new(:combination => combination, :bet_type => bet_type, :stake => stake.to_i, :order_id => @order.id, :race_id => @race.id) collector.enqueue(:PAHIMISE, :model => bet, :extra_params => { :race_name => [@race.to_s, @race.name, @race.place.name].join(' - '), :balance_before_checkout => current_user.balance, :balance_after_checkout => current_user.balance - bet.stake}) do bet.save! end See how the closure is acting? We don't actually save the bet in the controller, but we save the context of the bet inside a ```Collector::Handler``` instance. If the vaulting went ok, then we can execute that code, and the ```bet``` will be saved. Ruby has some magic inside sometimes, because the variable "bet" only live in this piece of code, and the closure block context. As long as the collector object is alive, the context is. Note, that the model (ie: the ```bet```) in the transaction is set before being changed, this is required because existing model will be used for the rollback in case of failure. Remember we should vault only when the GP returned a valid response. Literally, this means that the ```bet``` was committed to the DB, so we can't use DB transactions to rollback in case of vaulting error (it would be too simple ;°) ). New request from the the Collector queue calls ```enqueue``` method: def enqueue(trace_type, options={}, &block) ct = CollectorTransaction.new do |ctransaction| ctransaction.trace_type = trace_type.to_s ctransaction.model = options[:model] if options[:model] ctransaction.extra_params = options[:extra_params].nil? ? {} : options[:extra_params] end ct.block = block @queue << ct if options[:after] options[:after].on_complete << ct ct.dont_enqueue = true end ct end ```block``` is executed when the collector will try to vault the whole transaction. Request parameters from the player are also saved in this model. ## Fire! Then, finally, vaulting is fired, which means "dequeuing" the handler: ActiveRecord::Base.transaction do @queue.each do |ctransaction| ... # Transaction is created ctransaction.request = create_request(ctransaction) # And the process responses ctransaction.request.on_complete do |response| if response.success? and response.headers['IDE'] ... Rails.logger.info "Vaulting successfull for transaction ##{ctransaction.id} (#{ctransaction.trace_type})" @queue.delete ctransaction if ctransaction.sent? # transactions left in the queue are failures else if ctransaction.max_attempts_tripped? # we have tried enough, it's time to give up Rails.logger.info "Vaulting failed for transaction ##{ctransaction.id} (#{ctransaction.trace_type}) with code=#{ctransaction.response_code}" ctransaction.rollback # Rollback the transaction, and cancel dependent ones (cf. CollectorTransaction model) else # let's retry! ctransaction.attempt += 1 # one more try @hydra.queue ctransaction.request end end end end end ## Conclusion Thanks to ruby magic closures, we could face and solve a very complex problem in a short coding time. This system was audited and validated by the ARJEL in 2011. We did our best to ensure the maximum of features, but keeping simplicity in mind. The whole vaulting system can be seen as a queueing system, with specific transaction system for rollbacking. Less code means less bugs, and easy maintenance, always keep that in mind! The complete collector code is available is in this [gist](https://gist.github.com/3272178).

#DFS #deep-first search #web gambling development #state_machine #ruby #Typhoeus