Documentation

Documentation/Architecture.txt

@@ -0,0 +1,170 @@
+    Okay, this is a little rough but here goes.  There are two main hubs 
+of data flow involved in this system:
+
+    The first is local to the embedded system and is implemented by the 
+ipc_server daemon (source located in the commhub/ directory).
+
+    The second is a WAN based synchronization service that is responsible
+for striving for the best possible balance between always-ready and eventual
+consistency between the database state stored on the bus (the client) and
+the master database state maintained by the garage (the server). 
+
+---------------------------------- Interprocess Communication Hub -----------
+
+    This module provides majordomo-like subscription based interprocess 
+communication between local modules using a UNIX domain socket in 
+SOCK_SEQPACKET mode.  This server functions as a smart router/switch 
+allowing each module to address any/all other modules with any
+of the following addresing modes:
+
+  1) Mailbox Name (this will deliver to every process that has subscribed to the named mailbox).
+  2) "_BROADCAST" (this will deliver to all connected clients)
+  3) ">nnn" (this will deliver the message to the client with PID == nnn)
+  4) ":module" (this will deliver to all clients registered with the supplied module name).
+
+    The server also manages a number of "special" mailboxes that execute
+specific communication flow management functions.  These mailboxes are as
+follows (these are #define'd string constants that resolve to the actual
+special mailbox names):
+
+  1) MAILBOX_SUBSCRIBE	(subscribes the sending client to the mailbox named in the payload)
+  2) MAILBOX_UNSUBSCRIBE (unsubscribes the sending client to the specified mailbox)
+  3) MAILBOX_BROADCAST (sends the message to all connected clients)
+  4) MAILBOX_WIRETAP (payload == "ON" to select wiretap mode, anything else to deselect).
+
+    When a client select wiretap mode it will receive an unmodified copy of
+each and every message from that point forward (until wiretap mode is
+deselected).  This includes all special management messages and PID or
+module addressed message.  This allows for easy message flow debugging
+(including debugging the communication hub itself) and for robust on-the-fly
+selectable logging in the field without any need to recompile or even
+interrupt the system.  This allows us to attach to a running system in any
+state (including failure states) and transparently observe data flow between
+modules.
+
+    The server listens on a socket located at /tmp/commhub.  Clients connect
+using the connect_to_message_server() function which returns a file
+descriptor that represents the connection to the IPC server.  This function
+makes the socket connection and registers the client process with the server
+by both PID and module name.
+
+    Once a client is registered, messages can be sent to the server by
+defining a message structure and populating it with the prepare_message()
+function call and then calling the send_message() function to dispatch the
+message to the server.
+
+    Incoming messages can be retrieved with the get_message() function after
+a call to poll() indicates POLLIN on the fd for the connection to the
+server. If there are no other file descriptors to poll, or a quick
+non-blocking check is desirable, there is a provided poll wrapper called
+message_socket_status() which can be passed the fd associated with the
+connection and a mask of MSG_SEND | MSG_RECV which will specify whether a
+message can be sent and/or whether there is a message waiting to be received 
+on that connection at the present moment. 
+
+    An important feature of this architecture is that it allows for the
+modules communicating through the hub to be started, stopped, and replaced
+independently and transparently.  If the modules are designed with sensible
+opaque interfaces this allows for a remarkable about of flexibility in
+implementation and testing as modules can be fashioned to function as test
+jigs to test other individual modules or complex subsystems of modules. 
+Test jigs can even be built to emulate fault conditions to test system
+recovery and failure-workaround.
+
+    Modules can also be built to hide hardware-servicing routines.  For
+instance, there could be a module that was responsible for communicating
+with every peripheral that lives in the passenger interface unit (RFID,
+Magstripe, Cash Vault, and Passenger Facing Display).  This module can
+simply multiplex and demultiplex the serial communications with those
+peripherals and create mailboxes to receive any data that needs to be
+transmitted to those peripherals and send any input received from those
+peripherals to some interface-specified mailboxes (one per peripheral and/or
+one per message type).  The beauty of this abstraction layer is two-fold. 
+First, it allows things like rebooting the PIU to happen without the need to
+interrupt or reload any other process (minimizing collateral damage from
+unneccesary depdendencies).  The second place this pays off is in the case
+where a hardware change causes a peripheral to migrate from one subsystem to
+another:  In this case, only the modules that directly service I/O to those
+peripherals need to be aware of this change... all other modules will
+continue to use the agreed upon message passing interface and the fact that
+the (for instance) cash-valut relay has been moved from the Passenger
+Interface Unit to the Driver Interface Unit will not matter to any of the
+other modules that need to make use of that service.
+
+    This design was heavily influenced by the elegance of the Erlang
+ERTS framework.  My goal here to make a very lightweight and portable
+C-based system that provides a similar type of modularization and
+abstraction such that we can have maximum ongoing implementation flexibility
+and robust testing similar to that provided by the ERTS framework without
+incurring the overhead (and hassle) of maintaining out own cross-compiled
+ARM port of ERTS (and without compelling all the other developers to learn
+Erlang (which with such a tight development schedule would be madness)).
+
+
+-------- Client (bus) to Server (garage) Synchronization System ---------
+
+
+    This module actually consists of a server component in the garage which
+is responsible for maintaining a master database which is considered the
+'absolute truth' and always contains the canonical system state.  There is a
+second component which lives on the bus (the client) which is intermittently
+connected to the garage (as limited by network availability).  The goal of
+these two modules to to maintain synchronization with as great a degree of
+accuracy as is practical while taking the following constraints into
+account:
+
+  1) Connectivity is spotty.  A client may loose touch with the server at
+any time, and therefore must ALWAYS be in a state where it can function
+autonomously with a reasonable degree of predictability (it must not confuse
+or thwart the drivers or riders).  This means there is a possibility of
+accepting a fare (for instance) on a pass that may have been used up during
+the communication outage, or other such conditions.  We must always be sure
+to err in favor of permitting 'normal' system functions.  It acceptable
+to occasionally give out what turns out to be a free ride on a recently
+expired pass, it is not acceptable to EVER reject a fare on a valid pass.  
+
+  2) Due to the multitude of busses and the intermittent nature of the
+connectivity, each client bus can only make relative declarations (Rider X
+has used 1 ride, decrement their pass) and the burden of aggregating and
+calculating the resulting system state lies on the server.  Each bus may
+individually update its own state, but it must always allow an update from
+the server to override any local changes.  An example scenario:
+
+   2a) Rider-X boards Bus-Y with an n-Ride pass with 7 rides left.
+   2b) Bus-Y decrements its LOCAL count for that pass to 6 rides.
+   2c) Bus-Y transmits a message to the server "Rider-X's pass has used 1 ride"
+   2d) Rider-X' (Rider-X's evil twin) gets on Bus-Z with another copy of the same pass.
+   2e) Bus-Z decrements its LOCAL count for that pass to 6.
+   2f) Bus-Z transmits a message to the server "Rider-X's pass has used 1 ride"
+   2g) The server receives both decrement messages and transmits to ALL
+       busses the message "Rider-X's pass now has 5 rides left"
+   2h) Bus-Y overwrites its LOCAL copy of Rider-X's pass count with the new
+       one from the server (5).
+   2i) Bus-Y overwrites its LOCAL copy of Rider-X's pass count with the new
+       one from the server (5).
+
+  3) For this method to work, the server must serialize all incoming events
+that have been transmitted from the individual busses and apply each change
+as a transaction in the master database.  This is a related but distinct
+process from the next step.
+
+  4) In incrementing serial number must be kept for each change in the
+master database.  Each time a client bus checks in with the server it should
+supply the serial number of the last successfully integrated transaction (a
+transaction is counted as successfully integrated by the client once the
+local database has been updated and the changes have been successfully
+sync'd to secondary storage).  The server mush then supply that client with
+a batch of all changes with a serial number greater than the supplied key.
+The client will then integrate those changes, synchronize to secondary
+storage, and then increment its own key.  In this manner a client can go an
+arbitrary amount of time without checking in and then receive a batch of all
+messages that need processing in the mean time.
+
+  5) Either end (server or client) always has the right to request a full
+synchronization where the client database is wiped and replaced wholesale by
+the server copy, and the serial number is updated to the newest in the
+transmitted 'full' copy.  This can be invoked if a client hasn't received
+incremental updates in some time, or it can be used as a troubleshooting
+measure if there is a suspected transmission or storage error whereby the
+client appears to have in incomplete or incorrect database.
+

Documentation/Module_Overview.txt

@@ -0,0 +1,182 @@
+
+	There are two sets of modules involved in this system, one set lives
+on each bus (we will call them Client Side), and those that live on the
+central server (we will calal them Server Side).
+
+--------------------------------------------------
+		Client Side Modules
+--------------------------------------------------
+
+Module Name:		PPP Dialer
+
+Module Function:	
+
+	Maintains connection with the server as much as possible using pppd
+to set up a gprs internet connection, and then using an ssh client to set up
+secure tunnels for each of the underlying Client->Server protocols
+implemented by the other modules.
+
+Module Files:		(these files live in /etc/ppp/)
+
+	ppp-dialer.sh	(script to manage ppp connection and SSH tunnels)
+	pap-secrets	(password file for bogus user that modem needs)
+	chap-secrets	(same as above)
+	ip-down.local  	(script to run when pppd link does down)
+	ip-up.local	(script to run when pppd link comes up)
+	options		(global pppd options)
+	chat/disconnect	(modem disconnect sequence)
+	chat/gprs	(modem init sequence, gprs specific dial string)
+	peers/gprs	(modem location, gprs specific pppd settings)
+
+Module Outputs:
+	
+	/tmp/network-is-up	(this file is present when ppp0 is up)
+	/tmp/tunnel-is-up	(this file is present when SSH tunnel is up)
+	/tmp/ssh_tunnel.pid	(pidfile for ssh client implementing tunnel)
+
+-------------------
+
+Module Name:		Client Supervisor
+
+Module Function:
+
+	This module serves a function for our application (as a set of
+client modules) as init servers for UNIX.  It is responsible for spawning
+a supplied set of other modules and gathering any log messages they may
+generate and routing them to the correct logging location.  It also performs
+"aliveness" monitoring for all of the child modules it spawns, resetting any
+module that has hung (and logging it), as defined by not getting PONG
+messages back from a module that's been sent a PING message through the IPC
+hub.  This provides a sanity check that will log (for debugging/improvement)
+any crashed process, but will also respawn it to maintain maximum system
+functionality.
+
+Module Files:		(these files live in the bin directory)
+
+	client_supervisor	(the binary executable)
+
+-------------------
+
+Module Name:		IPC Hub
+
+Module Function:
+
+	This module serves as a communication hub for all of the other
+modules to send notifications, requests, and data structures to one another
+in an efficient manner.	 It provides a UNIX domain socket in /tmp which
+client processes can connect to and then subscribe to mailboxes, as well as
+send messages to mailboxes.  Every message sent to a mailbox gets delivered
+to each process with an active subscription to that mailbox.  This module
+also provides a wiretap feature which allows a debug client to attach to a
+running system and watch the messages as they flow through the hub allowing
+live trouble shooting.
+
+Module Files:
+
+	/tmp/commhub	(UNIX domain socket to which clients connect)
+	ipc_server	(this is the executable for this module)
+	debug_client	(this is the wiretap client for live debug)
+
+-------------------
+
+Module Name:		Billing Database Client
+
+Module Function:
+
+	This module collects submissions of billing and diagnostic log
+messages from other modules through its IPC mailbox and stores them in a
+transaction synchronous local database.  When an SSH tunnel is available to
+the central server, it will attempt to flush any message that hasn't yet
+been confirmed by the server, retransmitting as needed.  
+
+Module Files:
+
+	billing.mem	(mmap()-able file containing database of log entries)
+	billdb		(billing database client)
+
+-------------------
+
+Module Name:		Pass Database Client
+
+Module Function:
+
+	This module maintains a transaction synchronous local database of
+riders (rider ID, Magstripe, RFID, Rule, Rule Parameter) which is kept up to
+date with the server whenever there is an SSH tunnel available.
+	This module also listens to the mailboxes for incoming credentials
+from the passenger interface unit and looks them up in the database.  If a
+rider record is found for that credential, the associated rule is applied
+which will generate a billing log entry (and possibly passenger interface
+and driver interface messages).
+
+Module Files:
+
+	passes.mem	(mmap()-able file containing database of riders)
+	passdb		(buspass database client)
+	init.scm	(library initialization for scheme interpreter)
+	rules.scm	(scheme definitions for rules attached to riders)
+
+-------------------
+
+Module Name:		Passenger Interface Unit
+
+Module Function:
+
+	This module communicates with the Passenger Interface Unit connected
+by serial port and reports any credential presentation events to the
+appropriate mailbox on the IPC hub.  This module also listens to a mailbox
+on the IPC hub for commands instructing it to update the text on the
+passenger facing display.
+
+Module Files:
+
+	piu_minder	(executable file for PIU module)
+
+
+-------------------
+
+Module Name:		AVLS Client
+
+Module Function:
+
+	This module provides Automatic Vehicle Location Service data to the
+server as connectivity allows, sending the current vehicle location,
+heading, velocity, and information about who is logged in and what route is
+being driven to the server whenever there is an SSH tunnel available.
+
+Module Files:
+
+	avls		(executable for AVLS client)
+
+
+--------------------------------------------------
+                Server Side Modules
+--------------------------------------------------
+
+Module Name:		Bus Pass Server
+
+Module Function:
+
+	This module responds to client busses when they ask for rider and
+pass updates.  It maintains a unique sequence number for each update and
+when requested gathers a list of the latest change to each rider after any
+given sequence number.  This constitutes enough information to bring a
+client up to date.  If a client is too far out of date, or if the client
+requests it explicitly, the server will send a ZFLUSH command, followed by a
+compressed list of all valid riders, followed by a ZFLUSHDONE command.  This
+allows a stale client to efficiently get to an up-to-date state.  
+
+-------------------
+
+Module Name:		Billing Log Server
+
+Module Function:
+
+	This module responds to a client's submitted log entry (billing or
+diagnostic log) by responding with (ACK, DUP, or IGN) messages followed by
+the MD5 checksum of the submitted message.  This response allows the client
+to be sure that the message has been logged in the central database, and may
+therefore be purged from the local database where storage space is at a
+premium.
+
+

Documentation/billdb-to-server.txt

@@ -0,0 +1,56 @@
+
+	The billdb module keeps track of the local copy of the billing log
+database by reciving billing log entries from other modules and storing
+them in a local database. Each message is a single line of ASCII text 
+with tab delimited fields (there is no valid reason to allow a tab in
+any of the fields, any tabs inside the fields should be translated to single
+spaces) and terminated by a newline.
+
+	When the client wants to commit new records, it 
+issues a commit request in this format:
+
+
+
+BILLINGENTRY
+BILLINGENTRY
+...
+
+where BILLINGENTRY is a record formated like this:
+ 
+0)  equip_num     INT\t
+1)  driver        INT\t
+2) paddle        INT\t
+3)  route         INT\t
+4)  trip          INT\t
+5)  stop          INT\t
+6)  ride_time TIMESTAMP\t
+7)  latitude      DOUBLE\t
+8)  longitude     DOUBLE\t
+9)  action VARCHAR(16)\t
+10)  rule  VARCHAR(24)\t
+11)	 ruleparam VARCHAR(24)\t
+12)  reason VARCHAR(32)\t
+13)  credential VARCHAR(32)\t
+14)  cash_value INT\t
+15)	 stop_name	VARCHAR(64)\n
+
+OR a log entry formatted like this:
+[!*#]some data....\n
+
+	The serevr will reply to EVERY message with either:
+	
+DUP	md5sum
+
+or
+
+ACK	md5sum
+
+or
+
+IGN	md5sum
+
+	Either if which will allow the client to remove its local copy of 
+the log entry.  Do not reply with ACK or DUP until the row has been stored
+to the server-side database.  Reply with IGN if you have been sent garbage
+and want the client to stop retrying that garbage string.
+

Documentation/passdb-to-server.txt

@@ -0,0 +1,117 @@
+
+	The passdb module keeps track of the local copy of the bus pass 
+database by polling the server from time to time to ask for updates
+newer than the latest locally recorded sequence number.  Each message
+is a single line of ASCII text with tab delimited fields (there is no
+valid reason to allow a tab in any of the fields, any tabs inside the
+fields should be translated to single spaces) and terminated by a
+newline.
+
+	When the client wants to inquire about any new records, it 
+issues a query request in this format:
+
+QUERY	1234567890
+
+	Where QUERY is the command, then a tab character, then the
+newest sequence number which the local client has in its database,
+followed by a newline.  The QUERY record has the following fields:
+
+0) Command 		= QUERY
+1) Sequence Number 	= Newest locally stored sequence number.  If
+			  this number is zero, it indicates a table 
+			  flush to the server, triggering a flush 
+			  marker to be sent before the first reply
+			  record.
+
+	The server will respond with a sequence of bus pass updates
+with the following allowable formats:
+
+UPDATE	1234567891	54321	2:771231232	35:111:2357	TCAT-NRIDE	7
+DELETE	1234567892	938773
+
+	The UPDATE record has the following fields:
+
+0) Command 		= UPDATE
+1) Sequence Number 	= Sequence number of this update.
+2) Rider ID		= Unique ID of this rider's record.
+3) Magstripe Token	= The magnetic stripe token in the format of
+			  track:track_data by which this rider is
+		          recognized.
+4) RFID Token		= The RFID token in the format of
+			  n_bits:site:id by which this rider is
+			  recognized.
+5) Rule Name		= The name of the rule to apply to this rider.
+6) Rule Parameter	= An (optionally blank) parameter to pass to the
+			  rule that validates this rider.
+
+	The DELETE record has the following fields:
+
+0) Command		= DELETE
+1) Sequence Number	= The sequence number of this deletion.
+2) Rider ID		= Unique ID of this rider's record.
+
+
+	If the client requests a query from Sequence Number 0, the
+server will send a flush marker which will prompt the client to flush
+its tables before processing any further records.  This is sent to
+prevent lag from causing a previous request's records from polluting
+a clean refresh.  It looks like this:
+
+ZFLUSH	523498
+<523498 octets of binary data>
+
+0) Command 		= ZFLUSH
+1) Data Size		= How many bytes of zlib compressed (with deflate)
+			  binary data follow before the command stream picks
+			  up again.
+
+	A flush is also triggered by the server if a query is sent with
+an extremely old sequence number, or a sequence number greater than the
+newest sequence number on the master server.
+
+	After a flush in completed, the server MUST send a ZFLUSHDONE
+command so the client can become aware that it is no longer in ZFLUSH mode.
+Doing so will trigger the client to actually apply the compressed data which
+it received in the ZFLUSH blob.
+
+This looks like this:
+
+ZFLUSHDONE
+
+0) Command		= ZFLUSHDONE
+
+
+NOP
+
+0) Command		= NOP
+
+	If a QUERY message results in NO data sent back to the requesting
+client, an NOP command will be sent to indicate that the connection is still
+alive.  This is useful for debugging, but has little value for end-user
+display.
+
+-----------------------OBSOLETE----------------------------------------
+-	If the client requests a query from Sequence Number 0, the
+-server will send a flush marker which will prompt the client to flush
+-its tables before processing any further records.  This is sent to
+-prevent lag from causing a previous request's records from polluting
+-a clean refresh.  It looks like this:
+-
+-FLUSH
+-
+-0) Command 		= FLUSH
+-
+-	A flush is also triggered by the server if a query is sent with
+-an extremely old sequence number, or a sequence number greater than the
+-newest sequence number on the master server.
+-
+-	After a flush in completed, the server should send a FLUSHDONE
+-command so the client can become aware that it is no longer in FLUSH mode.
+-This looks like thise:
+-
+-FLUSHDONE
+-
+-0) Command		= FLUSHDONE
+
+
+

experiment/prototype/cleanup-all

@@ -35,8 +35,9 @@ function ngc_position {
 
 ./svg2ngc "back-plate.svg"
 ./svg2ngc "top-plate.svg"
+./svg2ngc "aux-plate.svg"
 
 
 ngc_position "back-plate.ngc"
 ngc_position "top-plate.ngc"
-
+ngc_position "aux-plate.ngc"