README
Connected Sets
High-Performances Reactive Web Application Framework
567 tests
npm install excess
Liberating your Creativity, while fighting Climate Change
Teaser
Displaying a reactive <table> which DOM container id is sales_table, ordered by date,
for the year 2014, from a source sales dataflow coming from a socket.io server,
pulling the minimum amount of data from the server and updating the table as soon as some
data is available from the server
(for witch the complete working code including http server is available at
examples/teaser):
client.js
xs.socket_io_server()
.flow ( 'sales' )
.filter( [ { year: 2014 } ] )
.order ( [ { id: 'date' } ] )
.table ( $( '#sales_table' ), sales_columns )
;
How does this work?
sales_table is updated reactively in realtime whenever sales are updated on the server.
xs.socket_io_server() connects the client to Connected Sets socket.io server.
flow( 'sales' ) declares that the sales dataflow is needed from the server.
[ { year: 2014 } ] is a filter query, it controls how much sales data will be
pulled from the server therefore reducing both bandwidth usage and latency.
Latency is further reduced by displaying the table as soon as the first sales come from the server, improving user experience.
This query can be a dataflow, updated by DOM controls and automatically pulling the minimum amount of data from the server.
[ { id: 'date' } ] is an organizer, it can also be a dataflow dynamically
updated by DOM controls, or any other application source.
The sales_columns dataflow controls table's columns. When it is updated, columns
are reactively added or removed in realtime without any addtitional programing required.
sales_columns can be defined in the client or also come from the socket.io server
using the following declarative code:
var sales_columns = xs
.socket_io_server()
.flow( 'sales_columns' )
;
The above code automatically shares the same socket.io connection with the previous code,
reducing resource usage on both clients and servers while only pulling from the server
the additional sales_columns dataflow.
Table updates are optimized to add and remove the minimum set of rows and columns, improving client responsiveness, battery life and user experience.
What does this mean?
The Connected Sets program above is expressed in one third the words required to express the problem in plain english replacing thousands of lines of complex and error-prone code.
Connected Sets programs have no loops, and no ifs, dramatically reducing the likelyhood of bugs and hence improving productivity by orders of magnitude. Under the hood, Connected Sets provides all the optimized and comprehensively tested loops and ifs you will ever need.
These same Occam's razor declarative techniques can be applied on the server side delivering a full stack scallable and secure framework with highest performances featuring reactive database and fine-grained authorization model.
The bottom line is that Connected Sets allows you to write the minimum amount of code while resulting in the highest performance reactive applications you could only dream of writing liberating your creativity and allowing you to finaly beat large teams of developpers.
Introduction
Connected Sets (XS in short, pronounced excess) is a high-productivity, high-performances, scalable, reactive web application framework aiming at massively improving productivity, reducing servers' environmental footprint, and increasing mobile clients battery life by making an optimal use of server, network and client resources.
Development Stage
XS is already very reliable thanks to its comprehensive test suite and is currently used to deliver a production web application featuring reactive photo albums for a 3D modeling and rending company.
Some features are still work in progress and some APIs are subject to changes meaning that XS should be considered Alpha at this time.
XS should provide a functionally-rich framework by version 0.6 in March 2015 including reasonably complete documentation extracted from the code where it currently stands, and reasonably stable API.
Our Team
Connected Sets is developped by a team of experienced and passionate back-end and front-end developpers.
We are well founded, not looking for additional founding, with well enough resources to complete this project.
If you are an experienced functional javascript programmer, understand the power of reactive dataflow programming and are would like to join our team, contact us.
Why yet-another JavaScript Web Application Framework?
The short answer is because we are not satisfied with the productivity, performances, and authorization models, of existing frameworks.
Internet servers are consuming an increasily significant share of worldwide exlectricity production, contributing to climate change and threatening Internet growth as the availability of cheap fosil fuels decreases due to population growth and per capita consumption growth.
The power of Internet server is now mostly increasing through the addition of CPU cores, meaning that the key to efficient usage of server resources must shift from raw-single-thread performence to high concurency and parallelism performance. This in turn requires new programming patterns to keep, or increase programmers' productivity.
Also, one must realize that the bulk of the vast majorty of today's web applications is about controling the motion of data throughout the network. Such data is no-longer limited to strictly public or strictly private informtation, requiring complex authorization schemes. This calls for a framework that allows to greatly simplify the management of user authorizations well beyond all-or-nothing authentication.
What do you mean by performances?
Our first priority is high-performances, because we believe that performance is the key to better user experience, lowest operational costs, and a lower environemental footprint.
We are fighting simultaneously against:
- CPU cycles that consume energy to run and cool-down servers, slow-down mobile clients, and drain mobile batteries faster than anyone desires
- Latency decreasing the responsiveness of applications and user experiences
- Bandwidth usage that consume energy, and increase latency over lower-bandwidth networks
We also want to keep good performances at scale. Most frameworks either do-not-scale or scale with much lower per-server performances further increasing the need for cash while increasing environemental footprints.
The bottom-line is that we want to be in better business faster, with less cash, and a lower environemental footprint that current technologies allow.
Connected Sets addresses all of these issues thanks to its unique Subscribe / Push reactive dataflow model that works accross web browsers and nodejs servers, as well as just-in-time code generators and other optimizations.
What's the big deal about authorizations?
Writing a complex application is hard-enough, add to this any significantly-complex authorization scheme and the whole thing breaks appart, slows-down to a crawl, clutters the code with plenty of unspotted security holes throughout every part of the application, and every now and then exposes end-users' data to unauthorized users.
Most companies try to get away with it by sweeping each leak under the carpet and promissing end-users that this will never happen again, or better yet, that this never happened. Internally, this usually ends-up with more meetings and paperwork, less things done for a system that although marginally improved, will at best remain unproven.
Because it is so hard, most frameworks take a this-is-not-our-problem approach to authorizations by stating that you should use third-party libraries or plugins to deal with it, all of which have shortcomming and usually will not fit the complexity of any real-world application let alone provide acceptable performances at scale.
Connected Sets provides a simple yet highly-efficient dataflow authorization model and system architecture that delivers Reactive UI updates on authorization changes at scale.
Now, you might consider that you don't need this, that end-users can refresh their page on authorization changes. But the reality is that we can do this because we provide a model that works in all cases, without requiring you to write a single additional line of code, so that you can sleep at night knowing that end-user data cannot be exposed by some piece of code that forgot to test a role in a corner-case.
How do you improve Productivity?
By allowing you to describe what you want in a declarative style, instead of how-the-hell this could ever be accomplished.
Figuring-out how this should a) work securely, b) scale and c) have best possible performances as stated above, is hard, really hard. So hard that there is not a single company today able to achieve that without throwing millions of dollars at the problem, and/or not struggling with bugs, bottlenecks and hard-to-work-around architecture limitations.
The only thing you need to know to understand XS programs is about XS Pipelets.
xs.upstream_pipelet( parameter, ... )
.a_pipelet( ... )
.downstream_pipelet( ... )
;
A pipelet is implemented as a JavaScript function() that:
- Maintains a dataset state, e.g. in memory, mass storage, the DOM, or virtually
- Subscribes to data events from upstream pipelets
- Reacts to events emitted by upstream pipelets
- Processes these upstream events to update the state of its dataset
- Emits events to downstream pipelets
- Has no side effects on other pipelets upstream or downstream
- Has a name that describes what it provides or does
- Is syntactically connected to upstream and downstream pipelets using the JavaScript '.' operator, or as parameters when there is more than one upstream or downstream pipelet connected
- Is optimized to process large amounts for data events reactively
- Can be composed with other pipelets to provide a new pipelet
An XS program is a JavaScript program where one can mix imperative-style programming with XS declarative-style programming.
The following provides an example of a non-trivial, high-performance, data server with reactive updates on everything including authorization changes, in 60 lines of code, comments included:
var xs = require( 'excess' ); // Loads XS core pipelets, returns xs head pipelet
require( 'excess/lib/server/file.js' ); // Loads file server pipelets
require( 'excess/lib/server/http.js' ); // Loads http server pipelets
require( 'excess/lib/server/socket_io_clients.js' ); // Loads socket.io server pipelets
var database = xs.file_json_store( 'data_store.json' ); // * Input/Output dataflows to/from datastore, one-line, no external database required
var users = database.flow( 'users' ); // Dataflow of users' credentials:
var clients = xs
// Define a set of web servers
.set( [
{ ip_address: '0.0.0.0', port: 80 }, // http://example.com/
{ ip_address: '0.0.0.0', port:443, key: '***', cert: '***' } // https://example.com/
] )
.http_servers() // Start http and https servers
.socket_io_clients() // Dataflow of socket.io client connections
.authenticate_users( users ) // * Dataflow of authenticated users' connections providing user_id
;
var authorizations = database.flow( 'authorizations' ); // Dataflow of all users' authorizations
database
.dispatch( clients, client ) // Serve 64k simultaneous user connexions over one core
._add_destination( database ) // Directs output of the dispatcher to the database pipelet
;
// Individual client composition
function client( source ) { // source refers to the output of the database here
var user_id = this.user_id; // id of authenticated user
var get_query = authorizations
.filter( [ { user_id: user_id, get: true } ] ) // Get authorizations for this user
.remove_attributes( [ 'user_id', 'get', 'set' ] ) // Strip unwanted query attributes
;
var set_query = authorizations
.filter( [ { user_id: user_id, set: true } ] ) // Set authorizations for this user
.remove_attributes( [ 'user_id', 'get', 'set' ] ) // Strip unwanted query attributes
;
var socket = this.socket; // Socket to exchange data with web browser
source // Dataflows from the database through dispatch()
.filter( get_query ) // delivers only what this user is authorized to get
._add_destination( socket ) // Send data to web browser
;
return socket // Receive data from web browser
.filter( set_query ) // only collects from client unauthorized writes
;
}
Connected Sets' unique Subscribe / Push reactive dataflow model allows to solve the how so that you don't have to deal with it.
To make it easier, the API describes what you want in plain JavaScript without requiring a graphical UI to glue hard-coded and hard-to-comprehend xml or json "nodes" and "links" together as many other dataflow libraries require.
XS reactive dataflow model provides higher level abstractions handling under the hood both subscribe dataflows and information push dataflows that allow to move the least amount of information possible between clients and servers reactively.
XS Subscribe / Push Dataflow Model
The following describes implementation details implemented at XS low level. Application Architects do not need do program anything for this to happen as it is entirely hidden by XS pipelets. Understanding of the underlying model helps understand why XS is so efficient and how it scales.
Dataflow libraries usually implement one of two models:
- push: all data is pushed downstream as it happens, allowing realtime updates
- pull: data is pulled upstream when needed, allowing lazy programming, pulling only what is required, when required
For web applications' communications between servers and clients these two models are usually not acceptable for these reasons:
- The pull method is not reactive, introducing average latency of the polling period. Worse it and can consume large amounts of bandwidth if the polling period is too small. It can nonetheless be used efficiently on the client side along with requestAnimationFrame() to prevent over-updating the DOM between refreshes.
- The push method pushes all data regardless of what the downstream many need. This can result in the transmission of large amounts of unused data, usually introducing unacceptable latency and bandwidth charges.
Connected Sets implements a more sophisticated Subscribe / Push model where downstream pipelets subscribe to the subset of data they are interested in and subsequently receive all updates in a push fashion only for that subset. This allows XS to move the least amount of data between clients and servers while remaining realtime with the lowest possible latency.
XS stateless pipelets also use a lazy model where they will not subscribe to anything from upstream unless initial data is fetched by a downstream stateful pipelet. This again allows to transmit only what is really used by the application at any given time.
A subscription is done using a query dataflow that represents a kind of filter on the upstream dataflow. Because the query is itself a dataflow, the subcription can change over time.
When tens of thousands of downstream pipelets subscribe to a single pipelet using different queries, XS provides a query tree that routes data events very efficiently in O( 1 ) time (i.e. that does not depend on the number of connected clients) therefore providing a more scalable solution within a single server. Sending actual data to n clients out of N connected clients is O( n ) so actual performances depends on the application (i.e. whether n << N or not).
A network of Connected Sets servers can be arranged in a tree-like fashion to provide unlimited size query trees, e.g. to dispatch data to millions of simultaneous clients. Each server subscribes to its upstream server the subset of data it dispatches to downstream servers and clients. This allows efficient and low-latency routing thanks in part to the high performances of each individual server query tree.
Data Events, Operations, Stateless Sets and Pipelets
Internally, Connected Sets dataflows represent the evolution of data sets over time where each event modifies a set. These dataflows are therefore reactive sets change flows.
Each event carries an opperation name such as add or remove and an array of values to add to, or remove from, a set.
Stateless pipelets process values which are not materialized either in memory or other storagem their state is vitual.
Stateless pipelets process data events independently of all other events and values in the set allowing very fast operations and lowest memory footprints.
Stateless pipelets can therefore process events out of order, much like Internet Protocol packets can be routed through various paths within the Internet and may arrive at their destinations in any order.
Stateful Pipelets
A Stateful pipelet maintains the state of a set either in memory, in mass storage, or any other API that provides a storage behavior.
User Interface pipelets are stateful as these present the state of a set through the DOM.
Much like the TCP protocol in the Internet which is responsible for end-to-end communications consistency, Stateful pipelets may receive data events in any order and are responsible for maintaining an application-consistent state.
Stateful pipelets are implemented thanks to the stateful set() pipelet that is typically used as a base pipelet for all stateful pipelets.
Also, much like the TCP protocol, stateful pipelets are found at the edges of a Connected Sets network of stateless pipelets.
Horizontal Distribution
Allowing out-of-order data events is a key feature of Conneceted Sets which greatly eases horizontal distribution of workloads and charding, because no synchronization is needed between chards that may be processed in parallel either over a number of threads, processes, or servers in a true share-nothing architecture.
Incremental Processing
Incremental sets processing allows to split large sets into optimal chunks of data rendering data to end-users' interface with low-latency, dramatically improving end-user experience. Data events update sets in real-time, on both clients and servers.
Incremental aggregates allow to deliver realtime OLAP cubes suitable for realtime data analysis and reporting over virtually unlimited size datasets.
Loops, Just-In-Time Code Generation
XS data events contain arrays of values which are typically processed in loops. In a traditional programming environement, one typically writes code that processes values in loops. With Connected Sets, architects do not write loops because these are absracted away as sets processed by pipelets.
This greatly simplifies programming while removing the likelihood for common programming errors.
Highest performances are provided thanks to Just-In-Time code generators delivering performances only available to compiled languages such as C or C++. Unrolling nested loops provide maximum performance while in turn allowing JavaScript JIT compilers to generate code that may be executed optimally in microprocessors' pipelines.
XS Pipelet Programming
At the lower level, XS Pipelets use a JavaScript functional programming model eliminating the typical callback hell of assynchronous request-response programming models.
Unlike the promises model, XS exceptions and errors may be carried-out out-of-band through global dataflows that can be processed, for error recovery, end-user feedback, and logging.
Error dataflows originating on clients can easily be routed to servers to allow proactive debugging of errors while in production, and effective service quality monitoring.
Transactions allow to group related operations over time and allow synhronization of concurrent dataflows. This synchronization becomes implicit at the architect level removing the possibility of race condition errors that have been traditionaly very hard to debug.
Service Architecture
With Connected Sets, services are typically composed of three different services:
- A stateful network of persistent database pipelets
- A stateless network of event dispatchers, acting as a marshalled multicasting network for dataflows
- A stateful network of client widgets delivering applications to end-users
For small applications with few simultaneous users the first two typically reside in a single server, while complex applications with large number of active users will be running on different servers. Because pipelets share no state they can easily be distributed.
A company could run multiple services through a single network of stateless event dispatchers, acting as web service aggregator.
The different nodes of an XS network communicate using the XS protocol that provides the Subscribe / Push service over a reliable transport (such as Sockets, WebSockets, ...) but not necessarily guarantying the order of packets. So XS could also work over a protocol that would only guaranty the delivery of packets.
The XS protocol therefore provides a much higher level alternative to existing web services protocols such as SOAP and REST, allowing to build efficiently complex real-time applications with no additional code.
Realtime Data Portability, Business Opportunities
A network of services sharing the same event dispatcher network enables to effectively separate XS Data Providers from XS Application Providers increasing business opportunities arising from the portability of reactive dataflows updated in real-time and as authorized by end-users.
Within an XS network, end-users no longer need to duplicate their personal data endlessly and updates are propagated to all applications in realtime putting an end to today's world of out-of-date data between services.
People will now expose their data, using a variety of services to view, edit, and publish their data to other people.
Using only stateless pipelets, this architecture will reach internet-scale very efficiently, delivering a Marshalled Subscribe / Push multicasting data exchange for services to share data among many service providers, while representing a business opportunity for XS Network Providers much like today's CDNs but for marshalled dynamic real-time content solving caching issues thanks to the immutability of data events.
To participate in this network, service providers only need to publish dataflows and/or subscribe to third-party dataflows.
End-users may use these services to backup their own data either on owned servers or using third-party XS Data Providers.
End-Users control access to their own data through XS Authorization reactive dataflows providing additional business opportunities for XS Authorization Management Providers helping end-users manage authorizations for all their data accross all their XS Applications.
Monetization of dataflows and applications can be controlled through XS reactive authorizations by XS Monetization Providers.
Disruptive new business opportunities arrising from XS Realtime Data Portability will prove much stronger than the current closed, data-within-application model, resulting in more data and more services available to more users and businesses.
Ecosystem
XS backend runs on Node.js providing a scalable database, web server, validation, and authorizations.
On the frontend, XS provides reactive controlers and views driven by reactive dataflows.
XS can optionally be coupled with any other framework but we recommend using reactive libraries such as AngularJS, Ember, Bacon.js, React, which model is closer to XS.
For responsive layouts, we recommand Bootstrap that we use it for our reactive Carousel and Photo Albums.
For DOM manipulation one can use any library, or none at all, as XS core has no dependencies.
XS can either be used to gradually improve existing applications on the back-end or front-end, or as a full-stack framework.
Integrated database and model
XS features a chardable document database with joins, aggregates, filters and transactions with eventual consistency allowing both normalized and denormalized schemes.
*Persistance will be implemented in version 0.4 and charding in version 0.7.
Demonstration Site
A demonstration and beta test site is available here.
The source code for this demonstration site is in the GitHub repository ConnectedSets / demo.
Documentation
This readme provides an introduction to Connected Sets.
The bulk of the documentation is currently embedded in the code of lib/pipelet.js
and other sources for the core as well as individual pipelets' sources.
We plan on extracting and completing this documentation to provide the following manuals:
- A Tutorial
- ConnectedSets Application Architect Manual
- ConnectedSets Pipelet Developper's Guide
- A Reference Manual for all available pipelets by module
Automated Tests, Continuous Integration
We have curently developped 567 automated tests for the XS core pipelets that run after every commit on Travis CI under node versions 0.8, 0.10. We no longer test version 0.6 since Travis had an issue with it around January 2014. Version 0.11 is not officially supported because ui tests using zombie cannot pass at this time.
Our continuous integration process also requires that before each commit the developper runs these tests so travis usually passes all tests. In the event that a test does not pass the top priority is to fix the test before anything else.
We also do manual testing on the following web browsers: Chrome (latest), Firefox (latest), IE 8, 9, and 10 but enventually plan on dropping support for IE8.
We publish to npm regularily, typically when we want to update the demonstration site or each time we increment the minor version. If you need more updates just let us know.
Contributions
Contributions are welcome but difficult at this time because XS is still in alpha, APIs may still change, and low-level documentation is incomplete requiring a lot of support on our part. We however welcome full-time contributors disiring to join our team.
Installation
From npm, latest release:
# npm install excess
Some image manipulation pipelets require ImageMagick that you can download here.
Running tests
# npm install -g coffee-script
# npm install -g mocha
# npm install expect.js
# npm install mocha-unfunk-reporter
#
# git clone https://github.com/ConnectedSets/ConnectedSets.git
#
# cd ConnectedSets
# ./run_tests.sh
Full test results are in test.out
-> passed 567 of 567 tests (4196ms)
#
# less -R test.out # for tests detailed traces
Example of complete client and server application
Application retrieving sales and employees from a server, aggregates these by year and displays the results incrementally in an html table.
This example also shows how to produce in realtime minified all.css and all-min.js. All assets content
is prefetched in this example for maximum performance. The less css compiler is also used to compile in real
time .less files. The same could be done to compile coffee script or use any other code compiler.
index.html
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Connected Sets - Aggregate Sales by Year and Employee</title>
<link rel="stylesheet" href="all.css" />
<script src="all-min.js"></script>
<script src="/socket.io/socket.io.js"></script>
</head>
<body>
<div id="sales_table"></div>
<script>client()</script>
</body>
</html>
client.js
"use strict";
function client() {
var xs = XS.xs // the start object for XS fluent interface
, extend = XS.extend // used to merge employees and sales
, sales = [ { id: 'sales' } ] // Aggregate sales measures
, by_year = [ { id: 'year' } ] // Aggregate dimensions
// Table columns order by year and employee name
, by_year_employee = [ { id: 'year' }, { id: 'employee_name' } ]
// Define table displayed columns
, columns = [
{ id: 'year' , label: 'Year' }, // First column
{ id: 'employee_name', label: 'Employee Name' },
{ id: 'sales' , label: 'Sales