As 2005 approached many businesses were investigating ways to switch their applications over to an open WS* model that could be used to allow those applications to expose and consume services. In essence, the XML revolution allowed applications running inside enterprises the ability to communicate and receive data using protocols not unlike that used by the web itself. In fact the html language syntax is itself a subset of the XML family of structured languages, based on the Standard Generalized Mark Up Language (SGML). web 2.0 saw its flowering in the form of consumer sites that utilized it to the fullest with the emergence of youtube, the continued growth of google, the transformations of yahoo, ebay and amazon and in the emergence of countless new 'startups' that employed very simple ideas implemented with unique dynamic elements enabled by either AJAX or by using the now mature Flash technology from Adobe that had finally reached sufficient performance on the newer more powerful computers to enable creation of rich internet applications. While the front end and application technologies changed so did the preference for the back end development platforms used to produce web applications. The weight and expense of the full content management systems like Vignette could not be supported on the tight budgets of the couples of college students that wanted to build web applications. For them , it was much easier to coddle together a site using the new server side scripting language platforms that emerged to allow easy back end development without the need of a heavy application server. Python, PHP and Ruby were the languages of choice for this set and soon these languages had "platforms" designed around them to enable them to be used much like the old content management applications were but without the expense. Plone, Drupal and Ruby on Rails became the foundation upon which many simple web 2.0 web applications were put together. Ruby on Rails in particular brought together the dynamic page creation (data side with the database and interface side with AJAX) and page management elements necessary to create fast sites. Unfortunately, these new platforms lacked the stability and scalability that would be necessary for them to serve as the base for massive growth. The web 2.0 era has been dotted with spectacular successes after much difficulty on the back end (twitter, digg) as the hodge podge of technologies put together to define these sites were optimized to scale to tens of thousands to millions of simultaneous users.
However, despite the success of many of these sites today, there is much room for improvement. The scalability that many of them currently have comes at the cost of large amounts of server resources and massive arrays of disks, not to mention the expense of running those servers, hiring managers for those servers and in general managing all the employees of the rapidly growing companies that they work for. The ability to design a truly scalable, secure and managable web application framework could never be found as the web iterated from 1.0 to 2.0, the next generation of web applications will take advantage of this deep synthesis of efficiency in the back end code design from the ground up to enable extremely efficient and dynamic web applications and sites using any of the desired technologies of the previous generation to build even more resource intensive sites but without the massive expense for hardware resources, managing users and employees. Enter the next step...
The economic slowdown of 2008 began with troubling signs in the housing market that soon snowballed into the collapses of several institutions (Fannie Mae , Freddie Mac) for home mortgages and several large and old banks (Leaman Brothers, Bear Stearns, Washinton Mutual) shutting their doors over night. The cascade of failed banks led to a tightening in the credit markest which again as in the 70's and the early 2000's made it extremely difficult for entrepreneuers to acquire venture capital. I consider web 3.0 a defining moment because the type of software that I believe that will characterize it is what I've been working on for the last 7 years. In order to realize the extreme efficiencies of low resource costs under massive scalability, the design time necessary to engineer a generalized solution had to be greater. When I was laid off from TheStreet.com in 2001, I decided that the best way to pursue my dream of running my own business was to start it then. I had a brain full of experience working in a highly challenging environment , where I engineered solutions using the emerging xml , content management and dynamic UI technologies. Before the end of 2001, I began the design of a generalized Entity management system. This system would be completely different from previous content management systems in that it would not specialize on any particular type of content. Content managent up to that time had been dominated by management only of certain types of content important for web applications or sites. Stories and Articles , Images, Video and Audio files were the ones that made sense within the context of web applications by 2005, management systems broadened their scope to include any file type but the systems failed to manage the most critical elements of a running business that had failed to be properly encapsulated into a combined content management system. Business processes. A truly generalized management system would include Users , Servers and other structural elements of the very application under the management paradigm. If articles , stories and images could be updated, created or deleted so should servers, users and any other conceivable entity of the business given objects in the system. This is what an Entity Management System would do. The task is a difficult one as it is an attempt to solve a problem landscape that is multi dimensional but at the same time still is easily managed both as new Entity types are added to the system and as new instances of all those types proliferate on the system.
Around 2004, when the bulk of my Entity Management API had been built I happened across an article on the J2EE java technology platform. I looked through a few documents and noticed immediately the use of "entity" to describe a generalized management type in the J2EE framework. However, continued reading of the technology quickly led to head aches, the reason was that J2EE was over complex. It tried to solve the entity management problem but not holistically. It had many different possible persistence API's that could be used, it had a strict class inheritance scheme that placed many requirements on the client programmers that didn't seem natural during the class design. After reading a bit on J2EE I was happy that I designed my API using core java from scratch. For a generalized web platform the most critical components were deemed as follows:
- Compositional xml based type (Entity) framework.
- Scoped security model based on hiearchical granular permissions.
- Twin hieararchy for Entity base class to optimize object instance sizes.
- Custom persistence API for optimal mediation from datastore.
- Action oriented Workflow for all Entities.
- DB Vendor agnostic database API.
- XML based bootstrap system.
- compiled software distribution for easy installation.
- Extensible during run time to add new Entities and thus new functionality.
- dynamic request load distribution across participating cluster nodes.
- dynamic request load distribution between branches of nodes for out of the box, geographic scalability
- "Manages itself" requirement, all templates and scripts that compose the framework are managed by the framework.
- secure web 2.0 Collaboration API to enable users to communicate in real time concerning actions over all Entities.
- Content delivery Protocol (ssh, sftp,file..etc.) and File type (.doc, .pdf, .html..etc) API
A little bit about these requirements and why they were deemed critical to a web 3.0 web application platform. The use of xml as a compositional core is what allows the content types that are based on standard content formats managed in traditional CMS systems to be easily handled by the new platform. XML based representations for articles, stories or even advertisements could be related to one another in the datastore and the xml of the composed object generated using a global "toXml()" method. Transformation of the composed xml could then follow quickly using the Xsl transformation language templates that can be added for that purpose. Note there is no further delineation of what is being composed, the point is any compositional relationship between business entities can be extracted as xml after being composed dynamically in the framework or as was retreived from the datastore. This allows the final rendering of content types from the composed xml to any desired file type using any desired delivery protocol.
Scoped permissions allow any action against any Entity to generate a unique permission id on the system. Each scope indicates the hieararchy of application for the permission. There are permissions that apply to all Entities, all instances of a single Entity or a single instance of a single Entity. Instance permissions are generated on demand, thus only permissions for objects in use are actually available to disceminate ensuring minimal granted rights for the entire entity landscape over the life of a cluster.
Twin hieararchy for the base classes allows Entities to be tailored to their use on the system. Entities that have compositional relationships with other entities have additional machinery from the baseclass for iterating through the "payload" of child objects and perform important iteration, skip and flow functions that such relationships usually employ. Entities without compositional relationships tend to be structural or system entities, the additional base class machinery is ommitted allowing these objects to be optimal managed in the system.
Custom persistence API, maps data base access objects to the twin hieararchy of base classes mentioned before. The abstract base class for the persistence API ensures necessary methods are implemented while providing any necessary method implemenations to inheriting classes. Ensures that the access objects are optimal in size for each entity type. Optimal sized entities allow more efficient use of application running memory. This translates to faster loading objects because they are optimal sized for their functions, more objects in smaller total memory space translates to doing more with that space. The more objects that fit in a memory, the less memory is needed to satisfy a given amount of application load, this translates directly to cost of memory and number of servers and that reduces operating costs for the business using the software.
Action oriented workflow is a User/System delegation paradigm that creates a lose coupling between the actions associated with Entity objects and the Users who may be able to perform them. By allowing the creation of workflow stages that collect Users with different permissions but delegated for a given action execution on a requested entity type by another User, commitment of an action can be made to converge on a User that has the permission for the action requested. Action centric rather than permission centric resolution of changes to Entity objects. This eliminates the need to create access control lists by allowing businesses to map their real world processes very easily into the system itself. The lose coupling enables a massive business process landscape to be covered by the paradigm. Also, actions can be delegated to the system to perform at any desired time either at action origination or by the action commiter.
A db vendor agnostic api allows the framework to be easily installed to use one of many popular RDBMS vendors and if desired to have the system extended to use custom vendors, so long as those vendors are ANSI sql compliant which forms the bulk of data management systems in use at the enterprise.
The XML based bootstrap, allows the detailed configuration of the entire cluster and of individual nodes and branches in the cluster to be managed using xml files. The framework has a permission scoped UI that allows GUI management of attributes in these files but the same files are also manageable through a GUI to users with the requisite system permissions. Making changes to the system can be done on a per node/branch or total cluster scope depending on the permissions of the commiting User and what is desired.
Run time extensibility, allows new Entity classes and associated db access classes and corresponding RDBMS tables to be added to the system via GUI or programmatically using a system XML script. Detailed instructions are provided to make the process of adding the new class packages, database tables and necessary management templates an efficient and fast process. A cluster of nodes can be extended in this way without requiring much beyond page refreshes once the relevant classes, tables and templates are in place.
Dynamic request load distribution, allows each node to determine if it should handle an incoming request for authentication to the system based on node specific criteria indicated in the system xml for each node in a cluster. The system can be adjusted so that requests beyond a certain percentage of utilized system memory or database connections are redirected to other nodes in the cluster or failing that denied. Since every node participates in this simple action, the entire cluster behaves in such a way as to push requests toward the least loaded nodes over time. This allows the most efficient handling of incoming load spikes by those machines in the cluster that are able to best handle them. The system allows warning and redirection events to notify designated Users, allowing constant awareness of the load on the live system to be had by User administrators. (Note, because of the permission system and the fact that the system manages itself...system administrators are Users just like content editors or writers, only their permissions distinguish what they can see or do on the system)
Geographic scale dynamic load distribution allows nodes to pass requests out to nodes located in other data centers across a WAN. The algorithm for load distribution biases redirections to nodes in the same branch as a redirecting node but if all nodes are redirecting a second request will be routed out to a node in another branch, this allows spikes in load that are geographically local to be difused to other geographies where requests that would other wise go denied can be handled at the cost of a slight penalty in latency. This is what allows clusters of the framework to scale smoothly even when the nodes are geographically dispersed.
Real time collaboration in the form of instant messaging, group chat, outbound email notifications for events or collaboration requests, collaboration feeds for User contact groups are all part of the system. User can control the scope of their collaboration to include only invited Users on the system or they can invite anonymous or ip specific internet guests. This enables Users to communicate with clients or partners or interact with customers. Again, the core idea of generality was built into the collaboration API allowing it to be maximally useful to any applications that are added later to the system. It is important to note, that once the design of the core framework was complete, adding functionality such as the content management API and the collaboration API proceeded as if the framework were being extended following the design paradigm that was the goal of the project in the first place.
There is a GUI managed API for setting up delivery objects for xml based generated content. New Protocol and FileTypes can be added by extending the corresponding API baseclasses , imported into the system and applied to existing Entity xml content set up to use the new protocols or file types. Content can be rendered to pdf and sent via ssh to a remote server, or the same content can be rendered to a doc file and sent via ftp to another location. There is no limit on the combination's as they are orthogonal in application.
So web 3.0 will be highlighted by frameworks of the kind I've designed. The applications built on web 3.0 platforms will be geographically scalable, implicitly load balanced, run time extensible and secure systems for building web applications and managing them with in the context of an existing or desired business workflow(s). Separate software for managing users(human resources) independent of the work they are tasked to perform is not needed, software for managing tasks performed by a user, when those tasks are performed, who collaborated on the changes are all implicitly managed by the system. Allowing managers to determine User efficiency by looking at reports of when actions are submitted and then committed. A single application can be managed by a handful of Users or even one User, delegation of actions through workflows can give this User sole control over performance of critical actions that other Users request. Agents can be delegated to perform actions from any web accessible location, enabling business to occur 24/7 and without need for physical offices. This allows massive savings on costs in time and money for setting up such processes using the previous web 2.0 model. Implicit load balancing and security allows the business application designers to focus on the business logic and not worry about scalability which is a forgone conclusion of the frame work, this enables them to engineer better applications that allow the business to be more competitive or dominant in the sector they are competing in while spending less money and hiring less people to do it. The parallel move to "cloud" based resources for file storage , e-commerce and even databases will allow platforms designed like my framework to become extremely flexible and cost efficient while allowing massive amounts of Users and business functions to be efficiently managed by small numbers of Users. Thus the applications built on web 3.0 technologies like my framework will be some of the most efficient software every designed allowing continued productivity gains to be had to the business community as they come online.