29 June, 2009

Yet another theory of dreams.

I recently engaged a discussion on Facebook that piqued my interest in the area of sleep and dream research. The conversation started on other topics and soon moved toward sleep and dreams and the possible reasons for their existence, both from an evolutionary perspective and from an individual physiological perspective. I think a proper theory of sleep and dreams must include what we know concerning the development of our species in the ancient past and how the increased cortical development that our ancestors experienced might have led to the selection of the ability to dream as a key survival advantage. First let's talk a bit about what we know about human cortical development.

Development of the human cortical mind

There are many related theories as to why our brains evolved into the large masses that they occupy today , relative to our cranium size. Compared to other primates humans have the largest brain size to cranium ratio, in fact it is believed that the gyri and sulci that make up the convolutions on the brain allow the extensive surface area of the higher cortex to fit into our relatively small brain cases. From a developmental perspective this is amazing as it hints at a runaway process of evolution of the brain that far outstripped the rate at which other structures (the skull itself) could evolve to match. So it is plausible to conclude that the changes that occurred in cortical development are relatively recent ones on this evidence alone. Fortunately, we don't have to rely on this modern interpretation of human physiology as it compares to other primates alone. We can also look at fossil finds from primate relatives going back millions of years, many of these finds presenting perfect cranial domes for us to determine the changes that occurred to their brains as they developed. More important than cross lineage comparative analysis though, is the ability to compare within our lineage of hominin ancestors. When this is done we see that the human brain went from the very chimpanzee like capacity of individuals of Austraipithicus Aferensis ("Lucy" made this ancestor famous) to the progressively larger cranial capacities of subsequent iterations of the hominin line that led directly to us according to the paleoanthropological evidence of morphological continuity. From A. Aferensis , H. Habilis, H. Ergaster, H. Erectus, archiac H. Sapien and modern H. Sapiens Sapiens. Observations of the various fossils show a marked acceleration in brain size between H. Habilis and H. Erectus.

The theories suppose that the shift of those ancestors from a predominantly herbivorous diet to ones including more protein and meat as the Rift Valley ecosystem continued to be denuded of the former dense canopy that had afforded previous ancestors protection, ushered in a growth of the brain , which itself is a large mass of proteins. The selection pressures during such a time of transition included the need to develop ways to maintain alertness to nearby dangers as absent a canopy, attack could could from many areas. An interesting correlation of the acceleration in brain growth is that it coincides with the emergence of up right walking ancestors. It is theorized that upright walking was more efficient to survival in an environment consisting of Savanna rather than forest as it allowed individuals to attain a higher position on the horizon and thus be better able to see or hear threats as they approach and it allowed escape to cover should a threat approach in a manner that our previous hominid ancestors could not match. That said, the shift in diet and the upright walking gait in a changing environment favored individuals that used those advantages to survive. As tool use coupled with mental development synergized a run away process of development occurred, particularly as it was now possible for individuals to use their hands to do things other than for the purpose of locomotion. It is likely that this aided in the transmission of knowledge through instruction performed using the hands. Individuals that mastered tools were able to train their progeny and relations and that fostered development of the brain areas necessary to retain such training a cycle of growth enabled by the serendipitous formation of the rift valley, beginning about 5 million years ago that led to the modern human mind. Homo. Erectus is the first homo species to make it out of Africa as result and in continued development in other parts of the world went on to develop actual culture and drawing ability as learned from finds in the Neander Valley in Germany.

Meanwhile, back in Africa the development continued as ancestral H. Erectus continued to evolve with the changing ecology, continued to develop their brains. In such a time, where the ability to think symbolically would have surely been an advantage in determining uses for the new elements of the changing environments, the brain may have found it useful to conserve energy after long periods of use and thus necessitated sleep periods to perform this conservation. We know that humans have a requirement for sleep that becomes progressively mandatory the longer we are deprived of it. Any theory of dreams must take into account this interesting correlation between sleep deprivation and the reduced function of the mind.

Sleeping versus dreaming

Given the correlation that exists between sleep deprivation and reduced mental and physical performance, a theory of dreams must in some way explain why sleep is necessary to restore the mental and physical function of a normal waking state. To this end I put forward yet another hypothesis on the purpose of sleep and dreams, as described by the REM state episodes that characterize their occurrence. The idea is that sleep serves to repair damage done to the cortical mind during continuous use, be it in the day time or at night (since our biorhythms vary across the globe these points are specific to the day/night cycles of the individual in a subjective sense)
This cortical damage requires the sleep phase to perform repair at which time incoming signals are unavailable to confound the repair process this leads to a first assertion of the theory.

a) The majority of repairs performed during sleep are performed on those regions that are active exclusively when the individual is awake. Thus the regions of the brain that shuttle real time sensation from the body into the processing regions of the brain for those sensations undergo some repair.

There are many other theories of dreams but many of them lack an explanation for why sleep deprived individuals experience gradual inability to perform, and posit only internal processes of the brain ,such as shuttling memories from short to long term (Continual Activation theory) or pruning abberant data from the mind (Junk removal theory ) as the cause for dreams. However, without addressing the facilitator of dreams (sleep) and the reasons for its occurrance we are only able to synthesize part of a solution to the puzzle. When we add sleep and recognize sleep deprivation as being a key cause of deficits, we must modify the dream theory to address this occurrance. This leads to the second assertion of this theory:

b) Sleep is required to reverse the damage of continual use of the cortical mind, it shuts down and repairs regious formerly in continuous use while organizing data in the form of memories stored for each of these sensory regions. An analogy from electronics expresses this assertion best. In a computer the ability to read and write data has an inverse correlation with speed, if we wish to read data quickly the data design required necessitates that writing will be slow, if we want to write data quickly then data design necessitates that reading data will be slow. A waking individuals storage mechanism is more like a system that needs to write quickly and read slowly, if the brain is acquiring billions of bits of new data every waking moment it must store that data away immediately in order to continue to acquire new data, in the background low level processes organize the incoming data but the flood of new data retards this ability. As time goes on the ability to organize incoming data becomes swamped by the incoming data and the system must shut down to continue the organization process. In computer hard disk drives new data is written to sectors of the disk such that the data is written to the nearest free memory cells, as old data is expunged this leads to gaps of free data that are then overwritten with new data, because this data is not written contiguously, subsequent reading of the written data requires more time. However, reading of the data can be accelerated, if a consolidation process called defragmentation is performed on the drive. Defragmenation requires that the portions of the drive undergoing the optimization is temporarily unavailable , much like how the sensory processing regions of the brain are unavailable when we sleep. Sleeping could be the defragmentation analog for our brains.

If sleeping is a repair process that reorganizes acquired sensory data for more efficient retrieval in the next waking state then dreaming characterized by REM states is the process of testing the reals. Playback of previously acquired data in order to calibrate the memory repair process previously performed. However, this testing phase may have had a beneficial side effect. The data acquired is played in way that is not random and may aid in determining solutions to events and situations experienced during the wake state. The REM state could serve to enhance creativity and that would easily be a selected attribute for survival for those individuals that could dream while undergoing the brain repair during sleep. Thus over time individuals that engaged in REM sleep and derived benefit could get the best of both worlds, using their brains during the wake state to survive and using it for a portion of the sleep state to engineer solutions to problems faced in the wake state. This hypothesis would be difficult if not impossible to test.

Testable hypothesis

Hypothesis, sleep deprivation experiments geared to induce sleep but prevent REM states will prevent the calibration from taking place and thus prevent individuals from restoring optimal performance during the wake state. Experiments to REM deprive individuals during sleep can explore this assertion.

Hypothesis, if sleep is a repair process, then neurotransmitters unique to the process must become active during sleep in the regions that are formerly active during the wake state. Analysis of brain scans should find such correlations between brain region and neurotransmitters.

Hypothesis, isolation and extraction of the repair neurotransmitters may allow the development of sleep deprivation aids that are more efficient than the use of stimulants, which don't really address the problem (required repair of continual use of sensory areas)

I welcome alternative ideas or corrections to this theory, feel free to weigh in the comment section below.






17 June, 2009

From the Lab: numeroom.com updates.

The beta launch of numeroom.com last week has been instructive in many ways , it has allowed me to finally get over the launch jitters and the many questions of "is it ready" to now be in a proactive mode of "what can I make better?". That said, I have identified several areas that can enhance the experience of new users and I will detail them as they are being identified and engineered for deployment to the production servers. These posts will serve as an inside look into the problems that users have that the site builders rarely anticipate and how they are solved.

Problem 1: Numeroom must be separately created.

When I first launched the site, I had a feeling I should write the registration logic so that it would automatically create a default Numeroom for the Users, this way they could get going immediately with inviting people, sharing files but more importantly , harnessing the suite of following features enabled uniquely by the Numeroom ecosystem in the form of file sharing events, category shift events and jump to events. These allow numeroom users insight into the actions of their contacts that no other collaboration system provides enalbing them to collaborate with a sense of immediacy here to for unknown. Unfortunately, I was focused on hitting the launch and decided to launch with the numeroom creation as a separate action from account creation, big mistake for several reasons.

  • If you are offering something new to people, then talking to them about it won't lead to the "ah ha!" moment. You have to show them, and the less you require they have to actively do to be shown the magic the better.
  • People get frustrated quickly when looking at a new UI , even if it is "logical" to you. Do as much as you can for your users leaving only the pure experience (sharing , chatting, following) left for them to decide over.

So, earlier this afternoon I started plotting the strategy for solving this problem, it can be solved quite literally in minutes, tested on the development machine (the same one I write this blog post on) and deployed to the production cluster in Texas using a numeroom, of course. ;) I'll update the blog when it is done.

Problem 2: Numeroom creator settings not accessible when User is in room.

This was something I meant to do before the launch but didn't get to in time, if a User is chatting in a room , allowing them to find out "who owns this room?" and as well be able to see that owners list of jump to rooms, send that user an IMail or a contact join request is key to quickly enabling them to build a network of contacts with rooms of interest. The faster I allow users to connect the faster the entire ecosystem builds value from those real time relationships. I performed the change which simply amounted to hyperlinking the room's name as displayed in the upper left corner of an active room window. Clicking the link now takes the User to the owners settings page where they can perform the aformentioned actions. Production rollout to follow.

So that is what is in the works for deployment to production, mostly likely tomorrow afternoon.

15 June, 2009

The difference between numeroom.com and google wave

I was recently asked this question by Juliette Powell, and thought it would be the great subject for a detailed blog answer as provided below:

From an architectural view , the differences lie in approach. Where as, Google is tacking on real time or near real time collaboration to their existing messaging infrastructure in the form of Google Wave; Numeroom ties a set of collaboration functions into a secure package that is useful to both individual users AND to businesses.

I designed it noticing the trend in the proliferating collaboration options about 3 years ago. Web 2.0 technology (XmlHttpRequest object to the clued in) enabled real time experiences that matched previous stand alone software but enabled user interaction that wasn't possible using decentralized stand alone software. The emergence of platforms as the new buzzword and engine behind a given service gave rise to many collaboration options. Meebo, Userplane, Campfire, Parachat, MeGlobe, LivePerson , Provide Support...etc. I also saw the services moving toward offering specific options but not others due to inherent strengths and weaknesses in the chosen underlying platforms and due to business choices of the management teams for the companies. Case in point:

Meebo - Goal, web based group chat and IM, link sharing to make money from ad revenue.

Userplane - Goal, web based group chat and IM, flash video and audio to make money from ad revenue and through dedicated installations and integrations.

Campfire - Goal, secure web based group chat and collaborative file sharing for small business collaboration, integrates with other 37Signals products like BaseCamp for project management but why are their two products for this?

MeGlobe - Goal, free ad supported attempt at real time translated chat.

ProvideSupport - Goal, Dedicated customer service dashboard for small businesses to enable customer support on their live web pages using a dispersed team of agents.

This snapshot of providers is only a small set of what is out there, each providing certain collaboration features but not others. As a proof of concept to the http://www.apriority.biz/aesplash.html AgilEntity platform (which I am keeping proprietary rather than open source unlike all the other platforms) I decided that numeroom.com should answer all the service options listed above and add what is missing in all of them. Namely the real time following and notification that makes Facebook and microblogging services like Twitter so addictive and useful. Adding this functionality also opened up a new landscape of potential metrics with regards to the discussions that are occurring inside chat rooms and how those discussions evolve. The "following" features to which I refer are:

  • Public,Private, Inactive Rooms -- Users can control who sees their room, public rooms are open to the world, private rooms open to invitation (email, ip, username) and inactive rooms (invisible to numeroom directory) open to contacts only when a pass is enabled.
  • "Jump To" connections -- Browse the global conference hall categories, find rooms of interest and generate a "jump to" link to the room instantly. Subsequently the room is available for access in a jump to section of the contacts display area.
  • dynamic categories -- Unlike all over chat services that force you to stay in a particular category once it is created, numerooms can be easily switched from one category to the next and the room is automatically then available for searches in the new category.
  • dynamic file sharing -- Files can be dropped into rooms, pulled off rooms and added as attachments to offline mail (called IMail on numeroom) and TimeLine feed notifications. Files can even be sent directly to the tables of users that allow this T2TFT: Table to Table file transfers.
  • broadcasting -- The jump to , file adding and room category shifting actions can trigger broadcasts to all the contacts of the room owner. The contacts can selectively choose to recieve these messages in the form of notification emails enabling a new form of real time awareness to how people are collaborating that is user controlled.
  • tuning -- Since broadcasting can enable the creation of many events for actions performed by contacts, tuning allows specific contacts to be "listened" to, options exist to enable this "tune set" while online or offline. Much like the third party applications provided by Twitter, tune sets allow Users to filter the broadcasts they recieve to only recieve the ones most salient to their collaboration needs at the time.

These 5 abilities open a new paradigm in real time collaboration that I alway felt was missing and that enables a brand new ability to mine the interactions of users in real time. Where Twitter allows filtering of the trends occuring in the Zeitgeist from the end users perspective, the "following" capabilities of numeroom enable trends in the zeitgeist to be monitored both by the end users as they desire and as well by the Numeroom ecosystem itself. For example, in classic chat room services (all those mentioned in the listing above and all the previous stand alone installed versions) the only way to know where a friend who leaves a room has gone was to ask them (if they were on your buddy list) often as has been discovered in Facebook and Twitter, people want to anonymously follow their friends through a content exploration. They do this by tuning into what those friends are posting about at any given moment (say using Twitter and the various third party filtering tools) , the same is possible on Facebook by adjustement of ones Wall feed, however a similar function was never added to real time chat until numeroom.com. The main difficulty lay in architecture, following is a "chirp" intensive function that many architectures can not handle at scale. The AgilEntity platform that numeroom.com is built on was designed for geographic scale, enabling distributed operation of running nodes making following a reality for the already chirp heavy real time collaboration function.

So where Google Wave allows the flowering of conversations around particular topics of interest to small groups of interacting individuals related by email (all on googles gmail service at first) Numeroom starts with conversation and then reveals an ability for users to discover and follow other individuals without constraint to email address today. The final and probably the biggest difference between the two however lies in how the Numeroom solution enabled by the underlying AgilEntity platform, enables the management of any number of Numeroom ecosystems for licensing by small to medium sized businesses. Each licensed portal operates as its own universe of Numeroom functions, separate from the commercial site. Thus a business can license one of these options and immediately:

  1. create its own Users
  2. Delegate permissions to created users or up-level to the Numeroom system.
  3. manage their interactions in accordance with custom designed workflow tools
  4. manage and create their own numerooms
  5. brand their site portal
  6. make their portal public or private to the internet
  7. create its own categories
  8. manage security for portal access such as the use of Captcha's and enabling automatic registration.
  9. Manage Site specific logs and system data.
  10. Monitor changes to the lifecycle of any managed entity: Users, categories, workflows..etc. through the use of real time notifications.

Thus, the numeroom ecosystem reveals itself to be designed much like a series of nested dolls, with self similar symmetry at all scales. Exposing this collaboration infrastructure to businesses allows them to keep the metrics gathered over the collaboration interactions occuring in their business private, and that level of privacy to collaboration interactions is the exact opposite of what Google wants by making Google Wave available late this year. They rely on the data and metrics gathered so that they can then better target advertisements for their other properties.
Thus a key difference in philosophy lies in the fact that Google banks on the open access to user data and interaction metrics for all Users, Numeroom only does it for the free and individual plan licensing Users of the service allowing Users that license the secure site plan options to retain sole knowledge of the following metrics of their private business.

Agility: and the design of Relational applications.

Since building the AgilEntity platform I've noticed the tactics that many companies claiming to use agile development methodologies use to market their own development platforms built over the last 7 years. I've isolated these tricks and list them below.

Any change to existing software is called "an application".

One of the tricks that I've seen used by many of the stand alone or web based providers is to generalize the definition of the well known term "application" to include updates and modifications to existing applications. This is a subtle point , as it allows them to describe changes such as adding new views of data in existing database tables using supposedly advanced visual modification tools, as deployments of entire applications. In reality, these are not application deployments, they are application modifications. The systems are not building entire layers of back end data base tables , business logic and front end UI components but instead are adding or modifying elements to an existing application. The underlying datastore remains a unique constraint to the flexibility of application redeployment. Any development platform that purports to redeploy an application is also implying that they are some how destroying and rebuilding database tables that conformed to an existing schema, and this is never done as , destroying a table destroys its data unless a complex migration process is set up and in that case , such change can never be performed "instantly" or "in a matter of minutes" as many of these products like to indicate. A few that come to mind that use the "modification as application" trick are Outsystems Agility platform, Zoho.

Deploy an application in minutes!

This claim is constant and for the platforms that use it, the reason why is seen in the demo. videos they often compile to "prove" their case. The applications that are deployed are always highly linear applications with no relational mappings between database tables , such as one to many or many to many relationships between extracted data. The most efficient of developed applications will involve such relationships ...unfortunately for "deploy in minutes!" claims these applications are difficult to design and built in a UI that is not aware of the precise relationship between the tables and more importantly , how the underlying datastore expresses those relationships most efficiently. In most cases the use of entity relations tables in the design establish these relationships on the data end and then more complex business logic for relating created tables to one another in the required ways must be designed as well...however this process can not be performed and deployed in minutes. It is true, that changes to an established application of this level of complexity can be performed rapidly but that is more a hallmark of the design rather than the UI tools used to create and map the relational changes. Often the demos show examples of creating User tables and retrieving or updating data from such tables...obviously this is easy to do in minutes as no relationships between the user table and any other table are established in the demo...however what happens if the changes require addition of new columns to an existing User table with 5,000 records? Now more complex actions are required on the datastore and these often take the process outside of the simple development automation created by the platform.

Claims of redundancy that are not true.

One important aspect of runtime applications that are accessible to a varied user community is the ability to scale smoothly. Often the development platform providers claim that scalability is easy but hide the fact that this often involves patches and script additions beyond simply just installing the development tools on a new machine. Others claim that their systems lack single points of failure but their design diagrams clearly show this is not a generally true claim. Outsystems Agility package only lacks single points of failure in the execution environment, the deployment server which is unique could suffer a failure and put the application development process in a disarray. They are careful to qualify their statement but they don't address the real dangers of failure onf the deployment server. I've only given cursory examination of the many available "platforms" but I am sure they are equally filled with wild claims cleverly hidden behind slick web page designs and many demos narrated by attractive young women.

Why the AgilEntity platform is proprietary

Recently I was asked by a fellow technology colleauge why I was choosing to keep the AgilEntity platform proprietary and not making the platform available for sales and marketing as the platforms indicated above have been doing. The main reason has to do with protection of my intellectual property in the form of an advantage derived from the tight integration of all development elements of the AgilEntity platform. Aside from maintaining the competitive advantage of the hyper efficient design for my own use, is the fact that no platform could ever truly achieve the promise of fast development and high application complexity. Salesforce.com started with the same sort of claims of fast RAD development and now with their force.com services are essentially an online version of the complex development shops of old. So rather than deploy endless options for building custom solutions, AgilEntity will be used internally to design specific vertical applications for complex problem spaces that will work more efficiently than anything being produced on the any of the other platforms. The numeroom.com commercial web site is a case in point, it is a multi-tenant , secure, distributed and scalable application. It employs a complex relational design that is most efficient for the problem of real time collaboration and simply could not be done on RAD development environments. The site is a proof of concept of the efficient design methodology of the underlying closed sourced AgilEntity platform, the proof of the design to be in how the varied unique features enabled by it are harnessed by the growing community while being allowed to scale as needed by the platform. I think this tactic is the best route to take to testing the true agility of the platform, and time will tell the tale.

01 June, 2009

Public beta of Numeroom.com

I am proud to announce the availability of the numeroom.com website and services. It's been a few months of very little sleep, lots of coffee and lots of cold food (that should have been eaten while it was hot but had to wait a pressing problem) ;)

numeroom.com provides an integrated and efficient suite of collaboration services in an AJAX based interface that allows individuals and small to medium sized businesses to quickly set up their own real time presence on their personal pages or on their business pages without any integration hassles.

Head over to the site and create a free "evangelist" account, accounts created during this period will be given additional room creation rights and additional file sharing and participant count enhancements.

Many video tutorials describe each feature starting with the one indicated below, how to create your new account and first numeroom in 3 minutes. check it out.