The OpenCog AtomSpace is an in-RAM knowledge representation (KR) database, an associated query engine and graph-re-writing system, and a rule-driven inferencing engine that can apply and manipulate sequences of rules to perform reasoning. It is a kind of in-RAM generalized hypergraph (metagraph) database. Metagraphs offer more efficient, more flexible and more powerful ways of representing graphs: a metagraph store is literally just-plain better than a graph store. On top of this, the Atomspace provides a large variety of advanced features not available anywhere else.
The AtomSpace is a platform for building Artificial General Intelligence (AGI) systems. It provides the central knowledge representation component for OpenCog. As such, it is a fairly mature component, on which a lot of other systems are built, and which depend on it for stable, correct operation in a day-to-day production environment.
It is now commonplace to represent data as graphs; there are more graph databases than you can shake a stick at. What makes the AtomSpace different? A dozen features that no other graph DB does, or has even dreamed of doing.
But, first: five things everyone else does:
A key difference: the AtomSpace is a metagraph store, not a graph store. Metagraphs can efficiently represent graphs, but not the other way around. This is carefully explained here, which also gives a precise definition of what a metagraph is, and how it is related to a graph. As a side-effect, metagraphs open up many possibilities not available to ordinary graph databases. These are listed below. Things are things that no one else does:
Newcomers often struggle with the AtomSpace, because they bring preconceived notions of what they think it should be, and its not that. So, a few things it is not.
It's not JSON. So JSON is a perfectly good way of representing
structured data. JSON records data as
key:value pairs, arranged
hierarchically, with braces, or as lists, with square brackets.
The AtomSpace is similar, except that there are no keys! The
AtomSpace still organizes data hierarchically, and provides lists,
but all entries are anonymous, nameless. Why? There are performance
(CPU and RAM usage) and other design tradeoffs in not using explicit
named keys in the data structure. You can still have named values;
it is just that they are not required. There are several different
ways of importing JSON data into the AtomSpace. If your mental model
of "data" is JSON, then you will be confused by the AtomSpace.
It's not SQL. It's also not noSQL. Databases from 50 years ago
organized structured data into tables, where the
key is the label
of a column, and different
values sit in different rows. This is
more efficient than JSON, if you have many rows: you don't have to
store the same key over and over again, for each row. Of course,
tabular data is impractical if you have zillions of tables, each with
only one or two rows. That's one reason why JSON was invented.
The AtomSpace was designed to store unstructured data. You can
still store structured data in it; there are several different ways
of importing tabular data into the AtomSpace. If your mental model
of "data" is structured data, then you will be confused by the AtomSpace.
It's not a vertex+edge store. (Almost?) all graph databases decompose graphs into lists of vertexes and edges. This is just fine, if you don't use complex algorithms. The problem with this storage format is locality: graph traversal becomes a game of repeatedly looking up a specific vertex and then, a specific edge, each located in a large table of vertexes and edges. This is non-local; it requires large indexes on those tables (requires a lot of RAM), and the lookups are CPU consuming. Graph traversal can be a bottleneck. The AtomSpace avoids much of this overhead by using (hyper-/meta-)graphs. This enables more effective and simpler traversal algorithms, which in turn allows more sophisticated search features to be implemented. If your mental model of graph data is lists of vertexes and edges, then you will be confused by the AtomSpace.
What is it, then? Most simply, the AtomSpace stores immutable, globally unique, typed s-expressions. The types can be thought of as being like object-oriented classes, and many (not all) Atom types do have a corresponding C++ class. Each s-expression is called "an Atom". Each Atom is globally unique: there is only one copy, ever, of any given s-expression (Atom). It's almost just that simple, with only one little twist: a (mutable) key-value database is attached to each Atom. Atoms can be used to define (hyper-/meta-)graphs. It's fun to think of these graphs as defining "plumbing"; whereas the Values stored in the associated key-value database is like the "fluid" in these pipes.
The AtomSpace borrows ideas and concepts from many different systems, including ideas from JSON, SQL and graph stores. The goal of the AtomSpace is to be general: to allow you to work with whatever style of data you want: structured or unstructured. As graphs, as tables, as objects. As lambda expressions, as abstract syntax trees, as prolog-like logical statements. A place to store relational data obeying some relational algebra. As a place to store ontologies or mereologies or taxonomies. A place for syntactic (BNF-style) productions or constraints or RDF/OWL-style schemas. In a mix of declarative, procedural and functional styles. The AtomSpace is meant to allow general knowledge representation, in any format.
The "special extra twist" of immutable graphs decorated with mutable values resembles a corresponding idea in logic: the split between logical statements, and the truth values (valuations) attached to them. This is useful not only for logic, but also for specifying data processing pipelines: the graph specifies the pipeline; the values are what flow through that pipeline. The graph is the "code"; the values are the data that the code acts on.
All this means that the AtomSpace is different and unusual. It might be a bit outside of the comfort zone for most programmers. It doesn't have API's that are instantly recognizable to users of these other systems. There is a challenging learning curve involved. We're sorry about that: if you have ideas for better API's that would allow the AtomSpace to look more conventional, and be less intimidating to most programmers, then contact us!
As it turns out, knowledge representation is hard, and so the AtomSpace has been (and continues to be) a platform for active scientific research on knowledge representation, knowledge discovery and knowledge manipulation. If you are comfortable with extremely complex mathematical theory, and just also happen to be extremely comfortable writing code, you are invited -- encouraged -- to join the project.
The AtomSpace is not an "app". Rather, it is a knowledge-base platform. It is probably easiest to think of it as kind-of-like an operating system kernel: you don't need to know how it works to use it. You probably don't need to tinker with it. It just works, and it's there when you need it.
End-users and application developers will want to use one of the existing "app" subsystems, or write their own. Most of the existing AtomSpace "apps" are focused on various aspects of "Artificial General Intelligence". This includes (unsupervised) natural-language learning, machine-learning, reasoning and induction, chatbots, robot control, perceptual subsystems (vision processing, sound input), genomic and proteomic data analysis, deep-learning neural-net interfaces. These can be found in other github repos, including:
Documentation is on the OpenCog wiki. Good places to start are here:
The OpenCog Brainwave blog provides reading material for what this is all about, and why.
The AtomSpace is a mashup of a large variety of concepts from mathematical logic, theorem proving, graph theory, database theory, type theory, model theory and knowledge representation. Its hard to provide a coherent overview without throwing around a lot of "big words" and "big concepts". We're trying to get a lot of things done, here, and there's no particularly simple or effective way of explaining it without a lot of foundational theory.
There are pre-defined Atoms for many basic knowledge-representation and computer-science concepts. These include Atoms for relations, such as similarity, inheritance and subsets; for logic, such as Boolean and, or, for-all, there-exists; for Bayesian and other probabilistic relations; for intuitionist logic, such as absence and choice; for parallel (threaded) synchronous and asynchronous execution; for expressions with variables and for lambda expressions and for beta-reduction and mapping; for uniqueness constraints, state and a messaging "blackboard"; for searching and satisfiability and graph re-writing; for the specification of types and type signatures, including type polymorphism and type construction. See Atom types.
Because of these many and varied Atom types, constructing graphs to represent knowledge looks kind-of-like "programming"; the programming language is informally referred to as "Atomese". It vaguely resembles a strange mash-up of SQL (due to queriability), prolog/datalog (due to the logic and reasoning components), lisp/scheme (due to lambda expressions), Haskell/CaML (due to the type system) and rule engines (due to the graph rewriting and forward/backward chaining inference systems). This "programming language" is NOT designed for use by human programmers (it is too verbose and awkward for that); it is designed for automation and machine learning. That is, like any knowledge representation system, the data and procedures encoded in "Atomese" are meant to be accessed by other automated subsystems manipulating and querying and inferencing over the data/programs. See Atomese.
Aside from the various advanced features, Atomese also has some very basic and familiar atom types: atoms for arithmetic operations like "plus" and "times", conditional operators, like "greater-than" or "equals", control operations like "sequential and" and "cond", as well as settable state. This makes Atomese resemble a kind of intermediate language, something you might find inside of a compiler, a bit like CIL or Gimple. However, it is both far more flexible and powerful than these, and also far less efficient. Adventurous souls are invited to create a compiler to GNU Lighting, CIL, Java bytecode or the bytecode of your choice; or maybe to a GPU backend, or even more complex data-processing systems, such as TensorFlow.
In its current form, Atomese was primarily designed to allow the generalized manipulation of large networks of probabilistic data by means of rules and inferences and reasoning systems. It extends the idea of probabilistic logic networks to a generalized system for algorithmically manipulating and managing data. The current, actual design has been heavily influenced by practical experience with natural-language processing, question answering, inferencing and the specific needs of robot control.
The use of the AtomSpace, and the operation and utility of Atomese, remains a topic of ongoing research and design experimentation, as various AI and knowledge-processing subsystems are developed. These include machine learning, natural language processing, motion control and animation, deep-learning networks and vision processing, constraint solving and planning, pattern mining and data mining, question answering and common-sense systems, and emotional and behavioral psychological systems. Each of these impose sharply conflicting requirements on the AtomSpace architecture; the AtomSpace and "Atomese" is the current best-effort KR system for satisfying all these various needs in an integrated way. It is likely to change, as the various current short-comings, design flaws, performance and scalability issues are corrected.
Active researchers and theoreticians are invited to join! The current codebase is finally clean and well-organized enough that a large number of possibilities have opened up, offering many different and exciting directions to pursue. The system is clean and flexible, and ready to move up to the next level.
One of the primary conceptual distinctions in Atomese is between "Atoms" and "Values". The distinction is made for both usability and performance. Atoms are:
By contrast, Values, and valuations in general, are:
Thus, for example, a piece of knowledge, or some proposition would be stored as an Atom. As new evidence accumulates, the truth value of the proposition is adjusted. Other fleeting changes, or general free-form annotations can be stored as Values. Essentially, the AtomSpace looks like a database-of-databases; each atom is a key-value database; the atoms are related to one-another as a graph. The graph is searchable, editable; it holds rules and relations and ontologies and axioms. Values are the data that stream and flow through this network, like water through pipes. Atoms define the pipes, the connectivity. Values flow and change. See the blog entry value flows as well as Atom and Value.
The primary documentation for the atomspace and Atomese is here:
The main project site is at https://opencog.org
Most users should almost surely focus their attention on one of the high-level systems built on top of the AtomSpace. The rest of this section is aimed at anyone who wants to work inside of the AtomSpace.
Most users/developers should think of the AtomSpace as being kind-of-like an operating system kernel, or the guts of a database: its complex, and you don't need to know how the innards work to use the system. These innards are best left to committed systems programmers and research scientists; there is no easy way for junior programmers to participate, at least, not without a lot of hard work and study. Its incredibly exciting, though, if you know what you're doing.
The AtomSpace is a relatively mature system, and thus fairly complex. Because other users depend on it, it is not very "hackable"; it needs to stay relatively stable. Despite this, it is simultaneously a research platform for discovering the proper way of adequately representing knowledge in a way that is useful for general intelligence. It turns out that knowledge representation is not easy. This project is a -good- excellent place to explore it, if you're interested in that sort of thing.
Experience in any of the following areas will make things easier for you; in fact, if you are good at any of these ... we want you. Bad.
Basically, Atomese is a mash-up of ideas taken from all of the above fields. It's kind-of trying to do and be all of these, all at once, and to find the right balance between all of them. Again: the goal is knowledge representation for general intelligence. Building something that the AGI developers can use.
We've gotten quite far; we've got a good, clean code-base, more-or-less, and we're ready to kick it to the next level. The above gives a hint of the directions that are now open and ready to be explored.
If you don't have at least some fair grounding in one of the above, you'll be lost, and find it hard to contribute. If you do know something about any of these topics, then please dive into the open bug list. Fixing bugs is the #1 best way of learning the internals of any system.
Looking ahead, some key major projects.
One of the development goals for the 2021-2023 time frame is to gain experience with distributed data processing. Currently, one can build simple distributed networks of AtomSpaces, by using the StorageNode to specify a remote AtomSpace. However, it is up to you as to what kinds of data these AtomSpace exchange with one-another. There are no pre-configured, suggested communications styles.
Because the AtomSpace can hold many different representatioinal styles, it is relatively easy to import data into the AtomSpace. The low-brow way to do this is to write a script file that imports the data. This is fine, but leads to data management issues: who's got the master copy?
The goal of data bridges is to create new Atoms that allow live
access into other online systems. For example, if an SQL database
holds a table of
(name, address, phone-number), it should be
possible to map this into the AtomSpace, such that updates not
only alter the SQL table, live and on line, but also such that
a query performed on the AtomSpace side translates into a query on
the SQL database side. This is not hard to do, but no one's done it
Similarly, a live online bridge between the AtomSpace and popular graph databases should also be possible. It's not clear if this should use the StorageNode API mentioned above, or if it needs something else.
The new Value system seems to provide a very nice way of working with fast-moving high-frequency data. It seems suitable for holding on to live-video feeds and audio streams and piping them through various data-processing configurations. It looks to be a decent API for declaring the structure and topology of neural nets (e.g. TensorFlow). However, it is more-or-less unused for these tasks. Apparently, there is still some missing infrastructure, as well as some important design decisions to be made. Developers have not begun to explore the depth and breadth of this subsystem, to exert pressure on it. Ratcheting up the tension by exploring new and better ways of using and working with Values will be an important goal for the 2021-2024 time-frame. See the value flows blog entry.
A particularly important first step would be to build interfaces between values and an audio DSP framework. This would allow AtomSpace structures to control audio processing, thus enabling (for example) sound recognition (do I hear clapping? Cheers? Boos?) without having to hard-code a "cheer recognizer". This opens the door to using machine learning to learn how to detect different kinds of audio events.
There is no particular need to limit oneself to audio: other kinds of data is possible (e.g. exploring the syntactic, hierarchical part-whole structure in images) but audio is perhaps easier!?
Many important types of real-world data, include parses of natural language and biochemical processes resemble the abstract mathematical concept of "sheaves", in the sense of sheaf theory. One reason that things like deep learning and neural nets work well is because some kinds of sheaves look like tensor algebras; thus one has things like Word2Vec and SkipGram models. One reason why neural nets still stumble on natural language processing is because natural language only kind-of-ish, partly looks like a tensor algebra. But natural language looks a whole lot more like a sheaf (because things like pre-group grammars and categorial grammars "naturally" look like sheaves.) Thus, it seems promising to take the theory and all the basic concepts of deep learning and neural nets, rip out the explicit tensor-algebra in those theories, and replace them by sheaves. A crude sketch is here.
Some primitive, basic infrastructure has been built. Huge remaining work items are using neural nets to perform the tensor-like factorization of sheaves, and to redesign the rule engine to use sheaf-type theorem proving techniques.
Current work is split between two locations: the "sheaf" subdirectory in this repo, and the generate repo.
The Atomspace runs on more-or-less any flavor of GNU/Linux. It does not run on any non-Linux operating systems (except maybe some of the BSD's). Sorry!
There are a small number of pre-requisites that must be installed before it can be built. Many users will find it easiest to use the install scripts provided in the ocpkg repo. Some users may find some success with one of the opencog Docker containers. Developers interested in working on the AtomSpace must be able to build it manually. If you can't do that, all hope is lost.
To build the OpenCog AtomSpace, the packages listed below are required. Essentially all Linux distributions will provide these packages.
apt-get install libboost-dev
apt-get install cmake3
sudo make installat the end.
apt-get install guile-2.2-dev
apt-get install cxxtest
The following packages are optional. If they are not installed, some
optional parts of the AtomSpace will not be built. The
during the build, will be more precise as to which parts will not be built.
apt-get install cython
apt-get install postgresql postgresql-client libpq-dev
Be sure to install the pre-requisites first! Perform the following steps at the shell prompt:
cd to project root dir mkdir build cd build cmake .. make -j4 sudo make install make -j4 test
Libraries will be built into subdirectories within build, mirroring the structure of the source directory root.
To build and run the unit tests, from the
./build directory enter
(after building opencog as above):
make -j4 test
Most tests (just not the database tests) can be run in parallel:
make -j4 test ARGS=-j4
The database tests will fail if run in parallel: they will step on one-another, since they all set and clear the same database tables.
Specific subsets of the unit tests can be run:
make test_atomese make test_atomspace make test_guile make test_join make test_matrix make test_persist_sql make test_python make test_query make test_sheaf
After building, you MUST install the atomspace.
sudo make install
Atomese -- that is -- all of the different Atom types, can be thought of as the primary API to the AtomSpace. Atoms can, of course, be created and manipulated with Atomese; but, in practice, programmers will work with either scheme (guile), python, C++ or haskell.
The simplest, most complete and extensive interface to Atoms and the
Atomspace is via scheme, and specifically, the GNU Guile scheme
implementation. An extensive set of examples can be found in the
/examples/atomspace and the
Python is more familiar than scheme to most programmers, and it offers
another way of interfacing to the atomspace. Unfortunately, it is not
as easy and simple to use as scheme; it also has various technical issues.
Thus, it is significantly less-used than scheme in the OpenCog project.
None-the-less, it remains vital for various applications. See the
/examples/python directory for how to use python
with the AtomSpace.