Sparse distributed memory and Salem International

Sparse distributed memory (SDM) is a mathematical model of human long-term memory introduced by Pentti Kanerva in 1988 while he was at NASA Ames Research Center. It is a generalized random-access memory (RAM) for long (e.g., 1,000 bit) binary words. These words serve as both addresses to and data for the memory. The main attribute of the memory is sensitivity to similarity, meaning that a word can be read back not only by giving the original write address but also by giving one close to it, as measured by the number of mismatched bits (i.e., the Hamming distance between memory addresses).

SDM implements transformation from logical space to physical space using distributed data storing. A value corresponding to a logical address is stored into many physical addresses. This way of storing is robust and not deterministic. A memory cell is not addressed directly. If input data (logical addresses) are partially damaged at all, we can still get correct output data.

The theory of the memory is mathematically complete and has been verified by computer simulation. It arose from the observation that the distances between points of a high-dimensional space resemble the proximity relations between concepts in human memory. The theory is also practical in that memories based on it can be implemented with conventional RAM-memory elements.

Contents 1 Definition 1.1 The binary space N 1.2 SDM as neural network 1.2.1 Neuron model 1.2.2 Neuron as address-decoder 1.3 Memory Location 1.3.1 Address Pattern 1.3.2 Data Pattern 1.3.3 Writing 1.3.4 Reading 1.3.5 Pointer chains 1.4 Critical Distance 2 Probabilistic Interpretation 3 Biological plausibility 4 Quantum-mechanical interpretation 5 Applications 5.1 The Best Match Search 5.2 Speech recognition 5.3 "Realizing forgetting" 5.4 Genetic memory 5.5 Statistical Prediction 5.6 Artificial General Intelligence 5.7 Reinforcement learning 5.8 Object indexing in computer vision 6 Extensions 7 Related patents 8 Related models 9 References


Human memory has a tendency to congregate memories based on similarities between them(although they may not be related), such as "firetrucks are red and apples are red". Sparse distributed memory is a mathematical representation of human memory, and uses high-dimensional space to help model the large amounts of memory that mimics that of the human neural network. An important property of such high dimensional spaces is that two randomly chosen vectors are relatively far away from each other, meaning that they are uncorrelated.

The underlying idea behind a SDM is the mapping of a huge binary memory onto a smaller set of physical locations, so-called hard locations. As a general guideline, those hard locations should be uniformely distributed in the virtual space, to mimic the existence of the larger virtual space as accurately as possible. Every datum is stored distributed by a set of hard locations, and retrieved by averaging those locations. Therefore, recall may not be perfect, accuracy depending on the saturation of the memory.

Kanerva’s proposal is based on four basic ideas: 1. The boolean space , or points in dimensions, exhibits properties which are similar to humans’ intuitive notions of relationships between the concepts. This means that it makes sense to store data as points of the mentioned space where each memory item is stored as an n-bit vector. 2. Neurons with n inputs can be used as address decoders of a random-access memory 3. Unifying principle: data stored into the memory can be used as addresses to the same memory. Distance between two points is a measure of similarity between two memory items. The closer the points, the more similar the stored vectors. 4. Time can be traced in the memory as a function of where the data are stored, if the data are organized as sequences of events The binary space N

The SDM works with n-dimensional vectors with binary components. Depending on the context, the vectors are called points, patterns, addresses, words, memory items, data, or events. This section is mostly about the properties of the vector space N = . Let n be number of dimensions of the space. The number of points, or possible memory items, is then . We will denote this number by N and will use N and to stand also for the space itself.

Concepts Related to the space N: Origin, 0: The point with all coordinates 0 is called the origin, 0 = 000...00. Complement, 'x: The complement, or opposite, of point x is the n-tuple that has ones where x has zeros and vice versa. Norm, |x|: The norm of point x is the number of ones in its binary representation. Difference, x − y: The difference of two points x and y is the n-tuple that has ones where x and y differ and zeros elsewhere. It is the bitwise 'exclusive or': x − y = x ⊕ y. The difference commutes: x − y = y − x. Distance, d(x, y) The distance between two points x and y is the number of dimensions at which x and y differ. It is called the Hamming distance (its square root is the Euclidean distance) and is expressed in bits. Distance is the norm of the difference: d(x, y) = |x − y| Betweenness, x:y:z: Point y is between points x and z if and only if the distance from x to z is the sum of the distances from x to y and from y to z; that is, x:y:z ⇔ d(x, z) = d(x, y) + d(y, z). It is easily seen that every bit of a point in between is a copy of the corresponding bit of an endpoint. Orthogonality, x ⊥ y: Point x is orthogonal to point y, or the two are perpendicular or indifferent, if and only if the distance between the two is half the number of dimensions: x ⊥ y ⇔ d(x, y) = n/2. The distance n/2 is called indifference distance of space N. If x is orthogonal to y, it is also orthogonal to its complement 'y (x is halfway between y and 'y). Circle, O(r,x) A circle with radius r and center x is the set of points that are at most r bits from x: O(r,x) = {y | d(x, y) ≤ r}.

Properties of the space N:

The space N can be represented by the vertices of the unit cube in n-dimensional Euclidean space. The vertices lie on the surface of an n-dimensional sphere with (Euclidean-metric) radius . This gives rise to the sphere analogy. We will call a space spherical if 1. any point x has a unique opposite 'x, 2. the entire space is between any point x and its opposite 'x, and 3. all points are "equal" (meaning that for any two points x and y there is a distance preserving automorphism of the space that maps x to y, so that from any of its points the space "looks" the same).

The surface of a sphere (in Euclidean 3d-space) clearly is spherical. According to definition, N is also spherical, since y ⊕ x ⊕ (…) is an automorphism that maps x to y. Because N is spherical, it is helpful to think of it as the surface of a sphere with circumference 2n. All points of N are equally qualified as points of origin, and a point and its complement are like two poles at distance n from each other, with the entire space in between. The points halfway between the poles and perpendicular to them are like the equator.

Distribution of the Space N

The number of points that are exactly d bits form an arbitrary point x (say, from the point 0) is the number of ways to choose d coordinates from a total of n coordinates, and is therefore given by the binomial coefficient:

The distribution of N thus is the binomial distribution with parameters n and p, where p = 1/2. The mean of the binomial distribution is n/2, and the variance is n/4. This distribution function will be denoted by N(d). The normal distribution F with mean n/2 and standard deviation is a good approximation to it: N(d) = Pr{d(x, y) ≤ d} ≅ F{(d − n / 2)/ }

Tendency to Orthogonality

An outstanding property of N is that most of it lies at approximately the mean (indifference) distance n/2 from a point (and its complement). In other words, most of the space is nearly orthogonal to any given point, and the larger n is, the more pronounced is this effect. SDM as neural network

The SDM may be regarded either as a content-addressable extension of a classical random-access memory (RAM) or as a special type of three layer feedforward neural network. The main SDM alterations to the RAM are: The SDM calculates Hamming distances between the reference address and each location address. For each distance which is less or equal to the given radius the corresponding location is selected. The memory is represented by counters (where n is number of locations and m is the input data length) instead of single-bit storage elements. Writing to the memory, instead of overwriting, is as follows: if the i-bit of the input data is 1, the corresponding counters (counters in the selected locations (rows) and in the i-th columns) are incremented, if the i-bit of the input data is 0, the corresponding counters are decremented. Reading (or recall) from the memory is similar: The contents of the selected locations are summed columnwise. Each sum is thresholded. If the sum is greater than or equal to the threshold value the corresponding output bit is set to 1, in the opposite case it is cleared. Note that the thresholds may be zero, if the training input vectors are closed to orthogonal ones.

Salem International and Sparse distributed memory

This article is about an organization. For the college, see Salem International University.

Salem International is an international humanitarian and environmental organisation actively working for peace throughout the world. Originally founded in 1957 by Gottfried Müller and a circle of friends in Stuttgart, Germany, to provide shelter for the homeless, Salem can now be found in many different countries, on 5 continents. Salem is a non-governmental organisation (NGO) whose belief is based on the Bible and human rights.

Contents 1 History 1.1 Beginnings 1.2 Creation of children's and youth villages 1.3 Health and the protection of nature and environment 2 Fields of work 2.1 Social work 2.1.1 Children's and youth villages 2.1.2 Public health services 2.1.3 Education 2.2 Nature 2.3 Neurofeedback 2.4 Organic farming 2.5 Guesthouse 3 See also 4 References 5 External links

History Beginnings

Gottfried Müller founded the Brotherhood Salem in 1957 in Stuttgart, Germany and devoted his life to people, hence fulfilling a promise he made to himself in a death cell during World War II captivity.

He started by giving away all of his money to two disabled Jewish women. Soon with the help of some friends he established a home for homeless people and ex-convicts in Stadtsteinach (?) and quickly more of these Salem homes were built in several German cities.

After several years of this social work it turned out, however, that the help for these people had mostly started off too late and that many could not permanently be rehabilitated to home life having been uprooted many times.

He found the solution during conversations with the people he has cared for. Many of them reported that they had grown up in orphanages. Creation of children's and youth villages

He went to the core of the issue, the cause of uprooting - the children being from disadvantaged family backgrounds. Gottfried Müller said,

"We must find a way to create a better future for the 'home' children, to educate them to be members or even representatives of a valuable community."

He gathered together educationalists and co-workers to realize with him a new kind of education closer to 'families' and created the Salem children's and youth welfare organization.

In all Salem institutions children from disadvantaged family backgrounds were accommodated to give them a lasting home. Successively, the Salem children's and youth villages in Stadtsteinach/Northern Bavaria, Höchheim/Northern Bavaria and Kovahl/Northern Germany were built. The model character of these villages was the reason for building further Salem institutions in the USA, Israel, Uganda, Russia and Ecuador. Health and the protection of nature and environment

What followed was the engagement of medical welfare work, vegetarian nutrition, holistic medicine, animal and environmental protection, organic agriculture, re-afforestation projects, and the Salem institute for the investigation of alternatives to animal experiments. Meanwhile Salem has grown into an international welfare organization offering a home for many discriminated children and juveniles. Fields of work Social work Children's and youth villages

The children in these villages and homes live together in extended families, about 6 to 8 children with adults. Each family is led by a couple of parents or an experienced mother with an educational professional training. Some villages offer residential homes for elderly people, too.

In 1980, 200 children were living in the long-established villages in Germany. But Muller has said that "Salem is only the model"; he and other organizers have not been interested in increasing the number of the villages belonging to the organization, instead seeing them as models. Public health services

In many projects the social commitment goes beyond the children's village. For example, in the Salem village in Uganda, there is a fully equipped hospital with a laboratory and an operating theatre. It provides medical care to an area of around 45,000 inhabitants. Furthermore, there is a training facility of health advisors and traditional midwives. The clinic in Togo, being supported by Salem, provides medical treatment freely or at a very low cost to approximately 42,000 people of the area. Education

The foundations of the education are the Ten Commandments and Jesus’ Sermon on the Mount from the Bible. The Christian faith is in the same way part of education as knowledge from pedagogics, psychology and sociology as well. Salem provides professional and academic apprenticeships, as well as continuing education, in different areas. For example, the apprenticeship workshop in Uganda and the school for children with behavioural disorder in New England, USA. Nature

A healthy interaction with nature is the basis of Salem's work. The activities range from animal protection and environmental education in Germany to reafforestation programmes in Africa. Neurofeedback

As an alternative Training Centre for ADD/ADHD Salem provides biofeedback training in the form of individual sessions and children's summer camps. It also offers practice-oriented seminars about Neurofeedback / EEG-biofeedback with internationally renowned experts and clinicians. (Web site only in German) Organic farming

Organic agriculture and gardens belong to most of the children's villages. In Germany these are farmed in accordance with the national guidelines for organic agriculture. In tropical countries other methods such as Permaculture and Agroforestry stand in the foreground. There are gardens which grow many herbs, fields to grow crops and vegetarian whole-grain diet based kitchens in these villages. Guesthouse

The guesthouse "Lindenhof Salem" in Stadtsteinach/ Northern Bavaria offers a vegetarian organic restaurant as well as naturopathic treatment. See also Maryland Salem Children's Trust
15/511 13 14 16 17 18