%\VignetteIndexEntry{nnlib and nnlib2Rcpp: An Introduction} %\VignetteAuthor{Vasilis Nikolaidis} %\VignetteEngine{R.rsp::tex} %\VignetteKeyword{R} %\VignetteKeyword{package} %\VignetteKeyword{vignette} %\VignetteKeyword{introduction} %\VignetteKeyword{LaTeX} %\VignetteTangle{no} %% tested using pdfLatex only \documentclass{article} %% -- LaTeX packages and custom commands --------------------------------------- \usepackage[a4paper]{geometry} \usepackage{graphicx} \usepackage{hyperref} %% bibliography %\usepackage[english]{inputenc} \usepackage{natbib} \bibpunct{(}{)}{;}{a}{}{,} \bibliographystyle{apalike} %% new custom commands \newcommand{\code}[1]{\texttt{#1}} \newcommand{\class}[1]{'\texttt{#1}'} \newcommand{\fct}[1]{\textbf{\texttt{#1()}}} \newcommand{\pkg}[1]{\texttt{#1}} \newcommand{\proglang}[1]{#1} \newcommand{\CRANpkg}[1]{\texttt{#1}} %% -- Article metainformation (author, title, ...) ----------------------------- \author{Vasilis N. Nikolaidis} \title{The \pkg{nnlib2} C++ library and \pkg{nnlib2Rcpp} R package for Artificial Neural Networks} \begin{document} {\Large \center The \pkg{nnlib2} C++ library and \pkg{nnlib2Rcpp} R package for Artificial Neural Networks.\par} {\center Vasilis N. Nikolaidis\par} {\tiny \center ORCID: 0000-0003-1471-8788 , email: vnnikolaidis@gmail.com\par} {\tiny \center Revision: 0.4.5\par} %% -- Introduction ------------------------------------------------------------- \section{Introduction} \label{sec:intro} Two interrelated projects are discussed in this document, \pkg{nnlib2} and \pkg{nnlib2Rcpp}. \subsection{The \pkg{nnlib2} C++ library} The \pkg{nnlib2} library is a small collection of C++ classes for implementation of neural networks (NN). It contains base classes and class-templates that model typical NN parts and components (processing nodes, connections, layers, groups of connections etc) as well as a class for composing these parts in complete NN. Users of this library can combine predefined parts and components with custom ones they can create by sub-classing the ones available. In all cases, considerable predefined functionality is provided, s.a methods for building the NN models from the components, presenting data and retrieving output, encoding and mapping data, serialization of the model to and from files etc. The produced models can be included and used in any C++ application. \newline The \pkg{nnlib2} code can be found on GitHub at \href{https://github.com/VNNikolaidis/nnlib2}{https://github.com/VNNikolaidis/nnlib2}. \newline For more information on \pkg{nnlib2}, go to Section \ref{sec:nnlib2}, below. \subsection{The \pkg{nnlib2Rcpp} R package} The \pkg{nnlib2Rcpp} package provides an interface between \proglang{R} and \pkg{nnlib2} NN components and entire models. It contains the entire \pkg{nnlib2} library and source code, as well as \proglang{R} functions that employ ready-to-use models implemented with \pkg{nnlib2}. It also contains a special \proglang{R} module (\code{'NN'} which allows the instantiation of individual parts and components (processing nodes, connections, layers, groups of connections etc) in custom NN configurations which can be used (trained, controlled, monitored etc) in R. Thus, new types of \pkg{nnlib2} NN parts can be developed using R-related tools (Rtools and RStudio), and employed in R (inside an 'NN' R module); if needed, the same components can be transferred and used in a pure C++ application (inside a \pkg{nnlib2} 'nn' C++ class object). \newline Stable version of the package (along with source and reference manual) can be found on CRAN at \href{https://cran.r-project.org/web/packages/nnlib2Rcpp}{https://cran.r-project.org/web/packages/nnlib2Rcpp}. \newline Development version can be found on GitHub at \href{https://github.com/VNNikolaidis/nnlib2Rcpp}{https://github.com/VNNikolaidis/nnlib2Rcpp}. \newline For more information on \pkg{nnlib2Rcpp}, go to Section \ref{sec:nnlib2Rcpp} , below. \pagebreak \section{The \pkg{nnlib2} C++ library} \label{sec:nnlib2} The \pkg{nnlib2} library is a collection of \proglang{C++} classes and templates that provide simple predefined base components useful for implementing and using NN. The \pkg{nnlib2} library may interest NN students and experimenters who prefer implementing new NN components and models using a small collection of base classes and templates whose purpose is clear, have simple interfaces, follow familiar NN concepts, and allow significant control. A small collection of ready-to-use NN components and models are also implemented and included in \pkg{nnlib2}. The \pkg{nnlib2} library requires a standard C++ compiler (has been tested with various GNU and Microsoft Visual Studio versions), and produces lightweight, standalone NN that can be invoced within any C++ application. Being written a compiled language, the produced NN are relatively fast and can be used in real applications and problems of small data size and NN complexity. \subsection{The \pkg{nnlib2} class structure} \label{sec:nnlib2cls} The \pkg{nnlib2} library consists of several \proglang{C++} class and class-template definitions, most of which match typical NN parts, sub-components and components (processing nodes, connections, layers, groups of connections, complete NN etc). Sub-components include processing elements (PEs a.k.a. nodes) and connections; these are grouped in components (layers of PEs, sets of connections, and complete NN) to provide typical NN functionality that can be inherited, overridden and/or extended when implementing a specific new NN model behavior. Two important virtual methods are provided by all component and sub-component classes: \fct{encode} invoked when the NN is trained (training stage), and \fct{recall} applied when retrieving data from the model (mapping stage), and thus should contain the core instructions for data processing. Some of the classes in \code{namespace nnlib2} are briefly outlined below. A brief (and somewhat simplified) outline of the most significant classes in \pkg{nnlib2} is also shown in Figure~\ref{figure:nClassDiagram}. Significant classes are: \begin{itemize} \item Class \class{pe} for nodes, processing elements (PEs). Provides typical PE (node) functionality and placeholders for internal input, activation, and threshold functions. All objects of this class maintain typical PE internal values s.a. input, bias, output etc. and inherit functionality for collecting inputs, applying the internal PE functions, state serialization etc. If left unmodified such objects will be referred to as generic PEs. \item Class \class{connection}. Provides typical connection functionality for communicating data between two PEs. Objects of this class maintain source and destination PE information as well as functions and values (including weights) needed to modify the transferred value. If used without modifications, objects of this class will be referred to as generic connections. \item Class \class{component} for a component of the NN topology, such as layers, sets of connections, control components, etc. Provides a common interface and functionality shared by all components (for processing, streaming, etc.). Component-type objects may be registered (added) to the NN’s \code{topology} structure (discussed later) creating complex topologies. Sub-classes that inherit \class{component}, include the following: \item Class \class{layer}, a \class{component} for a layer of PEs. It maintains the layer’s \class{pe} objects, and provides functionality to initialize, interface with, trigger processing and in general, manipulate the layer’s PEs. A template where generic or model-specific \class{pe} types can be used (as well as a 2-d variation) is provided; it can be sub-classed to define new types of \class{layer} classes with specific behavior. \item Class \class{connection\_set}, a \class{component} for a set of connections between any two \class{layer}s (can be the same layer), and a template where generic or model-specific \class{connection} types can be used and \class{connection} objects are maintained. It includes functionality to create connections between two PEs, initialize, serialize, trigger processing and in general, manipulate a set of connections; it too can be sub-classed to define new types of specialized \class{connection\_set} objects. \item Class \class{nn} a \class{component} for a neural network. While a \class{component} itself, its main purpose is to maintain other \class{component}s and control them. This is done in its \code{topology}, implemented as a double-linked list-based structure that maintains the \class{component} objects (of any type) which constitute the NN. By default the order of components in the \code{topology} corresponds to the order of processing performed when a NN executes a typical feed-forward operation (or feed-backward if in reverse order), but this can be modified by overriding the \class{nn} \fct{encode} and \fct{recall} methods. Alternatively, for simple NN topologies, a developer may choose to not use the \code{topology} structure, define the \class{component} objects as member variables and manipulate them in custom code. However, new components that are registered to the \code{topology} are subsequently handled by the default \class{nn} predefined methods: encode/recall revocations, display, serialization, deletion etc. may be performed with little or no extra code (subject to the specifics of the particular NN model implemented). Registering the components in the \code{topology} structure also allows implementation of dynamic and/or multilayer NN models, with complex topologies and “deep(er)”-learning NN configurations. As already mentioned, \class{nn} class objects are also derived from class \class{component}, allowing embedment of a NN inside the topology of another NN. \end{itemize} In addition to the above classes, \pkg{nnlib2} includes a collection of several secondary classes. For example, \class{aux\_control} components are classes of objects that can be used to provide auxiliary, user-defined functionality (e.g. control, display output, user-break, handle other components or sub-components [create, delete, communicate, call methods], perform data operations [filter, mutate, which\_max, feature expansion] etc.). Being themselves \class{component} objects, they can be added to NN's topology, thus be activated during the NN’s data processing sequence via their respective \fct{encode} and \fct{recall} methods. Other secondary classes include helper objects for communicating data, sharing run-time error information etc. Some ready-to-use NN model implementations are also included in \pkg{nnlib2}, such as versions of Learning Vector Quantization (\class{lvq\_nn} class, supervised, subclass of \class{nn}) and Self-Organizing Map (\class{som\_nn} class, unsupervised, subclass of \class{lvq\_nn}), Back-Propagation multilayer perceptron (in \class{bp\_nn} class, supervised, subclass of \class{nn}), Autoencoder (in \class{bpu\_autoencoder\_nn} class, unsupervised, a subclass of \class{bp\_nn}), and MAM (in \class{mam\_nn}, supervised, subclass of \class{nn}). \begin{figure}[htbp] \centering \includegraphics[scale=0.75]{nClassDiagram} \caption{Significant nnlib2 classes.} \label{figure:nClassDiagram} \end{figure} \subsection{Expanding the library with new NN parts and models}\label{sec:nnlib2Example} This section discusses how a new NN and its parts can be defined in \pkg{nnlib2}. The implementation of a simple Matrix Associative Memory (MAM) NN using \pkg{nnlib2} classes will be presented in this section as an example. MAM (and its building blocks) already exist in the library, thus they do not need to be defined again. However, it may be instructional to discuss the steps taken in defining a simple NN s.a. MAM for users who need to add new or customized NN components or models that are not currently implemented. MAM is a simple supervised model trained by Hebbian learning that stores input-output vector pairs $(x,z)$ by computing their tensor product and storing it in a $d$ x $c$ matrix $M$ (where $d$ and $c$ is the dimensionality (length) of $x$ and $z$ vectors respectively). Encoding is a non-iterative process, where\begin{equation} \label{eq:mamenc}M=x^T z \end{equation}is computed. Recall is also done in a single step where, given input $x$ and matrix $M$, \begin{equation}\label{eq:mamdec1}xM = xx^Tz\end{equation}is computed, which simplifies to\begin{equation}\label{eq:mamdec2}(x_1^2+x_2^2+\dots+x_d^2 )z\end{equation}i.e vector $z$ multiplied by a number. Ideally, if this number equals to 1, a perfect recall of $z$ is performed. Multiple vector pairs can be stored in a single matrix $M_s$ which is the sum of the matrices $M_i$ resulting for each vector pair $i$. MAM is the basis for several other associate memory models, but even the simple version described here has interesting properties: MAMs are bidirectional ($x$ can be recalled from $z$ as well as $z$ from $x$) and allow deletion of stored vector pairs. However, this MAM has limited and variable storage capacity which may be enhanced by proper encoding and normalization of the input and output data. Another solution to the storage issue is to employ a system of multiple MAMs, adding new MAMs when the current ability to store a vector pair is exhausted. While the MAM model is probably best implemented as a series of vector and matrix operations, it can also be realized as a simple NN. Only two layers of PEs, for $x$ and $z$ vectors respectively, are needed. These layers are fully connected (each PE in the first layer is linked to all PEs in the second) with connections whose weights collectively form matrix $M_s$. PEs apply simple functions: composition of inputs (the PE \emph{input function}) is summation of the incoming values, and the result is typically copied to the output (no \emph{activation function} is used, while a \emph{threshold function} is optional in MAM NN PEs). During the MAM NN feed-forward encode process, the $x$ feature values are input to the corresponding layer PEs, similarly $z$ is input to the other layer, and connection weights (which are initially 0) are adjusted by \begin{equation}\label{eq:mamnnenc}w_{ij} =w_{ij} + ( x_i z_j )\end{equation} where $w_{ij}$ is the weight of the connection between the i-th PE in the first layer (having input equal to $x_i$ [and same output, being the sum of the single input value]) to the j-th PE in the second layer (having input $z_j$). To retrieve vector $z$ given vector $x$, $x$ is presented as input to the corresponding layer and a single feed-forward recall step is performed towards the other layer: PEs in the first layer transfer their inputs unmodified (again being the sum of the single input value) and pass them to the connections which multiply their weights and input the results to the second layer PEs which sum them (as mentioned earlier) and output values that collectively form the NN's output vector (ideally resembling $z$). To create the components of this simple NN (if they did not already exist) or define any new NN model behavior using \pkg{nnlib2}, the user has to subclass and modify \class{component} classes (the NN's \class{layer} and \class{connection\_set} classes), sub-component classes (\class{pe} and \class{connection} etc) with the model-specific processing behavior. In the first case, the new components should have their \fct{encode} and \fct{recall} functions overridden (if the default behavior does not suffice); if all required processing is defined in these functions, such components could even contain only generic, unmodified \class{pe} and/or \class{connection} objects. In the second case, custom \class{pe} and \class{connection} classes are created and placed in (possibly unmodified) \class{layer} and \class{connection\_set}-based objects. This second approach is used below, where MAM-specific \class{pe} and \class{connection} classes containing the functionality specific to a simple MAM NN (called sMAM to distinguish it from the one already included in \pkg{nnlib2}), as follows: \newline \newline (a) PEs. The generic (unmodified) \class{pe} class suffices for sMAM and no new \class{pe}-based type is needed. Its default \fct{encode} and \fct{recall} functions invoke three methods, with the result of each (a single value) passed to the next, namely \fct{input\_function}, \fct{activation\_function} and \fct{threshold\_function}. This last \fct{threshold\_function} produces the final PE output value. The default \fct{input\_function} is summation, while the default \fct{activation\_function} and \fct{threshold\_function} are both set to perform the identity function. Therefore generic, unmodified, \class{pe} objects output the sum of their inputs. Modifying these methods and/or \class{pe} \fct{encode} and \fct{recall} customizes PE behavior. Any \class{pe} object collects and temporarily stores the individual values received as input (via its \fct{receive\_input\_value} method) and uses \fct{input\_function} to process them and produce a single final \code{input} value; alternatively, this final \code{input} value can be accessed and computed directly (bypassing the collection of individual values and invocation of \fct{input\_function}, as briefly discussed later). While this fits perfectly the sMAM example, other NN models may require one or more PE types with modified behavior; to illustrate how this is done, a \class{pe}-type (\class{sMAM\_pe}) where a \fct{threshold\_function} applies \fct{sin} to its incoming data will be defined and used in the sMAM example below: \begin{verbatim} class sMAM_pe : public pe { DATA threshold_function (DATA value) { return sin(value); } }; \end{verbatim} Note: header \code{nnlib2.h} contains various important definitions that depend on the target platform. \code{DATA} is one of them and is usually defined as \code{double}. Another useful macro is \code{TEXTOUT} which can be used to stream text output to standard output (\code{cout} when the target in a C++ console application, \code{Rout} i.e. R console in R, etc. \newline \newline (b) Connections. Unlike MAM PEs, connections do need to be modified to provide the MAM-specific processing described earlier. Thus a \class{connection}-based class (\class{sMAM\_connection}) is defined and its \fct{encode} and \fct{recall} functions modified. Here the \class{connection} methods \fct{source\_pe} and \fct{destin\_pe} (which return the source and destination PEs linked by the connection) are also useful: \begin{verbatim} class sMAM_connection : public connection { public: void encode() { weight() = weight() + source_pe().output * destin_pe().input; } void recall() { destin_pe().receive_input_value( weight() * source_pe().output ); } }; \end{verbatim} Function \fct{encode} effectively performs (\ref{eq:mamnnenc}), while \fct{decode} sends the input multiplied by weight to the output layer PEs (via their \fct{receive\_input\_value}) to be summed during their \fct{decode} step (as described earlier). \newline \newline (c) Define the sMAM component types from component templates. \class{Layer} and \class{Connection\_Set} (note the upper-case letters) are templates for the corresponding base classes, and can be defined to contain objects of the classes created above: \begin{verbatim} typedef Layer sMAM_layer; typedef Connection_Set sMAM_connection_set; \end{verbatim} If the defined layers were to contain generic \class{pe} objects instead of \class{sMAM\_pe}s, the above \code{sMAM\_layer} definition would be: \begin{verbatim} typedef Layer sMAM_layer; \end{verbatim} While these suffice for sMAM, in other more complex NN models multiple \class{layer} and/or \class{connection\_set}-type classes may need to be defined. Also (as mentioned earlier), in a NN implementation the processing details could be defined at component objects (such as \class{layer} and \class{connection\_set}) instead of modifying sub-components (\class{connection} and/or \class{pe}). This is sometimes dictated by the model’s algorithm, and unavoidable, or could be useful for certain optimizations. To implement this approach in the sMAM example, components would sub-classed from templates \class{Layer} and \class{Connection\_Set} containing only generic \class{pe} and \class{connection} objects. Their \fct{encode} and \fct{recall} functions would need to be modified to provide the needed behavior. For example, a MAM-specific \class{connection\_set} class could have its \fct{recall} function modified to perform for each connection \code{c} in the set: \begin{verbatim} destin_pe(c).input = destin_pe(c).input + c.weight() * source_pe(c).output; \end{verbatim} To allow data processing be defined at component level, the internal variables that sub-components maintain (including \code{weight} for \class{connection} or \code{input} and \code{output} for \class{pe}s are accessible from components. Here it was also chosen to bypass the destination \class{pe} \fct{input\_function} and directly modify its final \code{input} value. \newline \newline (d) finally, create the class for the actual MAM NN objects, based on \class{nn}. Here the specific components will be created and the topology defined. In the sMAM case, only a constructor is needed; once the components are (dynamically) created and registered to the \code{topology}, the default \class{nn} functions manipulating them suffice: \begin{verbatim} class sMAM_nn : public nn { public: sMAM_nn(int input_length, int output_length) :nn("MAM Neural Network") { topology.append(new sMAM_layer("Input layer", input_length, my_error_flag())); topology.append(new sMAM_connection_set); topology.append(new sMAM_layer("Output layer", output_length, my_error_flag())); connect_consequent_layers(); set_ready(); } }; \end{verbatim} A common, local to the NN, flag (\fct{my\_error\_flag}) is shared by the NN and its components to communicate run-time errors between them; the \class{nn} method \fct{connect\_consequent\_layers} is called to detect sequences of layers and setup their internal connection sets (fully connecting them with 0 weights - other options, including random or pre-computed weights are available); finally, \fct{set\_ready} sets a flag indicating that the \class{nn} is ready to encode or decode. \newline \newline Once defined \code{sMAM\_nn} objects can be created and used in the \proglang{C++} project. To create one that maps input vectors of length 3 to output vectors of length 2: \begin{verbatim} sMAM_nn theMAM(3,2); \end{verbatim} Two functions provided by parent \class{nn} class, namely \fct{encode\_u} and \fct{encode\_s} can be used for unsupervised and supervised training respectively, presenting data to the NN and triggering data encoding for the entire NN topology. The first, \fct{encode\_u} by default presents a single data vector to the NN and initiates its encoding, while the second \fct{encode\_s} is similar but presents a pair of vectors (input and desired output). Data recall functions are also provided by \class{nn} class. To encode an input-output vector pair (in corresponding vectors \code{input} and \code{output} of length 3 and 2 respectively): \begin{verbatim} theMAM.encode_s( input, 3, output, 2 ); \end{verbatim} and similarly, to get the sMAM output for given \code{input}: \begin{verbatim} theMAM.recall( input, 3, output_buffer, 2 ); \end{verbatim} where \code{output\_buffer} is a buffer (of length 2) to receive the NN’s output. \newline \newline To summarize, the entire code needed to define a simple MAM NN type is shown below. It can be found in \pkg{nnlib2} file \code{nn\_mam.h}. \begin{verbatim} #include "nn.h" namespace nnlib2 { class mam_connection: public connection { public: void encode() { weight() = weight() + source_pe().output * destin_pe().input; } void recall() { destin_pe().receive_input_value ( weight()*source_pe().output ); } }; typedef Connection_Set mam_connection_set; class mam_nn : public NN_PARENT_CLASS { public: mam_nn() :nn ("MAM Neural Network") {} bool setup(int input_length,int output_length) { reset(); add_layer( new Layer < pe > ( "Input layer" , input_length ) ); add_connection_set( new mam_connection_set ); add_layer( new Layer < pe > ( "Output layer", output_length ) );; connect_consecutive_layers(); return no_error(); } }; } \end{verbatim} A (very minimal) C++ application that uses this \code{mam\_nn} could be as follows: \begin{verbatim} #include "nn_mam.h" int main() { DATA input[4][3] = { { 1, 1, -1 }, // row 0 { -1, -1, -1 }, // row 1 { 1, -1, 1 }, // row 2 { 1, 1, 1 }, // row 3 }; DATA output[4][2] = { { 0, 1 }, // encode type 1 for row 0 { 1, 0 }, // encode type 0 for row 1 { 0, 1 }, // encode type 1 for row 2 { 0, 1 }, // encode type 1 for row 3 }; mam_nn theMAM(3,2); // train (encode) the input-output pairs to MAM... for(int i=0;i<4;i++) theMAM.encode_s(input[i],3,output[i],2); // retrieve output from MAM for input #0 DATA nn_output_buffer[2]; theMAM.recall(input[0],3, nn_output_buffer,2); // ... return 0; } \end{verbatim} \pagebreak \section{The \pkg{nnlib2Rcpp} R package}\label{sec:nnlib2Rcpp} The \pkg{nnlib2Rcpp} \proglang{R} package includes \pkg{nnlib2} in its source code, and uses in two ways: (a) to provide wrapper \proglang{R} functions for some of the predefined NN models in \pkg{nnlib2}, and (b) to provide an R module for creating NN that contain components (s.a. layers and sets of connections) defined using \pkg{nnlib2}. Thus, package \CRANpkg{nnlib2Rcpp} provides a small collection of ready-to-use neural network models that can directly be used in moderately sized problems, and also some tools to aid the experimentation with new neural network components and models (which may be useful for prototyping custom NN models or for educational purposes). Below is a brief discussion of how the predefined NN collection is used, while the implementation of new types of NN components and models is discussed in later sections. \subsection{Using predefined neural network models in \pkg{nnlib2Rcpp}} The package contains ready-to-use R functions and modules for several NN types. These currently include versions of an auto-encoding NN (\code{Autoencoder}, for PCA-like dimensionality reduction or expansion), Back-Propagation (\code{BP}, for input-output mappings), unsupervised Learning Vector Quantization (\code{LVQu}, for clustering), supervised LVQ (\code{LVQs}, for supervised classification), and Matrix Associative Memory (\code{MAM}, for storing vector pairs). All implemented models accept and process numerical data, usually in the form of vectors or matrices. Details and information for each function can be found in the package \href{https://cran.r-project.org/web/packages/nnlib2Rcpp/nnlib2Rcpp.pdf}{reference manual} or by invoking the package built-in documentation (i.e. calling \code{help(package='nnlib2Rcpp')} in R). Two brief examples follow below. \subsubsection{An unsupervised example} Functions are provided for the predefined NN models that employ an unsupervised training approach (are not trained using a second dataset of desired output); such are the Auto-encoder and Unsupervised-LVQ. For example, placing the \code{iris} data set (from R package \pkg{datasets}) in 3 clusters using Unsupervised LVQ can be done by invoking the related \code{LVQu} function, in a manner similar to: \begin{verbatim} LVQu( iris_s, 3, number_of_training_epochs = 100, neighborhood_size = 1 ) \end{verbatim} This will return a vector of cluster id numbers (0, 1 or 2) indicating the cluster assigned to each \code{iris} case. Before doing so, however, some data pre-processing must be done. LVQs require numerical data only, and this data needs to be in [0,1] range, so the complete example is: \begin{verbatim} # Create data to use in examples below (scale iris features to 0..1 range): iris_s <- as.matrix( iris [ 1 : 4 ] ) c_min <- apply( iris_s, 2, FUN = "min" ) c_max <- apply( iris_s, 2, FUN = "max" ) c_rng <- c_max - c_min iris_s <- sweep( iris_s, 2, FUN="-", c_min ) iris_s <- sweep( iris_s, 2, FUN="/", c_rng ) # Apply unsupervised LVQ ids <- LVQu( iris_s, 3, 100, 1 ) \end{verbatim} which results in: \begin{verbatim} > ids [1] 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 [39] 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 [77] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1 2 2 1 2 1 2 2 1 2 1 [115] 1 2 1 2 2 1 2 1 2 1 2 2 1 1 2 2 2 2 2 1 1 2 2 2 1 2 2 2 1 2 2 2 1 2 2 1 \end{verbatim} \subsubsection{A supervised example} Models that use supervised training (such as BP, Supervised-LVQ and MAM) are implemented as modules. This allows them to be placed in R variables and structures (such as vectors) and thus maintain state, be trained, used, saved to and restored from files, later retrained or applied to new data etc. For example, \code{"LVQs"} module is a Supervised Learning LVQ. To train it with \code{iris\_s} data, a proper known classification id, i.e. the desired output indicating the correct species for each \code{iris\_s} case, is required. These ids should be integers from 0 to n-1, where n the number of classes (here n=3 species): \begin{verbatim} iris_desired_class_ids <- as.integer( iris$Species ) - 1 \end{verbatim} A supervised learning LVQ (\code{LVQs}) object is created and stored in variable \code{lvq} as follows: \begin{verbatim} lvq <- new( "LVQs" ) \end{verbatim} To encode input-output pairs in the \code{lvq} object (here for 100 training epochs): \begin{verbatim} lvq$encode( iris_s, iris_desired_class_ids, 100 ) \end{verbatim} Once trained, the model can be used to recall the class id for given data by using its \code{recall} method. To reclassify the original \code{iris\_s} data and plot results (Figure~\ref{figure:nLVQs}) : \begin{verbatim} lvq_recalled_class_ids <- lvq$recall( iris_s ) plot( iris_s, pch = lvq_recalled_class_ids + 1) \end{verbatim} In addition to \code{encode} and \code{recall}, all supervised NN modules provide methods to display their internal state (\code{print}) and store it (\code{save}) or retrieve it (\code{load}) using files. The ability to store a NN to file allows a trained model to be applied on new data without retraining it, or interrupt training and later resume it, or share a trained model with other \pkg{nnlib2Rcpp} users (or even \pkg{nnlib2} C++ apps), or include and invoke a trained model on \pkg{Shiny} web apps etc. \begin{figure}[htbp] \centering \includegraphics{nLVQs} \caption{Iris classes recalled by a Supervised LVQ NN.} \label{figure:nLVQs} \end{figure} \subsection{Expanding the library of NN components} The next sections outline the process of creating new, not currently implemented, or custom neural network parts and models in \pkg{nnlib2Rcpp} using the included \pkg{nnlib2} classes. Much of the information provided here is discussed in greater detail earlier (see section \ref{sec:nnlib2}). While many users of the package may choose to skip these sections and only use predefined parts, components and/or models, these sections provide insight on the underlying classes needed to create new ones which, possibly combined with existing NN parts, can be employed in \class{NN} R module objects (discussed later) to form NNs. Implementing new components does currently (as of version 0.1.4 described herein) require the package source code, package \CRANpkg{Rcpp}, Rtools and some familiarity with the C++ language, but since significant NN functionality is provided by the included \pkg{nnlib2} C++ base classes, this last requirement may be minimal. All predefined neural network models in \pkg{nnlib2Rcpp} are implemented using a collection of C++ base classes and templates for creating NN. This class library, called \pkg{nnlib2}, is included in the package source and interfaced to R via package \CRANpkg{Rcpp}. It contains classes that correspond to the basic NN building elements described in classic practical related literature. Thus dissimilar NN can be defined using a scheme common for all models, which breaks each model into the same components and sub-components, organizes them in a network "topology" and controls the processing performed by each part during encoding or recalling (outputting) data. The \pkg{nnlib2} base classes can be used for implementing different NN parts and models with code that follows a common pattern, allowing functionality reuse and helping code readability. Being written in a compiled language, the produced models are also relatively fast (at least for limited data sizes and NN complexities) and, since \pkg{nnlib2} is actually a separate standalone target-independent C++ library, they can be included in various other types of C++ projects. A brief (and somewhat simplified) outline of the most significant classes in \pkg{nnlib2} is shown in the class-diagram of Figure~\ref{figure:nClassDiagram}, and is as follows: All NN are based on class \code{"nn"} which maintains one or multiple \code{"component"}-class objects in its topology (an internal ordered list of NN components). Such components may include \code{"layer"} objects, \code{"connection\_set"} objects (sets of connections between two layers), entire other \code{"nn"} objects, special-purpose components etc. The class \code{"layer"} defines components that are layers of processing nodes and provides predefined layer-related functionality; objects of this class internally maintain the nodes (here called 'processing elements') which are objects based on class \code{"pe"}. Similarly, a \code{"connection\_set"} is a set of connections between two \code{"layer"} objects, and maintains objects inherited from class \code{"connection"} which connect particular nodes in the layers. To simplify the creation of layers and sets of connections containing custom \code{"pe"} and \code{"connection"} types respectively, class templates \code{"Layer"} (or \code{"Layer2D"} for 2-d layers) and \code{"Connection\_Set"} can be used (note that template names use capital initial letters). New \code{"layer"} and \code{"connection\_set"} types or sub-classes can also be defined based on these templates. All aforementioned classes have an '\code{encode}' method, invoked when the NN is trained (training stage), and a '\code{recall}' method invoked when data is retrieved (mapping stage). These two methods contain the core instructions for a single step of data processing (for example a single encoding step in an iterative encoding process). Calling the \code{encode} method of a \code{"nn"} object (a neural network) probably -and by default- triggers a sequence of \code{encode} invocations in its components which, in turn, invoke the \code{encode} method of their sub-components (\code{"pe"} objects in a \code{"layer"}, \code{"connection"} objects in a \code{"connection\_set"}). In any case, new NN parts and models can be defined by sub-classing these base classes and overloading the methods (especially constructors and '\code{encode}'/'\code{recall}') to modify their functionality. Some examples follow below. \subsubsection{Defining new types of nodes and layers} Assume a layer is needed with a new (rather useless) type of nodes that output the sum of their inputs plus 10 when recalling data. First, the new node class needs to be defined, based on\code{"pe"}. The \code{"pe"} base class provides a method for receiving multiple incoming values (\code{receive\_input\_value}) and an overridable method for initial processing of these values (\code{input\_function}) whose result is stored in its internal variable '\code{input}'. Subsequent internal processing will produce a final value, and place it in variable '\code{output}'. By default, unmodified (generic) \code{"pe"} objects simply output the sum of all incoming values. The new type of nodes called \code{"JustAdd10\_pe"} could be implemented as shown below, first invoking the base-class \code{recall} and then adding 10 (note: this example may be already implemented in the package "\code{additional\_parts.h}" file on the development version on GitHub i.e. at \href{https://github.com/VNNikolaidis/nnlib2}{this link}): \begin{verbatim} class JustAdd10_pe : public pe { public: void recall() { pe::recall(); output = output + 10; } }; \end{verbatim} Below, the \code{"Layer"} template is used, to create (in variable r) a layer containing 25 such nodes labeled "test layer": \begin{verbatim} Layer< JustAdd10_pe > r ( "test layer", 25 ); \end{verbatim} The same template can be used to define new \code{"layer"} component types, or sub-classes with customized layer functionality. For example to define a type (named \code{"JustAdd10\_layer"}) : \begin{verbatim} typedef Layer < JustAdd10_pe > JustAdd10_layer; \end{verbatim} \subsubsection{Defining new types of connections and connection sets} To illustrate the definition of a new type of connections (and, later, of an entire NN that uses them), the code for the (particularly simple) MAM NN (as included in file \code{"nn\_mam.h"} of the package source) is analyzed below. The MAM connections encode data by multiplying the values of the nodes they connect, adding the result to their (initially zeroed) weights. During recall, the connection multiplies the incoming value to its weight and sends the result to the connected (destination) node. With base-class \code{"connection"} methods \code{source\_pe} and \code{destin\_pe} providing access to the connected source and destination nodes, a class for MAM-specific connections may be defined as: \begin{verbatim} class mam_connection: public connection { public: void encode() { weight() = weight() + source_pe().output * destin_pe().input; } void recall() { destin_pe().receive_input_value ( weight() * source_pe().output ); } }; \end{verbatim} Below, the \code{"Connection\_Set"} template is used to create (in variable q) an empty set of such connections labeled "test connections" (the actual connections will be added later): \begin{verbatim} Connection_Set < mam_connection > q ( "test connections" ); \end{verbatim} The same template can be used to define new \code{"connection\_set"} component types or sub-classes with customized functionality. For example to define a type (named \code{"mam\_connection\_set"}): \begin{verbatim} typedef Connection_Set < mam_connection > mam_connection_set; \end{verbatim} \subsection{Defining new types of neural networks (with bare Rcpp bindings)} The obvious (but not particularly R-friendly) way to create a new NN model is to sub-class the related \pkg{nnlib2} \class{nn} C++ class, modify it for the desired model-specific functionality and expose it to R via \CRANpkg{Rcpp}. This approach was taken for the predefined models, and results in the best run-time performing NN implementations. Since the NN class is the one actually presented to the user, its implementation may vary. For example, the code for the MAM NN (taken from file mam\_nn.h in the package source with only headers and comments removed) is shown below. MAMs have a 2-layer, fully connected topology (each node in one layer has connections with all nodes in the other); the simplest version of MAM (implemented here) uses two layers of generic nodes with a set of MAM connections (as defined the previous section) between them: \begin{verbatim} class mam_nn : public nn { public: mam_nn():nn("MAM Neural Network") {} bool setup(int input_length,int output_length) { reset(); add_layer( new Layer < pe > ( "Input layer" , input_length ) ); add_connection_set( new mam_connection_set ); add_layer( new Layer < pe > ( "Output layer", output_length ) ); connect_consecutive_layers(); return no_error(); } }; \end{verbatim} The \code{setup} method above adds the three components to the NN topology, i.e. two layers containing generic \code{"pe"} and a set of MAM-specific connections between them; layers must be setup before creating connections between their nodes, so \code{"nn"} method \code{connect\_consecutive\_layers} is called last, which (by default) populates the set with all possible connections between the nodes of the two layers, fully connecting them. For this very simple NN model no other code is required, except the "glue code" exposing this class (including the default \code{encode} and \code{recall} methods inherited from parent \code{"nn"} class) as a module to R using typical \CRANpkg{Rcpp} methodology (see \cite{Rcpp3}). \subsection{Creating neural networks with the \class{NN} R module} A more versatile, R-based way to combine layers and connections into a neural network is provided by the \code{NN} module which \CRANpkg{nnlib2Rcpp} package includes (as of version 0.1.4). This module maintains a neural network object (with initialy empty topology) and provides an interface to build it from components, monitor it, input to and output data from it, train and generally use it from R. Any of the component types (such as layers and connection sets) created for the predefined NN in the package, or any other custom components defined by the user can be added to its topology. The module can be used to create NN models with mixed component types, recursive or reverse-direction connections, unusual encoding or recalling sequences and, generally, aids experimentation by allowing significant control of the models from R. Once a \code{NN} module object is created, it can be manipulated by methods such as: \begin{itemize} \item\code{add\_layer}: to create (and append to the topology) a new layer containing a given number of nodes; the type of this layer (and thus also of the nodes it contains) is defined by '\code{name}' parameter, with names available for several predefined layer types while additional names can be supported for user-defined components. \item\code{add\_connection\_set}: to create (and append to the topology) a new empty set of connections. It does not connect any layers (as they may not be setup yet) nor contains any connections. The type of this set (and thus also of the connections it will contain) is defined by '\code{name}' parameter, with names available for several predefined types of such sets; again additional names can be supported for user-defined ones. \item\code{create\_connections\_in\_sets}: to fill connection sets with connections, fully connecting adjacent layers by adding all possible connections between their nodes. \item\code{connect\_layers\_at}: to insert a new empty set of connections (whose type is specified by '\code{name}' parameter) between two layers and prepare it to connect them (no actual connections between layer nodes are created). \item\code{fully\_connect\_layers\_at}: as above, but also fills the set with all possible connections between the nodes of the two layers. \item\code{add\_single\_connection}: to add a single connection between two nodes. \item\code{remove\_single\_connection}: to remove a single connection between two nodes. \item\code{input\_at}: to input a data vector to a component in the topology. \item\code{encode\_at}: to trigger the encoding operation of a component in the topology. \item\code{recall\_at}: to trigger the recall (mapping, data retrieval) operation of a component in the topology. \item\code{encode\_all}: to trigger the encoding operation of all components in the topology, in forward (first-to-last) or backward (last-to-first) order. \item\code{recall\_all}: to trigger the recall (mapping, data retrieval) operation of all components in the topology, in forward or backward order. \item\code{get\_output\_from}: to get the current output of a component. \item\code{get\_input\_at}: to get the current input at a component. \item\code{get\_weights\_at}: to get the current weights of the connections in a \code{connection\_set} component. \item\code{get\_weight\_at}: to get the current weights of a particular connection in a \code{connection\_set}. \item\code{set\_weight\_at}: to set the current weight of a particular connection in a \code{connection\_set}. \item\code{print}: to print the internal NN state, including the state of each component in topology. \item\code{outline}: to show a summary of the NN topology. \item...and others. More information is available in the package documentation (type \code{?NN} in R). \end{itemize} For example, the \code{NN} module can be used to create a NN similar in functionality to the simple MAM described earlier; To define a NN that accepts input-output pairs of 3 element vectors, first a \code{NN} object is created (in variable \code{m}) and then two generic layers of 3 nodes are added to its topology: \begin{verbatim} m <- new( "NN" ) # create NN object in variable m m$add_layer( "generic" , 3 ) # add a layer of 4 generic nodes m$add_layer( "generic" , 3 ) # add another layer of 3 generic nodes # next, add a full set of MAM connections between the two # layers which currently are at positions 1 and 2 in topology: m$fully_connect_layers_at(1, 2, "MAM", 0, 0) \end{verbatim} As specified by its parameters, the last step where \code{fully\_connect\_layers\_at} is called effectively inserts a \code{"mam\_connection\_set"} component between positions 1 and 2 of the topology and connects all corresponding layer nodes by adding connections having weights initialized to 0. The final topology contains three components, a layer currently in topology position 1, a set of 9 MAM connections in 2, and another layer in position 3. This can be verified by using the \code{outline} method, which results in: \begin{verbatim} > m$outline() ------Network outline (BEGIN)-------- Neural Network (Ready - No Error) Current NN topology: @1 (c=0) component (id=67) is Layer : generic of size 3 @2 (c=1) component (id=69) is Connection Set : MAM (Fully Connected) 67-->68 of size 9 @3 (c=2) component (id=68) is Layer : generic of size 3 --------Network outline (END)-------- \end{verbatim} (note that component ids are assigned at run-time, and may differ). This NN stores input-output vector pairs. Two such vector pairs are encoded below (this simple MAM is not very powerful in mapping data, so ideal examples were selected): \begin{verbatim} m$input_at(1, c( 1, 0, 0 ) ) # input 1st vector of a pair to layer at 1 m$input_at(3, c( 0, 0, 1 ) ) # input 2nd vector of a pair to layer at 3 m$encode_all( TRUE ) # encode, adjusting weights (fwd-direction) m$input_at(1, c( 0, 0, 1 ) ) # input 1st vector of a pair to layer at 1 m$input_at(3, c( 1, 0, 0 ) ) # input 2nd vector of a pair to layer at 3 m$encode_all( TRUE ) # encode, adjusting weights (fwd-direction) \end{verbatim} To recall the second vector given the first: \begin{verbatim} m$input_at(1, c( 1, 0, 0 ) ) # input vector to layer at topology position 1 m$recall_all( TRUE ) # recall (fwd-direction) m$get_output_from( 3 ) # get 2nd vector of the pair from layer at 3 \end{verbatim} which returns: \begin{verbatim} [1] 0 0 1 \end{verbatim} and similarly, \begin{verbatim} m$input_at(1, c( 0, 0, 1 ) ) # input another vector to layer at position 1 m$recall_all( TRUE ) # recall (fwd-direction) m$get_output_from( 3 ) # get 2nd vector from layer at position 3 \end{verbatim} which returns: \begin{verbatim} [1] 1 0 0 \end{verbatim} In the next example, a back-propagation-based auto-encoding NN is created, with a the network topology composed mostly of predefined back-propagation (BP) components: \begin{verbatim} a <- new( "NN" ) # create a NN object in variable a a$add_layer( "generic", 4 ) # 1. a layer of 4 generic nodes a$add_connection_set( "BP" ) # 2. a set of BP connections a$add_layer( "BP-hidden", 3 ) # 3. a layer of 3 BP pes a$add_connection_set( "BP" ) # 4. another set of BP connections a$add_layer( "BP-hidden", 2 ) # 5. another layer of 2 BP pes a$add_connection_set( "BP" ) # 6. another set of BP connections a$add_layer( "BP-hidden", 3 ) # 7. another layer of 3 BP pes a$add_connection_set( "BP" ) # 8. another set of BP connections a$add_layer( "BP-output", 4 ) # 9. a layer of 4 BP output pes a$create_connections_in_sets ( 0, 1 ) # Populate sets with actual connections \end{verbatim} This defines a network of 5 layers (sized 4, 3, 2, 3, and 4 nodes respectively) and sets of BP connections (with random initial weights in [0 1]) between them. The data encoding example shown below (for the scaled \code{iris\_s} data defined earlier) presents each data vector to the first layer and performs a recall. It then presents the same vector to the last layer as the correct (desired) value, and performs encoding in all the components (from last to first), where discrepancies between recalled and desired output values are used to adjust connection weights, a functionality provided by the BP components used. The process is repeated for 1000 epochs: \begin{verbatim} for(e in 1:1000) # for 1000 epochs for(r in 1:nrow(iris_s)) # for each data case { a$input_at( 1, iris_s[ r , ] ) # present data at 1st layer a$recall_all( TRUE ) # recall (fwd direction, entire topology) a$input_at( 9, iris_s[ r , ] ) # present data at last layer a$encode_all ( FALSE ) # encode, adjusting weights (bwd-direction in topology) } \end{verbatim} Once the data is encoded (or auto-encoded since input and desired output are the same), new composite variables for the data can be collected at an intermediate layer. Below, the layer of 2 nodes (in position 5 of the topology) is used, so a set of 2 variables will be collected: \begin{verbatim} result <- NULL for(r in 1:nrow(iris_s)) # for each data case { a$input_at( 1, iris_s[ r , ] ) # present data at 1st layer a$recall_all( TRUE ) # recall (in fwd direction) entire topology z <- a$get_output_from( 5 ) # collect output from layer at position 5 result <- rbind( result, z ) # append this to a 'result' matrix, to plot it etc } plot( result, pch = unclass( iris$Species ) ) \end{verbatim} The plot of a typical output resulting from the above, is shown in Figure~\ref{figure:nNNAutoenc}, with corresponding \code{iris} species used for symbols (due to random initial weights, output may differ). \begin{figure}[htbp] \centering \includegraphics{nNNAutoenc} \caption{Results from a custom auto-encoding NN on Iris data.} \label{figure:nNNAutoenc} \end{figure} \subsection{Using custom NN components in \class{NN} module} In addition to predefined ones, new neural network component types can be defined and used by \code{NN} objects. Currently, the definition of new components must be done in C++, using \pkg{nnlib2} classes as discussed in several above sections of this document. Thus it requires the \CRANpkg{nnlib2Rcpp} package source code (which includes the \pkg{nnlib2} base classes) and the ability to compile it. In particular: \begin{enumerate} \item New component types can be added to a single header file called "\code{additional\_parts.h}" (which is included in the package source). All new components to be employed by the \class{NN} R module must be defined in this file (or be accessible from \code{generate\_custom\_layer()} and \code{generate\_custom\_connection\_set()} functions in the file). \item The new \code{"pe"}, \code{"layer"}, \code{"connection"} or \code{"connection\_set"} definitions must (at least loosely) comply to the \pkg{nnlib2} base class hierarchy and structure, and follow the related guidelines outlined earlier (minimal examples can be found in the "\code{additional\_parts.h}" file itself). \item A name must be reserved for the new \code{"layer"} and \code{"connection\_set"} types or classes, to be used as parameter in \class{NN} module methods that require a name to create a component. This can be as simple as a single line of code where, given the textual name, the corresponding component object is generated and returned. This code must be added (as appropriate) to either \code{generate\_custom\_layer()} or \code{generate\_custom\_connection\_set()} functions found in the same "\code{additional\_parts.h}" header file. \end{enumerate} In an earlier example, a custom layer component type (called \code{"JustAdd10\_layer"}) was defined; it contains \code{"JustAdd10\_pe"} nodes, which \code{'recall'} the sum of their inputs plus 10. Should the definitions be placed in the "\code{additional\_parts.h}" header file, the new layer type can be used in \class{NN} objects. The only other modification required, is to register a name for such layers, which can be done by adding the following line of code to function \code{generate\_custom\_layer} (also in "\code{additional\_parts.h}"): \begin{verbatim} if(name == "JustAdd10") return new JustAdd10_layer (name,size); \end{verbatim} (note: this example may be already implemented in the package "\code{additional\_parts.h}" file on the development version on GitHub i.e. at \href{https://github.com/VNNikolaidis/nnlib2}{this link}. With these two steps completed and the modified package compiled, specifying the name \code{"JustAdd10"} when creating a layer in \class{NN} objects will result in a layer of \code{"JustAdd10\_pe"} nodes: \begin{verbatim} x <- new( "NN" ) # create NN object in variable x x$add_layer( "JustAdd10" , 3 ) # add layer of 3 'JustAdd10_pe' nodes x$add_connection_set( "pass-through" ) # add set of pass-through connections x$add_layer( "JustAdd10" , 1 ) # add layer of 1 'JustAdd10_pe' nodes x$create_connections_in_sets ( 1, 1 ) # fill sets with connections \end{verbatim} The network, with 3 nodes at its first layer, a set of 3 connections (that pass data unmodified) and a single node in the last layer, will effectively output \code{sum( i + 10 ) + 10} for any 3 element input vector \code{i}. For input \code{c( 0, 10, 20 )} output is expected to be 70. To verify this: \begin{verbatim} x$input_at( 1, c( 0, 10, 20 ) ) # present data at 1st layer x$recall_all( TRUE ) # recall (fwd entire topology) \end{verbatim} If output at 1st layer is checked, the initial values are increased by 10: \begin{verbatim} > x$get_output_from(1) [1] 10 20 30 \end{verbatim} while at the last layer (3rd topology component), summation is done on these incoming values, and the result also increased by 10, producing the final output: \begin{verbatim} > x$get_output_from(3) [1] 70 \end{verbatim} \subsubsection{A step-by-step example using Rtools and RStudio} An example of the steps needed to add a new custom NN component to the package (and then use it in \class{NN} objects) is outlined below. \newline (a) Preparation. \begin{enumerate} \item Tools required for this example are R, Rtools, and RStudio. Download and install them, if necessary. In R, install package \pkg{Rcpp} and its dependencies. You may also want to install package \pkg{devtools} and its dependencies. \item Download the \CRANpkg{nnlib2Rcpp} source code from either CRAN or GitHub. Extract if necessary, placing all files in a directory of your choice. You will be building your version of the package (with your custom components added) from this directory. \item Just to verify all is ok, rebuild the package. Go to the directory containing file "\code{nnlib2Rcpp.Rproj}" (an RStudio project file) and open it in RStudio. Next, rebuild the package (select \code{Build / Clean and Rebuild} on the RStudio menu). If successful, the package will be installed and loaded. \end{enumerate} (b) Create your own types of NN parts (nodes, layers, connections etc). \begin{enumerate} \item When creating a new NN part, you will have to consider how it will cooperate with other parts you plan to include in the model. This is unavoidable in NN design, where all components cooperate with each other towards the final result. Traditionally -but not necessarily-, each part of a NN has two different modes of operation, one performed while encoding (training, learning) and another performed when recalling (mapping) data. To add your new NN part, you will subclass \CRANpkg{nnlib2} C++ classes to define new parts (nodes, node layers, connections, sets of connections etc) with customized behavior. The particulars will determine whether you need to add extra variables to your class, or overload any of the base methods (usually methods \fct{setup}, \fct{encode} and \fct{recall}). Only minimal experience with C++ is required. The \CRANpkg{nnlib2} base classes are already available in \CRANpkg{nnlib2Rcpp}, under its \code{src} directory and you may place your creations is the \code{additional\_parts.h} file. This file already contains some minimal examples of NN parts (nodes, layers etc). Alternately, you can create your own source code files (again in \code{src} directory) and include appropriate header files in \code{additional\_parts.h}. In any case, new user-defined components should be accessible from this file, and exist inside \code{namespace nnlib2} (similarly to the examples). For this short guide, all changes (below) will be done exclusively inside \code{additional\_parts.h} file. \item Define a new NN part, for example a new type of \class{pe}s or \class{connection}s. In the example below we will create a type of \class{connection}s and call them \class{MEX\_connection}s (\code{MEX\_} for Manual EXample). The definition must be inside \code{namespace nnlib2}. The appropriate base class (\class{connection}) will be sub-classed, and only \fct{encode} and \fct{recall} will be modified (as often the case in many new NN parts regardless of type). This being a \class{connection} it connects a source node (\code{source\_pe()}) to a destination node (\code{destin\_pe()}) and can send or receive data to them via their methods. \begin{verbatim} class MEX_connection: public connection { public: // model-specific behavior during mapping stage: void recall() { destin_pe().receive_input_value(pow( source_pe().output - weight() , 2) ); } // model-specific behavior during training stage: // in this example, only the current connection weight (i.e. weight()) // and incoming value from the source node (i.e. source_pe().output) are // used in a series of calculations that eventually change the // connection weight (see last line). void encode() { double x = source_pe().output - weight(); double s = .3; double m = weight(); double pi = 2*acos(0.0); double f = 1/(s*sqrt(2*pi)) * exp(-0.5*pow((x-m)/s,2)); double d = (f * x) / 2; weight() = weight() + d; } }; \end{verbatim} Next, you may define a new NN component for your type, in this example a \class{connection\_set} that will contain this new type of connections. Below this is done using the template \code{Connection\_Set} which is a template for defining sets containing only a given \code{connection} type. We will call the new type \class{MEX\_connection\_set}: \begin{verbatim} typedef Connection_Set < MEX_connection > MEX_connection_set; \end{verbatim} \item Finally, locate the \fct{generate\_custom\_connection\_set} in \code{additional\_parts.h} file and add the following line: \begin{verbatim} if(name == "MEX") return new MEX_connection_set(name); \end{verbatim} This will create a set of \class{MEX\_connection}s when the text "MEX" is specified as parameter to \class{NN} methods such as \code{add\_connection\_set}. Note: the \code{name} passed as parameter in the right side of the above simply copies it (i.e. "MEX") as the name of the new set (this is a label used only for display purposes and can be changed at any time). \textit{[A side-note: the above steps are similar to those needed for defining new nodes (or processing elements , \code{pe}s) and \code{layer}s of such nodes. Below is a simple example where a new type of \class{pe} and a \class{layer} containing such nodes is defined (new names used also start with \code{MEX\_}, for Manual EXample): } \begin{verbatim} class MEX_pe : public pe { public: // during mapping, do the default 'pe' behavior (which adds incoming values) // and then take its square root for final output. void recall() { pe::recall(); output = sqrt(output); } }; typedef Layer < MEX_pe > MEX_layer; \end{verbatim} \textit{ There is a corresponding \fct{generate\_custom\_layer} function in \code{additional\_parts.h} file where the following line could be added to make this new type of layer available in R, s.a.:} \begin{verbatim} if(name == "MEX") return new MEX_layer(name, size); \end{verbatim} \textit{ Thus a layer of this specific type will be used when the text \code{"MEX"} is passed as parameter to \class{NN} methods such as \code{add\_layer}.]} \item Build the package. In RStudio, select \code{Build / Install and Restart} menu option. Once finished your new part definition will be available in the package. From now on you can directly go to (c) below where your new parts (along with other available components) can be combined in R to form NNs. \end{enumerate} (c) Use your NN parts in R. As mentioned earlier, you only do (a) and (b) once to 'install' your user-defined type in your version of the package. From now on you can bypass them and directly go to R to use your new parts (along with other available components) in a NN. You can now create NNs in R that employ your newly defined parts (along with others already available in the package). This is done using \class{NN} R objects (type \code{?NN} for more information, or see earlier section). An example using the \code{iris} data set and the \code{MEX\_} types created in (b) above. Output is shown in Figure~\ref{figure:nNNMEX}: \begin{figure}[htbp] \centering \includegraphics{nNNMEX} \caption{Results from a NN with custom parts on Iris data.} \label{figure:nNNMEX} \end{figure} \begin{verbatim} library(nnlib2Rcpp) set.seed(123456) # scale and shuffle iris data rr <- sample(nrow(iris)) suffled_iris <- iris[rr,] iris_s <- as.matrix( suffled_iris [ 1 : 4 ] ) c_min <- apply( iris_s, 2, FUN = "min" ) c_max <- apply( iris_s, 2, FUN = "max" ) c_rng <- c_max - c_min iris_s <- sweep( iris_s, 2, FUN="-", c_min ) iris_s <- sweep( iris_s, 2, FUN="/", c_rng ) iris_s <- (iris_s*2)-1 # create a nn with the new components: ann <- new("NN") ann$add_layer("generic",4) ann$add_connection_set("MEX") ann$add_layer("MEX",4) ann$create_connections_in_sets(-.5,.5) # (optional) show the NN topology ann$outline() # (optional) show the NN internal state ann$print() # train it. Present data to component at position 1, # do 200 epochs, encode from first to last component: ann$encode_dataset_unsupervised(iris_s,1,200,TRUE) # use it to recall (map) the iris data and collect output recalled_values <- ann$recall_dataset(iris_s,1,3,TRUE) # in this toy-model the output node with smallest output is # the 'winner' (similar to LVQ). Find it and plot results find_best<-apply(recalled_values,1,which.min) plot(iris_s, pch = find_best, col=find_best) pairs(iris_s, pch = find_best, col=find_best) \end{verbatim} \section{One last note} Please consider submitting any useful NN parts, components, or entire models you define, so they may enrich future versions of \CRANpkg{nnlib2} and \CRANpkg{nnlib2Rcpp}. For more information, contact the author of this document. \section{Summary} This document introduces \CRANpkg{nnlib2}, a collection of C++ classes for defining neural network components and models. This document also introduces \CRANpkg{nnlib2Rcpp}, an R package that provides a module (\class{NN}) to employ \CRANpkg{nnlib2} components (s.a. layers, connections etc) in arbitrary network topologies and processing sequences. Both also contain predefined, ready-to-use NN parts and models. \bibliography{intro} \end{document}