Artificial Neural Network

What is Neural Network?

The absolute most effective profound learning strategies include counterfeit neural systems. Fake neural systems are roused by the 1959 natural model proposed by Nobel laureates David H. Hubel and Torsten Wiesel, who discovered two sorts of cells in the essential visual cortex: straightforward cells and complex cells. Numerous counterfeit neural systems can be seen as falling models of cell sorts roused by these natural perceptions. 

Fukushima's Neocognitron presented convolutional neural systems somewhat prepared by unsupervised learning with human-coordinated components in the neural plane. Yann LeCun et al. (1989) connected regulated backpropagation to such structures. Weng et al. (1992) distributed convolutional neural systems Cresceptron for 3-D object acknowledgment from pictures of jumbled scenes and division of such questions from pictures. 

A conspicuous requirement for perceiving general 3-D articles is slightest movement invariance and resilience to twisting. Max-pooling had all the earmarks of being initially proposed by Cresceptron to empower the system to endure little to-substantial twisting progressively, while utilizing convolution. Max-pooling helps, yet does not ensure, shift-invariance at the pixel level. 

With the appearance of the back-engendering calculation taking into account programmed separation, numerous scientists attempted to prepare directed profound simulated neural systems without any preparation, at first with little achievement. Sepp Hochreiter's confirmation proposition of 1991 formally recognized the explanation behind this disappointment as the vanishing inclination issue, which influences numerous layered feedforward systems and intermittent neural systems. Repetitive systems are prepared by unfurling them into profound feedforward systems, where another layer is made for every time venture of an information grouping handled by the system. As blunders spread from layer to layer, they recoil exponentially with the quantity of layers, blocking the tuning of neuron weights which depends on those mistakes. 

To conquer this issue, a few techniques were proposed. One is Jürgen Schmidhuber's multi-level progressive system of systems (1992) pre-prepared one level at once by unsupervised adapting, calibrated by backpropagation. Here every level takes in a packed representation of the perceptions that is sustained to the following level. 

Another strategy is the long fleeting memory (LSTM) system of Hochreiter and Schmidhuber (1997). In 2009, profound multidimensional LSTM systems won three ICDAR 2009 rivalries in associated penmanship acknowledgment, with no earlier information about the three dialects to be educated. Sven Behnke in 2003 depended just on the indication of the inclination (Rprop) when preparing his Neural Abstraction Pyramid to take care of issues like picture recreation and face confinement. 

Different strategies additionally utilize unsupervised pre-preparing to structure a neural system, making it first learn by and large valuable element identifiers. At that point the system is prepared further by administered back-spread to characterize marked information. The profound model of Hinton et al. (2006) includes taking in the circulation of an abnormal state representation utilizing progressive layers of double or genuine esteemed dormant variables. It utilizes a confined Boltzmann machine (Smolensky, 1986) to display each new layer of more elevated amount highlights. Each new layer ensures an expansion on the lower-bound of the log probability of the information, consequently enhancing the model, if prepared legitimately. Once adequately numerous layers have been scholarly, the profound design might be utilized as a generative model by replicating the information when inspecting down the model (a "familial go") from the top level component initiations. Hinton reports that his models are compelling component extractors over high-dimensional, organized information. 

The Google Brain group drove by Andrew Ng and Jeff Dean made a neural system that figured out how to perceive more elevated amount ideas, for example, felines, just from watching unlabeled pictures taken from YouTube recordings. 

Different strategies depend on the sheer preparing force of cutting edge PCs, specifically, GPUs. In 2010, Dan Ciresan and colleagues[74] in Jürgen Schmidhuber's gathering at the Swiss AI Lab IDSIA demonstrated that regardless of the aforementioned "vanishing slope issue," the prevalent preparing force of GPUs makes plain back-engendering practical for profound feedforward neural systems with numerous layers. The strategy beat all other machine learning methods on the old, renowned MNIST written by hand digits issue of Yann LeCun and associates at NYU. 

At about the same time, in late 2009, profound learning feedforward systems made advances into discourse acknowledgment, as set apart by the NIPS Workshop on Deep Learning for Speech Recognition. Concentrated shared work between Microsoft Research and University of Toronto analysts showed by mid-2010 in Redmond that profound neural systems interfaced with a concealed Markov model with connection subordinate expresses that characterize the neural system yield layer can definitely lessen blunders in vast vocabulary discourse acknowledgment undertakings, for example, voice seek. The same profound neural net model was appeared to scale up to Switchboard assignments around one year later at Microsoft Research Asia. Much prior, in 2007, LSTM prepared by CTC began to get magnificent results in specific applications. This technique is currently generally utilized, for instance, in Google's incredibly enhanced discourse acknowledgment for all cell phone clients. 

Starting 2011, the cutting edge in profound learning feedforward systems exchanges convolutional layers and max-pooling layers, topped by a few completely associated or scantily associated layer took after by a last characterization layer. Preparing is typically managed with no unsupervised pre-preparing. Since 2011, GPU-based usage of this methodology won numerous example acknowledgment challenges, including the IJCNN 2011 Traffic Sign Recognition Competition, the ISBI 2012 Segmentation of neuronal structures in EM stacks challenge, the ImageNet Competition,and others. 

Such managed profound learning strategies additionally were the main fake example recognizers to accomplish human-focused execution on specific undertakings. 

To beat the hindrances of frail AI spoke to by profound learning, it is important to dive past deep learning structures, in light of the fact that natural brains use both shallow and profound circuits as reported by mind life systems showing a wide assortment of invariance. Weng contended that the cerebrum self-wires to a great extent as indicated by sign insights and, in this way, a serial course can't get all major measurable conditions. ANNs could promise shift invariance to manage little and expansive characteristic items in substantial messed scenes, just when invariance stretched out past movement, to all ANN-learned ideas, for example, area, sort (object class name), scale, lighting. This was acknowledged in Developmental Networks (DNs) whose epitomes are Where-What Networks, WWN-1 (2008)through WWN-7 (2013).

Introduction to Deep Learning: Fundamental Concepts

Deep learning algorithms are based on distributed representations. The underlying assumption behind distributed representations is that observed data are generated by the interactions of factors organized in layers. Deep learning adds the assumption that these layers of factors correspond to levels of abstraction or composition. Varying numbers of layers and layer sizes can be used to provide different amounts of abstraction.

Deep learning exploits this idea of hierarchical explanatory factors where higher level, more abstract concepts are learned from the lower level ones. These architectures are often constructed with a greedy layer-by-layer method. Deep learning helps to disentangle these abstractions and pick out which features are useful for learning.

For supervised learning tasks, deep learning methods obviate feature engineering, by translating the data into compact intermediate representations akin to principal components, and derive layered structures which remove redundancy in representation.

Many deep learning algorithms are applied to unsupervised learning tasks. This is an important benefit because unlabeled data are usually more abundant than labeled data. Examples of deep structures that can be trained in an unsupervised manner are neural history compressors[16] and deep belief networks.

Introduction to Deep Learning: Definition

There are a number of ways that the field of deep learning has been characterized. For example, in 1986, Rina Dechter introduced the concepts of first order deep learning and second order deep learning in the context of constraint satisfaction. Later, deep learning was characterized as a class of machine learning algorithms that: 

• use a cascade of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. The algorithms may be supervised or unsupervised and applications include pattern analysis (unsupervised) and classification (supervised).
• are based on the (unsupervised) learning of multiple levels of features or representations of the data. Higher level features are derived from lower level features to form a hierarchical representation.
• are part of the broader machine learning field of learning representations of data.
• learn multiple levels of representations that correspond to different levels of abstraction; the levels form a hierarchy of concepts.

These definitions have in common (1) multiple layers of nonlinear processing units and (2) the supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features. The composition of a layer of nonlinear processing units used in a deep learning algorithm depends on the problem to be solved. Layers that have been used in deep learning include hidden layers of an artificial neural network and sets of complicated propositional formulas. They may also include latent variables organized layer-wise in deep generative models such as the nodes in Deep Belief Networks and Deep Boltzmann Machines.

Deep learning algorithms transform their inputs through more layers than shallow learning algorithms. At each layer, the signal is transformed by a processing unit, like an artificial neuron, whose parameters are 'learned' through training. A chain of transformations from input to output is a credit assignment path (CAP). CAPs describe potentially causal connections between input and output and may vary in length. For a feedforward neural network, the depth of the CAPs, and thus the depth of the network, is the number of hidden layers plus one (the output layer is also parameterized). For recurrent neural networks, in which a signal may propagate through a layer more than once, the CAP is potentially unlimited in length. There is no universally agreed upon threshold of depth dividing shallow learning from deep learning, but most researchers in the field agree that deep learning has multiple nonlinear layers (CAP > 2) and Schmidhuber considers CAP > 10 to be very deep learning.

What is Deep Learning

Deep learning (also known as deep structured learning, hierarchical learning or deep machine learning) is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using a deep graph with multiple processing layers, composed of multiple linear and non-linear transformations.

Deep learning is part of a broader family of machine learning methods based on learning representations of data. An observation (e.g., an image) can be represented in many ways such as a vector of intensity values per pixel, or in a more abstract way as a set of edges, regions of particular shape, etc. Some representations are better than others at simplifying the learning task (e.g., face recognition or facial expression recognition). One of the promises of deep learning is replacing handcrafted features with efficient algorithms for unsupervised or semi-supervised feature learning and hierarchical feature extraction.

Research in this area attempts to make better representations and create models to learn these representations from large-scale unlabeled data. Some of the representations are inspired by advances in neuroscience and are loosely based on interpretation of information processing and communication patterns in a nervous system, such as neural coding which attempts to define a relationship between various stimuli and associated neuronal responses in the brain.

Various deep learning architectures such as deep neural networks, convolutional deep neural networks, deep belief networks and recurrent neural networks have been applied to fields like computer vision, automatic speech recognition, natural language processing, audio recognition and bioinformatics where they have been shown to produce state-of-the-art results on various tasks.

Deep learning has been characterized as a buzzword, or a rebranding of neural networks.

Algorithms For Pattern Recognition II

Ensemble learning algorithms (supervised meta-algorithms for combining multiple learning algorithms together)

Here are some algorithm that based on Ensemble learning algorithms:

• Boosting (meta-algorithm)
• Bootstrap aggregating ("bagging")
• Ensemble averaging
• Mixture of experts, hierarchical mixture of experts
• General algorithms for predicting arbitrarily-structured (sets of) labels
• Bayesian networks
• Markov random fields

Multilinear subspace learning algorithms (predicting labels of multidimensional data using tensor representations)

Unsupervised:

• Multilinear principal component analysis (MPCA)
• Real-valued sequence labeling algorithms (predicting sequences of real-valued labels)

Supervised :

• Kalman filters
• Particle filters
• Regression algorithms (predicting real-valued labels)

Regression Algorithms

Supervised:

• Gaussian process regression (kriging)
• Linear regression and extensions
• Neural networks and Deep learning methods

Unsupervised:

• Independent component analysis (ICA)
• Principal components analysis (PCA)

Sequence labeling algorithms (predicting sequences of categorical labels)

Supervised:

• Conditional random fields (CRFs)
• Hidden Markov models (HMMs)
• Maximum entropy Markov models (MEMMs)
• Recurrent neural networks

Unsupervised:

• Hidden Markov models (HMMs)

Algorithms For Pattern Recognition

Algorithms for pattern recognition depend on the type of label output, on whether learning is supervised or unsupervised, and on whether the algorithm is statistical or non-statistical in nature. Statistical algorithms can further be categorized as generative or discriminative.

Classification algorithms (supervised algorithms predicting categorical labels)

In Classification algorithm for pattern recognition, it have two kinds of alogrithm: parametric and non-parametric.

Parametric:

• Linear discriminant analysis
• Quadratic discriminant analysis
• Maximum entropy classifier (aka logistic regression, multinomial logistic regression): Note that logistic regression is an algorithm for classification, despite its name. (The name comes from the fact that logistic regression uses an extension of a linear regression model to model the probability of an input being in a particular class.)

Nonparametric:

• Decision trees, decision lists
• Kernel estimation and K-nearest-neighbor algorithms
• Naive Bayes classifier
• Neural networks (multi-layer perceptrons)
• Perceptrons
• Support vector machines (svm)
• Gene expression programming

Clustering algorithms (unsupervised algorithms predicting categorical labels)

Clustering algorithm for pattern recognition have some algorithm. They are:

• Categorical mixture models
• Deep learning methods
• Hierarchical clustering (agglomerative or divisive)
• K-means clustering
• Correlation clustering
• Kernel principal component analysis (Kernel PCA)

The Using of Pattern Recongition

Within medical science, pattern recognition is the basis for computer-aided diagnosis (CAD) systems. CAD describes a procedure that supports the doctor's interpretations and findings.

Other typical applications of pattern recognition techniques are automatic speech recognition, classification of text into several categories (e.g., spam/non-spam email messages), the automatic recognition of handwritten postal codes on postal envelopes, automatic recognition of images of human faces, or handwriting image extraction from medical forms. The last two examples form the subtopic image analysis of pattern recognition that deals with digital images as input to pattern recognition systems.

Optical character recognition is a classic example of the application of a pattern classifier, see OCR-example. The method of signing one's name was captured with stylus and overlay starting in 1990. The strokes, speed, relative min, relative max, acceleration and pressure is used to uniquely identify and confirm identity. Banks were first offered this technology, but were content to collect from the FDIC for any bank fraud and did not want to inconvenience customers.

Artificial neural networks (neural net classifiers) and Deep Learning have many real-world applications in image processing, a few examples:

• identification and authentication: e.g., license plate recognition, fingerprint analysis and face detection/verification;
• medical diagnosis: e.g., screening for cervical cancer (Papnet) or breast tumors;
• defence: various navigation and guidance systems, target recognition systems, shape recognition technology etc.
• For a discussion of the aforementioned applications of neural networks in image processing.

In psychology, pattern recognition (making sense of and identifying objects) is closely related to perception, which explains how the sensory inputs humans receive are made meaningful. Pattern recognition can be thought of in two different ways: the first being template matching and the second being feature detection. A template is a pattern used to produce items of the same proportions. The template-matching hypothesis suggests that incoming stimuli are compared with templates in the long term memory. If there is a match, the stimulus is identified. Feature detection models, such as the Pandemonium system for classifying letters (Selfridge, 1959), suggest that the stimuli are broken down into their component parts for identification. For example, a capital E has three horizontal lines and one vertical line.