Neural and Evolutionary Computing
New submissions
[ showing up to 2000 entries per page: fewer  more ]
New submissions for Tue, 19 Oct 21
 [1] arXiv:2110.08631 [pdf, other]

Title: Learning Continuous Chaotic Attractors with a Reservoir ComputerAuthors: Lindsay M. Smith (1), Jason Z. Kim (1), Zhixin Lu (1), Dani S. Bassett (1 and 2) ((1) University of Pennsylvania, (2) Santa Fe Institute)Comments: 9 pagesSubjects: Neural and Evolutionary Computing (cs.NE); Chaotic Dynamics (nlin.CD)
Neural systems are well known for their ability to learn and store information as memories. Even more impressive is their ability to abstract these memories to create complex internal representations, enabling advanced functions such as the spatial manipulation of mental representations. While recurrent neural networks (RNNs) are capable of representing complex information, the exact mechanisms of how dynamical neural systems perform abstraction are still not wellunderstood, thereby hindering the development of more advanced functions. Here, we train a 1000neuron RNN  a reservoir computer (RC)  to abstract a continuous dynamical attractor memory from isolated examples of dynamical attractor memories. Further, we explain the abstraction mechanism with new theory. By training the RC on isolated and shifted examples of either stable limit cycles or chaotic Lorenz attractors, the RC learns a continuum of attractors, as quantified by an extra Lyapunov exponent equal to zero. We propose a theoretical mechanism of this abstraction by combining ideas from differentiable generalized synchronization and feedback dynamics. Our results quantify abstraction in simple neural systems, enabling us to design artificial RNNs for abstraction, and leading us towards a neural basis of abstraction.
 [2] arXiv:2110.08741 [pdf, ps, other]

Title: Minimal Conditions for Beneficial Local SearchAuthors: Mark G WallaceComments: 31 pages plus 18 pages of appendixSubjects: Neural and Evolutionary Computing (cs.NE); Logic in Computer Science (cs.LO)
This paper investigates why it is beneficial, when solving a problem, to search in the neighbourhood of a current solution. The paper identifies properties of problems and neighbourhoods that support two novel proofs that neighbourhood search is beneficial over blind search. These are: firstly a proof that search within the neighbourhood is more likely to find an improving solution in a single search step than blind search; and secondly a proof that a local improvement, using a sequence of neighbourhood search steps, is likely to achieve a greater improvement than a sequence of blind search steps. To explore the practical impact of these properties, a range of problem sets and neighbourhoods are generated, where these properties are satisfied to different degrees. Experiments reveal that the benefits of neighbourhood search vary dramatically in consequence. Random problems of a classical combinatorial optimisation problem are analysed, in order to demonstrate that the underlying theory is reflected in practice.
 [3] arXiv:2110.08858 [pdf, other]

Title: Backpropagation with Biologically Plausible SpatioTemporal Adjustment For Training Deep Spiking Neural NetworksSubjects: Neural and Evolutionary Computing (cs.NE); Computer Vision and Pattern Recognition (cs.CV)
The spiking neural network (SNN) mimics the information processing operation in the human brain, represents and transmits information in spike trains containing wealthy spatial and temporal information, and shows superior performance on many cognitive tasks. In addition, the eventdriven information processing enables the energyefficient implementation on neuromorphic chips. The success of deep learning is inseparable from backpropagation. Due to the discrete information transmission, directly applying the backpropagation to the training of the SNN still has a performance gap compared with the traditional deep neural networks. Also, a large simulation time is required to achieve better performance, which results in high latency. To address the problems, we propose a biological plausible spatial adjustment, which rethinks the relationship between membrane potential and spikes and realizes a reasonable adjustment of gradients to different time steps. And it precisely controls the backpropagation of the error along the spatial dimension. Secondly, we propose a biologically plausible temporal adjustment making the error propagate across the spikes in the temporal dimension, which overcomes the problem of the temporal dependency within a single spike period of the traditional spiking neurons. We have verified our algorithm on several datasets, and the experimental results have shown that our algorithm greatly reduces the network latency and energy consumption while also improving network performance. We have achieved stateoftheart performance on the neuromorphic datasets NMNIST, DVSGesture, and DVSCIFAR10. For the static datasets MNIST and CIFAR10, we have surpassed most of the traditional SNN backpropagation training algorithm and achieved relatively superior performance.
 [4] arXiv:2110.09332 [pdf, other]

Title: Result Diversification by Multiobjective Evolutionary Algorithms with Theoretical GuaranteesComments: 46 pages, 2 figuresSubjects: Neural and Evolutionary Computing (cs.NE); Computational Complexity (cs.CC); Machine Learning (cs.LG)
Given a ground set of items, the result diversification problem aims to select a subset with high "quality" and "diversity" while satisfying some constraints. It arises in various realworld artificial intelligence applications, such as webbased search, document summarization and feature selection, and also has applications in other areas, e.g., computational geometry, databases, finance and operations research. Previous algorithms are mainly based on greedy or local search. In this paper, we propose to reformulate the result diversification problem as a biobjective maximization problem, and solve it by a multiobjective evolutionary algorithm (EA), i.e., the GSEMO. We theoretically prove that the GSEMO can achieve the (asymptotically) optimal theoretical guarantees under both static and dynamic environments. For cardinality constraints, the GSEMO can achieve the optimal polynomialtime approximation ratio, $1/2$. For more general matroid constraints, the GSEMO can achieve the asymptotically optimal polynomialtime approximation ratio, $1/2\epsilon/(4n)$. Furthermore, when the objective function (i.e., a linear combination of quality and diversity) changes dynamically, the GSEMO can maintain this approximation ratio in polynomial running time, addressing the open question proposed by Borodin et al. This also theoretically shows the superiority of EAs over local search for solving dynamic optimization problems for the first time, and discloses the robustness of the mutation operator of EAs against dynamic changes. Experiments on the applications of webbased search, multilabel feature selection and document summarization show the superior performance of the GSEMO over the stateoftheart algorithms (i.e., the greedy algorithm and local search) under both static and dynamic environments.
Crosslists for Tue, 19 Oct 21
 [5] arXiv:2110.08259 (crosslist from cs.LG) [pdf, other]

Title: Training Neural Networks for Solving 1D Optimal Piecewise Linear ApproximationSubjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
Recently, the interpretability of deep learning has attracted a lot of attention. A plethora of methods have attempted to explain neural networks by feature visualization, saliency maps, model distillation, and so on. However, it is hard for these methods to reveal the intrinsic properties of neural networks. In this work, we studied the 1D optimal piecewise linear approximation (PWLA) problem, and associated it with a designed neural network, named lattice neural network (LNN). We asked four essential questions as following: (1) What are the characters of the optimal solution of the PWLA problem? (2) Can an LNN converge to the global optimum? (3) Can an LNN converge to the local optimum? (4) Can an LNN solve the PWLA problem? Our main contributions are that we propose the theorems to characterize the optimal solution of the PWLA problem and present the LNN method for solving it. We evaluated the proposed LNNs on approximation tasks, forged an empirical method to improve the performance of LNNs. The experiments verified that our LNN method is competitive with the startoftheart method.
 [6] arXiv:2110.08465 (crosslist from cs.LG) [pdf, other]

Title: A Heterogeneous Graph Based Framework for Multimodal Neuroimaging Fusion LearningSubjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Neurons and Cognition (qbio.NC)
Here, we present a Heterogeneous Graph neural network for Multimodal neuroimaging fusion learning (HGM). Traditional GNNbased models usually assume the brain network is a homogeneous graph with single type of nodes and edges. However, vast literatures have shown the heterogeneity of the human brain especially between the two hemispheres. Homogeneous brain network is insufficient to model the complicated brain state. Therefore, in this work we firstly model the brain network as heterogeneous graph with multitype nodes (i.e., left and right hemispheric nodes) and multitype edges (i.e., intra and interhemispheric edges). Besides, we also propose a selfsupervised pretraining strategy based on heterogeneou brain network to address the overfitting problem due to the complex model and small sample size. Our results on two datasets show the superiority of proposed model over other multimodal methods for disease prediction task. Besides, ablation experiments show that our model with pretraining strategy can alleviate the problem of limited training sample size.
 [7] arXiv:2110.08598 (crosslist from eess.AS) [pdf, other]

Title: A Variational Bayesian Approach to Learning Latent Variables for Acoustic Knowledge TransferComments: Submitted to ICASSP 2022Subjects: Audio and Speech Processing (eess.AS); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Sound (cs.SD)
We propose a variational Bayesian (VB) approach to learning distributions of latent variables in deep neural network (DNN) models for crossdomain knowledge transfer, to address acoustic mismatches between training and testing conditions. Instead of carrying out point estimation in conventional maximum a posteriori estimation with a risk of having a curse of dimensionality in estimating a huge number of model parameters, we focus our attention on estimating a manageable number of latent variables of DNNs via a VB inference framework. To accomplish model transfer, knowledge learnt from a source domain is encoded in prior distributions of latent variables and optimally combined, in a Bayesian sense, with a small set of adaptation data from a target domain to approximate the corresponding posterior distributions. Experimental results on device adaptation in acoustic scene classification show that our proposed VB approach can obtain good improvements on target devices, and consistently outperforms 13 stateoftheart knowledge transfer algorithms.
 [8] arXiv:2110.08771 (crosslist from cs.LG) [pdf]

Title: An LSTMbased Plagiarism Detection via Attention Mechanism and a Populationbased Approach for PreTraining Parameters with imbalanced ClassesAuthors: Seyed Vahid Moravvej, Seyed Jalaleddin Mousavirad, Mahshid Helali Moghadam, Mehrdad SaadatmandComments: 12 pages, The 28th International Conference on Neural Information Processing (ICONIP2021), BALI, IndonesiaSubjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)
Plagiarism is one of the leading problems in academic and industrial environments, which its goal is to find the similar items in a typical document or source code. This paper proposes an architecture based on a Long ShortTerm Memory (LSTM) and attention mechanism called LSTMAMABC boosted by a populationbased approach for parameter initialization. Gradientbased optimization algorithms such as backpropagation (BP) are widely used in the literature for learning process in LSTM, attention mechanism, and feedforward neural network, while they suffer from some problems such as getting stuck in local optima. To tackle this problem, populationbased metaheuristic (PBMH) algorithms can be used. To this end, this paper employs a PBMH algorithm, artificial bee colony (ABC), to moderate the problem. Our proposed algorithm can find the initial values for model learning in all LSTM, attention mechanism, and feedforward neural network, simultaneously. In other words, ABC algorithm finds a promising point for starting BP algorithm. For evaluation, we compare our proposed algorithm with both conventional and populationbased methods. The results clearly show that the proposed method can provide competitive performance.
 [9] arXiv:2110.08966 (crosslist from math.OC) [pdf, other]

Title: Computing Semilinear Sparse Models for Approximately Eventually Periodic SignalsAuthors: Fredy VidesSubjects: Optimization and Control (math.OC); Neural and Evolutionary Computing (cs.NE); Systems and Control (eess.SY)
Some elements of the theory and algorithmics corresponding to the computation of semilinear sparse models for discretetime signals are presented. In this study, we will focus on approximately eventually periodic discretetime signals, that is, signals that can exhibit an aperiodic behavior for an initial amount of time, and then become approximately periodic afterwards. The semilinear models considered in this study are obtained by combining sparse representation methods, linear autoregressive models and GRU neural network models, initially fitting each block model independently using some reference data corresponding to some signal under consideration, and then fitting some mixing parameters that are used to obtain a signal model consisting of a linear combination of the previously fitted blocks using the aforementioned reference data, computing sparse representations of some of the matrix parameters of the resulting model along the process. Some prototypical computational implementations are presented as well.
 [10] arXiv:2110.09138 (crosslist from cs.LG) [pdf, other]

Title: StateSpace Constraints Improve the Generalization of the Differentiable Neural Computer in some Algorithmic TasksSubjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
Memoryaugmented neural networks (MANNs) can solve algorithmic tasks like sorting. However, they often do not generalize to lengths of input sequences not seen in the training phase. Therefore, we introduce two approaches constraining the statespace of the network controller to improve the generalization to outofdistributionsized input sequences: state compression and state regularization. We show that both approaches can improve the generalization capability of a particular type of MANN, the differentiable neural computer (DNC), and compare our approaches to a stateful and a stateless controller on a set of algorithmic tasks. Furthermore, we show that especially the combination of both approaches can enable a pretrained DNC to be extended post hoc with a larger memory. Thus, our introduced approaches allow to train a DNC using shorter input sequences and thus save computational resources. Moreover, we observed that the capability for generalization is often accompanied by loop structures in the statespace, which could correspond to looping constructs in algorithms.
 [11] arXiv:2110.09217 (crosslist from cs.CV) [pdf]

Title: Color Image Segmentation Using MultiObjective Swarm Optimizer and Multilevel Histogram ThresholdingComments: 11 pages, 6 figuresSubjects: Computer Vision and Pattern Recognition (cs.CV); Neural and Evolutionary Computing (cs.NE)
Rapid developments in swarm intelligence optimizers and computer processing abilities make opportunities to design more accurate, stable, and comprehensive methods for color image segmentation. This paper presents a new way for unsupervised image segmentation by combining histogram thresholding methods (Kapur's entropy and Otsu's method) and different multiobjective swarm intelligence algorithms (MOPSO, MOGWO, MSSA, and MOALO) to thresholding 3D histogram of a color image. More precisely, this method first combines the objective function of traditional thresholding algorithms to design comprehensive objective functions then uses multiobjective optimizers to find the best thresholds during the optimization of designed objective functions. Also, our method uses a vector objective function in 3D space that could simultaneously handle the segmentation of entire image color channels with the same thresholds. To optimize this vector objective function, we employ multiobjective swarm optimizers that can optimize multiple objective functions at the same time. Therefore, our method considers dependencies between channels to find the thresholds that satisfy objective functions of color channels (which we name as vector objective function) simultaneously. Segmenting entire color channels with the same thresholds also benefits from the fact that our proposed method needs fewer thresholds to segment the image than other thresholding algorithms; thus, it requires less memory space to save thresholds. It helps a lot when we want to segment many images to many regions. The subjective and objective results show the superiority of this method to traditional thresholding methods that separately threshold histograms of a color image.
Replacements for Tue, 19 Oct 21
 [12] arXiv:2108.09455 (replaced) [pdf, other]

Title: Natural Evolution Strategy for Unconstrained and Implicitly Constrained Problems with Ridge StructureComments: accepted at IEEE Symposium Series on Computational Intelligence (IEEE SSCI 2021)Subjects: Neural and Evolutionary Computing (cs.NE)
 [13] arXiv:2106.12423 (replaced) [pdf, other]

Title: AliasFree Generative Adversarial NetworksAuthors: Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, Timo AilaSubjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE); Machine Learning (stat.ML)
[ showing up to 2000 entries per page: fewer  more ]
Disable MathJax (What is MathJax?)
Links to: arXiv, form interface, find, cs, recent, 2110, contact, help (Access key information)