Segmented-memory recurrent neural networks.
Conventional recurrent neural networks (RNNs) have difficulties in learning long-term dependencies. To tackle this problem, we propose an architecture called segmented-memory recurrent neural network (SMRNN). A symbolic sequence is broken into segments and then presented as inputs to the SMRNN one symbol per cycle. The SMRNN uses separate internal states to store symbol-level context, as well as segment-level context. The symbol-level context is updated for each symbol presented for input. The segment-level context is updated after each segment. The SMRNN is trained using an extended real-time recurrent learning algorithm. We test the performance of SMRNN on the information latching problem, the "two-sequence problem" and the problem of protein secondary structure (PSS) prediction. Our implementation results indicate that SMRNN performs better on long-term dependency problems than conventional RNNs. Besides, we also theoretically analyze how the segmented memory of SMRNN helps learning long-term temporal dependencies and study the impact of the segment length.
Chen
Jinmiao
J
School of Computer Engineering, Nanyang Technological University, Singapore 639798, Singapore. chenjinmiao@pmail.ntu.edu.sg
Chaudhari
Narendra S
NS
D000465:Algorithms; D001185:Artificial Intelligence; D005544:Forecasting / Q000379:methods; D006801:Humans; D008568:Memory; D016571:Neural Networks, Computer; D017433:Protein Structure, Secondary; D011506:Proteins / Q000737:chemistry / Q000235:genetics; D020539:Sequence Analysis, Protein
D011506:Proteins