title: Efficient algorithms for accurate hierarchical clustering of huge datasets: tackling the entire protein space.
abstract: BACKGROUND
  UPGMA (average linking) is probably the most popular algorithm for hierarchical data clustering, especially in computational biology. However, UPGMA requires the entire dissimilarity matrix in memory. Due to this prohibitive requirement, UPGMA is not scalable to very large datasets.
  
  
  CONCLUSIONS
  We present a novel class of memory-constrained UPGMA (MC-UPGMA) algorithms. Given any practical memory size constraint, this framework guarantees the correct clustering solution without explicitly requiring all dissimilarities in memory. The algorithms are general and are applicable to any dataset. We present a data-dependent characterization of hardness and clustering efficiency. The presented concepts are applicable to any agglomerative clustering formulation.
  
  
  RESULTS
  We apply our algorithm to the entire collection of protein sequences, to automatically build a comprehensive evolutionary-driven hierarchy of proteins from sequence alone. The newly created tree captures protein families better than state-of-the-art large-scale methods such as CluSTr, ProtoNet4 or single-linkage clustering. We demonstrate that leveraging the entire mass embodied in all sequence similarities allows to significantly improve on current protein family clusterings which are unable to directly tackle the sheer mass of this data. Furthermore, we argue that non-metric constraints are an inherent complexity of the sequence space and should not be overlooked. The robustness of UPGMA allows significant improvement, especially for multidomain proteins, and for large or divergent families.
  
  
  BACKGROUND
  A comprehensive tree built from all UniProt sequence similarities, together with navigation and classification tools will be made available as part of the ProtoNet service. A C++ implementation of the algorithm is available on request.
authors.affiliation: School of Computer Science and Engineering, Institute of Life Sciences, The Hebrew University of Jerusalem, Israel.
authors.forename: Yaniv
authors.initials: Y
authors.lastname: Loewenstein
authors.forename: Elon
authors.initials: E
authors.lastname: Portugaly
authors.forename: Menachem
authors.initials: M
authors.lastname: Fromer
authors.forename: Michal
authors.initials: M
authors.lastname: Linial
mesh_terms: D000465:Algorithms; D000595:Amino Acid Sequence; D001185:Artificial Intelligence; D016000:Cluster Analysis; D030562:Databases, Protein; D008969:Molecular Sequence Data; D010363:Pattern Recognition, Automated / Q000379:methods; D011506:Proteins / Q000737:chemistry*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D016415:Sequence Alignment / Q000379:methods*; D020539:Sequence Analysis, Protein / Q000379:methods*
chemical_list: D011506:Proteins