title: A framework for exploration and cleaning of environmental data--Tehran air quality data experience.
abstract: BACKGROUND
  Management and cleaning of large environmental monitored data sets is a specific challenge. In this article, the authors present a novel framework for exploring and cleaning large datasets. As a case study, we applied the method on air quality data of Tehran, Iran from 1996 to 2013.
  
  
  METHODS
  The framework consists of data acquisition [here, data of particulate matter with aerodynamic diameter ≤10 µm (PM10)], development of databases, initial descriptive analyses, removing inconsistent data with plausibility range, and detection of missing pattern. Additionally, we developed a novel tool entitled spatiotemporal screening tool (SST), which considers both spatial and temporal nature of data in process of outlier detection. We also evaluated the effect of dust storm in outlier detection phase.
  
  
  RESULTS
  The raw mean concentration of PM10 before implementation of algorithms was 88.96 µg/m3 for 1996-2013 in Tehran. After implementing the algorithms, in total, 5.7% of data points were recognized as unacceptable outliers, from which 69% data points were detected by SST and 1% data points were detected via dust storm algorithm. In addition, 29% of unacceptable outlier values were not in the PR.  The mean concentration of PM10 after implementation of algorithms was 88.41 µg/m3. However, the standard deviation was significantly decreased from 90.86 µg/m3 to 61.64 µg/m3 after implementation of the algorithms. There was no distinguishable significant pattern according to hour, day, month, and year in missing data.
  
  
  CONCLUSIONS
  We developed a novel framework for cleaning of large environmental monitored data, which can identify hidden patterns. We also presented a complete picture of PM10 from 1996 to 2013 in Tehran. Finally, we propose implementation of our framework on large spatiotemporal databases, especially in developing countries.
authors.affiliation: 1)Department of Epidemiology and Biostatistics, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran. 2)Center for Air Pollution Research (CAPR), Institute for Environmental Research (IER), Tehran University of Medical Sciences, Tehran, Iran. 3)Non-Communicable Diseases Research Center, Endocrinology and Metabolism Population Sciences Institute, Tehran University of Medical Sciences, Tehran, Iran.yunesian@tums.ac.ir.
authors.forename: Mansour
authors.initials: M
authors.lastname: Shamsipour
authors.affiliation: 3)Non-Communicable Diseases Research Center, Endocrinology and Metabolism Population Sciences Institute, Tehran University of Medical Sciences, Tehran, Iran. 4)Endocrinology and Metabolism Research center, Endocrinology and Metabolism Research Institute, Tehran University of Medical Sciences, Tehran, Iran.
authors.forename: Farshad
authors.initials: F
authors.lastname: Farzadfar
authors.affiliation: 5)Department of Biostatistics, Faculty of Paramedical Sciences, Shahid Beheshti University of Medical Sciences, Tehran, Iran. 3)Non-Communicable Diseases Research Center, Endocrinology and Metabolism Population Sciences Institute, Tehran University of Medical Sciences, Tehran, Iran.
authors.forename: Kimiya
authors.initials: K
authors.lastname: Gohari
authors.affiliation: 1)Department of Epidemiology and Biostatistics, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran. 3)Non-Communicable Diseases Research Center, Endocrinology and Metabolism Population Sciences Institute, Tehran University of Medical Sciences, tehran, Iran.
authors.forename: Mahboubeh
authors.initials: M
authors.lastname: Parsaeian
authors.affiliation: 6)Department of Epidemiology and Public Health, Swiss Tropical and Public Health Institute (Swiss TPH), Basel, Switzerland.7)University of Basel, Basel, Switzerland. 8)Kurdistan Environmental Health Research Center, Kurdistan University of Medical Sciences, Sanandaj, Iran.
authors.forename: Hassan
authors.initials: H
authors.lastname: Amini
authors.affiliation: Isfahan Cardiovascular Research Center, Cardiovascular Research Institute, Isfahan University of Medical Sciences, Isfahan, Iran.
authors.forename: Katayoun
authors.initials: K
authors.lastname: Rabiei
authors.affiliation: 2)Center for Air Pollution Research (CAPR), Institute for Environmental Research (IER), Tehran University of Medical Sciences, Tehran, Iran. 10)Department of Environmental Health Engineering, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran.
authors.forename: Mohammad Sadegh
authors.initials: MS
authors.lastname: Hassanvand
authors.affiliation: 1)Department of Epidemiology and Biostatistics, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran. 3)Non-Communicable Diseases Research Center, Endocrinology and Metabolism Population Sciences Institute, Tehran University of Medical Sciences, tehran, Iran.
authors.forename: Iman
authors.initials: I
authors.lastname: Navidi
authors.affiliation: Department of Epidemiology and Biostatistics, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran.
authors.forename: Akbar
authors.initials: A
authors.lastname: Fotouhi
authors.affiliation: 2)Center for Air Pollution Research (CAPR), Institute for Environmental Research (IER), Tehran University of Medical Sciences, Tehran, Iran. 10)Department of Environmental Health Engineering, School of Public Health, Tehran University of Medical Sciences, Tehran, Iran.
authors.forename: Kazem
authors.initials: K
authors.lastname: Naddafi
authors.affiliation: Isfahan Cardiovascular Research Center, Cardiovascular Research Institute, Isfahan University of Medical Sciences, Isfahan, Iran.
authors.forename: Nizal
authors.initials: N
authors.lastname: Sarrafzadegan
authors.affiliation: 5)Department of Biostatistics, Faculty of Paramedical Sciences, Shahid Beheshti University of Medical Sciences, Tehran, Iran. 3)Non-Communicable Diseases Research Center, Endocrinology and Metabolism Population Sciences Institute, Tehran University of Medical Sciences, tehran, Iran.
authors.forename: Anita
authors.initials: A
authors.lastname: Mansouri
authors.affiliation: 2)Center for Air Pollution Research (CAPR), Institute for Environmental Research (IER), Tehran University of Medical Sciences, Tehran, Iran. 11)Center for Water Quality Research (CWQR), Institute for Environmental Research (IER), Tehran University of Medical Sciences, Tehran, Iran.
authors.forename: Alireza
authors.initials: A
authors.lastname: Mesdaghinia
authors.affiliation: Endocrinology and Metabolism Research center, Endocrinology and Metabolism Research Institute, Tehran University of Medical sciences, Tehran, Iran.
authors.forename: Bagher
authors.initials: B
authors.lastname: Larijani
authors.affiliation: 2)Center for Air Pollution Research (CAPR), Institute for Environmental Research (IER), Tehran University of Medical Sciences, Tehran, Iran. 10)Department of Environmental Health Engineering, School of Public Health, Tehran University of Medical Sciences,tehran, Iran.
authors.forename: Masud
authors.initials: M
authors.lastname: Yunesian
mesh_terms: D000393:Air Pollutants / Q000032:analysis*; D000397:Air Pollution / Q000032:analysis / Q000706:statistics & numerical data*; D000465:Algorithms; D003627:Data Interpretation, Statistical; D016208:Databases, Factual; D004784:Environmental Monitoring / Q000379:methods* / Q000706:statistics & numerical data; D007492:Iran; D015233:Models, Statistical; D052638:Particulate Matter / Q000032:analysis*; D012107:Research Design; D062211:Spatio-Temporal Analysis; D013997:Time Factors
chemical_list: D000393:Air Pollutants; D052638:Particulate Matter