Overview

Welcome to the Data Validator webpage. This tool allows you to validate data interactively by uploading a dataset and rules file. To get started, go to the validator tab on the left.

Contribute

Join our team to build this tool!

Github

Win Cowger

Moore Institute for Plastic Pollution Research

    Shelly Moore

    Moore Institute for Plastic Pollution Research

      Hannah Hapich

      Moore Institute for Plastic Pollution Research

        Scott Coffin

        State Water Resources Control Board

          Tony Hale

          San Francisco Estuary Institute

            Diana Lin

            San Francisco Estuary Institute

              Rebecca Sutton

              San Francisco Estuary Institute

                Gabriel Daiess

                  Leah Thornton Hampton

                  Southern California Coastal Water Research Project

                    Richard Nelson

                    California State Water Resources Control Board

                      Haig Jack Minasian

                      CSU Long Beach

                        Holden Ford

                        CSU Long Beach

                          Anja Oca

                          CSU Long Beach

                            Libby Heeren

                            Moore Institute for Plastic Pollution Research

                              Hannah Sherrod

                              Moore Institute for Plastic Pollution Research

                                Nicholas Leong

                                Moore Institute for Plastic Pollution Research

                                  Instructions for Preparing Your Data

                                  Contents

                                  INTRODUCTION                                                                                                                             

                                  SAMPLE TABLE                                                                                                                             

                                  SAMPLE FILE VALIDATION CHECKS                                                                                                      

                                  METHODOLOGY TABLE

                                  METHODOLOGY FILE VALIDATION CHECKS                                                                            

                                  PARTICLE TABLE STRUCTURE                                                                                                   

                                  PARTICLE FILE VALIDATION CHECKS                                                                                     

                                  DATA LIBRARIES                                                                                                                              

                                  DOWNLOAD PDF OF THE INSTRUCTIONS


                                   

                                  INTRODUCTION

                                   

                                  Data reporting is often an arduous and difficult task; however, ensuring consistent and clean data when many data collectors are involved is necessary. This section describes the data tables necessary to produce understandable and easily analyzed data. The information in this report on what data is best to collect is based on a recent paper entitled “Reporting requirements to increase the reproducibility and comparability of research on microplastics” by Cowger et. al (in press). While it may seem complex, it is can be broken down into simple tables. There are three main data tables and six additional metadata tables (Figure 1). The three main tables include information directly related to the sample, with the Sample Results table containing the main results relative to the amount, shape and type of microplastics contained within each sample. The Lab Information, Instrument Information and related instrument type tables include metadata on how procedures were done in the lab to prepare and extract the microplastics from various matrices, as well as how the instruments were used to analyze the microplastics samples. Recording this information is extremely important to determine what factors may alter the results from a given sample.

                                  Figure 1. Simplistic diagram of microplastic data tables and flow.

                                   

                                  Data will be entered into the different tables through a Microsoft Excel template workbook. The sheets within the workbook will include the table names and necessary fields to fill in within each table (Figure 2). The plan is to have all project participants submit their data through a web-based data system that will check the data for form and consistency. This section of the study plan is meant to provide guidance on how to enter the data, show the relationships between the tables, and define each field and the values that go into them.

                                   

                                  Figure 2. Entity relationship diagram showing microplastic data tables and structures.

                                   

                                  Each required table has its own section below with a description of the purpose of each table as well as a table of the field names, types, sizes, and a definition. The order of the tables is 1) Sample, 2) Particle, and 3) Methodology.

                                   

                                  SUBMISSION GUIDELINES

                                   

                                  The file must be a comma separated file (*.csv) and can be named at the discretion of the user.

                                  A website will be created by XXXX for data submission. Users will drag and drop their file over the website and the file will be checked for errors. Errors include those associated with required fields that are not populated, fields that include the wrong data type, and values within the fields that do not match the list of allowed values. More information on data submission will be posted as the date nears for data submission.

                                  DATA RECORDING GUIDELINES

                                  Data will be entered into tables as described below. Each data table is outlined in a table that consists of field names, data types, data sizes, descriptions of the field names, and whether the field is required. In addition, lookup lists are provided in the Appendix for fields that have set values to ensure consistency.  While some fields are not required you are encouraged to populate all that you are able to in order to provide any information that might influence the results.

                                  Field names are required to match the lists below for each table. The type represents the data type need in the field, which includes Text (field can be up to 255 characters), Integer (whole number), Decimal (continuous number), Date/Time (for dates and/or times) and Yes/No. The required field is filled in with Yes, No or Conditional. Conditionally required fields are dependent on how other fields are filled out (e.g. if one field is a yes, another field may be required to fill out to describe the yes answer). Size is relative to the number of characters needed for answers for a given field. Typically, text fields are limited to a maximum of 255, but smaller limits are preferred if the choices for a given field can be smaller in size. Comments fields are usually 255 characters, but it is recommended that comments be as brief and to the point as possible and also relate directly to information necessary to the record. A brief description is provided for each field to help the data provider determine the values to put into each filed. Where possible, constrained lookup lists are provided in the appendix. These lookup lists provide consistency in values and ensure proper spelling, etc.

                                  DATA SUBMISSION TOOL

                                   

                                  SAMPLE TABLE

                                  The purpose of this table is to document information regarding the lab receiving the samples. Information on the types of samples, when they were received and who received them will be collected. This table can be connected to the Particle table via the SampleID field.

                                  PRIMARY KEY: To ensure each record is unique within the data set the following fields will be used as a primary key:

                                       SampleID

                                   

                                  Sample table structure (primary key field is indicated with bold text).

                                  FIELD NAME

                                  TYPE

                                  REQUIRED

                                  SIZE

                                  DESCRIPTION

                                  SampleID

                                  Text

                                  Yes

                                  25

                                  The ID assigned to the sample by the lab performing the analysis.

                                  OwnerOrganization

                                  Text

                                  Yes

                                  25

                                  Organization that owns the data.

                                  AnalysisOrganization

                                  Text

                                  Yes

                                  25

                                  Organization that analyzed the sample.

                                  ReportingOrganization

                                  Text

                                  Yes

                                  25

                                  Organization reporting the data.

                                  Latitude

                                  Decimal

                                  No

                                   

                                   

                                  Longitude

                                  Decimal

                                  No

                                   

                                   

                                  CollectionDate

                                  Date

                                  Yes

                                   

                                  The date the sample was collected.

                                  SampleVolume

                                  Date

                                  Yes

                                   

                                   

                                  Comments

                                  Text

                                   

                                  N

                                  255

                                  Additional remarks relative to taking and/or receiving the sample.

                                   

                                   

                                  SAMPLE FILE VALIDATION CHECKS

                                  The purpose of this table is to document everything associated with sample preparation within the laboratory. This includes homogenization, digestion and extraction procedures, as well as any dyes used to stain microplastics.

                                  This table can be connected to the Sample Receiving, Sample Results and Sample Results Summary tables via a combination of the SampleID and LabID fields.

                                  PRIMARY KEY: To ensure no duplicates within the data set the following fields will be used as a primary key to create unique records:

                                       SampleID

                                       LabID

                                   

                                  Sample Preparation table structure (primary key fields are indicated with bold text).

                                  NAME

                                  VALID_EXAMPLE

                                  SEVERITY

                                  RULE

                                  SampleID

                                  23jreh334

                                  ERROR

                                  SampleID must be unique and in the SampleID List.

                                  OwnerOrganization

                                  Moore Institute for Plastic Pollution Research

                                  ERROR

                                  OwnerOrganization must be in the OrganizationList.

                                  AnalysisOrganization

                                  Moore Institute for Plastic Pollution Research

                                  ERROR

                                  AnalysisOrganization must be in the Organization List.

                                  ReportingOrganization

                                  Moore Institute for Plastic Pollution Research

                                  ERROR

                                  ReportingOrganization must be in the Organization List.

                                  Latitude

                                  33.746974

                                  ERROR

                                   

                                  Longitude

                                  -118.115372

                                  ERROR

                                   

                                  CollectionDate

                                  1/30/2023

                                  ERROR

                                  The method used to split the sample.

                                  SampleVolume

                                  100

                                  ERROR

                                   

                                   

                                   

                                  METHODOLOGY TABLE

                                  The purpose of this table is to document everything associated with the sample results for microplastics that are counted by size class but are not individually assessed for polymer type. Each record in this table represents a given size class, color and morphology category.

                                  This table can be connected to the Sample Receiving, Sample Preparation and Sample Results Summary tables via a combination of the SampleID and LabID fields.

                                  PRIMARY KEY: To ensure no duplicates within the data set the following fields will be used as a primary key to create unique records:

                                       SampleID

                                       LabID

                                       SizeClass

                                       Color

                                       MorphologyCategory

                                   

                                  Sample Results Binned table structure (primary key fields are indicated with bold text).

                                  FIELD NAME

                                  TYPE

                                  REQUIRED

                                  SIZE

                                  DESCRIPTION

                                  SampleID

                                  Text

                                  Yes

                                  25

                                  The ID assigned to the sample by the lab performing the analysis.

                                  LabID

                                  Text

                                  Yes

                                  25

                                  The ID assigned to the lab to identify them as the lab performing the analysis.

                                  AnalysisDate

                                  Date/Time

                                  Yes

                                  -

                                  Date the sample was analyzed. MM/DD/YYYY

                                  InstrumentID

                                  Text

                                  Yes

                                  25

                                  The ID assigned to the instrument by the lab performing the analysis.

                                  SizeClass

                                  Text

                                  Yes

                                  25

                                  Size Class the particles belong to (e.g. 1-20µm, 20-100µm, 100-300µm).

                                  Color

                                  Text

                                  Yes

                                  25

                                  The color of the microplastic particle (e.g. red, orange, yellow, green, blue, purple, pink, black, brown, or white).

                                  MorphologyCategory

                                  Text

                                  Yes

                                  25

                                  The shape of the particle (e.g. fragment, fiber, fiber bundle, sphere, foam, film, rubber, pellet).

                                  Count

                                  Integer

                                  Yes

                                  -

                                  Number of particles counted in the size class listed above.

                                  Mass(g)

                                  Decimal

                                  No

                                  -

                                  Mass of sample in grams.

                                  PhotoID

                                  Text

                                  Yes

                                  25

                                  Unique identification assigned to a photo of the particle.

                                  SampleType

                                  Text

                                  Yes

                                  10

                                  The type of sample this particle represents (e.g. Result or LabBlank).

                                  Comments

                                  Text

                                  No

                                  255

                                  Additional remarks relative to the sample preparation.

                                  METHODOLOGY FILE VALIDATION CHECKS

                                  The purpose of this table is to document everything associated with the sample results. Each record in this table represents an individual particle and its associated characteristics of polymer type, shape, size and color.

                                  This table can be connected to the Sample Receiving, Sample Preparation and Sample Results Summary tables via a combination of the SampleID and LabID fields.

                                  PRIMARY KEY: To ensure no duplicates within the data set the following fields will be used as a primary key to create unique records:

                                       SampleID

                                       LabID

                                       ParticleID

                                   

                                  Sample Results Subset table structure (primary key fields are indicated with bold text).

                                  FIELD NAME

                                  TYPE

                                  REQUIRED

                                  SIZE

                                  DESCRIPTION

                                  SampleID

                                  Text

                                  Yes

                                  25

                                  The ID assigned to the sample by the lab performing the analysis.

                                  LabID

                                  Text

                                  Yes

                                  25

                                  The ID assigned to the lab to identify them as the lab performing the analysis.

                                  AnalysisDate

                                  Date/Time

                                  Yes

                                  -

                                  Date the sample was analyzed. MM/DD/YYYY

                                  InstrumentID

                                  Text

                                  Yes

                                  25

                                  The ID assigned to the instrument by the lab performing the analysis.

                                  ParticleID

                                  Text

                                  Yes

                                  25

                                  The ID assigned to the particle by the lab performing the analysis.

                                  MorphologyCategory

                                  Text

                                  Yes

                                  25

                                  The shape of the particle (e.g. fragment, fiber, fiber bundle, sphere, foam, film, rubber, pellet).

                                  Color

                                  Text

                                  Yes

                                  25

                                  The color of the microplastic particle (e.g. red, orange, yellow, green, blue, purple, pink, black, brown, or white).

                                  SizeClass

                                  Text

                                  Yes

                                  25

                                  Size Class the particles belong to (e.g. 1-20µm, 20-100µm, 100-300µm).

                                  Length(mm)

                                  Integer

                                  Yes

                                  -

                                  Length of the particle along its longest axis in millimeters.

                                  Width(mm)

                                  Integer

                                  Yes

                                  -

                                  Length of the particle along its shortest axis in millimeters.

                                  PolymerType

                                  Text

                                  Yes, for FTIR, Raman and Py-GCMS

                                  100

                                  Polymer type as identified during machine analysis.

                                  Mass(g)

                                  Decimal

                                  No

                                   

                                  Mass of particle in grams.

                                  PhotoID

                                  Text

                                  Yes

                                  25

                                  Unique identification assigned to a photo of the particle.

                                  SampleType

                                  Text

                                  Yes

                                  10

                                  The type of sample this particle represents (e.g. Result or LabBlank).

                                  Comments

                                  Text

                                  No

                                  255

                                  Additional remarks relative to the sample preparation.

                                   

                                  PARTICLE TABLE STRUCTURE

                                  The purpose of this table is to document everything associated with the laboratory preparing the samples. Each record in this table represents a given time frame and the laboratory conditions during that time frame. The information in this table is meant to convey the measures taken to lessen the risk of contamination to the samples.

                                  This table can be connected to the Sample Preparation, Sample Results and Sample Results Summary tables via a combination of the LabID and AnalysisDate fields.

                                  PRIMARY KEY: To ensure no duplicates within the data set the following fields will be used as a primary key to create unique records:

                                       LabID

                                       StartDate

                                   

                                  Lab Information table structure (primary key fields are indicated with bold text).

                                  FIELD NAME

                                  TYPE

                                  REQUIRED

                                  SIZE

                                  DESCRIPTION

                                  LabID

                                  Text

                                  Yes

                                  25

                                  The ID assigned to the lab to identify them as the lab performing the analysis.

                                  StartDate

                                  Date/

                                  Time

                                  Yes

                                   

                                  Beginning date the information for the lab holds true for.

                                  EndDate

                                  Date/

                                  Time

                                  Yes

                                   

                                  End date the information for the lab holds true for.

                                  WaterType

                                  Text

                                  Yes

                                  25

                                  Type of water used in the lab for rinsing and mixing (e.g. RO, DI).

                                  AirFiltration

                                  Yes/

                                  No

                                  No

                                  3

                                  Is there an air filtration system in the lab?

                                  AirFiltrationType

                                  Text

                                  Yes, if AirFiltration is Yes

                                  100

                                  Type of filtration used in the lab (e.g. Hepa filter). Required if AirFiltration is Yes.

                                  SealedEnvironment

                                  Yes/

                                  No

                                  No

                                  3

                                  Is a sealed environment being used to minimize contamination during sample preparation?

                                  SealedEnvironmentType

                                  Text

                                  Yes, if SealedEnvironment is Yes

                                  100

                                  Type of sealed environment (e.g. laminar flow cabinet). Required if SealedEnvironment is Yes.

                                  ClothingPolicy

                                  Yes/

                                  No

                                  No

                                  3

                                  Is there a clothing policy in place in the lab?

                                  ClothingPolicyType

                                  Text

                                  Yes, if ClothingPolicy is Yes

                                  100

                                  Type of clothing policy (e.g. cotton required). Required if ClothingPolicy is Yes.

                                  FiltrationTechniqueforReagents

                                  Text

                                  Yes

                                  100

                                  Method for filtration of reagents.

                                  GlasswareRinsed

                                  Yes/

                                  No

                                  Yes

                                  3

                                  Is the glassware rinsed after each use?

                                  GlasswareRinseTemperature

                                  Decimal

                                  Yes

                                   

                                  Temperature of the water used to rinse glassware in Celcius.

                                  SamplesCovered

                                  Yes/

                                  No

                                  Yes

                                  3

                                  Are the samples covered while being processed or stored?

                                  Comments

                                  Text

                                  No

                                  255

                                  Any comments relative to the lab and procedures used.

                                   

                                  PARTICLE FILE VALIDATION CHECKS

                                  The purpose of this table is to document everything associated with the instrument(s) used to analyze microplastics samples. Each record in this table represents an instrument, its manufacturer, software and last calibration date.

                                  This table can be connected to the Sample Results and Sample Results Summary tables via the InstrumentID and LabID fields.

                                  PRIMARY KEY: To ensure no duplicates within the data set the following fields will be used as a primary key to create unique records:

                                       InstrumentID

                                       LabID

                                   

                                  Instrument Information table structure (primary key fields are indicated with bold text).

                                  FIELD NAME

                                  TYPE

                                  REQUIRED

                                  SIZE

                                  DESCRIPTION

                                  InstrumentID

                                  Text

                                  Yes

                                  25

                                  The ID assigned to the instrument by the lab performing the analysis.

                                  LabID

                                  Text

                                  Yes

                                  25

                                  The ID assigned to the lab to identify them as the lab performing the analysis.

                                  InstrumentType

                                  Text

                                  Yes

                                  100

                                  Type of instrument used for the analysis. (e.g. stereoscope, FTIR, Raman)

                                  Manufacturer

                                  Text

                                  Yes

                                  100

                                  Manufacturer of the instrument.

                                  Software

                                  Text

                                  Yes

                                  100

                                  Software and version used for the analysis on the instrument.

                                  CalibrationDate

                                  Date/

                                  Time

                                  Yes

                                  -

                                  Date the instrument was last calibrated.

                                  CalibrationFrequency

                                  Text

                                  Yes

                                  100

                                  How often is instrument calibrated during sample analysis (e.g. Daily).

                                  Comments

                                  Text

                                  No

                                  255

                                  Any comments relative to the instrument.

                                   

                                  DATA LIBRARIES

                                   

                                  Sample Matrix Lookup List

                                  SAMPLE MATRIX

                                  Clean Water

                                  Dirty Water

                                  Sediment

                                  Fish Tissue

                                  Lab Blank

                                   

                                  Shape Category Lookup List

                                  MORPHOLOGY CATEGORY

                                  Fragment

                                  Fiber

                                  Fiber bundle

                                  Sphere

                                  Foam

                                  Film

                                  Pellet

                                   

                                  Color Lookup List

                                  COLOR

                                  Black

                                  Blue

                                  Brown

                                  Clear

                                  Gold

                                  Green

                                  Grey

                                  Orange

                                  Pink

                                  Purple

                                  Red

                                  Silver

                                  White

                                  Yellow

                                   

                                   

                                  Download

                                  Tutorial

                                  Welcome to the Data Validator webpage. This tool allows you to validate data interactively by uploading a dataset and rules file. To get started, go to the validator tab on the left.

                                  Instructions and Examples

                                  Both the data and rules files must be in .csv or .xlsx format. Examples for how to structure and query the data and rules can be found below:

                                  Download Sample Rules Download Valid Sample Data Download Invalid Sample Data
                                  1. Uploaded the data and rules file on the validator tab.
                                  2. If your data is valid a popup will appear that allows you to upload to a remote repository if you have a key.
                                  3. If your data is valid you may download a certificate, this is saved on our end too to prove the event occured.
                                  4. In the event of invalid data, the description of the issue(s) to be resolved and severity will be displayed in the 'Issues Raised' panel.
                                  5. You can click on any of the descriptions to display the rows where the issue was found in the 'Issues Selected' panel.
                                  6. The 'Issues Raised' and 'Issue Selected' data sheets may be copied, or downloaded as CSV, Excel, or PDF.

                                  FAQs

                                  Where is my data going?

                                  Links shared in this tool for upload will be added to an S3 bucket to test that they are able to be downloaded. Data will not be shared externally unless you specify the data can be shared by inputting the security key.

                                  Is this open source web tool secure?

                                  The validator app is https encrypted and the source code is available on GitHub for security review.

                                  Contact Us

                                  Have any additional questions or concerns? Email us using the link below:

                                  Contact Us

                                  Please include in your email:

                                  (1) What should the app be doing?

                                  (2) What is the app doing instead?