Dataset: Subsurface Nitrospirota and Nitrospinota Origins

Summary of processing on the submitted dataset by the dataset's BCO-DMO data manager.

1. Modified submitted accessions file

Fixed two shifted cell values in the submitted file "Supplemental Data 1.csv" and saved it as "Supplemental Data 1 fixed.csv" by using Excel.

The original file has a row number 333 with some values shifted into the wrong columns. Part of an "IsolationPlot" value was shifted to the right into the "Sample" column. And the “Sample” value was shifted to the right into the “Location” column. This was fixed by cutting the value in the “Sample” column and copying it into the "IsolationPlot" column for a final value of “saline water (ENVO:00002010) including plankton (ENVO:xxxxxxxx) “ and then cutting the value SAMN06450621 from the “Location” column and copying it into the "Sample" column.

2. Updated this fixed accessions file with IMG and NCBI metadata

Python code in Jupyter notebooks was created by the data manager and run to add metadata to the submitted accessions file. The scripts were run using the fixed file "Supplemental Data 1 fixed.csv" as input.

Data manager GitHub code repo (last commit = 7e8e2462a on 12/30/2024 updating docs)
https://github.com/BCODMO/DM_scripts_by_dataset/tree/master/Orcutt_accessions/933610

Code run on 2024-12-29 using the following two Jupyter notebooks

https://github.com/BCODMO/DM_scripts_by_dataset/blob/master/Orcutt_accessions/933610/get_ncbi_accessions_with_api.ipynb

https://github.com/BCODMO/DM_scripts_by_dataset/blob/master/Orcutt_accessions/933610/create_updated_supplemental_dataset.ipynb

The code and and processing procedure are documented in markdown files in the GitHub repository
https://github.com/BCODMO/DM_scripts_by_dataset/blob/master/Orcutt_accessions/933610/README.md

https://github.com/BCODMO/DM_scripts_by_dataset/blob/master/Orcutt_accessions/933610/summary_of_processing_steps.md

Eight new columns were added using python Jupyter notebooks run on 12/29/2024 with the output saved to the file named “updated_pi_supplemental_table.csv”.

columns added to the fixed file "Supplemental Data 1 fixed.csv": IMG_genome_id, BioProject, Corrected_BioSample, GenBank_assembly, release_date, last_updated_date, publication_date, ncbi_taxon_id

column definitions

IMG_genome_id: IMG genome assembly id (extracted from values in “ID” column)
BioProject: NCBI BioProject accession
Corrected_BioSample: correct NCBI BioSample accession
GenBank_assembly: NCBI GenBank genome assembly accession
release_date: Date genome assembly was released to the public
last_updated_date: Last date BioSample was updated on NCBI
publication_date: Date genome assembly submitted to either IMG or NCBI
ncbi_taxon_id: NCBI taxon id of the sampled organism

Steps to retrieve extra metadata

Used a search on the IMG website and python Jupyter notebooks to add NCBI accessions, dates, and NCBI taxon id metadata columns to provide more context to the submitted file.

There are two sources for the new metadata. One source is a search query on the IMG website (Integrated Microbial Genomes and Microbiomes https://img.jgi.doe.gov/ https://img.jgi.doe.gov/) and the other source is NCBI API calls retrieving JSON reports.

The goal was to add corresponding BioProject accessions for each row in the ‘ID’ column of a submitted accessions file.

In the process of gathering supporting metadata by the dataset's BCO-DMO data manager, 15 BioSample values retrieved with the NCBI API using NCBI genome assembly accessions were found to be different than those in the submitted file’s “Sample” column, which contains BioSample values. The 15 new BioSample values in the column "Corrected_BioSample" in the final BCO-DMO dataset 933610_v1_nitrospirota_and_nitrospinota_genomes.csv were manually checked on their corresponding NCBI web pages. It was found that the BioSample metadata and the corresponding NCBI genome assembly value confirmed the BioSample values retrieved via NCBI API calls were correct. The original 15 BioSample values were manually checked on their corresponding NCBI web pages, and the BioSample metadata and NCBI genome assembly accession for that BioSample did not match those listed in the submitted file.

The incorrect BioSample values in the "Sample" column are the following:
SAMN03222684, SAMN03222684, SAMN02862043, SAMN03222686, SAMN03222686, SAMEA3140934, SAMEA3140928, SAMN01922995, SAMN03203005, SAMN03203000, SAMN10411419, SAMN12518157, SAMN04315473, SAMN06659639, SAMN07631077

The correct BioSample values are found in the column "Corrected_BioSample" in the primary dataset file 933610_v1_nitrospirota_and_nitrospinota_genomes.csv.

The new columns were found with results from a query on the IMG website (Integrated Microbial Genomes and Microbiomes at [https://img.jgi.doe.gov/](https://img.jgi.doe.gov/)) and python Jupyter notebooks using NCBI API calls and a series of joins with the fixed submitted accessions dataset.

To query the IMG website, a list of IMG genome ids was created using IMG id values in the “ID” column of the file "Supplemental Data 1 fixed.csv". The IMG genome ids in the “ID” column are prefixed with “IMG_” in the “ID” column and some are suffixed with an underscore followed by a number.

IMG query URL: https://img.jgi.doe.gov/cgi-bin/m/main.cgi?section=FindGenomes&page=genomeSearch

Downloaded the query results table from IMG (Integrated Microbial Genomes and Microbiomes https://img.jgi.doe.gov/) where the query used the list of IMG genome ids from the “ID” column. The results table contained NCBI accessions and other metadata associated with IMG genome ids.

Created a python Jupyter notebook to perform NCBI API calls using BioSample accessions, NCBI assembly accessions, and taxon names to retrieve data reports in JSON format. The reports JSON contained values for BioSample, BioProject, NCBI assembly accession, release date, last updated date, publication date, and NCBI taxon id.

The NCBI API calls used the NCBI version 2 REST API which is explained here https://www.ncbi.nlm.nih.gov/datasets/docs/v2/reference-docs/rest-api/. The following three API endpoints were used:

/genome/accession/{accession}/dataset_report
/biosample/accession/{accession}/biosample_report
/genome/taxon/{taxon_name}/dataset_report

The API endpoint /genome/accession/{accession}/dataset_report was called with NCBI genome assembly accessions from the “ID” column of the accessions dataset table.

The API endpoint /biosample/accession/{accession}/biosample_report was called with NCBI BioSample accessions from the “Sample” column of the accessions dataset table.

The API endpoint /genome/taxon/{taxon_name}/dataset_report was called using the phylum taxon names of Nitrospinota and Nitrospirota.

All the metadata results from the NCBI API endpoints were combined into one file with duplicates removed.

Created a python Jupyter notebook to run after attaining NCBI accessions and IMG genome ids and this retrieves and adds 8 extra metadata columns to the fixed submitted accessions dataset.

This notebook performed a series of joins on IMG genome ids, NCBI BioSamples and NCBI genome assembly accessions.

The first step was joining the IMG genome ids and its associated NCBI metadata on to the starting accessions dataset to create a new table.

The next steps were to perform a series of joins onto this new table.

Joined this new table with a table of NCBI accession values, which came from 3 NCBI API calls as discussed above, on the NCBI genome assembly values to create a new table.

A column containing NCBI BioSample values was created from this join. This column contains 15 BioSample values that are different than those in the original submitted accessions table.

A new column named ‘Fixed_BioSample’ was created by merging NCBI BioSample values found with the IMG query and NCBI BioSample values found with NCBI API calls.

Using the BioSample values in the ‘Fixed_BioSample’ column created by the Jupyter notebook, the NCBI API was called again and the new metadata values found for these corrected BioSample values was used in the final processed dataset. This retrieved metadata was used over any previously attained metadata. Any metadata values not retrieved using the ‘Fixed_BioSample’ column values are filled in with the original metadata values.

Renamed columns in the table with new metadata, 'Fixed_BioSample' was renamed 'Corrected_BioSample' and this table was saved to a file named “updated_pi_supplemental_table.csv”.

3. Processed the file “updated_pi_supplemental_table.csv” with the BCO-DMO dataset processing tool named laminar

Renamed columns to conform with the BCO-DMO parameter naming convention of replacing spaces with underscores, and starting parameter names with an alphabetical character. Renamed the parameter ““# predicted genes” to “num_predicted_genes”.

Renamed the parameter ncbi_taxon_id to NCBI_organism_taxid.

Converted UTC date parameters with the format %Y-%m-%dT%H:%M:%S.%f to the ISO UTC datetime format of %Y-%m-%dT%H:%M:%SZ.

The organism taxon id is an integer, so removed the trailing decimal 0 to convert it into an integer.

Reordered the parameters so that genome metadata is towards the end of the dataset table and accessions metadata is towards the start.

Saved this updated table as the final dataset named 933610_v1_nitrospirota_and_nitrospinota_genomes.csv