import_genome

Import genome-associated files into Arx folder structure, automatically generate metadata files.

Overview

The import_genome command is the primary tool for importing genome data into the Arx folder structure. It handles file organization, metadata generation, and can optionally rename locus tags during import. This is an improved version with better configuration options and file handling.

Basic Usage

If the annotation was performed using the proper organism name, genome identifier and taxonomic information (recommended), the import is straightforward because no files need to be renamed.

export FOLDER_STRUCTURE=/path/to/folder_structure
import_genome --import_dir=/prokka/out/dir

Optional sanity check parameters:

import_genome --import_dir=/prokka/out/dir --organism STRAIN --genome STRAIN.1

Required Files

These files need to be in import_dir:

• .fna: Assembly (FASTA)
• .gbk: GenBank file
• .gff: General feature format file

Optional Files

• .faa: Protein sequences (FASTA). If non-existent, it will automatically be generated from the .gbk file
• .ffn: Nucleotides file (FASTA). If non-existent, it will automatically be generated from the .gbk file
• .sqn: Required for submission to GenBank, not really used by Arx
• .emapper.annotations: Eggnog annotation file
• .XX: Custom annotation file (e.g. EC, .GO, etc.; any files with a suffix of two upper case letters are detected as custom annotations)
• _busco.txt: BUSCO output file, content will be added to genome.json
• genome.json: Content will be added to final genome.json, may be as simple as {"restricted": true}
• organism.json: Content will be added to final organism.json, may be as simple as {"assembly_tool": "SPAdes"}
• submol.yaml: PGAP submission file for additional metadata extraction

File Renaming During Import

Should the locus tags not start with the genome identifier, the files need to be renamed accordingly. The import_genome command can do this automatically using the --rename flag.

export FOLDER_STRUCTURE=/path/to/folder_structure
import_genome --import_dir=/prokka/out/dir --organism STRAIN --genome STRAIN.1 --rename

Renaming Limitations

The renaming is provided as-is, and was only tested on files produced by certain versions of prokka and PGAP. If there is an error, you must rename the files manually (with or without the help of the renaming scripts) and then import them as described in the basic usage section.

Example Result

After import, your folder structure will look like this:

folder_structure
└── organisms
    └── STRAIN
        ├── organism.json
        └── genomes
             └── STRAIN.1
                 ├── genome.json
                 ├── STRAIN.1.faa
                 ├── STRAIN.1.ffn
                 ├── STRAIN.1.fna
                 ├── STRAIN.1.gbk
                 ├── STRAIN.1.gff
                 ├── STRAIN.1.sqn
                 └── rest
                     ├── PROKKA_08112021.err
                     ├── PROKKA_08112021.fsa
                     ├── PROKKA_08112021.log
                     ├── PROKKA_08112021.tbl
                     ├── PROKKA_08112021.tsv
                     ├── PROKKA_08112021.txt
                     └── short_summary.specific.lactobacillales_odb10.FAM3228-i1-1_busco.txt

Configuration

Import Settings

It is possible to change where files end up in the folder structure. The behaviour is determined by a config file in JSON format that can be specified with the --import_settings parameter or the ARX_IMPORT_SETTINGS environment variable.

export ARX_IMPORT_SETTINGS=/path/to/import_config.json

Default Settings

{
    "organism_template": {},
    "genome_template": {},
    "import_actions": [
        {"type": "copy", "from": "*", "to": "{original_path}", "expected": true}
    ],
    "file_finder": {
        "fna": {"glob": "*.fna", "expected": 1},
        "gbk": {"glob": "*.gbk", "expected": 1},
        "gff": {"glob": "*.gff", "expected": 1},
        "faa": {"glob": "*.faa", "expected": false},
        "sqn": {"glob": "*.sqn", "expected": false},
        "ffn": {"glob": "*.ffn", "expected": false},
        "eggnog": {"glob": "*.emapper.annotations", "expected": false},
        "yaml": {"glob": "*.yaml", "expected": false},
        "busco": {"glob": "*_busco.txt", "expected": false},
        "custom_annotations": [
            {"glob": "*.GC", "anno_type": "GC", "expected": false},
            {"glob": "*.GP", "anno_type": "GP", "expected": false},
            {"glob": "*.EP", "anno_type": "EP", "expected": false},
            {"glob": "*.ED", "anno_type": "ED", "expected": false},
            {"glob": "*.EO", "anno_type": "EO", "expected": false},
            {"glob": "*.EC", "anno_type": "EC", "expected": false},
            {"glob": "*.KG", "anno_type": "KG", "expected": false},
            {"glob": "*.KR", "anno_type": "KR", "expected": false},
            {"glob": "*.GO", "anno_type": "GO", "expected": false},
            {"glob": "*.SL", "anno_type": "SL", "expected": false},
            {"glob": "*.OL", "anno_type": "OL", "expected": false}
        ]
    }
}

Alternative Configuration Example

For more complex file organization, you can use a custom configuration that maps specific file patterns to different locations:
Input (from Prokka):

$ tree outdir/
├── STRAIN.1.err
├── STRAIN.1.faa
├── STRAIN.1.ffn
├── STRAIN.1.fna
├── STRAIN.1.fsa
├── STRAIN.1.gbk
├── STRAIN.1.gff
├── STRAIN.1.log
├── STRAIN.1.sqn
├── STRAIN.1.tbl
├── STRAIN.1.tsv
└── STRAIN.1.txt

Command: import_genome --import_dir outdir --import_settings arx-import.config
With this custom arx-import.config file:

{
    "organism_template": {},
    "genome_template": {},
    "import_actions": [
        {"type": "copy", "from": "*.fna", "to": "1_assembly/{genome}.{suffix}", "expected": true},
        {"type": "copy", "from": "*.faa", "to": "2_cds/{genome}.{suffix}", "expected": false},
        {"type": "copy", "from": "*.gbk", "to": "2_cds/{genome}.{suffix}", "expected": true},
        {"type": "copy", "from": "*.gff", "to": "2_cds/{genome}.{suffix}", "expected": true},
        {"type": "copy", "from": "*.ffn", "to": "2_cds/{genome}.{suffix}", "expected": false},
        {"type": "copy", "from": "*.sqn", "to": "2_cds/{genome}.{suffix}", "expected": false},
        {"type": "copy", "from": "PROKKA_.*", "to": "2_cds/{original_path}", "expected": false},
        {"type": "copy", "from": "*.emapper.annotations", "to": "3_annotation/{genome}.eggnog", "expected": false},
        {"type": "copy", "from": "*.[A-Z]{2}", "to": "3_annotation/{genome}.{suffix}", "expected": false},
        {"type": "copy", "from": "*_busco.txt", "to": "3_annotation/{original_path}", "expected": false},
        {"type": "copy", "from": "genome.md", "to": "genome.md", "expected": false},
        {"type": "copy", "from": "organism.md", "to": "../../organism.md", "expected": false},
        {"type": "copy", "from": "*", "to": "rest/{original_path}", "expected": false}
    ],
    "file_finder": {
        "fna": {"glob": "1_assembly/*.fna", "expected": 1},
        "gbk": {"glob": "2_cds/*.gbk", "expected": 1},
        "gff": {"glob": "2_cds/*.gff", "expected": 1},
        "faa": {"glob": "2_cds/*.faa", "expected": false},
        "ffn": {"glob": "2_cds/*.ffn", "expected": false},
        "sqn": {"glob": "2_cds/*.sqn", "expected": false},
        "eggnog": {"glob": "3_annotation/*.emapper.annotations", "expected": false},
        "yaml": {"glob": "3_annotation/*.yaml", "expected": false},
        "busco": {"glob": "3_annotation/*_busco.txt", "expected": false},
        "custom_annotations": [
            {"glob": "3_annotation/*.GO", "anno_type": "GO", "expected": false}
        ]
    }
}

Output:

$ tree folder_structure/organisms/STRAIN/
├── organism.json
└── genomes
    └── STRAIN.1
        ├── genome.json
        ├── 1_assembly
        │   └── STRAIN.1.fna
        ├── 2_cds
        │   ├── STRAIN.1.faa
        │   ├── STRAIN.1.ffn
        │   ├── STRAIN.1.gbk
        │   ├── STRAIN.1.gff
        │   └── STRAIN.1.sqn
        └── rest
            ├── STRAIN.1.err
            ├── STRAIN.1.faa
            ├── STRAIN.1.ffn
            ├── STRAIN.1.fna
            ├── STRAIN.1.fsa
            ├── STRAIN.1.gbk
            ├── STRAIN.1.gff
            ├── STRAIN.1.log
            ├── STRAIN.1.sqn
            ├── STRAIN.1.tbl
            ├── STRAIN.1.tsv
            └── STRAIN.1.txt

Adding Custom Metadata

There are two ways to add custom metadata:

1. Per-import: Add a organism.json and/or genome.json file into import_dir (see Required Files)
2. Global: Set a global organism.json and/or genome.json file that is used as a basis for all future imports (see Configuration)

Parameters

• --import_dir: Folder with files to import (required)
• --folder_structure_dir: Path to the root of the Arx folder structure (optional, uses FOLDER_STRUCTURE environment variable)
• --organism: Name of the organism (optional, auto-detected from GenBank file)
• --genome: Identifier of the genome (optional, auto-detected from GenBank file)
• --rename: Automatically rename locus tags to match genome identifier (optional)
• --check_files: Check if locus tag prefixes match genome identifier (default: true)
• --import_settings: Path to import settings file (optional, uses ARX_IMPORT_SETTINGS environment variable)
• --pause: Wait after import actions, before file finding (optional)

Help

import_genome --help