support streamed ascii-save

[westram]

Motivation:

• reduce memory usage when saving database from silva pipeline

Implement interface object ArbDBwriter which allows to modify database during save providing the following methods:

• startSaveAs / finishSave
• saveFromTill(GBDATA*start,GBDATA*end)

Reimplement

GB_ERROR GB_MAIN_TYPE::save_as(const char *as_path, const char *savetype) {
    ArbDBwriter writer(this);
    GB_ERROR error = write.startSaveAs(as_path,savetype);
    if (!error) error = writer.saveFromTill(root_container,root_container);
    if (!error) error = writer.finishSave();
    return error;
}

Intended use:

GBDATA *storeNextSpecies(GBDATA *gb_species_container, GB_ERROR *error) {
    assert(!*error);
    GB_transaction ta(gb_species_container);
    GBDATA *gb_species = ...; 
    return gb_species;
}

    // ...
    ArbDBwriter writer(gb_main);
    GB_ERROR error = write.startSaveAs("xxx.arb", "a");
    GBDATA *gb_species_data = ...; 
    if (!error) error = writer.saveFromTill(gb_main,gb_species_data);
    if (!error) {
        for (GBDATA *gb_species = storeNextSpecies(&error); 
             gb_species && !error;
             gb_species = storeNextSpecies(&error)) 
        {
             error = writer.saveFromTill(gb_species,gb_species);
             if (error) break;
             {
                 GB_transaction ta(gb_main);
                 error = GB_delete(gb_species);
             }
             if (error) break;
        }
    }
    if (!error) error = writer.saveFromTill(gb_species_data,gb_main);
    if (!error) error = writer.finishSave();  

[westram]

promising ticket number

[westram]

by [14577]

@Chris:

• Interface for streamed ascii save is ​here.
• Example test code is ​here.
  It basically does what we've discussed:
  • save until species container
  • foreach (of 2) species
    • create species
    • stream-save it
    • delete species
  • save rest of database
• New flags to request compressed save are available and documented ​here (#665).

[westram]

mark changes that got fixed after arb 6.0.x

[westram]

Milestone renamed