The simulated communities were built according to the following design:

1. Databank: Biobank from which taxa were drawn to construct theoretical communities, either Silva (silva) or Utax (utax).
   
2. Number of OTUs: 20, 100, 200, 500 and 1000;
   
3. Abundance distribution: abundances of OTUs were either uniform (uniform) or sampled from a power distribution (power_law);
   
4. Dataset: Theoretical communities. Each dataset (5 for each combinaison of abundance distribution and number of OTUs) correspond to a unique ideal bacterial community specified by its own taxa set and corresponding vector of relative abundances.
   
5. Set Number: Biological replicates (10 for each dataset), i.e. communities created by sampling organisms with replacement in the theoretical communities.
   
6. Amplicon: variable region of the 16S rRNA used to produce the ampicon sequences, either the V3-V4 (V3V4) variable region or the V4-V4 (V4) variable region

This resulted in a total of 2 databanks \(\times\) 5 community sizes \(\times\) 2 abundance distribution \(\times\) 10 theoretical communities \(\times\) 10 replicates for each theoretical community \(\times\) 2 amplicons \(=\) 2000 samples (1000 per databank) (100 000 sequences per sample).

3.1.1 Divergence

The comparisons of divergence at the sample level in the scatterplots shows that on average, FROGS has comparable but better performances than MOTHUR, UPARSE and QIIME (SOP): most samples end up in the upper left corner (corresponding to the region “divergence FROGS < divergence competitor”) but no too far away from the first diagonal (grey line). It also has comparable performances to QIIME (MA) but in certain conditions, samples are not mostly contained in the upper left half of the graph, meaning that QIIME (MA) outperforms FROGS for those parameter values.

A

A more traditional representation using boxplot of the excess divergence of FROGS, with samples from all theoretical communities pooled together, confirms the results: FROGS has similar (compared to UPARSE) or lower (compared to MOTHUR) divergence for the vast majority of samples. Note that the y-range was reduced from \([-85, 31]\) to \([-15, 3]\) in order to exclude outliers (4% of outliers with excess divergence < -15 and 0.02% with excess divergence > 3) and zoom in on the boxplots. As expected, all methods perform quite similarly up to the order level and the main differences appear at the Family and Genus levels, where MOTHUR and QIIME (SOP) produces much larger divergences than competing methods. The only configuration where FROGS is consistently outperformed is complex communities (number of species > 200) with uniform abundances and sequenced on the V4 region. In that configuration, FROGS is outperformed by QIIME (MA).

B

Finally a focus on the accuracy of FROGS alone shows that divergence levels vary between 0 and 10% and as expected, is higher for fine classification (Genus) than for coarse ones (Phylum). Unsurprisingly, the V3V4 amplicon gives less distorded view of communities than the V4. Overall, FROGS recover community compositions very well expect at the genus level for complex communities (size > 200), with uniform abundances and sequenced using the V4 region.

C

3.1.2 False Positive and False Negative OTUs

We repeat the graphical exploration of the resutls with False Positive and False Negative OTUs. A first representation shows that use of the V3V4 amplicon leads to more false postive and less false negative than the V4. The graphics also highlight the gigantic number of false positive inferred by mothur (up to 20 times more than the real community size).

A focus on FROGS and UPARSE leads to similar patterns: FROGS always produces less false negatives than UPARSE but produces a bit more false positive under power law abundance distribution and a bit less under uniform abundance distribution.

The lower number of false positive under power law abundances could be due to the abundance based filters used in UPARSE.

3.2.1 Divergence

We present the results of the paired tests, either parametric (t-test, top) or non-parametric (signed rank test, bottom). Both tests show that FROGS perform as well or better as UPARSE and MOTHUR in most conditions. The only condition in which FROGS does worse than UPARSE is small community size (20). It also does better than QIIME (SOP) in all settings, and than QIIME (MA) in most settings with the notable exception of large communities (size > 200) with uniform abundance studied using the V4 region.

The real strength of FROGS lies in its ability ot give a more accurate view of large communities (size > 200) at fine scales (Species or Genus level).

3.2.2 Conditions where QIIME (MA) outperforms FROGS

A focus on the performance of QIIME (MA) reveals that QIIME (MA) indeed performs better than FROGS with uniform distribution when using the V4 amplicon region. However and although significant, the differences are small in that case: less than 2 percentage points in all cases and most marked at the Genus level where the divergences of both FROGS and QIIME (MA) are quite high.

3.2.3 False Positive and False Negative OTUs

The same paired test as in the previous section reveal that FROGS strictly outperforms MOTHUR in terms of both FP and FN taxas. It also produces less FN than UPARSE. Additionnally, it produces less FP than UPARSE for uniform distributions and more power law ones. Overall, FROGS produces less FP and less FN than either of UPARSE and MOTHUR for high community sizes (>200 for uniform distributions, >1000 for power law distributions). It also produces less FP and FN than QIIME under power law distribution and less FP but more FN than QIIME under uniform distribution.

4.1.1 Divergence

The comparisons of divergence at the sample level in the scatterplots shows that on average, FROGS has comparable but better performances than MOTHUR (MA), UPARSE (MA) and QIIME (MA): most samples end up in the upper left corner (corresponding to the region “divergence FROGS < divergence competitor”) but no too far away from the first diagonal (grey line).

A

A more traditional representation using boxplot of the excess divergence of FROGS, with samples from all theoretical communities pooled together, confirms the results: FROGS has similar (compared to UPARSE (MA) and QIIME (MA)) or lower (compared to MOTHUR (MA)) divergence for the vast majority of samples. Note that the y-range was reduced from \([-51, 41]\) to \([-15, 3]\) in order to exclude from outliers (1% of communities with low FROGS but very high MOTHUR (MA) divergence or high FROGS but low QIIME (MA) divergences) and zoom in on the boxplots. The only configuration where FROGS is consistently outperformed is complex communities (number of species > 200) with uniform abundances and sequenced on the V4 region. In that configuration, FROGS is outperformed by QIIME (MA).

B

Finally a focus on the accuracy of FROGS alone shows that divergence levels vary mostly between 0 and 15% and as expected, are higher for finer classifications (Species) than for coarse ones (Phylum). Unsurprisingly, the V3V4 amplicon gives less distorded view of communities than the V4.

C

4.1.2 False Positive and False Negative OTUs

We repeat the graphical exploration of the resutls with False Positive and False Negative OTUs. A first representation shows that use of the V3V4 amplicon leads to more false postive and less false negative than the V4. The graphics also highlight the gigantic number of false positives inferred by MOTHUR and QIIME (up to 20 times more than the real community size).

A focus on FROGS and UPARSE (MA) leads to similar patterns: FROGS always produces less false negatives than UPARSE (MA) but produces a bit more false positive under power law abundance distribution and a bit less under uniform abundance distribution.

The lower number of false positive in under power law abundances could be due to the abundance based filters used in UPARSE (MA).

4.2.1 Divergence

We present the results of the paired tests, either parametric (t-test, top) or non-parametric (signed rank test, bottom). Both tests show that FROGS perform as well or better as UPARSE (MA) and MOTHUR (MA) in most conditions. The only condition in which FROGS does worse than UPARSE (MA) is small community size (20). It also does better than QIIME (MA) in most settings, with the exception of large communities (size > 200) with uniform abundance studied using the V4 region.

The real strength of FROGS lies in its ability ot give a more accurate view of large communities (size > 200) at fine scales (Genus level).

4.2.2 False Positive and False Negative OTUs

The same paired test as in the previous section reveal that FROGS strictly outperforms MOTHUR (MA) in terms of both FP and FN taxas. It also produces less FN than UPARSE (MA) and less FP than QIIME (MA). Additionnally, it produces less FP than UPARSE (MA) for uniform distributions and more for power law ones.

Overall, FROGS produces less FP and less FN than either of UPARSE (MA) and MOTHUR (MA) for high community sizes (>200 for uniform distributions, >1000 for power law distributions) and less FP than QIIME (MA) at all sizes.