Test runtime is something certain people really care about, and we should probably support it cleanly.
It seems sensible to extend Test Artifacts with a new field, duration, so it can be reported on a per-result basis (eg. even for subtests), and Aggregators that don't care can simply ignore the field.
An Executor-created fallback result could include it too, measured simply between the test being started and its exit, in the case of FMFExecutor, it could be just $original_max_duration - $now, to include saving/restoring metadata-imposed duration limit.
It should probably always be in seconds, not as some ISO timestamp or h/m/s string. Probably in full seconds (integer), as Executor execution, even if running just local commands, is likely not precise enough for sub-second test durations.
Test runtime is something certain people really care about, and we should probably support it cleanly.
It seems sensible to extend Test Artifacts with a new field,
duration, so it can be reported on a per-result basis (eg. even for subtests), and Aggregators that don't care can simply ignore the field.An Executor-created fallback result could include it too, measured simply between the test being started and its exit, in the case of FMFExecutor, it could be just
$original_max_duration - $now, to include saving/restoring metadata-imposed duration limit.It should probably always be in seconds, not as some ISO timestamp or h/m/s string. Probably in full seconds (integer), as Executor execution, even if running just local commands, is likely not precise enough for sub-second test durations.