225 add a page to begin documenting standards for pipelines shared externally

[arielleleon]

Modified platform_support.md to edit "Platform requirements", add a "Data flow pipeline" blurb, and edit "Logging". Left everything else the same but we may want to edit some of the processing pipeline documentation to reference pipeline standards documentation.

processing_pipeline_standards is a new heading under "Policies and practices". It contains a sub-contents for:

• package management
• versioning pipelines (moved from platform_support.md)
• ai/ND processing pipelines
• processing and pipeline testing
• pipeline parameterization
• branch management

I left a couple headings within "Policies and practices" in case they do not deserve a full page

Closes #225

---

📚 Documentation preview 📚: https://scicomp-docs--238.org.readthedocs.build/en/238/

[alejoe91]

Thanks @arielleleon

Added some comments as talking points

[alejoe91]

Is this about how the pipeline GH repo should be organized? Naming conventions etc.?

I think it's a good place to start, but I would rename to: Repository organization and conventions

[alejoe91]

Suggested change

	Pipelines are a standardized series of processing steps that take raw data from a single modality and typically produce an NWB file as output. Pipelines are organized in a Nextflow pipeline with individual Code Ocean capsules performing internal steps. Platforms should use established pipelines for processing.
	Pipelines are a standardized series of processing steps that take raw data from a single modality and typically produce an NWB file as output. Pipelines are organized in a Nextflow pipeline with individual processes implemented as separate GitHub repositories performing internal steps. Platforms should use established pipelines for processing.

[alejoe91]

Suggested change

	version_pipelines.md
	packaging_and_versioning.md

I would merge the concept of version and branch management: I started implementing something for the ephys pipeline here, open for discussion: AllenNeuralDynamics/aind-ephys-pipeline#138

[alejoe91]

I understand now that pipeline_package_management is probably referring to container images etc. I think all of these concepts can be explained here.

[alejoe91]

Suggested change

	aind_processing_pipelines.md
	pipeline_and_process_testing.md
	pipeline_parameters.md
	pipeline_branch_management.md
	pipeline_parameters.md
	pipeline_testing.md
	aind_processing_pipelines.md

Just reordering in my personal favorite order of importance

[alejoe91]

Suggested change

	## Process standards (processes are wrappers around processing libraries)
	## Process standards

IMO the standard is that each process should correspond to a specific processing step, implemented as an individual GH repo with a main entry point (./run + run_capsulepy/main.py), potentially accepting JSON strings as parameters.

The pipeline has a clear map to repos and commits/versions for each step (as in https://github.com/AllenNeuralDynamics/aind-ephys-pipeline/blob/dev/pipeline/capsule_versions.env)

[alejoe91]

Suggested change

	## Nextflow version (UI will no longer be compatible)
	## Nextflow version

Here let's pin some concepts:

1. Use DSL2 and worflows
2. Accept a --params_file argument
3. Other shared design choices (e.g., have a process for validate_parameters/save_provenance...etc, these can be reaused across pipelines)