Skip to content
Merged

Dev #508

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@
__pycache__/
*.pyc
node_modules/
.DS_Store
2 changes: 1 addition & 1 deletion docs/FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The Openscope Databook is a great place to demonstrate the capabilities and some
As described in [This Jupyter blog post](https://blog.jupyter.org/mybinder-org-reducing-capacity-c93ccfc6413f), Binder no longer has the support of Google, and therefore shows reduced performance. Launches may fail or take a long time. There is no working solution to this except trying to launch again. An alternative would be to launch Databook notebooks with Google Colab.

**How can I store my work on the Databook and come back to it later?**\
You can fork the [GitHub repository](https://github.com/AllenInstitute/openscope_databook), and clone the repository to your local machine or [Dandihub](https://hub.dandiarchive.org). Running files locally and on Dandihub are explained in further detail on the [front page](https://alleninstitute.github.io/openscope_databook/intro.html). As you are working, be sure to commit your changes and push them to GitHub.
You can fork the [GitHub repository](https://github.com/AllenInstitute/openscope_databook), and clone the repository to your local machine. Running files locally is explained in further detail on the [front page](https://alleninstitute.github.io/openscope_databook/intro.html). As you are working, be sure to commit your changes and push them to GitHub.

**How do you recommend using the Databook?**\
The Databook can be used to reproduce analysis on files, as a starting point for investigating a dataset, or as an educational resource to get more familiar with NWB files or particular kinds of data. In all of these cases, the code can be modified, copied, and interactively run to gain a better understanding of the data. For educational use, the databook may be run remotely with Thebe, Binder, or Google Colab as simple demonstrations. For more advanced usage and analysis, it may behoove you to download an individual notebook and run it locally.
Expand Down
4,587 changes: 4,258 additions & 329 deletions docs/higher-order/GLM_pynapple_nemos.ipynb

Large diffs are not rendered by default.

11 changes: 3 additions & 8 deletions docs/intro.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

Reproducibility is a significant challenge in neuroscience, as analysis and visualization methods are often difficult to replicate due to a lack of accessible code, separation of code from published figures, or unavailability of code altogether. This issue may arise from the complex nature of neuroscience research, the use of diverse data formats and analysis techniques, and insufficient emphasis on open-source, collaborative practices. In addition, key neuroscience analyses are typically rewritten at the start of new scientific projects, slowing down the initiation of research efforts.

Four key components are essential for reproducible analysis: accessible data, accessible computational resources, a reproducible environment, and usage documentation. The OpenScope Databook, provided by the Allen Institute's OpenScope Project, offers a solution to these challenges by facilitating the analysis and visualization of brain data, primarily using [NWB files](https://www.nwb.org/) and the [DANDI Archive](https://dandiarchive.org/). Hosted on GitHub, the entire publication – including code, data access, text, references, and revisions from reviewers and contributors – is readily available for collaboration and version control, promoting transparency and collective knowledge growth. The OpenScope Databook addresses these components by leveraging a combination of open-source Python libraries, such as DANDI, [Binder](https://mybinder.org/), [Jupyter Book](https://jupyterbook.org/en/stable/intro.html), LaTeX references, Python scripts, Git versioning, and scientific revision through approved pull requests. The entire publication can be recreated by running the code locally, on distributed servers such as Binder, [DandiHub](https://hub.dandiarchive.org/), or on any host running Jupyter notebooks.
Four key components are essential for reproducible analysis: accessible data, accessible computational resources, a reproducible environment, and usage documentation. The OpenScope Databook, provided by the Allen Institute's OpenScope Project, offers a solution to these challenges by facilitating the analysis and visualization of brain data, primarily using [NWB files](https://www.nwb.org/) and the [DANDI Archive](https://dandiarchive.org/). Hosted on GitHub, the entire publication – including code, data access, text, references, and revisions from reviewers and contributors – is readily available for collaboration and version control, promoting transparency and collective knowledge growth. The OpenScope Databook addresses these components by leveraging a combination of open-source Python libraries, such as DANDI, [Binder](https://mybinder.org/), [Jupyter Book](https://jupyterbook.org/en/stable/intro.html), LaTeX references, Python scripts, Git versioning, and scientific revision through approved pull requests. The entire publication can be recreated by running the code locally, on distributed servers such as Binder, or on any host running Jupyter notebooks.

We cover several broadly used analyses across the community, providing a missing component for system neuroscience. Our key analyses are organized into chapters, including NWB basics such as downloading, streaming, and visualizing NWB files from data archives. We document essential analyses typically performed in all neuroscience laboratories, such as temporal alignment, alignment to sensory stimuli, and association with experimental metadata. We cover the two leading neuronal recording techniques: two-photon calcium imaging and electrophysiological recordings, and share example analyses of stimulus-averaged responses. Advanced first-order analyses include showing receptive fields, identifying optotagged units, current source density analysis, and cell matching across days.

Expand All @@ -22,14 +22,13 @@ This resource is actively maintained and can be updated by the community, provid
(Usage)=
## How Can I Use It?

There are four primary ways to run this code: **Locally**, with **Binder**, in **Google Colab**, or with **DandiHub**.
There are three primary ways to run this code: **Locally**, with **Binder**, or in **Google Colab**.

### Quick Start: Cloud Execution

Launch the Databook in the cloud with Binder or DANDIHub using the links below:
Launch the Databook in the cloud with Binder using the link below:

[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/AllenInstitute/openscope_databook/HEAD)
[![DandiHub](https://img.shields.io/badge/Open%20in-DandiHub-blue?logo=jupyter)](https://hub.dandiarchive.org/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2FAllenInstitute%2Fopenscope_databook&urlpath=lab%2Ftree%2Fopenscope_databook%2Fdocs)

Google Colab works a little differently. Colab can only open a specific `.ipynb` file, not the Databook landing page or the `docs/` directory as a whole. For that reason, Colab badges are added automatically to individual notebook pages when the project is built. Open a notebook page in the Databook and click its `Open in Colab` badge to launch that notebook in Colab.

Expand Down Expand Up @@ -103,10 +102,6 @@ Binder will automatically set up the environment with [repo2docker](https://gith
Google Colab is best suited for opening and running an individual notebook rather than the Databook as a whole. Each notebook page in the built Databook includes an automatically generated `Open in Colab` badge that links directly to that notebook's `.ipynb` file in GitHub. Clicking that badge opens the notebook in Colab, where you can run it in a hosted Python environment.


### DANDIHub
[DANDIHub](https://hub.dandiarchive.org/) is an instance of JupyterHub hosted by the [DANDI Archive](https://dandiarchive.org). To use it, navigate to `https://hub.dandiarchive.org` in your browser and sign in with your GitHub account. From there you will need to set up the environment yourself — there is no pre-built OpenScope image available. Follow the [Locally (uv)](#locally-preferred-uv) or [Locally (Pip-only fallback)](#locally-pip-only-fallback) instructions above after launching your server. Once the environment is ready, navigate to the `openscope_databook/docs` directory which contains the OpenScope notebooks.


## How Does It Work?

Reproducible analysis requires four components:
Expand Down
2 changes: 1 addition & 1 deletion docs/methods/environments.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Managing Environments on Multiple Platforms

As briefly mentioned in [Jupyter/Jupyter Book](./jupyter_book.md), Jupyter Book allows users to launch notebooks in the Databook onto other computing platforms. Namely, [Dandihub](hub.dandiarchive.org), [Binder](mybinder.org), and [Google Colab](colab.research.google.com). In the case of Binder, clicking the launch button automatically produces a docker container with the necessary environment installed by using the files [apt.txt](https://github.com/AllenInstitute/openscope_databook/blob/main/apt.txt), [setup.py](https://github.com/AllenInstitute/openscope_databook/blob/main/setup.py), and [postBuild](https://github.com/AllenInstitute/openscope_databook/blob/main/postBuild) from the root directory. In the case of Dandihub, a user can select the `OpenScope` image from the dropdown menu. As for Google Colab, the environment must be setup every runtime. If a notebook is being run locally, the environment may or may not be installed depending on the user.
As briefly mentioned in [Jupyter/Jupyter Book](./jupyter_book.md), Jupyter Book allows users to launch notebooks in the Databook onto other computing platforms. Namely, [Binder](mybinder.org) and [Google Colab](colab.research.google.com). In the case of Binder, clicking the launch button automatically produces a docker container with the necessary environment installed by using the files [apt.txt](https://github.com/AllenInstitute/openscope_databook/blob/main/apt.txt), [setup.py](https://github.com/AllenInstitute/openscope_databook/blob/main/setup.py), and [postBuild](https://github.com/AllenInstitute/openscope_databook/blob/main/postBuild) from the root directory. As for Google Colab, the environment must be setup every runtime. If a notebook is being run locally, the environment may or may not be installed depending on the user.

The measure we employed to solve this is to have an *"Environment Setup"* cell at the top of every notebook. This cell checks to see if the `databook_utils` package exists. databook_utils consists of merely a few small Python modules which handle DANDI operations that are used in nearly every Notebook in the databook. It can be presumed that the presence of databook_utils is a reliable indicator of whether or not the proper environment of the Databook is installed in the local environment of a notebook. Therefore, if importing databook_utils succeeds, the notebook continues. If it fails, the databook is cloned from GitHub and the necessary environment is installed from within the Databook's root directory using the command `pip install -e .`. This installs all the necessary pip dependencies as well as the databook_utils package which is stored within the Databook at [/databook_utils](https://github.com/AllenInstitute/openscope_databook/tree/main/databook_utils). However, the Jupyter kernel does not update its environment with the local environment until it has been restarted. In the case that the environment was installed by the Environment Setup, the Jupyter kernel must be restarted by the user and run again.

Expand Down
2 changes: 1 addition & 1 deletion docs/methods/jupyter_book.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ The Databook primarily functions as a repository comprised of [Jupyter notebooks

In its deployed form, the Databook is a website powered by [Jupyter Book](jupyterbook.org). Jupyter Book handles the web-based functionality of the Databook and manages a neat and simple UI. All Jupyter book files exist within the [/docs](https://github.com/AllenInstitute/openscope_databook/tree/main/docs) directory of the Databook. Jupyter Book is configurable with a set of yaml files, namely [_config.yml](https://github.com/AllenInstitute/openscope_databook/blob/main/docs/_config.yml) and [_toc.yml](https://github.com/AllenInstitute/openscope_databook/blob/main/docs/_toc.yml). To build the repository into the Jupyter Book, the command `jupyter-book build ./docs` is used within the root directory. The build generates a `_build` directory which contains all the HTML and Javascript files for a functional website, which is used automatically when the Databook is deployed.

Within the Table of Contents, we specified the Introductory page, various chapters of the Databook, and their respective notebook/markdown files. Within the config file, the project Title, logo, and links to the GitHub repository and GitHub issues were specified. Also through the config we specified the available launch buttons and their links as [Dandihub](hub.dandiarchive.org) (Jupyterhub), [Binder](mybinder.org), and [Google Colab](colab.research.google.com). We also included the option to allow for [Thebe](https://jupyterbook.org/en/stable/interactive/thebe.html) functionality.
Within the Table of Contents, we specified the Introductory page, various chapters of the Databook, and their respective notebook/markdown files. Within the config file, the project Title, logo, and links to the GitHub repository and GitHub issues were specified. Also through the config we specified the available launch buttons and their links as [Binder](mybinder.org) and [Google Colab](colab.research.google.com). We also included the option to allow for [Thebe](https://jupyterbook.org/en/stable/interactive/thebe.html) functionality.

Additionally, Jupyter Book has integrated citation and bibliography functionality, such that when a [references.bib](https://github.com/AllenInstitute/openscope_databook/blob/main/docs/references.bib) file is included and specified in the config, and citations appear in the markdown of the Databook's pages in the form `{cite}citation_name_here`, a bibliography page is generated which we put in the final chapter of the Table of Contents. As new works are cited in notebooks, they can be manually added to the references.bib file after which they will show up automatically in the bibliography when the Jupyter Book is built.
Loading
Loading