Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
770 changes: 378 additions & 392 deletions Manifest.toml

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -53,4 +53,4 @@ StatsPlots = "f3b207a7-027a-5e70-b257-86293d7955fd"
Turing = "fce5fe82-541a-59a6-adf8-730c64b5f9a0"

[compat]
Turing = "0.45"
Turing = "0.46"
2 changes: 1 addition & 1 deletion _quarto.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ website:
href: https://turinglang.org/team/
right:
# Current version
- text: "v0.45"
- text: "v0.46"
menu:
- text: Changelog
href: https://turinglang.org/docs/changelog.html
Expand Down
6 changes: 3 additions & 3 deletions core-functionality/index.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -582,10 +582,10 @@ Some of Turing.jl's default settings can be changed for better usage.

#### AD Backend

Turing is thoroughly tested with three automatic differentiation (AD) backend packages.
Turing is thoroughly tested with four automatic differentiation (AD) backend packages.
The default AD backend is [ForwardDiff](https://github.com/JuliaDiff/ForwardDiff.jl), which uses forward-mode AD.
Two reverse-mode AD backends are also supported, namely [Mooncake](https://github.com/compintell/Mooncake.jl) and [ReverseDiff](https://github.com/JuliaDiff/ReverseDiff.jl).
`Mooncake` and `ReverseDiff` also require the user to explicitly load them using `import Mooncake` or `import ReverseDiff` next to `using Turing`.
Three other backends are also supported: [Mooncake](https://github.com/chalk-lab/Mooncake.jl), [ReverseDiff](https://github.com/JuliaDiff/ReverseDiff.jl), and [Enzyme](https://github.com/EnzymeAD/Enzyme.jl).
These require the user to explicitly load them, for example `import Mooncake` next to `using Turing`.

For more information on Turing's automatic differentiation backend, please see the [Automatic Differentiation]({{<meta usage-automatic-differentiation>}}) article as well as the [ADTests website](https://turinglang.org/ADTests/), where a number of AD backends (not just those above) are tested against Turing.jl.

Expand Down
17 changes: 17 additions & 0 deletions tutorials/variational-inference/index.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,23 @@ To see the difference, we can compare the ELBO of the two:
We can see that `ELBO_q_avg` is slightly more optimal.
:::

:::{.callout-note}
## Automatic differentiation for VI

VI uses automatic differentiation in two places: the `adtype` keyword argument to `vi` selects the backend used to differentiate the model's log density (`AutoForwardDiff()` by default), while the first argument to `KLMinRepGradDescent`, `KLMinRepGradProxDescent`, and `KLMinScoreGradDescent` selects the backend used to differentiate the variational objective.
(The other `KLMin...` algorithms take no AD backend argument; they rely only on `vi`'s `adtype`.)
For larger models a reverse-mode backend such as `AutoMooncake()` is often faster than the `AutoForwardDiff()` used above.
Note that it should be supplied in both places: changing only the algorithm's backend leaves the model's own gradient — usually the expensive part — on `vi`'s default.
As with sampling, you must load the backend package yourself, for example `import Mooncake`.
See the [Automatic Differentiation]({{<meta usage-automatic-differentiation>}}) page for more on backends.

Note that `AutoReverseDiff(; compile=true)` is not supported for VI and raises an error, because a compiled tape can silently give incorrect gradients when reused across optimisation steps.
Use `AutoReverseDiff(; compile=false)` instead.

Since AdvancedVI 0.7, `KLMinScoreGradDescent` optimises in unconstrained (linked) space, consistent with the other `KLMin...` algorithms.
For a model with constrained parameters its results may differ slightly from earlier releases.
:::


`result.info` also contains information generated during optimisation that could be useful for diagnostics.
For the default setting, which is `RepGradELBO`, it contains the ELBO estimated at each step, which can be plotted as follows:
Expand Down
10 changes: 6 additions & 4 deletions usage/automatic-differentiation/index.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The gradient of the log probability density is used by various algorithms in Tur

The Julia ecosystem has a number of AD libraries.
You can switch between these using the unified [ADTypes.jl](https://github.com/SciML/ADTypes.jl/) interface, which for a given AD backend, provides types such as `AutoBackend` (see [the documentation](https://docs.sciml.ai/ADTypes/stable/) for more details).
For example, to use the [Mooncake.jl](https://github.com/compintell/Mooncake.jl) package for AD, you can run the following:
For example, to use the [Mooncake.jl](https://github.com/chalk-lab/Mooncake.jl) package for AD, you can run the following:

```{julia}
# Turing re-exports AutoEnzyme, AutoForwardDiff, AutoReverseDiff, and AutoMooncake.
Expand Down Expand Up @@ -49,7 +49,9 @@ There are two aspects to choosing an AD backend: firstly, what backends are avai

### Usable AD Backends

Turing.jl uses the functionality in [DifferentiationInterface.jl](https://github.com/JuliaDiff/DifferentiationInterface.jl) ('DI') to interface with AD libraries in a unified way.
Turing.jl computes gradients for sampling and variational inference through [AbstractPPL.jl](https://github.com/TuringLang/AbstractPPL.jl)'s [`prepare` / `value_and_gradient!!` interface](https://turinglang.org/AbstractPPL.jl/stable/evaluators/).
ForwardDiff and Mooncake have native support in this interface; any other backend is routed through [DifferentiationInterface.jl](https://github.com/JuliaDiff/DifferentiationInterface.jl) ('DI').
(Mode estimation is the exception: `maximum_likelihood` and `maximum_a_posteriori` delegate differentiation to [Optimization.jl](https://github.com/SciML/Optimization.jl).)
In principle, any AD library that DI provides an interface for can be used with Turing; you should consult the [DI documentation](https://juliadiff.org/DifferentiationInterface.jl/DifferentiationInterface/stable/) for an up-to-date list of compatible AD libraries.

Note, however, that not all AD libraries in there are thoroughly tested on Turing models.
Expand All @@ -59,8 +61,8 @@ Turing is most extensively tested with **ForwardDiff.jl** (the default), **Enzym
::: {.callout-note}
## Gradient preparation

Users of DifferentiationInterface.jl will have seen that it provides functions such as `prepare_gradient`, which allow you to perform a one-time setup to make subsequent gradient computations faster.
Turing will automatically perform gradient preparation for you when calling functions such as `sample` or `optimize`, so you do not need to worry about this step.
AbstractPPL's interface provides a `prepare` step that performs a one-time setup to make subsequent gradient computations faster (this wraps the equivalent preparation step of the underlying AD backend).
Turing will automatically perform this preparation for you when calling functions such as `sample` or `vi`, so you do not need to worry about this step.
:::

### ADTests
Expand Down
3 changes: 3 additions & 0 deletions usage/performance-tips/index.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,9 @@ However, it is important to note that if your model contains any branching code,
If you use this option for the (considerable) speedup it can provide, make sure to check your code for branching and ensure that it does not affect the gradients.
It is also a good idea to verify your gradients with another backend.

Note that `compile=true` is not supported for variational inference: passing `AutoReverseDiff(; compile=true)` to `vi` raises an error, because a compiled tape can silently give incorrect gradients when reused across optimisation steps.
For VI, use `AutoReverseDiff(; compile=false)` or a different reverse-mode backend such as `AutoMooncake()`.

## Ensure that types in your model can be inferred

For efficient gradient-based inference, e.g. using HMC, NUTS or ADVI, it is important to ensure the types in your model can be inferred.
Expand Down
2 changes: 1 addition & 1 deletion usage/vectorisation/index.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ ldf_ad = LogDensityFunction(model, getlogjoint_internal, LinkAll(); adtype=AutoF
:::{.callout-note}
## Gradient preparation

When constructing a `LogDensityFunction` with an `adtype`, gradient preparation (using DifferentiationInterface.jl) will be performed automatically.
When constructing a `LogDensityFunction` with an `adtype`, gradient preparation (through AbstractPPL's `prepare` interface) will be performed automatically.
This means that subsequent gradient calculations will be much faster, but also means that depending on the backend used, the construction of the `LogDensityFunction` may take a long time.
:::

Expand Down
Loading