ArgoLabs Dataflow is on hold - use Numaflow

[alexec]

We've put Dataflow on hold as we are planning on some significant changes and redesign. Stay tuned.

Dataflow has been rewritten and is now available as Numaflow

[Bouwdie]

@alexec we would love to start testing dataflow, do you maybe have some insights in what the changes will be? and how fundamental 👯

[alexec]

We're currently working a new version:

• It's focussed on providing ways to do windowing operations. This is something difficult for the old version.
• It's currently supports Kafka, does not provide support for the variety of source and sinks.
• It has many similar concepts and ideas as this version (e.g. pipelines are CRDs).
• It should be an easy migration from the older version to this version.
• It should be available in about 3mo.

With that in mind, any testing you do with old dataflow, will be extremely instructive and helpful.

[cdeboeser]

Already excited to see this unfold! Are you looking for contributors?

Also another question: Is there any aspiration to allow for batch jobs, ie reading a bounded source including fan-out and -in? Use case is finite data processing jobs, along ETL lines really.

[ant31]

Hello,
Why is it not built in the opensource, isn't argo a CNCF project?

[BenCoughlan15]

Hey, is this still in the works? I'm wondering if Nats streaming will still be supported?

[dafang]

Hey, is this still in the works? When this can be used in production?

[openrory]

Hey guys, any update on the re-launch date yet?

[louisnow]

Argo 3.4 rc1 has removed dataflow pipelines from the codebase

argoproj/argo-workflows#8998

From earlier in the thread, the idea was that it should be possible to upgrade easily .

It should be an easy migration from the older version to this version.

Will that still be the case @alexec

[alexec]

@edlee2121 can I please ask you to comment on this issue to give users insight into the future.

[edlee2121]

Hi all, dataflow is being reworked as part of a new project focused on operational analytics.
Please check it out.
https://github.com/numaproj/numaflow

[vigith]

Now that we have open-sourced numaflow - https://github.com/numaproj/numaflow, shall we close this issue?