Hypergraph-based system execution scheduler.

[Kuba663]

In this PR, I implemented a system scheduler that automatically decides which tasks can be ran in parallel.
The Scheduler works as follows:

1. A hypergraph is created, where the systems are vertices and their dependencies (components and resources) are edges.
2. A graph is created where each vertex is connected to all of the vertices that do not share an edge in the hypergraph.
3. The resulting disjointed graph is colored.
4. Vertices of each color are added topriority queues (one for each color).
5. The priority queue array is used to build a directed acyclic graph (this is not complete yet).

TODO:

• Make the Scheduler actually build the DAG.
• Write a more efficient Hypergraph implementation, the current one uses two multimaps.
• Handle systems deending on eachother.
• Use a Fibonacci Heap in the graph coloring function as it's faster.
• Refactor the coding style, as it's not appropriate.
• Write Unit Tests for Hypergraphs and Graph Coloring.
• Integrate into Kirino, currently it's just a thing that exists and does nothing.

[Kuba663]

I'm giving up on the Fibonacci Heap for now as I don't know how to implement removal without key decreases.

[Kuba663]

Bottleneck related to systems depending on eachother, currently the scheduler handles them by simply treating them as sharing a hyperedge, however this can potentially cause problems in the coloring part, as it's not guaranteed that depending systems will have the same color. A possible solution would be collapsing dependent vertex chains into single vertices representing sequential execution of several systems. This however raises another problem related to the vertices in two different branches that are required for executing another system sharing an hyperedgeedge, which could cause potential race conditions. The solution would require creating a separate system for dealing with dependencies that treats system/system relation differently from the typical system/resource relation.
Presented below is an ilustration showcasing the problem. The vertices cannot be collapsed due to a shared hyperedge representing a shared read-write/write-write to a resource (that is: a Resource or a Component).

Yet another alternative solution to this problem would be collapsing all vertices under the same hyperedges into a single vertex, until only vertices representing systems completely independent of eachother remain.

[tttsaurus]

I'm not sure if I fully grasp your modeling. Let me illustrate what I read from your wording first.

1 Build Hypergraph

Let's say

System A accesses Position
System B accesses Position
System C accesses Velocity

so hyperedges are

H1 (Position): { A, B }
H2 (Velocity): { C }

I deliberately ignore the granularity here, like read or write or their actual component access chain position.xyz.x or whatever

So the systems sharing a hyperedge may conflict.

2 Build Another Graph To Represent The Compatibility

Based on

H1: { A, B }
H2: { C }

• For A
  • A and B shares H1, don't connect A to B
  • A and C shares no edge, connect A to C
• For B
  • B and A shares H1, don't connect B to A
  • B and C shares no edge, connect B to C
• For C
  • C and A shares no edge, connect C to A
  • C and B shares no edge, connect C to B

As a result, we got

graph TD
    A <--> C
    B <--> C

Loading

No need to be directed. I simply can't remove the arrow.

And based on what you said

The resulting disjointed graph is colored.

I think the resulting graph isn't necessarily disjoint and I just created a counterexample.

connected to all of the vertices that do not share an edge in the hypergraph

Therefore, the resulting graph is a compatibility graph instead of a conflict graph.
Thus, the vertices that share the same color can't run together.

3 Compatibility Graph Coloring

graph TD
    A[Red] <--> C[Blue]
    B[Red] <--> C[Blue]

Loading

4 Add To Priority Queues

Q1: A[Red], B[Red]
Q2: C[Blue]

And the priority is set by the user right? Let's say A is greater than B here.

Just a quick check, B runs after A, which is correct as both of them wants Position, and Q1 and Q2 can be parallelized.

5 Build The DAG (aka SystemExeFlowGraph)

graph TD
    S[Start] --> A
    S --> C
    A --> B
    C --> E[End]
    B --> E

Loading

6 Your Previous Comment

Bottleneck related to systems depending on eachother, currently the scheduler handles them by simply treating them as sharing a hyperedge, however this can potentially cause problems in the coloring part

I'm confused about why systems need to depend on each other, like why do we want to add dependencies like A --> B explicitly to the graph? Isn't the priority handled by the priority queues, like users manually input the priority beforehand?

Even if you want the capability to declare dependencies like depends(A, B), you can maintain a DAG and compute the corresponding priority for the users. Just like a facade layer.

7 Granularity

Btw, the resource (or component) declaration is definitely too coarse at the moment. What I want is like

ComponentA:
  field1.field2 | read
  fieldX.fieldY | write

ComponentB:
  xyz | write

And there's definitely a way that you don't have to refactor too much, like doing it in a facade way.
Specifically, you can keep this part

System A accesses Position
System B accesses Position
System C accesses Velocity

But, treat Position, Velocity like constraint identifiers instead of the actual component resource representation.
Then, you generate these identifiers from the actual component resource representation:

System A
  Component Position:
    xyz.x | read

System B
  Component Position:
    xyz.y | write

System C
  Component Velocity:
    xyz | read

This is where the actual read/read, write/write, read/write analysis happens. Obviously, A doesn't conflict B;
A,B doesn't conflict C as well.

So the generated constraint identifiers will be:

System A accesses PositionStub0
System B accesses PositionStub1
System C accesses VelocityStub0

As a result, there will be three hyperedges, causing no conflict as needed.

[tttsaurus]

I was wrong about the compat graph coloring. We should find cliques intead OR color the conflict graph.

Counterexample

Compat graph:

graph TD
    A <--> B
    B <--> C
    C <--> D

Loading

Minimum coloring:

graph TD
    A[Red] <--> B[Blue]
    B[Blue] <--> C[Red]
    C[Red] <--> D[Blue]

Loading

Red queue: [A, C]
Blue queue: [D, B]

A is incompatible with D since there is no edge between A and D, but
the resulting queues want A & D to run in parallel.

Example: Clique Covering

graph TD
    A[A] <--> B[B]
    B[B] <--> C[C]
    C[C] <--> D[D]

Loading

We want cliques to cover all the edges.
This is only one way to organize cliques in this case. (there is no 3-node clique)

Clique family:

Clique 1: A B
Clique 2: B C
Clique 3: C D

We need to construct the clique traversal:

C_1 -> C_2 -> C_3

For every adjacent clique in the traversal, we want to compute some properties:

C_1 -> C_2

Shared: C_1 ∩ C_2 = {B}
Leaving: C_1 \ C_2 = {A}
New: C_2 \ C_1 = {C}

C_2 -> C_3

Shared: C_2 ∩ C_3 = {C}
Leaving: C_2 \ C_3 = {B}
New: C_3 \ C_2 = {D}

A conservative transition rule can be defined as:

• Run every system in a clique in parallel
• Wait Leaving ones to finish before entering the next clique
• Keep Shared ones running when it enters the next clique
• Run New ones for the next clique

The queues can therefore be derived

Run {A, B} for C_1
  append A to Q1
  append B to Q2

When {A} finishes, run {C} for C_2
  append {C} to Q1 (C runs after A)

When {B} finishes, run {D} for C_3
  append {D} to Q2 (D runs after B)

Notice: Leaving must finish before the traversal may admit New vertices of the next clique.
Otherwise, it might result in A and D running together.

Q1: A C
Q2: B D

The derived queues are totally valid as AB are compatible and CB are compatible. (reading the columns)

Algorithms

Cliques TO Clique Traversal (WIP)

For certain compat graphs like A-B-C-D, the traversal construction is easy and intuitive.

However, consider

A --- B
|      |
C --- D

The cliques will be C_1={A, B}, C_2={B, D}, C_3={D, C}, C_4={C, A}

The clique traversal construction is harder/impossible since it forms a cyclic structure.

A linear traversal covering every clique doesn't exist under certain situations, for example the 4-cycle above.

Clique Traversal TO Queues

The Shared/Leaving/New model is enough to generalize the process.

C_i -> C_j

Shared: C_i ∩ C_j = {...}
Leaving: C_i \ C_j = {...}
New: C_j \ C_i = {...}

A conservative transition rule can be defined as:

• Run every system in a clique in parallel
• Wait Leaving ones to finish before entering the next clique
• Keep Shared ones running when it enters the next clique
• Run New ones for the next clique
• Leaving ones must finish before the traversal may admit New vertices of the next clique

Clique Family Construction

General Idea:

• Given the compat graph G=(V, E), I want to use multiple cliques to cover G. Vertex overlaps must be minimized.

Let's formulate how to minimize the vertex overlaps.

Referene: https://www.researchgate.net/publication/273388215_Sigma_clique_covering_of_graphs

Clique family: 𝓒 = {C_1, ... , C_m}

𝓒 must satisfy that E ⊆ union over C_i edges

Sigma Clique Cover: scc(G) = min over 𝓒 (summation over every C∈𝓒, |C|)

Goal: find a clique family 𝓒 that attains scc(G). The value of scc(G) doesn't matter but the corresponding 𝓒 matters

Turns out whether the individual cliques are strictly the largest doesn't matter, but scc(G) implicitly tries to create large cliques already; nevertheless, our scheduler doesn't technically require the largest cliques. scc(G) is good enough for this situation.

We are able to derive some corollaries as well.

If there is no vertex overlap & G has no isolated vertices, scc(G)=|V|

And,

Define clique multiplicity of a vertex: val_𝓒(v) = |{C_i : v ∈ C_i }|

(summation over every C∈𝓒, |C|) = (summation over every v∈V, val_𝓒(v))

If there is no isolated vertex, then

(summation over every C∈𝓒, |C|) - |V|= (summation over every v∈V, val_𝓒(v) - 1)