Diffusion calculations

[stephenson-s]

Hypothesis 4, Test 3: Calculating diffusion of Black-majority clusters in Philly, Atlanta, Chicago, Dallas/Houston, Miami, LA, NY, DC, to see if it is happening in these large cities over time.

Dedicated branch, see readme for more details.

Steps:

• Define the areas of search (not a trivial q, so for now we work with a manually created list of tracts and plan to work of figuring out a suitable definition)
• Calculate each decade's centroid
• Calculate mass and spread
• Visualize

[stephenson-s]

Other approaches to diffusion to test:

1. Center of mass spread calculation.
2. Looking at the radius of an n-ball encapsulating 90% of the black population within the cluster. Comparing n across the years. If it goes up, diffusion is occurring.
3. Calculate the extent for each year separately. Note the total number of nodes in the cluster. If the number of nodes goes up, diffusion is occurring.

More suggestions:

1. Looking at the extent (tracts that are adjacent to cluster but outside of it), compare the extent's share of Black population in time. As the Black share on the extent grows from very low to mixed, we get higher share tracts next to higher share tracts (inside the cluster). So Moran goes up. But as the extent becomes more mixed Capy sees more X-Y connections and goes down.
2. Looking at the extent, compare the jump in Black shares between the extent tracts and their adjacent tracts inside the cluster. This will indicate if the border is getting smoother in time. Adjacent low-share outer nodes and high-share cluster nodes produce negative Moran and Capy sees few X-Y connections so it's high. When outer and inner tracts become more similar (smooth), more similar edges are connected, so Moran goes up, but Capy sees more X-Y connections and goes down.

[stephenson-s]

On choosing clusters as illustrations:

We're looking for a cluster(s) that would illustrate the Moran-Capy dynamic well: where Moran goes up, Capy goes down. A few cases we looked at are Chicago's South Side and Austin, Philly's Germantown and Chester. These cities were chosen because they generally show what we want:

However, once we zoom in on particular clusters, the picture changes:

• Chicago's South Side (as a cluster) is not exhibiting the Moran's dynamic that we're after. Its Moran goes down each year. After some slicing and testing how that changes, I think the most important factor that brings the cluster in accordance with the city is if we include the outer rim of the cluster and how many steps from the rim we include (buffer size, so to say). An example of Moran's behavior under different included buffers:

So you can see how Moran changes from mostly falling in time to fluctuating in time to going up, as the buffer around the cluster grows. So the behavior we're after comes not quite from the cluster itself but from either or both of: (1) including the rim (the jump from low-Black to high-Black tracts), (2) including the low-Black tracts more generally.

• The picture is somewhat similar for Austin: as more buffer nodes are added, its Moran stops falling in 1980-1990 and starts growing, like we see in the whole city of Chicago. Its 2000 is a weird year, because Moran always falls at least somewhat, and drastically when buffer=0. Apparently that happens because the share of Black population that year is 97%, and this affects other buffer values too.

• Philadelphia's Chester: Moran starts growing like on the lineplot once we add a 2-edge buffer, so that's pretty quick:

• Philadelphia's Germantown: exhibits the same behavior by 8-10 buffer steps:

Take-aways / proposed next steps:

(1) Clusters as they are defined now are not a great way to illustrate Moran's behavior, try another definition. Pick a larger cluster-area or maybe even a whole city (but then we'll need to think about how to get a diffusion score for the whole city, so maybe that's not great)
(2) Try the same diffusion exercise but including buffers in the cluster definition. 5-8 steps?

[stephenson-s]

Why Focusing on Cluster's Alone Doesn't Work

1. It Excludes the Contrast Between Cluster And Non-Cluster

The diffusion hypothesis comes from experiments of idealized diffusion:

Here we see that when a high rho cluster is surrounded by a large of low rho areas, diffusion increases Moran, however when the cluster is most of the region and diffusion happens, Moran precipitously drops. This can explain how the trend in Moran dramatically changes based on buffer size. For a small buffer Moran sees something like steps 6-9 in the simulation and accordingly drops. For a large buffer, it sees something like steps 1-3 and rises. For a medium buffer Moran sees something more in the middle of the process and accordingly you see little overall change but slow rise and then fall. In intuitive terms the pattern moran sees is not a cluster on its own but the contrast between a high rho cluster and its low rho neighborhood.

2. Abnormally High Rho

Looking above we see that for Austin Moran's I goes from .22 in 1990 to .06 in 2000 to .28 in 2010 all while the region looks pretty much the same. This is a problem with measuring Moran's I for extremely high values of rho.

This seems to flag another problem, focusing on just the cluster enormously increases rho, making Moran behave oddly. In this case The average node in Austin in 2000 has a rho of .93 meaning that even quite high rho (e.g. 85% black nodes) will read as checker boarding and significantly lower moran if they are surrounded by higher rho areas.. While a look at just population counts does not show any major differences between 1990, 2000 and 2010 (although a hole created by a 0 pop node was filled), a visualization of which nodes were above and below rho shows that in 2000 the below average areas were rarely clustered together in 2000. Below are the Austin graphs for the years 1990, 2000, and 2010 respectively. Blue nodes reflect below average rhos, red nodes reflect above average nodes.

[stephenson-s]

For a better illustration of the above point see the following image where we compare the same process of a 4x4 cluster of group x diffusing in a lattice as a subset of a 6x6, 8x8, or 15x15 lattice. Both dissimilarity and half edge both decrease monotonically whereas Moran ranges from monotonic increase in the case of the 15x15 lattice, slight increase followed by slight decrease in the 8x8 and a sharp almost monotonic decrease in the 6x6 case.

This seems to hold empirically for moran as well (but not for HalfEdge). Below are plots of Moran, HalfEdge and Dissimilarity over time for a boundary size of 0 through 10. Moran and Half-Edge are equally erratic with both having a U shape for low boundaries and slightly increasing. For larger values in the case of Moran or slightly decreasing in slightly decreasing and Moran slightly increasing for most boundaries. In all cases, the trend seems to converge to a certain shape as boundaries get bigger, suggesting we should expect similar results for a buffer between 4 and 10 nodes large.

[maria-milosh]

A few things to note from the maps:

1. Spread values always fall in 2010. Maybe some major revision of tracts took place in Census that year, and they merged a lot of tracts?
2. As buffer size grows, moran values stop falling in time and start growing across all clusters. That's something we saw above too
3. Dissimilarity and capy dynamic looks generally similar across all clusters, buffers, and years.
4. Picking a buffer to show in the paper: even rather low values of 2-3 look good - this is where Moran already starts showing the upward trend, while Capy and D fall.

[maria-milosh]

What can we say about spread and Moran values as the buffer size changes?

1. Spread values don't reverse the dynamic. It can become more pronounced but it doesn't change signs.
2. In contrast, Moran can jump around a lot as the buffer size grows.
3. For Chicago clusters, buffer=3 is around where the two clusters start appearing within the same buffer. I suppose that could explain the reversal of trends in the South Side, but Austin appears to be way less affected.
4. I'd love to make a more general statement about spread and Moran movements but they seem to be pretty chaotic.

[stephenson-s]

Some suggestions for measuring diffusion.

1. Instead of counting the number of nodes in the cluster we could calculate InnerEdges-BorderEdges. Where InnerEdges is the number of edges between cluster nodes and BorderEdges is the number of edges between cluster nodes and non cluster nodes

If its using Euclidean instead of graph distance is not too problematic we could:
2. Calculate the spread using euclidean distance
3. For time T and T+1 assign tract population to each tract's centroid as mass. Calculate the Wasserstein optimal transport path between the T and T+1 centroids. Diffusion is happening if transport tends to be from nodes with high rho or high mass to those with lower rho/mass (this needs to be operationalized)