Neural Network Sparsity Experiments

This repository contains a series of experiments on dense and sparse neural networks using PyTorch. We explore fully connected networks (DenseNet/SparseNet) on the Breast Cancer dataset and convolutional networks (DenseCNN/SparseCNN) on MNIST, investigating the effects of sparsity, pruning techniques, and accuracy trade-offs.

---

Table of Contents

1. Experiment 1: Fully Connected Networks on Breast Cancer
2. Experiment 2: CNNs on MNIST
3. Sparsity Comparison
4. Pruning Techniques Comparison
5. Visual Analysis
6. Key Observations and Takeaways
7. Conclusion

---

Experiment 1: Fully Connected Networks on Breast Cancer

Dataset

• Breast Cancer Wisconsin (Diagnostic) dataset
• 569 samples, 30 numeric features
• Binary classification: malignant (1) or benign (0)
• Added Gaussian noise to features to simulate a more challenging environment
• Standardized features

Models

1. DenseNet (Fully Connected, Dense)
   • Architecture: Input -> 128 -> ReLU -> 64 -> ReLU -> 1 -> Sigmoid
   • Trained for 20 epochs with Adam optimizer and BCELoss
2. SparseNet (Pruned Fully Connected)
   • Same architecture as DenseNet
   • Pruning: L1 unstructured pruning at 70%
   • Pruning applied before training

Results

Model	Total Params	Non-zero	Accuracy	Precision	Recall	F1 Score
DenseNet	235,146	235,146	0.9781	0.9781	0.9778	0.9779
SparseNet	235,146	117,770	0.9795	0.9798	0.9791	0.9794

Observations:

• SparseNet achieved slightly higher accuracy with 50% of weights pruned.
• The model retains predictive performance despite high sparsity, showing overparameterization in DenseNet.
• Confusion matrices confirmed balanced performance across classes.

---

Experiment 2: CNNs on MNIST

Dataset

• MNIST handwritten digits
• 28x28 grayscale images
• 10 classes (0–9)
• Standard transforms: normalization (0.1307, 0.3081)

Models

1. DenseCNN
   • Architecture:

     Conv2d(1,32,3)->ReLU->MaxPool2d(2)
     Conv2d(32,64,3)->ReLU->MaxPool2d(2)
     Flatten->Linear(64*7*7,128)->ReLU->Linear(128,10)
     

2. SparseCNN
   • Same as DenseCNN
   • Various sparsity levels (30%, 50%, 70%, 90%)
   • Three pruning techniques:
     • Unstructured L1 pruning: zero weights with smallest absolute values
     • Structured pruning: prune entire channels/neurons
     • Random pruning: randomly zero weights

Dense vs Sparse (Initial 50% pruning)

Model	Total Params	Non-zero	Accuracy	Precision	Recall	F1 Score
DenseCNN	421,642	421,642	0.9918	0.9918	0.9917	0.9917
SparseCNN	421,642	210,938	0.9892	0.9890	0.9893	0.9891

Observation:

• SparseCNN maintained very high accuracy (~0.989) despite 50% of weights pruned.
• DenseCNN and SparseCNN training curves (loss/accuracy) showed fast convergence.

---

Sparsity Comparison (DenseCNN vs SparseCNN at 30–90%)

Model	Sparsity	Non-zero	Accuracy	Precision	Recall	F1 Score
DenseCNN	0%	421,642	0.9890	0.9889	0.9889	0.9889
SparseCNN	30%	415,051	0.9923	0.9923	0.9922	0.9922
SparseCNN	50%	410,001	0.9900	0.9899	0.9900	0.9899
SparseCNN	70%	408,937	0.9909	0.9908	0.9908	0.9908
SparseCNN	90%	390,089	0.9891	0.9892	0.9890	0.9890

Insights:

• Accuracy is stable across 30–70% sparsity.
• Non-zero parameters drop moderately for unstructured pruning; true memory savings are limited.
• MNIST CNN is robust to weight pruning, confirming overparameterization.

---

Pruning Techniques Comparison (Best Sparsity 30%)

Technique	Sparsity	Non-zero	Accuracy	Precision	Recall	F1 Score
DenseCNN	0%	421,642	0.9879	0.9877	0.9879	0.9877
Unstructured L1	30%	414,851	0.9897	0.9896	0.9896	0.9896
Structured	30%	341,640	0.9918	0.9917	0.9917	0.9917
Random	30%	417,381	0.9894	0.9895	0.9893	0.9893

Observations:

• Structured pruning reduces parameters most drastically while retaining or slightly improving accuracy.
• Unstructured L1 pruning preserves accuracy but does not significantly reduce memory usage.
• Random pruning is surprisingly robust on MNIST but less effective at memory reduction.

---

Full Sparsity vs Techniques Table (30–90%)

Technique	Sparsity	Non-zero	Accuracy
Unstructured L1	30%	415,204	0.9907
Unstructured L1	50%	409,333	0.9915
Unstructured L1	70%	410,433	0.9869
Unstructured L1	90%	393,148	0.9893
Structured	30%	354,295	0.9917
Structured	50%	266,627	0.9881
Structured	70%	221,976	0.9869
Structured	90%	154,796	0.9849
Random	30%	413,002	0.9913
Random	50%	409,050	0.9920
Random	70%	397,981	0.9918
Random	90%	334,138	0.9862

---

Visual Analysis

• Accuracy vs Sparsity Plot: Shows accuracy stability across different sparsity levels and pruning techniques.
• Non-zero Parameters vs Sparsity Plot: Shows memory reduction achieved by each technique.
• Observations from Plots:
  • Structured pruning reduces parameters drastically but slightly impacts accuracy at high sparsity.
  • Unstructured and Random pruning maintain high accuracy even at 70–90% sparsity but do not reduce memory as effectively.

---

Key Observations & Takeaways

1. Sparse networks can maintain or even slightly improve accuracy due to regularization effects.
2. Structured pruning is most effective for true memory and computational reduction.
3. Unstructured L1 pruning is easy to apply and robust for low to moderate sparsity.
4. Random pruning is surprisingly robust on MNIST, but memory reduction is limited.
5. MNIST CNNs are overparameterized, enabling aggressive pruning without losing accuracy.
6. For deployment, structured pruning at moderate sparsity (~30–50%) gives the best memory-performance trade-off.

---

Conclusion

• Fully connected and convolutional networks are robust to pruning, especially on small datasets like MNIST and Breast Cancer.
• Pruning type and sparsity level significantly influence memory usage and, to a lesser degree, accuracy.
• This study demonstrates a complete end-to-end pipeline:
  Dense → Sparse → Pruning techniques → Multiple sparsity levels → Accuracy & memory trade-off → Visualization.
• These results provide guidance for deploying lightweight neural networks on resource-constrained devices without sacrificing performance.

---

References

• PyTorch Documentation: https://pytorch.org/docs/stable/index.html
• Scikit-learn: https://scikit-learn.org/stable/
• MNIST dataset: LeCun, Y., Cortes, C., & Burges, C. (1998)
• PyTorch pruning utilities: https://pytorch.org/docs/stable/nn.utils.prune.html