Working with Neural Nets

Can you Dance?

(a)

(b)

Figure 1: Bharat Natyam Poses

Introduction

One of our aims with Creative Coding is to of course make things interactive. Here we will apply the ml5.js library in p5.js to use an ML/DL algorithm called Classification to detect human poses in front of the camera. The code can then create unique experiences based on pose-detection with ML, and the subsequent code that responds to the user.

We will be following the ideas from here:

Adavu Detection

https://docs.ml5js.org/#/reference/bodypose

Mudra Detection

https://docs.ml5js.org/#/reference/handpose

Bhava Detection

https://docs.ml5js.org/#/reference/facemesh

But Wait! How does Classification Work?

Ah, peasants. Isn’t it enough that you can dance?

So, we can perform Classification based on Machine Learning (ML) structured and algorithms such as:

1. Random Forests. Also see Google Decision Forests. We will try to get an intuition into bootstrapping of variables in data, creating decision trees, and making random selections of variables from a dataset to create random forests.

2. And there are Deep Learning (DL) structured and algorithms that allow us to do the same things, perhaps in a more “black-box” manner. We will peep into:

   1. The Perceptron
   2. The Multilayer Perceptron
   3. Backpropagation
   4. Gradient Descent
   5. Convolutional Neural Nets (in a later course)

Here, we will also try to build an intuitive sense of some of the technical terminology involved: convolution, regression, activation, weighting…and such terms that generally elude peasants.

Wait, But Why?

Understanding the underlying math inside of Neural Nets can help us appreciate better how to apply them design with them, and even keep them as simple as needed.

How to Train your Dragon Neural Network

References

1. Colah’s Blog.(Apr 6, 2014). Neural Networks, Manifolds, and Topology. https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/. Very simple and readable article.
2. Machine Learning Tokyo: Interactive Tools for ML/DL, and Math. https://github.com/Machine-Learning-Tokyo/Interactive_Tool
3. https://developers.google.com/machine-learning.https://developers.google.com/machine-learning
4. The Neural Network Zoo - The Asimov Institute. http://www.asimovinstitute.org/neural-network-zoo/
5. It’s just a linear model: neural networks edition. https://lucy.shinyapps.io/neural-net-linear/

Convolutional Neural Networks

1. Digit Recognition with CNNs. Interactive! https://transcranial.github.io/keras-js/#/mnist-cnn
2. CNN Convoluter. https://pwwang.github.io/cnn-convoluter/
3. CNN Explainer: Learn Convolutional Neural Network (CNN) in your browser!. https://poloclub.github.io/cnn-explainer/
4. Deep Lizard. Understanding Convolution Operations in Neural Networks. https://deeplizard.com/resource/pavq7noze2
5. Andrej Karpathy. ConvNetJS: Deep Learning in your browser.https://cs.stanford.edu/people/karpathy/convnetjs/
6. Adit Deshpande. A Beginner’s Guide to CNNs. https://adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/
7. Anyone Can Learn AI Using This Blog. https://colab.research.google.com/drive/1g5fj7W6QMER4-03jtou7k1t7zMVE9TVt#scrollTo=V8Vq_6Q3zivl
8. Practical Deep Learning for Coders: An Online Free Course.https://course.fast.ai
9. Neural Networks Visual with vcubingx
   • Part 1. https://youtu.be/UOvPeC8WOt8
   • Part 2. https://www.youtube.com/watch?v=-at7SLoVK_I

LLMs

1. Brendan Bycroft.Visualizing LLMs. https://bbycroft.net/llm
2. Rohit Patel (20 Oct 2024). Understanding LLMs from Scratch Using Middle School Math: A self-contained, full explanation to inner workings of an LLM. https://towardsdatascience.com/understanding-llms-from-scratch-using-middle-school-math-e602d27ec876
3. AI-powered reporting and annotation for radiology. https://www.md.ai

Using R for DL

1. torch for R: An open source machine learning framework based on PyTorch. https://torch.mlverse.org
2. Torch Interactive Tutorial. https://mlverse.shinyapps.io/torch-tour
3. Geeks for Geeks. Convolutional Neural Nets in R. https://www.geeksforgeeks.org/convolutional-neural-networks-cnns-in-r/
4. David Selby (9 January 2018). Tea and Stats Blog. Building a neural network from scratch in R. https://selbydavid.com/2018/01/09/neural-network/
5. Akshaj Verma. (2020-07-24). Building A Neural Net from Scratch Using R - Part 1 and Part 2. https://rviews.rstudio.com/2020/07/20/shallow-neural-net-from-scratch-using-r-part-1/ and https://rviews.rstudio.com/2020/07/24/building-a-neural-net-from-scratch-using-r-part-2/
6. Ander Fernandez Jauregui. https://anderfernandez.com/en/blog/how-to-create-neural-networks-with-torch-in-r/
7. https://f0nzie.github.io/rtorch-minimal-book/

Textbooks

1. Michael Nielsen. Neural Networks and Deep Learning. Available Online
2. The Little Book of Deep Learning. Available Online
3. Simone Scardapane. Alice’s Adventures in Diffferentiable WonderLand: A Primer on Designing Neural Networks. https://www.sscardapane.it/alice-book/
4. Parr and Howard (2018). The Matrix Calculus You Need for Deep Learning.https://arxiv.org/abs/1802.01528
5. Zhang, Lipton, Li, Smola. Dive into Deep Learning. https://www.d2l.ai/
6. Sigrid Keydana. Deep Learning and Scientific Computing with R torch https://skeydan.github.io/Deep-Learning-and-Scientific-Computing-with-R-torch/

R Package Citations

Package	Version	Citation
keras	2.15.0	Allaire and Chollet (2024)
safetensors	0.1.2	Falbel (2023)
tensorflow	2.16.0	Allaire and Tang (2024)
torch	0.14.2	Falbel and Luraschi (2025)

Allaire, JJ, and François Chollet. 2024. keras: R Interface to “Keras”. https://CRAN.R-project.org/package=keras.

Allaire, JJ, and Yuan Tang. 2024. tensorflow: R Interface to “TensorFlow”. https://CRAN.R-project.org/package=tensorflow.

Falbel, Daniel. 2023. safetensors: Safetensors File Format. https://CRAN.R-project.org/package=safetensors.

Falbel, Daniel, and Javier Luraschi. 2025. torch: Tensors and Neural Networks with “GPU” Acceleration. https://CRAN.R-project.org/package=torch.