Packages

We will use the Python language with packages Pandas to load the data, Torch to train the neural network and M:w plotlib for plotting.

%matplotlib inline
import pandas as pd
import torch
from torch import nn
from matplotlib import pyplot as plt

The data

The data can be downloaded from various places, all derived from http://www.math.tamu.edu/~phoward/m442/modbasics.pdf. I load it from here. The dataset consists of observations of lynxes and Hare pelts (in thousands) across the years 1900 to 1920.

url = "https://raw.githubusercontent.com/cas-bioinf/statistical-simulations/master/hudson-bay-lynx-hare.csv"
df = pd.read_csv(url, sep=",\\s", skiprows=2, index_col=0)
df

The data follow a cyclical pattern in which increases in the population of hares are followed by increases in the lynx populations in the subsequent years.

df[["Hare", "Lynx"]].plot(figsize=(10,6), grid=True, xticks = df.index.astype(int)[::2], title="Hudson Bay Lynx-Hare Dataset", ylabel="Number of pelts (thousands)")

The Lotka-Volterra model

The Lotka-Volterra population model the change of predator and pray populations over time: - is the population size of the prey species at time t (the hare in our case), and - is the population size of the predator species (the lynx in our case).

By using such a model, some assumptions are made on the dynamics of these populations. We will not go into those here, but they can be found on the Wikipedia page. The population sizes over times of the two species are modelled in terms of four parameters, as

It is these four parameters we will estimate in this post. They have the following interpretations: - is the growth rate of the prey population, - is the rate of shrinkage relative to the product of the population sizes, - is the shrinkage rate of the predator population, - is the growth rate of the predator population as a factor of the product of the population sizes.

In the Stan article, the author obtains posterior mean point estimates

and in the Mathematical Modeling course, the author obtains

To begin, we first set up some plotting functionality for keeping track of the training progress.

def plot_result(i, t, y, yh, loss, xp=None, lossp=None, params=None):
    plt.figure(figsize=(8, 4))
    plt.xlim(0, 40)
    plt.ylim(0, 85)

    # Data
    plt.plot(t, y, "o-", linewidth=3, alpha=0.2, label=["Exact hare", "Exact lynx"])
    plt.gca().set_prop_cycle(None)  # reset the colors
    plt.plot(t, yh, linewidth=1, label=["Prediction hare", "Prediction lynx"])

    # Physics training ticks
    if xp is not None:
        plt.scatter(xp, 0*torch.ones_like(xp), s=3, color="tab:green", alpha=1, marker="|", label="Physics loss timesteps")

    # Legend
    l = plt.legend(loc=(0.56, 0.1), frameon=False, fontsize="large")
    plt.setp(l.get_texts(), color="k")

    # Title
    plt.text(23, 73, f"Training step: {i}", fontsize="xx-large", color="k")
    rmse_text = f"RMSE data: {int(torch.round(loss))}"
    if lossp is not None:
        rmse_text += f", physics: {int(torch.round(lossp))}"
    plt.text(23, 66, rmse_text, fontsize="medium", color="k")
    if params is not None:
        plt.text(23, 60, params, fontsize="medium", color="k")

    plt.axis("off")
    plt.show()

The inputs for the model are the timesteps corresponding to the years 1900-1920 and the outputs are the numbers of hares and lynxes.

df["Timestep"] = df.index - min(df.index)
t = torch.tensor(df["Timestep"].values, dtype=torch.float32).view(-1,1)
y = torch.tensor(df[["Hare", "Lynx"]].values, dtype=torch.float32)

First, we define a simple dense feed-forward neural network using Torch.

class NN(nn.Module):
    def __init__(self, n_input, n_output, n_hidden, n_layers):
        super().__init__()
        self.encode = nn.Sequential(nn.Linear(n_input, n_hidden), nn.Tanh())
        self.hidden = nn.Sequential(*[nn.Sequential(nn.Linear(n_hidden, n_hidden), nn.Tanh()) for _ in range(n_layers-1)])
        self.decode = nn.Linear(n_hidden, n_output)

    def forward(self, x):
        x = self.encode(x)
        x = self.hidden(x)
        x = self.decode(x)
        return x

We can train neural network on the dataset (predicting the number of lynx and hares based on timestep ) and see that it is complex enough to be able to fit the dataset well. This takes a couple of seconds on my machine.

torch.manual_seed(123)
model = NN(1, 2, 8, 2)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
for i in range(50000):
    optimizer.zero_grad()

    yh = model(t)
    loss = torch.mean((yh - y)**2)

    loss.backward()
    optimizer.step()
    if i%10000 == 0:
        yh = model(t).detach()

        plot_result(i, t, y, yh, loss, None)

Next, we use exactly the same architecture but add parameters alpha, beta, gamma and delta reflecting the parameters in the Lotka-Volterra equations. We initialize them randomly between 0 and 1, and add them to the list of model paramters so that they are updated in the backpropagation steps.

torch.manual_seed(123)
model = NN(1, 2, 8, 2)

alpha = torch.rand(1, requires_grad=True)
beta = torch.rand(1, requires_grad=True)
gamma = torch.rand(1, requires_grad=True)
delta = torch.rand(1, requires_grad=True)

extra_parameters = [alpha, beta, gamma, delta]
parameters = list(model.parameters()) + extra_parameters
optimizer = torch.optim.Adam(parameters, lr=1e-3)

We define t_physics; the range of timesteps on which we numerically evaluate the differential equations. In this case, we evaluate them every 0.1 year.

t_physics = torch.arange(0, len(t), 0.1, dtype=torch.float32).view(-1,1).requires_grad_(True)

Training the PINN is identical to training the normal neural network, except that the loss function consists of a combination of the data loss and the physics loss. The data loss is identical to the loss of the previous neural network. For the physics loss, note that we can rewrite the Lotka-Volterra equations as We calculate the mean of the square of the left-hand side across the values of in t_physics, thus punishing the model if the equations are far away from 0. We use torch.autograd to estimate the partial derivatives, which means that these partial derivatives are based on what the neural network has learned so far. We calculate a weighted sum of the physics and data loss, using a weight of lambda1 set to 0.01. The value of this weight can matter quite a lot and depends on the relative sizes of the errors. Intuitively, it captures how much you trust the data versus the physical laws.

lambda1 = 1e-2
for i in range(50000):
    optimizer.zero_grad()

    # Data loss
    yh = model(t)
    data_loss = torch.mean((yh - y)**2)

    # Physics loss
    yhp = model(t_physics)  # output of the model at the t_physics timesteps
    u, v = yhp[:, 0], yhp[:, 1]  # hare and lynx populations according to the neural network
    dudt  = torch.autograd.grad(u, t_physics, torch.ones_like(u), create_graph=True)[0].flatten() # time derivative of hare
    dvdt = torch.autograd.grad(v,  t_physics, torch.ones_like(v),  create_graph=True)[0].flatten() # time derivative of lynx
    dudt_loss = torch.mean((dudt - (alpha*u - beta*u*v))**2)
    dvdt_loss = torch.mean((dvdt - (delta*u*v - gamma*v))**2)
    physics_loss = dudt_loss + dvdt_loss

    loss = data_loss + lambda1*physics_loss
    loss.backward()
    optimizer.step()

    if i%10000 == 0:
        yh = model(t).detach()
        tp = t_physics.detach()

        a, b, c, d = [round(param.item(), 3) for param in extra_parameters]
        params = rf"$\alpha={a}, \beta={b}, \gamma={c}, \delta={d}$"
        plot_result(i, t, y, yh, loss, tp, physics_loss, params)

We see that the model is still able to fit well (albeit a little bit worse), and is able to estimate values for the parameters. Specifically, it estimates

which are very close to the results of the other two approaches mentioned above. Only the estimate for deviates significantly from the other estimates.

However, this approach is very simple to implement and requires no knowledge about the differential equations or Bayesian modelling software.

Thanks for reading!

Linear programming

A nice way to solve problems like this is by Integer Linear Programming. Like in my Trees and Tents post, I like to do this in Julia using the JuMP package.

The constraints we have are on the products of the numbers in the rows and the columns. Therefore the problem is not linear if we implement it naively (a variable for every cell denoting the number between 1 and 9 it should contain). We need to do something a little bit more clever. A natural way to look at this problem is via the prime factorizations of the row and the column products. There are four primes between 1 and 9, namely 2, 3, 5 and 7. Given the prime factorization of every row and column, we only need to distribute these primes over the cells. Since , we know that in the first row there should be a 2, a 3, no 5 and two 7s. Note that a cell can contain multiple primes, or none at all (meaning it is a 1).

We therefore need a variable for every cell and each of the 4 possible primes, the value of which denotes the multiplicity of this prime in this cell.

In Julia, we first import the required packages, and define the row and column products.

using JuMP, GLPK
using Primes: factor, primes
using DataStructures: DefaultDict

row_prods = [294, 216, 135, 98, 112, 84, 245, 40];
col_prods = [8_890_560, 156_800, 55_566];
prms = primes(1, 9);

n_prms = length(prms);
n_rows = length(row_prods);
n_cols = length(col_prods);

Now, we create lists of the multiplicities of the four primes in the rows and the columns. The [1, 1, 0, 2] denotes the multiplicities of 2, 3, 5 and 7 in the first row.

row_factors = factor.(Dict, row_prods);
col_factors = factor.(Dict, col_prods);

row_factors = [[DefaultDict(0, row_factors[row])[prms[i]] for i in 1:4] for row in 1:n_rows]
col_factors = [[DefaultDict(0, col_factors[col])[prms[i]] for i in 1:4] for col in 1:n_cols]

8-element Array{Array{Int64,1},1}:
 [1, 1, 0, 2]
 [3, 3, 0, 0]
 [0, 3, 1, 0]
 [1, 0, 0, 2]
 [4, 0, 0, 1]
 [2, 1, 0, 1]
 [0, 0, 1, 2]
 [3, 0, 1, 0]
3-element Array{Array{Int64,1},1}:
 [6, 4, 1, 3]
 [7, 0, 2, 2]
 [1, 4, 0, 3]

Now onto the actual problem definition. As stated, we need a variable for every cell and every prime. Every prime can occur 0 or more times, and its multiplicity is an integer.

model = Model(with_optimizer(GLPK.Optimizer))

@variable(model, 0 <= x[1:n_rows, 1:n_cols, 1:n_prms], integer=true)

Rows and column prime multiplicities should add up to the correct numbers.

for i in 1:n_rows, p in 1:n_prms
    @constraint(model, sum(x[i, :, p]) == row_factors[i][p])
end

for j in 1:n_cols, p in 1:n_prms
    @constraint(model, sum(x[:, j, p]) == col_factors[j][p])
end

Finally, we want every cell to be an integer between 1 and 9, which means , or , a constraint which is linear in its variables.

for i in 1:n_rows, j in 1:n_cols
    @constraint(model, sum(x[i, j, :] .* log.(prms)) <= log(9))
end

And that is all! We let JuMP do the hard work and come up with a feasible solution. As an example, we print out the number of times a 2 (the first prime) appears in every cell.

JuMP.optimize!(model)

solution = JuMP.value.(x);

solution[:, :, 1]

8×3 Array{Float64,2}:
 0.0  0.0  1.0
 0.0  3.0  0.0
 0.0  0.0  0.0
 0.0  1.0  0.0
 3.0  1.0  0.0
 0.0  2.0  0.0
 0.0  0.0  0.0
 3.0  0.0  0.0

The final solution to the riddle is then the product of every prime to the power of its multiplicity:

[[Int(prod(prms .^ solution[i, j, :])) for j in 1:n_cols] for i in 1:n_rows]

8-element Array{Array{Int64,1},1}:
 [7, 7, 6]
 [9, 8, 3]
 [9, 5, 3]
 [7, 2, 7]
 [8, 2, 7]
 [7, 4, 3]
 [5, 7, 7]
 [8, 5, 1]

And there’s the solution! I love how easy it is to have JuMP solve such a problem after coming up with a way to phrase it as a linear program.

Python

My initial python code was very hacky and unstable, so I decided to take a more structured approach, now that I knew I would be able to get it working. I found a nicely structured project by Michael Fogleman that does the same for the Makeblock XY Plotter. It defines an object that holds the connection to the plotter, and defines methods that send commands to it, like move(), pen_up() and pen_down(). It can render a drawing and store it as an image before sending it to the plotter, which makes it easy to assess an image virtually before physically drawing it. I forked the project, edited the device instructions and restructured it a bit, which resulted in EleksDrawPy.

This new code is a lot easier to use and more stable than my previous attempt. It also allows for usage from R via the reticulate R interface to Python.

R

My pen plotter was collecting dust when I stumbled upon fawkes, “an R interface to the AxiDraw plotter” by Thomas Lin Pedersen, who is part of RStudio’s tidyverse team. It uses reticulate to be able to use the AxiDraw python interface from R, and defines a device which one can write ggplot2 plots to, similar to R’s built-in png() or pdf() device. How cool would it be to be able to plot ggplot2 graphs on my plotter?

It turned out to be very straightforward to port his code to use my python code. The resulting code can be found on GitHub. I kept the original code intact, but added the eleks_dev() and the eleks_manual() functions, which are the counterparts of the axi_dev() and axi_manual() functions already present. The former can be used as a device like png(), the latter for interactive mode (i.e. sending individual commands to the device). I did not create a version of the axi_svg() function (that uses the AxiDraw svg plotting capabilities), since my EleksDraw python code does not have this functionality (yet).

The fawkes::ghost_dev() device that is also present in the package, makes it easy to get a feeling of what the resulting plot will look like. It shows not only the lines that will be drawn, but also the paths the device takes while the pen is in the air.

Now, we can plot ggplot2 graphs from R right onto a piece of paper! Below are some examples. All credits go to Thomas Lin Pedersen and Michael Fogleman, whose code I merely adjusted for my specific goal.

Results

The python module can be installed with pip install -r requirements.txt (after cloning the project), and the R package with remotes::install_github('basjacobs93/fawkes'). The packages we’ll use besides the aforementioned fawkes and ggplot2 are dplyr (for data manipulation) and sf (for using spatial data).

library(dplyr)
library(fawkes)
library(ggplot2)
library(sf)

Given a ggplot2 plot p, the plot can be previewed with the following,

gd <- ghost_dev('A6', portrait = FALSE, margins = 5, ignore_color = TRUE)
p
invisible(dev.off())
gd$preview(plot_air = TRUE)

and plotted with the EleksDraw using the below.

gd <- eleks_dev('A6', portrait = FALSE, margins = 5, ignore_color = TRUE)
p
invisible(dev.off())

For each of the below examples, we show the ggplot2 graph as a png, the output of the fawkes::ghost_dev() preview, and finally a picture of the plot on paper.

p <- ggplot(mtcars) +
  geom_point(aes(disp, mpg)) +
  facet_wrap(~ gear) +
  theme_bw(base_size = 6) +
  theme(
    plot.background = element_blank(),
    panel.background = element_blank()
  )

f1=jitter(sample(c(2,3),1));f2=jitter(sample(c(2,3),1));f3=jitter(sample(c(2,3),1));f4=jitter(sample(c(2,3),1))
d1=runif(1,0,1e-02);d2=runif(1,0,1e-02);d3=runif(1,0,1e-02);d4=runif(1,0,1e-02)
p1=runif(1,0,pi);p2=runif(1,0,pi);p3=runif(1,0,pi);p4=runif(1,0,pi)
harmonograph <- data.frame(t = seq(0, 80*pi, 80*pi/10000)) %>%
  transmute(
    x = exp(-d1*t)*sin(t*f1+p1) + exp(-d2*t)*sin(t*f2+p2),
    y = exp(-d3*t)*sin(t*f3+p3) + exp(-d4*t)*sin(t*f4+p4)
  )

p <- harmonograph %>%
  ggplot(aes(x, y)) +
  geom_path() +
  theme_bw(base_size = 6) +
  theme_void()

n_cols <- 12
n_rows <- 22

p <- expand_grid(col = 0:(n_cols-1),
            row = 0:(n_rows-1),
            tibble(x = c(-0.5, 0.5,  0.5, -0.5),
                   y = c( 0.5, 0.5, -0.5, -0.5))) %>%
  group_by(col, row) %>%
  mutate(angle = rnorm(1, sd = row/60),
         xn = x*cos(angle) - y*sin(angle),
         y = x*sin(angle) + y*cos(angle),
         x = xn) %>%
  select(-xn) %>%
  mutate(x = x + 1 + col,
         y = y + 1 - row,
         x = x + rnorm(1)*row/80, # jitter
         y = y + rnorm(1)*row/60) %>%
  mutate(xend = lag(x, default = last(x)),
         yend = lag(y, default = last(y))) %>%
  ungroup() %>%
  ggplot(aes(x = x, y = y, xend = xend, yend = yend)) +
  geom_segment() +
  coord_fixed() +
  theme_void()

Provinces of The Netherlands

We can use the excellent sf package to load and plot spatial data with ggplot2, like a map of the province boundaries of the Netherlands. Below, we load provinces data from CBS with sf::st_read(), simplify it to decrease the complexity of the plot, and plot it with ggplot2::geom_sf().

provinces <- st_read("https://geodata.nationaalgeoregister.nl/bestuurlijkegrenzen/wfs?request=GetFeature&service=WFS&version=1.1.0&typeName=bestuurlijkegrenzen:provincies")

p <- provinces %>% 
  st_transform(54032) %>% # Azimuthal equidistant
  st_simplify(dTolerance = 400) %>% # simplify
  st_transform(4326)  %>% # WGS84
  ggplot() +
  geom_sf(fill = NA) +
  theme_void()

Summary of method

In the paper, the authors start with an initial grid of points that defines a triangulation. This grid can either be a regular layout, in which the points are spaced evenly across the width and height of an image, or it can be made via importance sampling, in which ‘important’ regions of the image are found and points are sampled, weighted by this importance. Every triangle gets a constant color, calculated as the mean of the colors of its interior points in the original image.

As an example, consider this image of a beautiful bird called a Bearded reedling (Panurus biarmicus), taken from this YouTube video.

The two types of initial triangulation are the regular layout (in this case with 50 triangles),

and the triangulation based on importance sampling (in this case by sampling 50 points).

Starting with such a triangulation, the triangles are improved iteratively by gradient descent. For every point, a small change in the horizontal and vertical direction is tried, and a gradient is calculated by calculating the errors of the point’s adjacent triangles for each such configuration. The error of a triangle is defined as the mean of the differences of its interior points’ colors, compared to the original image. This way, the initial triangulation gets better every step, but it may converge to a local optimum, depending on the initial triangulation.

On top of gradient descent, every couple of steps, every triangle can be split into three by placing an additional point at its centroid. This way, large triangles that have a big error can be split up into smaller triangles with different colors and smaller errors. The triangles eligible for this splitting have to be large enough, should not be too narrow, and should have a large error. These thresholds, together with the number of steps, step size, and other parameters can be tweaked in the code, yielding different results.

An example output is the below, in which the initial grid was a regular layout, and which resulted in an image of 700 triangles.

For a more mathematically thorough explanation, I refer to the original paper.

More results

Another beautiful picture of the same species of bird is the following (source by Kaeptn chemnitz, CC BY 3.0, via Wikimedia Commons).

The resulting triangulations are more abstract than the above, but I think they are beautiful. The results with 100, 500, 1000 and 2000 triangles are below.

Two more examples of triangulations with 2000 triangles are below. They represent a bluethroat and a red-backed shrike.

More results (June 2022)

I have updated the code to make the drawing faster and to include more techniques from the paper (edge flips and node collapse).
Below is the new code demonstrated on the Lee Morgan album cover Lee-way after 50, 100, 120 and 250 steps.

And the same on Cannonball Adderley’s album Somethin’ Else.

Conclusion

Implementing part of the methodology of the paper in Julia was fairly straightforward, even though my experience with the language is limited. However, the performance is not as good as I hoped it would be. Generating the above images took a couple of minutes upto half an hour each. This makes it hard to fiddle around with the parameters, since a lot of time is spent waiting for the results. The authors of the paper wrote parts of their code in C++ because of performance, and parallelized and optimized the code such that it was faster. Although Julia code can be fast, my naive implementation still requires lots of loops (over the triangles, points, pixels), so it makes sense for it to be slow. That said, even though my implementation is naive, a pretty triangulation is only a key-press away and can be generated in a couple of minutes.

As stated, the notebook is available on GitHub, so feel free to try it out!

Simulation in R

The following R function simulates eating one bag of chocolates and outputs what the last chocolate was.

chocolates <- c(rep("m", 2), rep("d", 8))

eat_chocolates <- function() {
  # shuffle the chocolates around
  chocolates_left <- sample(chocolates)

  # eat the first chocolate
  last_taken <- chocolates_left[1]
  chocolates_left <- chocolates_left[-1]

  # eat the rest of the chocolates
  while (length(chocolates_left) > 0) {
    # take the first chocolate out of the bag
    to_eat <- chocolates_left[1]
    chocolates_left <- chocolates_left[-1]

    # put back and shuffle if differs from previous piece
    if (to_eat != last_taken) {
      chocolates_left <- sample(c(to_eat, chocolates_left))
    }

    last_taken <- to_eat
  }

  last_taken
}

We can now easily run, for example, 100,000 simulations and calculate the percentage in which the last chocolate was milk:

simulations <- replicate(1e5, eat_chocolates())
mean(simulations == "m")

[1] 0.45907

In a couple of seconds, this gives an estimate of a 46% probability of ending with a piece of milk chocolate. This is closer to 50% than I initially expected. Intuitively though, it does make sense, given the nature of the problem that the majority chocolate type will be eaten relatively quickly, since it is easier to get a ‘streak’ of those.

Running this (admittedly naive) simulation for more than, say, 1 million replications quickly becomes intractable. For a more accurate result that does run quickly, we can resort to recursion.

Recursion

The problem ca be written in a recursive form: given a number of milk and dark chocolates, we can create a set of possible next states and attach probabilities to those states. We can repeat this until we arrive at a bag of only one type of chocolate.

In order to do that, let’s first introduce some notation. Let be the probability of ending with a milk chocolate given that we currently have a bag of milk and dark chocolates and that the last chocolate we ate was of type . Denote by the chocolate type that is not . We then know the trivial cases and ; you can’t “fail” if you only have milk left, and you can’t “succeed” if you only have dark chocolates left.

Moreover, the probability of ending with a milk chocolate given a certain distribution of chocolates and a previous pick equals the probability of ending with a dark chocolate given that we have switched all dark and milk chocolates around. In mathematical terms, .

Now, given a state , you will pick a milk chocolate with probability , resulting in a state of , or you will pick dark with probability , resulting in . In other words, given ,

We can now immediately see a problem with programming this recursion as it is: the second term did not decrease the total number of chocolates. Were we to do another step, we could again arrive at the situation we started with, thus resulting in an endless loop. In other words, theoretically we could keep alternately picking and chocolates and never finish the whole bag.

We need to go deeper.

Let’s do another recursion step by expanding the right-most term in the above equation:

Filling this into the first equation,

which we can rearrange into

or

which gives

Now, by magic, this recursion does strictly decrease the total number of chocolates in the bag, thus making a program feasible. Together with the relationship , we can calculate our quantity of interest, which is the probability of ending with a milk chocolate given a starting bag and no previous chocolates eaten:

In R code, this recursion can be programmed as follows.

P <- function(m, d, previous = "") {
  if (m == 0) {
    return(0)
  } else if (d == 0) {
    return(1)
  }

  if (previous == "") {
    m/(m+d)*P(m-1, d, "milk") + d/(m+d)*P(m, d-1, "dark")
  } else if (previous == "milk") {
    (m^2+m*d)/(m^2+d^2+m*d)*P(m-1, d, "milk") + (d^2)/(m^2+d^2+m*d)*P(m, d-1, "dark")
  } else if (previous == "dark") {
    1 - P(d, m, "milk")
  }
}

This allows us to calculate some simple examples,

P(2,0)

[1] 1

P(0,2)

[1] 0

P(1,1)

[1] 0.5

but also the actual quantity of interest

P(2,8)

[1] 0.4615372

within a second. This agrees with the outcome of our simulation: the probability of ending with a milk chocolate is about 46.15%.

Other starting points

Now that we have code that runs quickly, we can explore the probabilities for other mixes of chocolates.

Say we start with 2 milk chocolates, but vary the number of dark chocolates.

library(tidyverse)

Warning: package 'dplyr' was built under R version 4.2.3

theme_set(theme_light())

simulations <- tibble(m = 2, d = 1:100) %>%
  rowwise() %>%
  mutate(p = P(m, d))

ggplot(simulations, aes(d, p)) +
  geom_line() +
  scale_y_continuous(labels = scales::percent_format()) +
  labs(title = "Probability of ending with a milk chocolate",
       subtitle = "Given 2 milk and a varying number of dark chocolates",
       x = "Number of dark chocolates",
       y = "Probability")

At first, the probability of success drops quickly, but at some point the streaks of dark chocolates will be so common that adding more dark chocolates does not decrease this probability significantly.

We can do the same thing in two dimensions by also varying the number of milk chocolates. Now, memoization comes in handy, since otherwise we calculate the same probability many times. The function P_mem calculates P but checks whether a certain probability has already been calculated in the list of matrices mem.

n_max <- 20
mem <- list(milk = matrix(nrow = n_max, ncol = n_max),
            dark = matrix(nrow = n_max, ncol = n_max))

P_mem <- function(m, d, previous = "") {
  if (m == 0) {
    return(0)
  } else if (d == 0) {
    return(1)
  }

  if (previous != "" && !is.na(mem[[previous]][m, d])) {
    return(mem[[previous]][m, d])
  }

  if (previous == "") {
    res <- m/(m+d)*P_mem(m-1, d, "milk") + d/(m+d)*P_mem(m, d-1, "dark")
  } else if (previous == "milk") {
    res <- (m^2+m*d)/(m^2+d^2+m*d)*P_mem(m-1, d, "milk") + (d^2)/(m^2+d^2+m*d)*P_mem(m, d-1, "dark")
    mem[[previous]][m, d] <<- res
  } else if (previous == "dark") {
    res <- 1 - P_mem(d, m, "milk")
    mem[[previous]][m, d] <<- res
  }

  res
}

simulations <- crossing(m = 1:n_max, d = 1:n_max) %>%
  rowwise() %>%
  mutate(p = P_mem(m, d))

ggplot(simulations, aes(m, d, fill = p)) +
  geom_raster() +
  scale_fill_continuous(labels = scales::percent_format()) +
  labs(title = "Probability of ending with a milk chocolate",
       x = "Number of milk chocolates",
       y = "Number of dark chocolates",
       fill = "Probability")

This graph shows that the more chocolates there are in the bag, the smaller the differences between different mixes become.

Problem definition

The puzzle can be formulated as a linear programming problem. There are three types of constraints:

• every tent must be next to a tree,
• the number of tents for every row and column is given,
• tents are not allowed to touch.

Even though we treat this puzzle as an optimization problem, there is no objective to be maximized or minimized in this case; any placements of tents satisfying the constraints is a valid solution. This is similar to the sudoku solutions in which the objective is also unimportant.

Julia’s JuMP package provides a clean interface to defining such a problem, and solving it with various solvers. In this case, we use GLPK since it’s free and works great for this problem. Solving the problem is a matter of writing down the constraints and calling JuMP.optimize.

using JuMP, GLPK

After loading the necessary packages, we need to define the puzzle displayed above. In the website’s html code, the puzzle is represented by three strings.

• lcpuzzle is a string of numbers, denoting the cells from left to right, top to bottom. The values represent whether the corresponding cell is blank (0), or contains a tree (1) or a tent (2). Note that this string contains the solution to the problem as well, so this allows us to check the solution that JuMP found afterwards.
• lcrownumbers is a string of numbers which denote the number of tents per row.
• lcrolnumbers is a string of numbers which denote the number of tents per column.

# from html, 0=blank, 1=tree, 2=tent
lcpuzzle = "001202011200021001020012000120001200020100100000010200200000201000001212102020000000000010021012201000000200102002020102000001010001001201201212"
lcrownumbers = "332123222404"
lccolnumbers = "222322230505"

We rewrite the above representation into something that is more easy to work with:

• trees is a list of (i,j) tuples corresponding to the locations of the trees.
• row_sums and col_sums are lists of integers denoting the totals per row and column.
• n is the size of the game, which is in this case 12.

puzzle = [parse(Int, i) for i in lcpuzzle]
n = Int(sqrt(length(puzzle)))

trees = [(i, j) for i in 1:n, j in 1:n if puzzle[(i-1)*n + j] == 1]
row_sums = [parse(Int, i) for i in lcrownumbers]
col_sums = [parse(Int, i) for i in lccolnumbers]

We initialize the solver, telling JuMP that we want to use GLPK. The variable that JuMP optimizes over is an matrix x, consisting of binary values, where a 0 means no tent for that cell, and a 1 means that the cell contains a tent.

model = Model(with_optimizer(GLPK.Optimizer))

@variable(model, x[1:n, 1:n], Bin)

Constraints

We are now ready to define the constraints. We start with the requirement that a tent be next to a tree.

First, a tent cannot be on top of a tree. Hence, if a tree is on position (row, col), then x[row, col] must be .

Next, for every cell (row, col) we check whether there is a tree next to this cell. If not, then this cell cannot contain a tent, hence x[row, col] should be .

for row in 1:n, col in 1:n
    # is not on a tree
    if (row, col) in trees
        @constraint(model, x[row, col] == 0)
    end

    # should be next to a tree
    found = false
    for tree in trees
        if abs(tree[1] - row) + abs(tree[2] - col) <= 1
            found = true
        end
    end
    if !found
        @constraint(model, x[row, col] == 0)
    end
end

The next constraint makes sure that the number of tents in the rows and columns add up to the right values. This can be done simply by summing the columns and rows of the matrix x, since this equals counting the number of ones.

for i in 1:n
    @constraint(model, sum(x[i, :]) == row_sums[i])
    @constraint(model, sum(x[:, i]) == col_sums[i])
end

Lastly, we need to encode the fact that the tents may not touch horizontally, vertically or diagonally. In other words, every square contains at most one tree, which is also easy to establish using the representation we chose.

for row in 1:(n-1), col in 1:(n-1)
    @constraint(model, sum(x[row:row+1, col:col+1]) <= 1)
end

Solution

Now that we have represented the puzzle as a Linear Program, we can simply tel JuMP to optimize this. Within a second, it spits out a solution. Since Julia supports Unicode characters, we can visualize the solution as a grid of emoji.

JuMP.optimize!(model)

solution = JuMP.value.(x)

for row in 1:n
    for col in 1:n
        if solution[row, col] == 1
            print("⛺")
        elseif (row, col) in trees
            print("🌲")
        else
            print("◻️")
        end
        print(" ")
    end
    println("")
end

◻️ ◻️ 🌲 ⛺ ◻️ ⛺ ◻️ 🌲 🌲 ⛺ ◻️ ◻️
◻️ ⛺ 🌲 ◻️ ◻️ 🌲 ◻️ ⛺ ◻️ ◻️ 🌲 ⛺
◻️ ◻️ ◻️ 🌲 ⛺ ◻️ ◻️ ◻️ 🌲 ⛺ ◻️ ◻️
◻️ ⛺ ◻️ 🌲 ◻️ ◻️ 🌲 ◻️ ◻️ ◻️ ◻️ ◻️
◻️ 🌲 ◻️ ⛺ ◻️ ◻️ ⛺ ◻️ ◻️ ◻️ ◻️ ◻️
⛺ ◻️ 🌲 ◻️ ◻️ ◻️ ◻️ ◻️ 🌲 ⛺ 🌲 ⛺
🌲 ◻️ ⛺ ◻️ ⛺ ◻️ ◻️ ◻️ ◻️ ◻️ ◻️ ◻️
◻️ ◻️ ◻️ ◻️ 🌲 ◻️ ◻️ ⛺ 🌲 ◻️ 🌲 ⛺
⛺ ◻️ 🌲 ◻️ ◻️ ◻️ ◻️ ◻️ ◻️ ⛺ ◻️ ◻️
🌲 ◻️ ⛺ ◻️ ◻️ ⛺ ◻️ ⛺ ◻️ 🌲 ◻️ ⛺
◻️ ◻️ ◻️ ◻️ ◻️ 🌲 ◻️ 🌲 ◻️ ◻️ ◻️ 🌲
◻️ ◻️ 🌲 ⛺ ◻️ 🌲 ⛺ ◻️ 🌲 ⛺ 🌲 ⛺

That’s it! I like the JuMP syntax a lot and will definitely use it to solve more puzzles!

More layers

Now let’s try a network with more layers. More specifically, let’s add two layers with both 10 nodes followed by sigmoid activations.

model <- keras_model_sequential()
model %>%
  layer_dense(10, input_shape = 2, activation = "sigmoid") %>%
  layer_dense(10, activation = "sigmoid") %>%
  layer_dense(3, activation = "sigmoid")

model %>% compile(
  loss = "mean_squared_error",
  optimizer = optimizer_adam(),
  metrics = "accuracy"
)

model %>% fit(
  X, Y,
  batch_size = 128,
  epochs = 20
)

df_out <- df
df_out[, 3:5] <- model %>% predict(X)
df_out <- df_out %>%
  gather(key = "cc", value = "value", -x, -y, convert = TRUE)

as.cimg(df_out) %>% list(im, .) %>%
  imappend("x") %>% plot(axes = F)

As can be seen, the image is already picking up some countours of the bird, together with some ‘rays’ coming from the top-left corner. If we go for an even deeper network, more of the bird’s features can be recognized from the image.

model <- keras_model_sequential()
model %>%
  layer_dense(100, input_shape = 2, activation = "tanh") %>%
  layer_dense(100, activation = "relu") %>%
  layer_dense(100, activation = "relu") %>%
  layer_dense(100, activation = "relu") %>%
  layer_dense(100, activation = "relu") %>%
  layer_dense(3) %>%
  layer_activation_relu(max_value=1)

model %>% compile(
  loss = "mean_squared_error",
  optimizer = optimizer_adam(),
  metrics = "accuracy"
)

model %>% fit(
  X, Y,
  batch_size = 128,
  epochs = 100
)

df_out <- df
df_out[, 3:5] <- model %>% predict(X)
df_out <- df_out %>%
  gather(key = "cc", value = "value", -x, -y, convert = TRUE)

as.cimg(df_out) %>% list(im, .) %>%
  imappend("x") %>% plot(axes=F)

Ways to improve this might include more layers, more epochs, different activation functions, different loss function. For now, I’m really happy with the result!

The data

The Netherlands have an open database containing elevation data for the whole country. It is called Actueel Hoogtebestand Nederland (Current Elevation database Netherlands) and can be viewed online. On average, it contains eight height measurements per square meter. The measurements are made using aircraft with LiDAR technology. Measuring the entire country takes sevaral years. In particular, I used the database called AHN2, which contains measurements made from 2007 to 2012. According to the website, 99.7% of the measured points are accurate up to 20 cm.
In the database, the country is divided into a grid of around 1300 cells, each having dimensions of 5x6.25 kilometers. For every cell, the measurements can be downloaded as a zipped GeoTIFF file of around 500 KB. Links to these files can be found in an Atom feed.

The code

Obtaining the links to all cells is as easy as reading the contents of the Atom feed and extracting all URLs from it. I chose to extract the URLs using a simple regular expression. Note that in R you need to escape the backslash with another backslash.

library(tidyverse)
feed <- read_file("http://geodata.nationaalgeoregister.nl/ahn2/atom/ahn2_5m.xml")
links <- str_extract_all(feed, "http[:/.\\w]+\\.zip")[[1]] %>% unique

This results in a vector of 1373 URLs.
For every url in this links vector, I download its contents into a temporary file, unzip it, read it into memory as a GDAL object, take its maximum over an axis and remove the temporary file again.

library(rgdal)

# Create temporary folder to store the files in  
temp <- tempfile(fileext = ".zip")
# Download the file and unzip it
download.file(url, temp, quiet = TRUE)
unzip(temp)
# Obtain .tif files contained in .zip
shp <- dir(tempdir(), "*.tif$")

# Convert to GDAL and project onto axis
y <- readGDAL(shp, silent = TRUE)
y <- project(y, axis)

# Remove temporary files
file.remove(temp)
file.remove(shp)

I wrap this into a function and run it for every url using plyr::llply(). I like to use llply because it has a progress bar functionality that I find useful when scraping. I set axis to 2, which means that I take the maximum value over the rows (longitude). In the end, obtain a data frame with 1250 values for every cell (6250 meter cell size, 5 measurements per meter). Since there are multiple cells within a row, I have to group this data frame by latitude and take the maximum over each group to get to the plottable result.
Plotting is now an easy task using ggplot.

ggplot(df, aes(x = x, y = elevation)) +
  geom_line() +
  theme_void()

This plot displays the country as if we were standing in Germany and looking at the Netherlands without the ability to see perspective. There are some weird values here and there, but overall it looks like can be expected. It is too rough to use as a logo; we should apply some smoothing to make it prettier.
Smoothing can be done in ggplot by simply replacing geom_line with geom_smooth. Under the hood, this is using using cubic regression splines to smooth the values. This yields a pretty picture better suited for using as a logo.

ggplot(df, aes(x = x, y = elevation)) +
  geom_smooth(se = FALSE) +
  theme_void()

Conclusion

That’s it! Once you have an idea, creating your own logo can be a couple of lines of code away.
The complete R code used can be viewed on my Github.

Buying and assembling

I ordered the EleksDraw from BangGood, which means that the product came from China and took around 3 weeks to arrive. It came in more than a hundred pieces and assembling it took me a couple of hours. The instructions on the website were just clear enough; they consisted of just a series of pictures with no text.

The EleksDraw consists of an arm that is connected via a cable to two motors which together can move the arm in the x and y direction. You can mount any pen or pencil to it which is held in place by a screw. After assembling it and hooking it up to my computer, I downloaded the EleksMaker software that can be used to convert an image to commands that this plotter can interpret. However, this software only works on Windows (I’m on a MacBook), and even after booting into Windows, it did not work as it should. Most of the menus didn’t show any text and when loading an image, I got an error that I couldn’t understand because it was in Chinese. I decided to boot back into MacOS and create some software myself. How hard could it be, right?

Hello, World!

The MacOS terminal has a screen command that allows you to send commands to an external device. In my case, typing screen /dev/wchusbserial1420 connects with the EleksDraw and opens a terminal in which I could write commands. The language that this plotter understands is called G-code. With this code, you can give the device the following commands:

• S0 M5 lifts the pen
• S1000 M3 lowers the pen
• G01 Xa Yb moves to position (a,b)

Note that G-code has many variants, so the above commands may not be exactly the same for other pen plotters. Weirdly, the plotter’s (0,0) position was in the top right corner, so I had to make all the coordinates negative to get it to work properly. Could be because I somehow wired it wrong, but I could not figure it out. Anyway, writing my own software allowed me to work around this very easily! Writing these instructions one by one, I could create all kinds of shapes consisting of straight lines. After every command I wrote, the device would either send back the message ok or an error. After this response, I could send it a new instruction. However, I wanted to automatically generate these commands and send them to the device using a script.

# Open serial port
s = serial.Serial('/dev/wchusbserial1420', 115200)

# Wake up
s.write("\r\n\r\n")
time.sleep(2)       # wait for grbl to initialize
s.flushInput()      # flush startup text in serial input

s.write("G21\n")   # units = mm
s.readline()

s.write("F5000\n") # speed = 5000 mm/min
s.readline()

s.write("S0 M5\n") # pen up
s.readline()

s.write("G01 X-10 Y-10\n") # go to (10, 10)
s.readline()

s.write("S1000 M3\n") # pen down
s.readline()

s.write("G01 X-20 Y-10\n") # go to (20, 10)
s.readline()

s.write("S0 M5\n") # pen up
s.readline()

Plotting SVGs

Back to my original idea: I wanted to be able to load an image, convert it to G-code commands and send these commands to the plotter automatically. Using SVG images as input seemed like the right choice since these are vector graphics and hence should be easy to convert to G-code. First step: writing an SVG parser.

import plotter as plt

class Line():
    # a line is defined by 2 points
    def __init__(self, point1, point2):
        self.P1 = point1
        self.P2 = point2

    def intersect(self, line2):
        # calculates the intersection point of the lines
        # solution is based on simple algebra
        a = self.P1.x - self.P2.x
        b = self.P1.y - self.P2.y

        u = (a*(line2.P2.y-self.P2.y) - b*(line2.P2.x-self.P2.x)) /\
            (a*(line2.P2.y-line2.P1.y) - b*(line2.P2.x-line2.P1.x))

        return (line2.P1.times(u)).plus(line2.P2.times(1-u))

    def perpendicular_at(self, P):
        U = Point(self.P2.y-self.P1.y, self.P1.x - self.P2.x)
        return Line(P, P.plus(U))

    def draw(self, canvas, color):
        pygame.draw.line(canvas, color,
                         (self.P1.x, canvas.get_height()-self.P1.y),
                         (self.P2.x, canvas.get_height()-self.P2.y))

    def plot_instructions(self):
        # assumes we are at P1
        return [plt.move(self.P2.x, self.P2.y)]

Drawing a harmonograph

Another blog post inspired me to try and plot some harmonographs. A harmonograph is a device that uses pendulums to draw trajectories on a piece of paper. The output of such a harmonograph can be described mathematically and we can simulate a random harmonograph by drawing random numbers and following a trajectory defined by sine waves. I followed the blog post mentioned above which draws 12 random numbers and plots the trajectory defined by

where runs from to . In his post, the author implemented the harmonograph in R, but since my plotter works from Python, I ported his code to Python.

import numpy as np

# draw the necessary random numbers
f1, f2, f3, f4 = np.random.randint(2, 4, size=4) + (np.random.random(size=4)-0.5) / 10
d1, d2, d3, d4 = np.random.uniform(0, 1e-02, 4)
p1, p2, p3, p4 = np.random.uniform(0, np.pi, 4)

# define the space and follow the trajectory
t = np.linspace(1, 100, num = 10**4)
xs = xt(t)
ys = yt(t)

After this, I could plot the curve by defining lines between subsequent points and sending the instructions to my plotter. Using the classes and functions I defined before, this was very easy:

objs = [shapes.Point(xs[0], ys[0])] +\
  [shapes.Line(
      shapes.Point(xs[i], ys[i]),
      shapes.Point(xs[i+1], ys[i+1])
   ) for i in range(len(xs)-1)]

s = wake_up_serial('/dev/tty.wchusbserial1410', 115200)
objs_to_plotter(objs, s)

Note that I started with a Point because the pen needs to move to the initial position before it can start drawing. If you look carefully, you can see that the figure was built up using small line segments. Increasing the number of steps in the np.linspace command above will make the result smoother. Nevertheless, I think the result looks very nice!

Conclusion

All in all, it was a nice journey. I never considered myself an artist, but now I kind of know what it feels like to be one, even though I did not design anything by hand. Someday, I want to learn Processing, a programming language designed specifically for creating computer generated art.