Bryan Brattlof

How to Proxy Git Connections

2022-04-02T00:00:00+00:00

One of the first things you will notice when you start developing open source software for a larger corporation is the company firewall or proxy blocking all outgoing traffic not routed through their network appliance for inspection and review first. This sadly includes all of our git fetch traffic.

Before we can do anything upstream we first need to configure Git to pull in updates through these corporate firewalls. This is where the multipurpose relay tool, socat, shines.

Proxy http(s) Remotes

Git uses the Smart HTTP protocol to download updates from a remote that starts with https://* URI. These type of connections have become, by far, the most popular way to use Git today and is the default protocol used by many popular forges like GitHub or GitLab.

Internally Git relies on the libcurl library to handle these HTTP connections which means Git comes with the ability to proxy these requests built-in.

It also means that Git will respect your system's http_proxy and https_proxy environment variables. Simply add something like this to your ~/.bashrc or execute these lines in your shell before you need to pull updates from your remote project and Git will proxy the requests through your corporate firewall appliance automatically.

$ grep 'https\?_proxy' ~/.bashrc
export http_proxy=http://mega.co.proxy.example.com:8080/
export https_proxy=http://mega.co.proxy.example.com:8080/

Alternatively, if you want to keep all your Git configurations in one place, you can add this to our ~/.gitconfig to configure Git's proxy settings:

[http]
    proxy = http://mega.co.proxy.example.com:8080

If you only need to proxy the repositories hosted outside of the company network, you can use a pattern like this to filter the domains that use the proxy configuration settings.

For example any repository hosted on kernel.org servers, use:

[http]
[http "https://git.kernel.org"]
    proxy = http://mega.co.proxy.example.com:8080

For more details on proxy-ing http based Git projects, check-out the --proxy option in the curl(1) man page, or the http.proxy entry in the git-config documentation.

Proxy Git Remotes

Another common way to fetch updates from a remote tree is by using their git://* URI which uses the Git Server Protocol. Often used to share projects that do not require any user authentication as it is the fastest of the three transfer protocols available.

To proxy these types of requests we will need Git to run a simple shell script, like the example below, any time we fetch a remote repository. This example script will proxy all repositories outside of the example.com domain.

$ cat ~/bin/gitproxy
#!/usr/bin/sh
if [ $1 == *example.com ]; then
    # the repository is outside the company network, proxy it
    socat tcp:mega.co.proxy.example.com:8080:$1:$2
else
    # no need to proxy internal network traffic
    socat tcp:$1:$2
fi

Then, to have Git run this script, we can set the gitProxy setting in our ~/.gitconfig like this:

[core]
    gitProxy = ~/bin/gitproxy

Because this is just a simple shell script, we can make this as complicated or as simple as we want. For example, you can add more logic if you use a laptop that may need work outside of the office.

Proxy SSH Remotes

Finally, if the remote starts with ssh://* or uses a typical ssh connection like git@git.sr.ht:* we are connecting via the SSH protocol. These connections are typically used when you maintain the remote repository or otherwise need to authenticate yourself before you can git push changes to the server.

In these situations, Git relies on SSH to handle the authentication and connection to the remote, which means we will need to edit our ~/.ssh/config file to proxy these types of connections.

Inside OpenSSH's configuration file, we can use the ProxyCommand option for all remote excluding any URI inside our company's network.

Host * !*.mega.co.example.com
    User git
    ProxyCommand socat tcp:mega.co.proxy.example.com:8080:%h:%p

Or we can manually add the remotes we want proxied. For example any remote at git.kernel.org:

Host git.kernel.org
    User git
    ProxyCommand socat tcp:mega.co.proxy.example.com:8080:%h:%p

For more information about OpenSSH's config file options, checkout the ssh_config(5) man page.

Wrapping Up

I hope these notes were of some use or helped you in some way. I recently accepted my first open source role working for a larger company (>10,000 people) and wanted to write these notes down in the hopes it would help others working in open source projects behind walled gardens.

As always if you see something that wasn't correct or I need to update or clarify, please feel free to send me a note any kind of way that is convenient to you.

~Bryan

My PGP Cheat Sheet

2021-11-02T00:00:00+00:00

I use gpg about as often as tar which is just enough time to forget how to use it.

What lies below are my (a Linux kernel contributor) notes on some of the less used commands to keep your gpg keys safe and your copy of the Linux Web of Trust up to date.

Please feel free to contact me if anything you read below is wrong, out of date, or find something I should expand on. :)

Helpful Jump To Points:

Why PGP?
Convert A PGP Key To An SSH Key
Locating A Public Key
Updating Your Keyring
Signing A Public Key
Extending An Expiration Date
Backing Up A GnuPG Directory

Why PGP?

Today, the Linux kernel development cycle is solely (for now) an email based work-flow involving pulling changes from over ~300 different repositories. [1] These changes will eventually make their way into Linus' main tree (the mainline) for the final approval before being distributed to the masses.

[1]	Not every repository needs to be pulled before each release - though the linux-next repository, the repository subsystem maintainers use to resolve merge conflicts before they reach Linus, tracks around 300.

With the existence of phishing, spoofing and other social-engineering attacks, any one of these pull requests could conceivably bring malicious code via a spoofed email, fooling either Linus or any one of the subsystem maintainers.

After the widespread credential stealing attack that compromised much of the core kernel.org infrastructure in 2011, the decision was made to develop a PGP based Web of Trust to give developers a way to independently verify other repositories without a central authority.

Now days, pull requests must have a signed tag from a PGP key the maintainer trusts not only to validate the pull request, but to attest to the changes being made. Like a chain of custody.

Convert A PGP Key To An SSH Key

Each PGP key can have one or more roles; Signing, Encryption, Authentication or Certification. One "neat" feature about authentication keys is they can be used as an OpenSSH key, allowing you to store your SSH and PGP keys together on the same NitroKey.

Any time you need to give out your public SSH key, just use this command:

$ gpg --export-ssh-key hello@bryanbrattlof.com
ssh-rsa AAAAB3NzaC1yc2E ...

Linode has a good writup on how to get started using your PGP key with SSH.

Locating A Public Key

Around June of 2019 [2], the public SKS Keyserver network used to publish public PGP keys was attacked in a way that could not be easily fixed. The attacker(s) attached thousands of valid but useless signatures to well known PGP keys to bloat the complexity of the Web of Trust graph and render gpg unusable for anyone unfortunate enough to download a "poisoned" key.

[2]	This was a known problem for more than a decade.

As a work-around, the Linux community published a repository of all the well known contributers to the kernel called pgpkeys.git, that we can use to send and receive updates to our PGP keys to the kernel community.

Before we can find a key, we need to clone the kernel's PGP keyring repository:

$ git clone https://git.kernel.org/pub/scm/docs/kernel/pgpkeys.git korg-keys

To find someone's PGP key, git grep for the key in the repository:

$ git grep -l torvalds *.asc
keys/79BE3E4300411886.asc

Then import the key into our keyring:

$ gpg --import keys/79BE3E4300411886.asc

Alternatively: we could import all keys currently in the repository:

$ gpg --import keys/*.asc

Done!

With your key imported, we will need a way to update these keys from the PGP repository. See the Updating Your Keyring section below.

Updating Your Keyring

After the public SKS keyserver network was abandoned and the Linux kernel developer's PGP keyring repository was created, any update to a kernel developer's PGP key should be uploaded to the repository so they can be shared with the community.

To submit changes to your keys, use this command to email your updates to the mailing list:

$ gpg -a --export [Email] | mail -s [Email] keys@linux.kernel.org

To download updates from the keyring repository, a helpful script was added, allowing us to download any new changes using cron, systemd, or invoking the script manually.

I prefer to let systemd run the script:

$ cat ~/.config/systemd/user/korg-refresh-keys.timer
[Timer]
OnCalendar=daily
Persistent=yes

[Install]
WantedBy=sockets.target

$ cat ~/.config/systemd/user/korg-refresh-keys.service
[Service]
ExecStart=%h/bin/korg-refresh-keys -q
Type=oneshot

So I can see how the script is doing at any time by running:

$ journalctl --user -fu korg-refresh-keys

Signing A Public Key

Once you feel confident signing someone's key or UID(s) and attesting to the validity of their PGP key, you can sign it with the following command:

$ gpg --ask-cert-level --ask-cert-expire --sign-key someone@example.com

Each situation is different, and everyone (and each community) has their own methods for signing someone's key. This is only the default arguments I use, and adapt to the situation.

`--ask-cert-level`
	Prompt for a certification level allowing you to specify how confident you are about this signature. Useful to signal the difference between the random person you met at a conference versus your workmate.
`--ask-cert-expire`
	Prompt for an expiration time, allowing your signature to expire after a set amount of time. I like to put a expiration date on my signatures when possible if only to stop an old email address having a valid signature from me.
`--sign-key <name>`
	Signs a public key with your secret certification key. This is a shortcut version of the subcommand "sign" from `--edit-key`.

If you're like me and store your certification key offline in an encrypted USB drive, (something I strongly encourage) this process will be a little more complicated.

Don't forget to update your backup after you finish.

Extending An Expiration Date

By default, master certification keys have an expiration date set two years from the date of their creation. This is for security reasons and to let obsolete keys disappear from the Web of Trust.

To add one year (from the current date) to the expiration, run:

$ gpg --quick-set expire hello@bryanbrattlof.com 1y

Make sure to backup your GnuPG directory as well as email the kernel developer's keyring mailing list to let everyone know about the changes you have made.

Backing Up A GnuPG Directory

Now that we live squarely in the hyper-connected information age, the more sensitive data we can store, encrypted and offline, the better. This includes keeping all the secret key material used in our PGP keys off our working computers.

How much effort you spend backing up your private keys is ultimately your decision. My goal is to have a good story to tell the mailing lists explaining how I lost control of my keys. I feel comfortable having two LUKS encrypted USB drives and a paperkey backup, storing all of my PGP keys, and allowing me to restore my Nitrokey if that device happens to die.

If you are like me and prefer to keep your main certification key safely stored in an encrypted drive and off our working computer, you will need to mount your encrypted device first:

$ mkdir /media/device/gnupg-backup
$ cryptsetup luksOpen [DEVICE] gnupg-backup-enc
$ mount /dev/mapper/gnupg-backup-enc /media/device/gnupg-backup

Then we can tell gpg to use it:

$ export GNUPGHOME=/media/device/gnupg-backup
$ gpg --list-secret-keys

You should now see that sec# has been replaced with sec indicating that both the public and the secret key material are available for your main certification key.

Now is a good time to make changes that require our main key like Extending An Expiration Date, Signing A Public Key or running commands like fsck on the decrypted volume.

Once you have finished making your changes, we need to import these changes back into our everyday working .gnupg directory:

$ gpg --export | gpg --homedir ~/.gnupg --import
$ unset GNUPGHOME
$ gpg --list-secret-keys

You should now see sec# and ssb> have returned.

Finally, unmount and close our encrypted volume, and return our USB drive to its safe place.

$ unmount /media/device/gnupg-backup
$ cryptsetup luksClose gnupg-backup-enc
$ rmdir /media/device/gnupg-backup

Wrapping Up

I will say gpg is not the easiest tool to use. After 30 years of development, the time and effort needed to maintain the Web of Trust seems to be more than many are willing to endure.

As difficult as it is, gpg is a great open source tool, helping developers from all around the world regardless timezone, language, or access to git-forges like Github the opportunity to work on one of the most widely used software projects in the world.

I hope these notes helped you in some way. If you read anything that needs to be updated or you feel like I should expand on something, please feel free to write me an email.

The Linux Kernel Coding Style

2021-05-28T00:00:00+00:00

Like the (very) old saying goes: "Beauty is in the eye of the beholder". This old (Greek maybe?) saying even holds true for the software we write. The Linux kernel, like any sufficiently large software project, with thousands of developers each having their own ideas of beautifully readable code, the issue of how the code looks eventually becomes a topic of discussion.

These coding style standards, while sometimes annoying, are essential to letting developers worry about developing rather than trying to decipher another developer's "beautifully" written code.

So for the 4^th challenge in The Eudyptula Challenge we'll return our focus on the Hello World Kernel Module we created in the 1^st challenge, bringing it up to the (charmingly unique) Linux kernel coding standard of what beautifully written C code looks like.

Enjoy!

Task No.4

Wonderful job in making it this far, I hope you have been having fun. Oh, you're getting bored, just booting and installing kernels? Well, time for some pedantic things to make you feel that those kernel builds are actually fun!

Part of the job of being a kernel developer is recognizing the proper Linux kernel coding style. The full description of this coding style can be found in the kernel itself, in the Documentation/CodingStyle file. I'd recommend going and reading that right now, it's pretty simple stuff, and something that you are going to need to know and understand. There is also a tool in the kernel source tree in the scripts/ directory called checkpatch.pl that can be used to test for adhering to the coding style rules, as kernel programmers are lazy and prefer to let scripts do their work for them...

And why a coding standard at all? That's because of your brain (yes, yours, not mine, remember, I'm just some dumb shell scripts). Once your brain learns the patterns, the information contained really starts to sink in better. So it's important that everyone follow the same standard so that the patterns become consistent. In other words, you want to make it really easy for other people to find the bugs in your code, and not be confused and distracted by the fact that you happen to prefer 5 spaces instead of tabs for indentation. Of course you would never prefer such a thing, I'd never accuse you of that, it was just an example, please forgive my impertinence!

Anyway, the tasks for this round all deal with the Linux kernel coding style. Attached to this message are two kernel modules that do not follow the proper Linux kernel coding style rules. Please fix both of them up, and send it back to me in such a way that does follow the rules.

What, you recognize one of these modules? Imagine that, perhaps I was right to accuse you of the using a "wrong" coding style :)

Yes, the logic in the second module is crazy, and probably wrong, but don't focus on that, just look at the patterns here, and fix up the coding style, do not remove lines of code.

Attachments:

hello_world.c - Note: This should be the module you submitted from the First Eudyptula Challenge. If you've been following along, this is the file we submitted.

coding_style.c

—Little Penguin

As the Little Penguin was saying, the Linux style guide is here not for your benefit, but to help other developers, maintainers, and reviewers understand and evaluate the code you write faster. The kernel works at a furious pace, receiving on average ~10 patches every hour, a pace that is only accelerating as more people and companies begin to depend on the project for the new device, radio, microwave or any other IoT gadget they're working on.

TLDR: The style guide helps everyone review your code which improves the chances of your patch being accepted.

Executive Summary of the Linux Kernel Coding Style

The style guide is a very straight forward document, the majority of it dealing with whitespace and naming things. (Something every project has to deal with regardless the number of developers it has) I'll list the "executive overview" here (for the SEO points 🙂) though you should really take a moment to skim the Linux coding style guide for the latest updates.

tabs are 8 characters, one statement per line. Historically tab keys were used to help tabulate charts on typewriters. Now they help maintainers keep track of your control blocks.

80 characters per line. An old throwback to when screens only displayed 80 characters per line. "Now-O-Days", with modern 48" 8k monitors, the 80 character hard limit is a bit softer. Today we can get away with a few lines under 100 characters if it improves readability. One possible exception to this is printk() statements, allowing us to easily grep for them when debugging.

brackets. Don't worry about brackets if you only need one line.

if (err)
        return -ENOMEM;

For multiple lines, place the bracket at the end of your if (switch, do, for, while, ...) statements.

if (err) {
        pr_dbg("Failed to allocate memory");
        return -ENOMEM;
}

Though function have brackets on the next new line.

int do_something_cool(void)
{
        return something_cool == DONE;
}

spaces. Most of the time, spaces go after each keyword. Some notable exceptions to this are sizeof, typeof, alignof, and __attribute__ which are treaded somewhat like functions in the kernel.
naming. Don't worry about descriptive variable names inside a function. tmp or err are perfectly acceptable. However global variables (to be used sparingly) should have descriptive names. Don't use MixedCase!
typedefs. Just Don't. Typedefs only hurt readability. You can't immediately tell if it's a struct or an integer of some specific size, or a pointer (to a pointer, to a pointer... to...).
commenting your code. If your explaining how the code works, you've written it wrong. Comments are for what your code does if the function's name didn't make this clear already.

These bullets take care of the majority of what a new kernel developer will deal with on a day-to-day basis, and if you're ever unsure about something you can always git grep <pattern> to see how others have coded similar things in the past.

scripts/checkpatch.pl

Fortunately, previous developers created a script to check for some of these style blunders and alleviate the burden put on maintainers to point out these minor coding style violations. This script is one of many helpful scripts inside the Linux source tree we downloaded in the second challenge. The one we're interested in and we will be using today is called scripts/checkpatch.pl.

At the root of the Linux source code, use the --help or --version options to see the usage instructions and get a complete list of all the options available.

$ ./scripts/checkpatch.pl --help

For this Eudyptula Challenge, we will remain focused on the --file or -f option to analyze the files the little penguin gave us (hello_world.c and coding_style.c) for style violations. I won't go into the specific changes we need to make (a task for you to complete). Instead I will go through how to use checkpatch.pl to find some of issues with the files the Little Penguin gave us.

We can check our file using the -f option followed by the file we want to check:

$ ./scripts/checkpatch.pl -f ~/eudyptupla/tasks/04/coding_style.c
...
total: 1 errors, 1 warnings, 1 checks, 35 lines checked

NOTE: For some of the reported defects, checkpatch may be able to
      mechanically convert to the typical style using --fix or --fix-inplace.

eudyptupla/tasks/04/coding_style.c has style problems, please review.

Then use you favorite editor to fix the issues checkpatch.pl printed out.

You can even use --strict or the --subjective option to enable more subjective tests and --codespell to check the file for misspelled words.

$ ./scripts/checkpatch.pl --strict --codespell -f ~/eudyptula ...

Once you've made your changes, and feel the code looks good (it's hard when you can remove any lines) use git format-patch HEAD~ and send in your changes.

Good Luck!

Wrapping Up

If you made it here, you may be surprised to see how little there was to this challenge. Essentially this "boils down" to reading the Linux style guide and practice using it. Though, so far in my software career, I have found that true for all coding standards.

PS: I made around 39 changes in all, though this may change depending on when you work on this. Hope that helps :)

Boston Parking Tickets

2021-02-09T00:00:00+00:00

Last year (mid December 2020) I created a Freedom of Information Act request for all parking tickets issued in Boston from 2011 to the end of 2020. Eventually I was given 40 CSV files that I've combined into a simple torrent you can download here:

boston-parking-tickets-2011-2020.tar.gz

Please feel free to send me an email if you don't wish to use BitTorrent, and I'll do my best to send you a copy using a different protocol.

These (very messy) files have data on every ticket, time and date it was issued, violation and fine total, how much was payed, the license plate number and state, including the car's make, style and color on every parking ticket. Also included is the hand-entered location of where the ticket was issued.

I say messy, because each ticket is manually entered on very small screens, often by parking attendants wearing gloves during winter while someone is telling them about their very bad day. Understandably there is a lot of typos and cleaning to do.

All of the code for these examples, along with the torrent file, is available in a git repository here. My goal is to implement various data cleaning techniques to see how well I can prepare this data for processing. What follows is the little bit of data expiration.

Tickets Issued in each Year

Boston police officers issued 13,023,114 parking tickets inside the Boston city limits between January 1^st 2011 and December 31^st 2020. If we exclude 2020 and its relaxed parking rules, Boston receives on average 1,367,606 (±51,848) parking tickets each year.

Surprisingly there wasn't a significant change in the number of tickets issued during each year even as Boston's population continues to grow.

I had assumed this would correlate with Boston's population growth, however a simple linear fit shows there are 3,358 fewer tickets being issued each year, well within the margin of error of the 1.4 million tickets issued on average.

Tickets Issued in each Month

Another interesting thing I noticed (for this southerner) if we plot the number of tickets issued by month, we can clearly see a dip in tickets during the winter months.

I have no evidence to support this, however, I assume the snow covered streets of Boston during the winter reduces the available metered parking spaces, or removes the 20,000 people willing to park their car when snow plows are actively roaming.

On average there is a decline of 20,000 tickets during the winter months. If we exclude 2020 again (shown here as red dots), there is on average 117,000 tickets issued each month.

Tickets Issued by Day of Month

Drilling further into when tickets are issued, if we look at each day in a month each ticket is issued (excluding the 31^st day of the 7 months that have 31 days) we can see a pretty steady ticketing rate. Again the red dots represent data from 2020 and are not included in the white shaded standard deviation range.

We can also clearly see the 572 days with under 1,000 tickets issued. As we'll see in the next section, this is mostly due (80% of the 572 day) to the relaxed parking rules on Sundays when most parking meters are turned off.

Tickets Issued by Day of Week

Like I was saying in the last section, parking on Sundays and City holidays is free. When we split the tickets issued by day of week we can see just how great this policy is for people who fail to feed their meters.

It's also interesting to see the reduction (roughly 1,000 on average) in tickets issued on Mondays. As of right now I don't have a good explanation as what could cause this.

Tickets by Violation

When we see what type of violations people are breaking on average each year, we can start to see why Sundays and City holidays have such a huge impact on the number of tickets issued each day.

First place with 25.7% or 3,348,515 of all tickets issued was from unpaid parking meters, most of which are disabled on Sundays.

Followed by an ever shrinking list of significantly less common violations.

The To-Do List

There is still a large amount of cleaning work I would like to do in the future. There is currently many misspelled states, vehicle makes and models, ticket locations or cross streets indicated with different symbols all of which makes classifying this data a fun and difficult task.

As of right now though, I'll publish this dataset with the promise to see what insights we can gleam from it in the future.

Cgit, Nginx & Gitolite: A Personal Git Server

2021-01-12T00:00:00+00:00

As I (and my responsibilities) continue to grow older, I find myself having less and less time to do the things I used to (write and maintain servers just for fun). And while I continue to want to do these things, I have to be realistic and prioritize the things I wish to do with the ever decreasing time I have to do them.

All of that to say, I've "sold-out" and transitioned to paying someone else (sourcehut) to worry about maintaining my publicly accessable repositories for me and allowing me to prioritize other exciting and novel things.

I've been on a "own my online presence" kick for more than a year now. So for this (overly protracted) essay, I thought I'd publish my notes on how I created my own Git server.

There are many open source projects like GitTea or GitLab to make hosting your own git projects effortless; however I wanted a much more simple (read: old school) setup. I ended up with something that uses many of the same projects that the Linux Organization uses to publish the Linux Kernel

The server (as of this writing) uses Ubuntu's 20.04.1 LTS (Focal Fossa) running on Digital Ocean's hardware (referral-link). I wholeheartedly support and recommend you chose a different setup. Diversity in people and in tech stack is always and will always be a great thing.

What lies below can be broken into 3 main topics:

The Start prepares a newly minted server for git hosting duties. Creating a new admin user, locking down the OpenSSH daemon, and installing fail2ban.
Gitolite installs and configures the server to allow us (and colleagues) to have more fine-grained control over who has access to git push|fetch on the server.
And Cgit, fcgiwrap, and Nginx to create a web-server to view our published projects.

In the end, you'll have a server much like this one.

Enjoy!

The Start

I often find security "best practices" are a lot like driving down the highway. Some people speeding past you are "obviously" just moments away from a major data breach, while the others you're passing are "clearly" so worried about the entire data-center burning down, they couldn't possibly get anything else done. Everyone thinks everyone else has lost their marbles.

So with that in mind, here are a few steps I took to secure my newly minted server. Please feel free to use only the "best practices" you deem appropriate for your mission.

Or just skip to the "installing Gitolite" part directly.

Admin User

For whatever reason, be it for security or protecting the server from my stupidity, one of the first things I do when creating a new server is add a new user for my general admin tasks.

Adding a new user is remarkably easy to do on a Ubuntu system:

$ adduser limb

You'll be prompted to answer a few questions, including creating a new UNIX password. This will be the password you'll need to sudo -i and gain root permissions, so make it a good one, or use tools like Pass, or BitWarden to help you remember.

Then give our new limb user sudo permissions:

$ usermod -G sudo limb

I've also largely eliminated all password based authentication when signing into servers, relying on open source smart cards like NitroKey for authentication. If interested, this requires we setup .ssh/authorized_keys for our limb user:

Just replace key with your public ssh key:

$ mkdir /home/limb/.ssh
$ echo "key" > /home/limb/.ssh/authorized_keys

Next, set the .ssh directory's file permissions so the ssh daemon can read the files:

$ chown -R limb:limb /home/limb/.ssh
$ chmod 700 /home/limb/.ssh
$ chmod 644 /home/limb/.ssh/authorized_keys

If everything worked, after you restart the ssh daemon (service sshd restart) you will now be able to login as the administrator user:

$ ssh limb@host

OpenSSH

Git and Gitolite (installed in the next sections) will need us to keep port 22 open, allowing us to git push from anywhere on the internet. This open port will eventually attract "a lot" of attention from bots who endlessly scour the internet looking for vulnerable servers, mindlessly stuffing passwords, hoping one password will eventually let them in.

We can eliminate all worry about weak or compromised passwords by disabling all password based authentication, relying solely on asymmetric cryptography, or "ssh keys". Just use your favorite text editor to open /etc/ssh/sshd_config and ensure these lines exist somewhere in it:

PubkeyAuthentication yes
PasswordAuthentication no

While we're here, a large majority [1] of these bots are interested in logging in as the root user. If you created a new admin account in the previous section and ensured you can login using your public key, you can also disable root logins entirely with this line in the config:

PermitRootLogin no

[1]	Some simple "bash-fu" on my `/var/log/auth.log` shows ~93.58% of the roughly 15,000 login attempts since I started this server, tried to login as `root` Second place was the user `git` (including legitimate logins) at ~1.82%.

If you uploaded your public key to your VPS provider, most of these changes should have already been configured for you. But in the off chance you had to make some changes, restart the ssh service to load the new config changes in:

$ service ssh restart

Uncomplicated FireWall

Depending on your VPS provider, they may also have a firewall system built into their admin panel allowing you to apply rules simply by adding tags to a server. However, I enjoy keeping all my firewall rules inside each box, if only for the same reason I keep all my socks on the left hand drawer, so everything stays organized and in the same place.

You can install ufw using the Advanced Packaging Tool:

$ apt install ufw

Right now, the only thing we have enabled is ssh which uses port 22. To allow port 22 through ufw just use the following command:

$ ufw allow ssh

and then turn the firewall on:

$ ufw enable

and viola! You have a firewall.

fail2ban

Even though we've turned off password based authentication in a previous section, we will still receive a significant amount of bots wasting our compute cycles trying to login. And while the likelihood of this being successful is zero when rounded to any order of magnitude, the bots will nevertheless continue to pilfer a non-zero amount of CPU if given the opportunity.

To stop the most brazen of these bots, tools like Fail2Ban, which creates temporary firewall rules to block IP address who repeatedly fail to authenticate with ssh, are a great compromise between usefulness and annoyance.

The Advanced Packaging Tool can again help us install fail2ban.

$ apt install fail2ban

Once installed, the ssh "jail" will come pre-enabled for you. If you wish to make any changes, you will need to make a copy of the fail2ban config file:

$ cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Then add your changes to jail.local so they will persist after an upgrade.

fail2ban does a great job documenting what each option does in the config file. Some of the changes I made are:

because I use ufw to manage my firewall, I changed banaction = ufw.
enabled bantime.increment to increase the duration of a ban based on how many times the IP address has been banned previously.
enabled bantime.rndtime to "randomize" the length of a ban, preventing bots from knowing exactly when they can resume their assault.
enabled bantime.maxtime so I won't need to unban IP addresses (if you're unfortunate enough to share an IP with a bot).
lowered bantime, findtime and maxretry allowing me to issue small bans that increase in severity as the IP address continues to antagonize.

Once you're satisfied with your changes, start fail2ban using systemd.

$ systemctl enable fail2ban
$ systemctl start fail2ban

And Done!

Depending on how "popular" you are on the internet, you should start to see NOTICE lines in /var/log/fail2ban.log of misbehaving bots and the equivalent firewall rules in ufw.

$ cat /var/log/fail2ban.log | grep 'NOTICE' | tail -1
...  [125553]: NOTICE  [sshd] Ban 156.155.159.161

$ ufw status
Status: active

To                         Action      From
--                         ------      ----
Anywhere                   REJECT      156.155.159.161
22/tcp                     ALLOW       Anywhere
22/tcp (v6)                ALLOW       Anywhere (v6)

Gitolite

Installing Gitolite is amazingly simple, there are no binaries to compile or daemons to monitor.

At its core, Gitolite is just a collection of Perl scripts that run after someone signs into the server using the ssh daemon we configured in the previous sections. Once installed, Gitolite will give us more fine-grained-control over who has git push|fetch permissions to each repository. I encourage you to checkout Gitolite's amazing documentation if only to see how capable Gitolite can be.

Step: 1 - Create The Git User

Before we install Gitolite, we'll need to create a new user for everyone to log into and to run Gitolite's Perl scripts. I typically use the username git for this, feel free to replace git with the username that you feel is more appropriate.

$ adduser --system --group --disabled-password --home /var/lib/git git

This creates a new system-user on the server called git. Because this is not a "normal" user, there will be no aging information in /etc/shadow, which is convenient when nobody will be monitoring this account.

We also used the --home option to set the $HOME variable to /var/lib/git. This is where we will eventually put Gitolite's configuration files and our Git repositories. Feel free to adjust this to where you prefer, I've seen many use /home/git.

I included the --disabled-password to disable any password based access into our new user. In the previous sections, we've disabled all password based authentication into the server and Gitolite requires ssh keys for authentication, so disabling passwords for our user is a smart move.

Step: 2 - Install Gitolite

Because Gitolite is just a bunch of Perl scripts, I prefer to install Gitolite from the source. As we will see, installing Gitolite from source also has the benefit of making upgrades and adding custom patches in the future extremely easy.

This also means we'll need to install Gitolite's dependencies ourselves:

$ apt install perl git

When we (or a colleague) signs into the server, using the git user, we will automatically run Gitolite's Perl scripts, which means these scripts must be executable by our git user. So, to make managing file permissions easier, we'll use our git user for the rest of the installation process.

Log into our git user with the "substitute user" command: (assuming you're the root user)

$ su - git

Then clone Gitolite's source code into the $HOME directory: (this should be /var/lib/git unless you changed it when we setup the git user above)

$ git clone https://github.com/sitaramc/gitolite

Step: 3 - Setup Gitolite

To setup Gitolite on the server, we'll need to assign Gitolite an admin that will have full control over editing Gitolite's configuration repository. This will most likely be you.

Still as the git user, use your favorite text editor to create a new file with your desired username in the $HOME directory and copy your public ssh key into it. For example, my file would be called bryanbrattlof.pub and look like this:

$ cat $HOME/bryanbrattlof.pub
ssh-ed25519 AAAAC3NzaC1l ...

Then run Gitolite's sanity checks and setup script to finish the installation:

$ $HOME/gitolite setup -pk bryanbrattlof.pub

Färdig!

If everything went well, you should now be able to clone Gitolite's configuration repository:

$ git clone git@host:gitolite-admin

I recommend consulting Gitolite's incredible documentation to understand how to properly configure access and add hooks to all of your projects.

Step: 4 - Add ~/.profile

While not technically needed for Gitolite to function properly, I find adding gitolite to our git user's $PATH is a great quality of life improvement on the rare days I need to play system administrator.

First, as our git user, create a new $HOME/bin directory:

$ mkdir -p $HOME/bin

Next, create the $HOME/.profile file and add the $HOME/bin directory to our $PATH.

PATH=$HOME/bin:$PATH

The born again shell will automatically run $HOME/.profile when someone starts a new session.

Now, we can use Gitolite's install script to add a symbolic link inside our $HOME/bin folder:

$ gitolite/install -ln $HOME/bin

Done!

Logout and back in to the git user (or use source $HOME/.profile) to pick up the changes. If everything was done correctly, you won't need to type the full path to Gitolite anymore.

$ whereis gitolite
gitolite: /var/lib/git/bin/gitolite

Cgit

Cgit is a script (written in C) that uses the Common Gateway Interface (CGI) specification to give people a web view of our projects. Convenient when you don't have access to your terminal or just want to lookup (or showoff) some changes to a project.

It operates as a back-end (much like PHP) to a webserver (we'll install Nginx in the next sections) that will parse our repositories and return a web-page for our webserver to distribute.

To get an idea for what Cgit will look like, some of the more popular projects that use Cgit are the Linux and FreeBSD kernels, along with Wireguard and Cgit itself.

Step: 1 - Install Cgit

Just like with Gitolite, I prefer to install Cgit from source so I can add personal patches and quickly change what version is running on the server. This also means we'll need to install the dependencies ourselves:

$ apt install libc6 liblua5.1-0 zlib1g \
  python3-docutils python3-markdown python3-pygments

We'll also need the build-essential packages to install the gcc and make tools needed to compile Cgit after we've cloned the project:

$ apt install build-essential

Then, as the root user in the /root directory clone the Cgit project:

$ git clone https://git.zx2c4.com/cgit

Because Cgit uses parts of Git's source code, (included as a submodule) we'll need to use git submodule to download the remaining code from the Git project.

After you cd into cgit

$ git submodule init    # register the git submodule in .git/config
$ git submodule update  # clone/fetch and checkout correct git version

Step: 2 - Build Cgit

With a full copy of Cgit on the server, we can now create some patches to customize it for our use-case. We'll start with creating cgit.conf inside the cgit project we just cloned, to tell make where we want to install the Cgit binaries.

CGIT_SCRIPT_PATH = /var/www/html/cgit/cgi
CGIT_CONFIG = /var/www/html/cgit/cgitrc
CACHE_ROOT = /var/www/html/cgit/cache
prefix = /var/www/html/cgit
libdir = $(prefix)
filterdir = $(libdir)/filters

Because this is a version controlled project, we can commit our changes to save our work:

$ git add -f cgit.conf
$ git commit -m "installation path changes"

Some additional changes I made:

Updated the cgit.png and favicon.ico icons
Changed the pygments highlighting style to "algol_nu"
Removed the Data URI icons from the tab menu
Limited the max-width of readme pages to 95ch
Added padding: 1em; to code-blocks

When you're satisfied with your changes, use make to compile and install Cgit:

$ make && make install

If everything went well, when you execute Cgit from the terminal, a web-page should print out:

$ ./cgit
Content-Type: text/html; charset=UTF-8
Last-Modified: Tue, 12 Jan 2021 22:35:43 GMT
Expires: Tue, 12 Jan 2021 22:40:43 GMT

<!DOCTYPE html>

And Done!

We can further customize Cgit's behavior using the cgitrc file located at /var/www/html/cgit/cgitrc. Feel free to check out the man page for a complete description of what every option does.

Some of the options I used:

set scan-path to the location of our repositories /var/lib/git/repositories
set project-list to the location of the projects.list file Gitolite creates, adding descriptions and categories to the list of repositories on Cgit's index page

FastCGI Wrapper

Cgit, which uses code from Git, was designed to let users run a command (eg: git push) then exit, allowing our computers to reclaim the used resources between each call. Nginx uses a faster protocol (FastCGI) which calls the same program multiple times without exiting.

However because Cgit was designed to exit after every run, it will never give back its used resources and will continue to take more, quickly exhausting all of the computer's available resources. This is why we need fcgiwrap.

Thankfully this is easy to install. The Advanced Packaging Tool can, once again, help:

$ apt install fcgiwrap

Then just enable and start the service:

$ systemctl enable fcgiwrap
$ systemctl start fcgiwrap

And that's a wrap!

Nginx

With Cgit and the FastCGI Wrapper installed, we can now turn our attentions to Nginx, which can be installed using the Advanced Packaging Tool:

$ apt install nginx

Next, create a new configuration file in the /etc/nginx/sites-enabled directory, replacing git.bryanbrattlof.com with your domain. The minimum configuration file you'll need for Cgit to work will look something like this:

server {
    server_name  git.bryanbrattlof.com;

    listen [::]:80;
    listen 80;

    access_log  /var/log/nginx/cgit-access.log;
    error_log   /var/log/nginx/cgit-error.log;

    root /var/www/html/cgit/cgi;
    try_files $uri @cgit;

    location @cgit {
        include          fastcgi_params;
        fastcgi_param    SCRIPT_FILENAME /var/www/html/cgit/cgi/cgit.cgi;
        fastcgi_pass     unix:/run/fcgiwrap.socket;

        fastcgi_param    PATH_INFO    $uri;
        fastcgi_param    QUERY_STRING $args;
        fastcgi_param    HTTP_HOST    $server_name;
    }
}

Feel free to add more to this, I've added a custom 5xx page, caching headers, as well as recommendations from Mozilla Observatory.

Once satisfied, start the Nginx service and open port 80 in the firewall:

$ service nginx start
$ ufw allow http

If something went wrong, or if you ever change the configuration file, you can use nginx -t to check the configuration for errors and nginx -s reload to restart the Nginx server.

$ nginx -t && nginx -s reload

Wrapping Up

By now we should have a working git server. However like most creative things "90% done ... 90% left to go." There is truly an endless supply of things you can and should add or configure to make your server more secure and accessible. If you're the type that likes to learn, then you'll likely find this as fun and rewarding experience as I did.

Some of the extra things I added:

Installed certbot to install and manage a free SSL certificate from Let's Encrypt. This has largely been mandatory for any public server for around 5 years now
Created a Borg based backup script with a Borg specific subscription to rsync.net to backup my projects. Useful when that jackass we talked about finds a 0-day
Created a Git Daemon service to allow people to clone my projects using the git:// protocol, if they prefer
Placed a bunch of Healthchecks Pings in the scripts and service required to keep everything running. Fail2Ban, Borg Backup, Certbot, all will alert me when cron or systemd fall over

All of which should be given their own essay as they can be used in all your setups, not just in this fully open-source and free (as in libre) git server.

The Great Influenza

2020-12-07T00:00:00+00:00

When living in New Orleans (Louisiana) one of the many "iconic" things visiting guests would request to see (before pandemics existed of course) was the above ground cemeteries. I was regularly surprised during each visit by both how heavily touristed New Orleans' cemeteries are, and how much death the 1918 influenza season brought to the city.

New Orleans' hot and humid swamps had always brought disease to the historically significant port city. In 1853 alone an estimated 8,000 people died from Yellow Fever. And while 8,000 is a shockingly high number, there is something different to seeing the aisles of 3,489 vaults (roughly 1% of New Orleans' population in 1918) for the people who died from influenza. It's much like the real life version of "what if all C@\/!$-19 deaths happened in your neighborhood".

That experience together with reading John M. Barry's book The Great Influenza it's easy to see how humbling a pandemic can be. Even today, without a vaccine available, the best advice doctors can give is little more than wear a mask and avoid other people, the same advice given by doctors in 1918. It would take scientists another 15 years, until 1933 to discover that influenza was caused by a virus rather than a bacterium they has believed during the 1918 pandemic. Today, even with the amazing advances with mRNA, and the many other tools 100 years of research has given us, science is still much slower than we wish.

Though, after reading the book that inspired George W. Bush and his Health Secretary, Mike Leavitt, to work on and publish a multi-billion dollar pandemic-preparation bill, giving us a playbook on how to navigate today's pandemic, it's clear how capable we are when we collectively focus on a goal (and history).

Doctors, as Barry describes, who where just recently expected to have what we today would think of as a traditional medical school education, thanks to the amazing work of schools like Johns Hopkins University, and The Rockefeller University founded decades before the pandemic in 1918, showed up day after day and helped the millions of people too sick to feed themselves all while making huge leaps in scientific discovery, that would lay the groundwork for what we rely on during the current pandemic and expect in modern medical universities.

It was truly amazing (in a morbid way) to read the parallels between the 1918 influenza season play out in the book while watching the news today. And it was amazing to see how determined the first responders of the "lost generation" reacted to major cities all but consumed by the overwhelming death brought by influenza. One nurse was quoted in the book as saying:

She remembered that at the peak of the epidemic the nurses wrapped more than one living patient in winding sheets and put toe tags on the boys’ left big toe. It saved time, and the nurses were utterly exhausted. The toe tags were shipping tags, listing the sailor’s name, rank, and hometown. She remembered bodies “stacked in the morgue from floor to ceiling like cord wood.” In her nightmares she wondered “what it would feel like to be that boy who was at the bottom of the pile" ...

Only to witness people today say almost the same thing about our current pandemic.

At first I was a little uneasy starting this book in the middle of a pandemic and presidential election. I thought that the parallels in the news of relearning what we already knew 100 years ago would be too depressing for me to read. And while I learned a great deal about the beginning of "modern medical schools", the "Captain America" levels of bravery and sacrifice fighting a world war during the deadliest influenza season in history, the greatest take-a-way I had from this read was how capable we humans are when we put our minds to a goal, both good and bad.

Calling Bullshit

2020-10-30T00:00:00+00:00

Turn on the news, log into social media, or just start talking to someone at the store and you're sure to run into it.

Complete and utter bullshit.

Some of it is easy to spot: The earth is flat, Area 51 is experimenting on aliens, airlines are spraying chemicals on us, the loch ness monster or bigfoot exists, Mattress Firms are laundering money, Alaska is trying to control everyone's minds, and the list goes on … for a depressingly long time.

Other times, bullshit can be tricky. Rampant food-stamps fraud, kills 99.9% of germs, chocolate as a weight loss supplement, rap music is deadly. Outside of scientific or journalistic circles, you would be forgiven for believing some of these claims or falling for their bullshit.

And that's exactly what Carl Bergstrom's and Jevin West's new book, Calling Bullshit, is trying to fix. Carl Bergstrom is a professor of Biology at the University of Washington who studies how information flows through scholarly circles. Jevin West is a data scientist and associate professor at the University of Washington who focuses on misinformation in science and society. Together they created a fun and relevant "how-to" book on calling out the bullshit we run into in our everyday lives.

In their book, Carl and Jeven argue, rather than spending weeks into fact checking every claim you hear, all you need to "call bullshit" on the vast majority of these false claims is to remain skeptical, use basic logic, and occasionally a quick web search. Even the imitating bullshit that hides behind a wall of statistics uses the same basic fallacies that gives all bullshit away, we just need practice spotting it. The best part of Carl's and Jeven's techniques is you don't have to be a data scientist, professional statistician or math prodigy to use them. And with the rise of the attention economy this book couldn't have come at a better time.

This book was fun, timely (especially for the upcoming US election), and enlightening. The book walks through case studies ranging from debunking food-stamp fraud to papers claiming to have built AI to detect criminality. Guiding us through how to spot the various forms of "new-school", data saturated bullshit we see everywhere today, from misleading visuals, equating correlation and causation, unrepresentative data, selection biases, problems with small sample sizes, and more. They even setup a website to help us practice spotting bullshit, like the fake images generated with cutting edge machine learning tools.

Whatever we call bullshit, whether it fools us or not, lies, spin, fake news, conspiracy theories and bullshit is everywhere. It's been with us forever and will likely remain when it all ends. The reason it's so prevalent is fairly simple. As an Italian programmer famously tweeted:

The bullshit asymmetry: the amount of energy needed to refute bullshit is an order of magnitude bigger than to produce it.

—Alberto Brandolini (@ziobrando)

I really enjoyed this book. Even if you're not into math and statistics, this book does an amazing job imparting the skills needed to spot logical fallacies using relevant examples and a ton of humor, which makes the book such a fun read that I would recommend to everyone.

Adding OpenStreetMaps To Matplotlib

2020-10-21T00:00:00+00:00

OpenStreetMap uses expensive hardware amazingly donated by community members. Recently they've started to block requests that abuse this hardware which, unfortunately, includes this script.

As someone who enjoys hosting publicly accessible computers on the internet, I completly understand this position and wish to honnor it. Therefore these scripts will remain broken. You are welcome to fix them but I encourage you to read the tile server usage policy first to ensure you do not abuse this wonderful asset.

Adding a map to your visuals is a great way to quickly understand the geographic information you're trying to investigate. Thankfully there are quite a few packages and libraries (like geopandas, cartopy, smopy, folium, tilemapbase, or ipyleaflet) that can make creating these visuals fairly straightforward and easy in your jupyter notebooks or whatever stack you're using.

For this essay though, I'll walk through the process of adding a base-map from OpenStreetMap to you're matplotlib visuals without using any of these libraries. In the end, we'll have a visual much like this (very messy) scatter-plot of buses as they service route 16 in New Orleans.

How it Works

The ability to add our base-map to our matplotlib visuals relies on matplotlib's imshow() function, which internally uses the Pillow library to display images, or any other two dimensional scalar data we want (like numpy arrays).

For example, if we download my self-portrait, we can add the image to a plot using code like this:

from matplotlib import pyplot as plt

img = plt.imread('path/to/my/self-portrait.png')
plt.imshow(img)

plt.show()

Resulting in a matplotlib visual that looks like this:

Creating the Map

The easiest way to generate the base-map for plt.imshow() is to use a mapping service. These mapping services use enormous amounts of raw computing power to take the terabytes of map data and render a map for us. Today there are quite a few services available online. The one I enjoy working with (and the one we will be using in this essay) is a free, community maintained service called OpenStreetMap.

Tile Servers

To make updating and sharing their work easier, OpenStreetMap (and virtually all other mapping services) have split their maps into billions of tiny (256 pixel) sections, called tiles, that we can download individually from their tile servers.

OpenStreetMap has quite a few tile servers that style or prioritize different map features with some having a slightly different API to request tiles. For example the Stamen Toner Map that I used in the first visual and prefer for it's simple color pallet.

For this essay though, we'll use the default tile server's API to request tiles:

URL = "https://tile.openstreetmap.org/{z}/{x}/{y}.png".format

This string formatting function will replace the {z}, {x}, and {y} with the tile coordinates and zoom level of the tile we want to download, where:

{z} is the "zoom" level ranging from 0 to 18. Zoom 0 being the most "zoomed out" and needs only one tile to depict the entire world at that level.
{x} is the number of tiles from the left most tile of the map.
and {y} is the number of tiles from the top most tile of the map.

Both {x} and {y} depend on the zoom level {z} we've chosen, with larger zoom levels requiring more tiles to render the map. To understand how to calculate which tiles we need for our data-set, we'll need to understand how mapping projections work.

Map Projections

Without going too deep into mapping projections, OpenStreetMap (along with many other mapping services) needed a way to convert (lat, lon) coordinates into planer (x, y) coordinates which work with their maps. Sadly there is no perfect way to do this.

Google (and everyone else eventually) settled on a variant of the Mercator Projection called the Web Mercator Projection which simplifies the conversion by assuming the earth is a perfect sphere (it's not). This can (and does) lead to confusion in the final visuals and why many official bodies refuse to accept this standard.

The advantage of assuming the earth is a perfect sphere is that the equation to convert our GPS coordinates into Web Mercator coordinates is fairly straightforward. The OpenStreetMap Wiki has the algorithm available in multiple programming languages. Here is the one for Python:

import math
TILE_SIZE = 256

def point_to_pixels(lon, lat, zoom):
    """convert gps coordinates to web mercator"""
    r = math.pow(2, zoom) * TILE_SIZE
    lat = math.radians(lat)

    x = int((lon + 180.0) / 360.0 * r)
    y = int((1.0 - math.log(math.tan(lat) + (1.0 / math.cos(lat))) / math.pi) / 2.0 * r)

    return x, y

Downloading A Tile

Now we can use the point_to_pixels() function to calculate the number of pixels from the top-left corner of the OSM map from the GPS coordinates in our data-set at any zoom level, for example the French Quarter of New Orleans:

zoom = 16
x, y = point_to_pixels(-90.064279, 29.95863, zoom)

Dividing the number of pixels by TILE_SIZE will then give us the {x} and {y} that we need for the URL() function we created a few sections ago for the OpenStreetMap API.

x_tiles, y_tiles = int(x / TILE_SIZE), int(y / TILE_SIZE)

That we can then use, along with the requests and Pillow libraries, to download a tile from the OpenStreetMap tile servers:

from io import BytesIO
from PIL import Image
import requests

# format the url
url = URL(x=x_tiles, y=y_tiles, z=zoom)

# make the request
with requests.get(url) as resp:
    resp.raise_for_status() # just in case
    img = Image.open(BytesIO(resp.content))

# plot the tile
plt.imshow(img)
plt.show()

Producing a tile of Jackson Square in the French Quarter of New Orleans:

Stitching Tiles Together

To download all the tiles needed for our visual, we'll need to calculate the limits of the data we'll be using in our visual. There are many ways we can do this, all of them are valid. For simplicity though, I'll calculate the min and max of both the lat and lon columns in my pandas DataFrame:

top, bot = df.lat.max(), df.lat.min()
lef, rgt = df.lon.min(), df.lon.max()

This gives us a bounding box (in GPS coordinates) that encompasses our entire data-set.

Next, just like we did in the last section, we'll use the point_to_pixels() function to convert our GPS coordinates into Web Mercator coordinates.

zoom = 13
x0, y0 = point_to_pixels(lef, top, zoom)
x1, y1 = point_to_pixels(rgt, bot, zoom)

That we can then divide by TILE_SIZE to calculate the minimum and maximum number of tiles we'll need to download for both the {x} and {y} arguments for the API:

x0_tile, y0_tile = int(x0 / TILE_SIZE), int(y0 / TILE_SIZE)
x1_tile, y1_tile = math.ceil(x1 / TILE_SIZE), math.ceil(y1 / TILE_SIZE)

As a precaution, we'll add an assert statement to limit the number of tiles we can download and save us from the embarrassment of accidentally burdening OpenStreetMap tile servers.

assert (x1_tile - x0_tile) * (y1_tile - y0_tile) < 50, "That's too many tiles!"

Now that we've calculated which tiles we need to download from OpenStreetMap, we can use the built-in itertools product() function to loop through every tile, downloading and saving the tiles to a single large pillow image using Pillow's paste() function:

from itertools import product

# full size image we'll add tiles to
img = Image.new('RGB', (
    (x1_tile - x0_tile) * TILE_SIZE,
    (y1_tile - y0_tile) * TILE_SIZE))

# loop through every tile inside our bounded box
for x_tile, y_tile in product(range(x0_tile, x1_tile), range(y0_tile, y1_tile)):
    with requests.get(URL(x=x_tile, y=y_tile, z=zoom)) as resp:
        resp.raise_for_status() # just in case
        tile_img = Image.open(BytesIO(resp.content))

    # add each tile to the full size image
    img.paste(
        im=tile_img,
        box=((x_tile - x0_tile) * TILE_SIZE, (y_tile - y0_tile) * TILE_SIZE))

plt.imshow(img)
plt.show()

Resulting in a plot like this:

The eagle-eyed among us will notice the image is too large for the visual we want to create. This is because of the math.ceil() and int() functions we used to round the pixel coordinates into {x} and {y} tiles we used above. To get our image back to size we'll need to crop out the fractions of tiles not inside our bounding box.

Cropping the Basemap

To help my human-eyed brethren, I added some lines to our previous graphic to help understand what's going on. Essentially some fraction of each tile we've downloaded (outlined in black) will be used in our final visual (outlined in red) that we calculated in the last section. Our goal for this section is to trim the fraction of tiles outside of our red square.

To curtail our oversize image, we'll use pillow's Image.crop() function, which takes a tuple (left, top, right, bottom) measured in pixels from the top left corner to crop our image.

From our work above, we know the pixel coordinates of the red square is defined as x0, y0 and x1, y1. We can then multiply the tile coordinates x0_tile, y0_tile by TILE_SIZE to find the pixel coordinates for the top-left corner of the current (oversize) basemap:

x, y = x0_tile * TILE_SIZE, y0_tile * TILE_SIZE

Now it's just a simple process of subtracting the edges of our red square from the pixel coordinates we just calculated for our oversize image to crop it to our desired size:

img = img.crop((
    int(x - x0),  # left
    int(y - y0),  # top
    int(x - x1),  # right
    int(y - y1))) # bottom

plt.imshow(img)
plt.show()

Resulting in our final (properly sized) basemap for our visual:

Plotting The Data

Finally, with our basemap created, we can plot our data just like any other visual with some key exceptions. We can start by setting a matplotlib subplots() and a scatter() plot for the lat and lon columns in our pandas DataFrames:

fig, ax = plt.subplots()
ax.scatter(df.lon, df.lat, alpha=0.1, c='red', s=1)

Then we'll add an extra argument to the imshow() function to properly locate our image in the final visual. The extent argument is used to move a image to a particular region in dataspace.

ax.imshow(img, extent=(lef, rgt, bot, top))

Next, we'll lock down the x and y axes to the limits we defined a few sections ago by using the set_ylim() and set_xlim() functions.

ax.set_ylim(bot, top)
ax.set_xlim(lef, rgt)

All of this work will produce a simple graphic with a (gorgeous) basemap of buses servicing New Orleans' Route 16.

Tribe

2020-09-08T00:00:00+00:00

We are currently living in the most connected time in human history. Today, half of the world's population has some type of access to the internet and the limitless information it contains. Together with the advent of social media, geography is no longer a limitation to finding and belonging to a community, allowing the most obscure groups to exist and thrive. In short, with the internet, it has never been easier to belong.

Surprisingly though, we are also living in the loneliest time in human history. An online study of 20,000 Americans above the age of 18 found that 43% of respondents feel they lack companionship, 43% feel their relationships are meaningless, 43% feel isolated from others, and 39% are no longer close to anyone. The study went on to find that, how frequently people have meaningful face-to-face interactions and how lonely they felt had the highest correlation, and contrary to popular belief, they found no correlation between how lonely you feel and how often you use social media. This suggests that while social media is a great benefit to the world, it cannot replace the benefits we get from belonging to an (IRL) tribe.

Sebastian Junger's book Tribe is a (thin, 138 page) thought-provoking book that blends anthropology and psychology to discuss the obstacles military veterans and other people returning from war-torn regions and other high stress environments face when returning home. Junger argues that the tight knit bonds formed in these intense environments is what we have historically depended on to survive and thrive as a species since the stone-age and what our, increasingly individualistic, society needs to encourage in order to reduce the obstacles our veterans face as they struggle to find the same bonds they formed during war. These "small groups defined by clear purpose and understanding" are what we need to feel purposeful and happy and unfortunately is in increasingly short supply in our modern world. Tribe essentially turns the question around from asking what is wrong with our veterans, to asking what is wrong with us.

I personally enjoyed this book. Junger, an award wining journalist and special corespondent to ABC News, has spent more time than most in intense environments allowing him to make the observations needed for the claims in his book. However I would have liked to see more examples taken from history, anthropology and psychology experts, or more interviews from people returning home, to give the book less of a journalistic story and more of an anthropological investigation. Overall the, very short, book makes a complex topic very accessible and introduces me to a new possible explanation as to why we are collectively struggling to belong when we have never been more "connected."

Why We Sleep

2020-08-27T00:00:00+00:00

If you're anything like me, the fundamental processes all living organisms need to sustain life do not come naturally to us. Things like knowing when to eat food, drink fluids, or just sleeping are often completely forgotten when our mind finds something else to occupy its time. It's this intense focus that allows me to characterize my relationship with sleep as quarrelsome.

After reading Matthew Walker's book, Why We Sleep, it's easy see the damage my adversarial relationship with sleeping causes. Walker, who is the director of the Human Sleep Science Center in UC Berkeley, lists some of the immediately recognizable symptoms from the aftermath of pulling an "all-nighter". In the short term, sleep deprivation destroys your creativity, problem solving, and decision-making skills, along with inhibiting your memory and your ability to learn. Chronic lack of sleep also has long term side-effects, devouring your heart, brain, and mental health, along with your emotional well-being, immune system, and ultimately your life span.

Throughout the book Walker does a great job of describing the why and how sleep can be such a cure-all for so many ailments, and why 8 hours in bed does not equal 8 hours of sleep. The goal of sleep is to bathe our brain in the restorative neurochemials generated by the different Rapid-Eye Movement (REM) and Non-Rapid-Eye Movement (NREM) stages of sleep. The 8 hour minimum suggested by doctors ensures you spend enough time in each stage to receive their full effect. If you (like me) have trouble sleeping throughout the night, you need to spend more than 8 hours in bed, giving your brain more opportunities to cycle through the different stages of sleep. He goes on to explain the importance of each stage of the sleeping process and why you will never be able to "catch up" from all the lost sleep on the weekend. A popular myth I had believed all throughout my college years.

Walker also does an amazing job describing the various systems our brain uses to control the many functions that regulate our lives, including when to sleep. My favorite section dealt with our Circadian Rhythm and how it changes as we age. Walker argues that teenagers, notorious for sleeping in, might have their brain chemistry to thank. Our circadian rhythm is a ancient internal time keeping process in our brain that regulates many things like our moods, emotions, urine output, core body temperature, metabolic rate, and numerous other hormones. As we age, our circadian rhythm adjusts as our brain develops, until we finally reach the typical split between morning people (morning larks) and evening dwellers (night owls).

For teenagers, as their brains continue to develop, their circadian rhythms are skewed to the evening, allowing them to naturally stay awake and alert throughout much of the night. This will make it almost impossible for them to wakeup before the irresponsibly set 7am school starting bell with a full 8 hours of much needed sleep. Walker describes this situation as if a fully grown adult where to go to bed at 5 or 6pm, and frustratingly try to sleep only to be woken at 1 or 2am.

Overall, Matthew Walker did an amazing job creating an enjoyably short introduction into the world of sleep science. Our brains are such a complex part of our body that we are only just beginning to understand. I thoroughly enjoyed reading this book, and I'm sure you will to.

Modify The Linux Kernel

2020-08-15T00:00:00+00:00

For the 3^rd task (this task) in The Eudyptula Challenge, we continue our work from the last challenge, compiling the absolute latest Linux Kernel from the source code. This time we focus on modifying the Makefiles and .config files that help us compile the kernel.

Before we begin, if you wish to work on The Eudyptula Challenge yourself before you read my notes (recommended), you can use my git repository, which has all 20 tasks and the code I used to complete each one.

Task No.3

Now that you have your custom kernel up and running, it's time to modify it!

The tasks for this round is:

take the kernel git tree from Task 02 and modify the Makefile to and modify the EXTRAVERSION field. Do this in a way that the running kernel (after modifying the Makefile, rebuilding, and rebooting) has the characters "-eudyptula" in the version string.

show proof of booting this kernel. Extra cookies for you by providing creative examples, especially if done in interpretive dance at your local pub.

Send a patch that shows the Makefile modified. Do this in a manner that would be acceptable for merging in the kernel source tree. (Hint, read the file Documentation/SubmittingPatches and follow the steps there.)

—Little Penguin

Depending on how literally we interpret "modifying the Makefile", there are multiple ways we can accomplish this challenge. One of which we briefly talked about in the last challenge was to override variables in our make command when compiling the kernel.

Override Directives

Just like with a lot of userspace projects, the Linux kernel uses GNU make to compile the various files into its final form. This means we can use make's ability to pass variable assignments as command line arguments.

From Chapter 6 of make's documentation, if we use a command line argument to set the EXTRAVERSION variable, then all other assignments to EXTRAVERSION within the Makefile will be ignored. For example, we can override EXTRAVERSION by using a make command like this:

$ make -j `getconf _NPROCESSORS_ONLN` EXTRAVERSION=-eudyptula

This will force make to set EXTRAVERSION to -eudyptula and ignore the value set in the kernel's Makefile, accomplishing our task for The Little Penguin.

Modify the Makefile

Our second option, if you want to take "modifying the Makefile" literally, is to do exactly that.

Simply open the kernel's Makefile, located in the root directory in the source code we copied from Linus in the last challenge, with your favorite text editor. The first five lines will have EXTRAVERSION somewhere in it. Change the value to -eudyptula and save.

-EXTRAVERSION = -rc1
+EXTRAVERSION = -eudyptula

Now that the kernel's Makefile will append -eudyptula to the kernel's version string by default, we can simplify our make command to build the kernel into:

$ make -j `getconf _NPROCESSORS_ONLN`

Modify menuconfig

Our third option and arguably the least accurate of the three ways to interpret "modifying the Makefile" is to use the ncurses configuration tool that we briefly talked about in the last challenge. I say this is the least accurate because this method modifies the CONFIG_LOCALVERSION variable and not the EXTRAVERSION variable, which will add -eudyptula to our kernel's version string, however not technically in the correct place.

We can start the ncurses menu with the following terminal command:

$ make menuconfig

where you should see a screen that looks something like this:

From here, navigate to the General setup option and press ENTER to move into the next menu. Next, use the arrow keys to highlight the Local version option and press ENTER. A new menu will appear letting you enter a value.

Type in -eudyptula and press TAB to move our cursor to the <OK> option and press ENTER to return back to the General setup menu. If everything went according to plan you should see "-eudyptula" set in the Local version menu:

From here, press TAB a few more times to move our cursor to highlight the <SAVE> option at the bottom. Then press ENTER to save the changes we made to our .config file.

After we've saved our changes press TAB a couple more times to highlight the <EXIT> option and ENTER to exit the ncurses menu.

Finally, with the changes to our .config file saved, we can re-compile our kernel with the same simpified command from above:

$ make -j `getconf _NPROCESSORS_ONLN`

Prepared

2020-08-09T00:00:00+00:00

When you think back to your primary or secondary school days was there something you would change? If you could change absolutely anything, create your own curriculum, add or remove some lectures, assign less homework, what would your dream school look like? Would you change school start times? Allowing students to get enough sleep and, in my case, preventing students from waking up at an un-Godly hour trying to catch a 6am bus.

Everyone involved in a child's education, including parents, teachers, administrators, and even the students themselves all want every student to succeed. As a country we have devised countless initiatives and tests to identify and help a student who falls behind on their journey to developing the tools needed for a successful life. Yet as of 2018, there were an estimated 2.1 million (~5.3%) American children, between 16 to 25 who would be labeled "dropout". A title, according to the Bureau of Labor Statistics, that gives them half the earning power as someone with a college degree and 5 times more likely to go without a job. Which asks the question, if everyone is helping our next generation succeed, why are so many failing to do so?

Diane Tavenner, the author of the book Prepared and founder of Summit Public Schools, which operates some of the top-performing schools in the nation, looks to have found the answer to ensuring every child has a successful life. Her goal with Summit is to go beyond teaching the math, writing, and science that students will need to get into college, but to also teach what students will need to live a great life. Life skills like the ability to learn and research new topics, manage their time wisely, and the self-confidence along with finding the initial direction they wish to pursue with their lives.

As she explains in her book, Summit is continuously refining and improving the teaching model that focuses on three key elements with their students:

self-directed learning: All students are responsible (with support from their teachers) for setting their own learning goals, planning how to learn the information, test their knowledge and assess their performance afterward.
project-based learning: A hands-on, problem oriented, self-discovery method of learning that allows students to deeply explore a topic they're curious about.
mentoring: Beyond the typical school guidance counselor, each student in Summit has a dedicated mentor that meets regularly with them to help and guide students as they achieve their personal and academic goals.

These skills are incredibly important in today's workforce. When all of humanity's information is just a fingertip away, we must learn more than just the reading, writing and arithmetic. We need the life skills like time-management, creative & critical thinking skills, along with the drive to accomplish our goals that let people truly succeed in this information saturated world.

The majority of the book is dedicated to the teachers and administrators overseeing our children's education. However, in the final section, Diane gives parents some advice on how to encourage their children's independent growth. Most of the section can be distilled into the belief that parents should be mentoring, not directing. If a child is into computers, support them. When they change their mind (and they will) support them. In this part of their lives they are exploring their world and finding their place in it. The best thing a parent can do is let them explore, as scary as that may be.

Building The Linux Kernel

2020-07-31T00:00:00+00:00

Previously, for the 1^st task in The Eudyptula Challenge, we built a Loadable Kernel Module (LKM) that adds "Hello World" to our kernel's message buffer any time the module is installed. You can read my notes all about that here if you want to learn more.

Or, if you wish to work on The Eudyptula Challenge yourself before you read my notes (recommended), you can use my git repository, which has all 20 tasks and the code I used to complete each one.

For this challenge, the 2^nd task of The Eudyptula Challenge, we will use the Kbuild system once again. This time we will use the Kbuild system to compile the latest version of the Linux Kernel. Then after compiling it (this takes some time), we install and boot from it.

Task No.2

Now that you have written your first kernel module, it's time to take off the training wheels and move on to building a custom kernel. No more distro kernels for you, for this task you must run your own kernel. And use git! Exciting isn't it! No, oh, ok...

The tasks for this round is:

download Linus's latest git tree from git.kernel.org (you have to figure out which one is his, it's not that hard, just remember what his last name is and you should be fine.)

build it, install it, and boot it. You can use whatever kernel configuration options you wish to use, but you must enable CONFIG_LOCALVERSION_AUTO=y.

show proof of booting this kernel. Bonus points for you if you do it on a "real" machine, and not a virtual machine (virtual machines are acceptable, but come on, real kernel developers don't mess around with virtual machines, they are too slow. Oh yeah, we aren't real kernel developers just yet. Well, I'm not anyway, I'm just a script...) Again, proof of running this kernel is up to you, I'm sure you can do well.

Hint, you should look into the 'make localmodconfig' option, and base your kernel configuration on a working distro kernel configuration. Don't sit there and answer all 1625 different kernel configuration options by hand, even I, a foolish script, know better than to do that!

After doing this, don't throw away that kernel and git tree and configuration file. You'll be using it for later tasks, a working kernel configuration file is a precious thing, all kernel developers have one they have grown and tended to over the years. This is the start of a long journey with yours, don't discard it like was a broken umbrella, it deserves better than that.

—Little Penguin

Build Tools

Just like mechanics, plumbers, and all the other skilled trades, having the right tools for the job makes a world of difference. This "tools maketh man" proverb rings true even for software developers, even though most of our tools are less tangible.

The tools we will need to properly build the kernel vary wildly depending on many factors, like your hardware, kernel version, how we plan to install the kernel, as well as the linux distribution on your computer. All the minimum requirements we will need to build the kernel are listed in the documentation here, chances are though you will need to use more.

The tools I needed to build version 5.8 of the kernel on my (charmingly retro) Lenovo T430, running Ubuntu's 18.04 LTS (Bionic Beaver) release, are easily installed with:

$ sudo apt install build-essential libncurses-dev bison flex libssl-dev libelf-dev

Cloning

When you first navigate to the kernel's git repositories, you will be greeted with hundreds of projects all having something to do with archiving or improving the Linux Kernel in some way. To help simplify the process of finding Linus' repository, which has the latest patches, we can search through the projects listed (using Ctrl+F) for "torvalds/".

Once you have found the URL, we can use git to download a copy of Linus' Linux repository using the typical git clone command. If you are unsure about how to clone a repository, the git documentation has a great article about getting started with git.

$ git clone https://git.kernel.org/pub/scm/linux ...

Configuring

With a copy of the kernel's source code in hand, our next task is to set the many thousands of configuration options needed for the Kbuild System to properly compile our kernel. Each configuration option, like the CONFIG_LOCALVERSION_AUTO the Little Penguin mentioned in the challenge message above, informs the Kbuild System of the drivers and features needed to be installed to function properly with our computer's hardware.

However, instead of setting roughly 8700 config options by hand, one-at-a-time (a very error prone, time consuming process), we can copy the configuration file from the linux distribution currently running on our computer as a sort of starting point.

Configuration Plagiarism

The first step in copying our distribution's configuration file is to find it in our /boot directory.

There are multiple configuration files inside our /boot directory. The one our computer is using depends on the kernel version our computer is using. For example my computer, using kernel version 4.15.0-112, will have a configuration file named config-4.15.0-112-generic in the /boot directory.

To copy your distribution's config file into the root directory of the kernel use the this command:

$ cp /boot/config-`uname -r` .config

Next, answer yes to any new configuration options added between the release of our distribution's kernel (they usually lag behind a few versions) and the bleeding edge we've downloaded from Linus.

$ yes "" | make oldconfig

Configuring By Hand

While not technically needed to complete this eudyptula challenge, this is a good place to talk about how we can customize our newly copied configuration file for our kernels. While there are a few, largely similar, commands that help us edit the .config file, the command I prefer is a ncurses based menu that can run on most systems or on remote servers if wanted.

$ make menuconfig

It produces a color menu, full of radio-lists and dialogues, that looks something like this:

Each menu option allows us to enable and disable certain features in our kernels, along with anything that depended on that option. For example, if we disabled Networking, all network-related options, like Amateur Radio, will also be disabled.

The full list of commands used to edit the kernel's configuration file is available to you in the kernel documentation, if you want to learn more. They largely use different GUIs to edit our .config file.

Building

With all the preparations complete, we can start to work on compiling the kernel into a compressed image ready for our computers. Just like in userspace applications, a simple make command is all that is needed to build our kernel:

$ make -j `getconf _NPROCESSORS_ONLN` CONFIG_LOCALVERSION_AUTO=y LOCALVERSION=-eudyptula

While a bare make command, without the added arguments, will also work to build our kernel, however, the kernel is such a large project, using the -j flag to compile multiple files at the same time will dramatically reduce the total time needed to compile.

We are also passing two extra configuration options with the make command above:

CONFIG_LOCALVERSION_AUTO=y LOCALVERSION=-eudyptula

Both of these configuration options will add information to our kernel's version string, helping us prove to The Little Penguin we installed the latest kernel published by Linus Torvalds.

The CONFIG_LOCALVERSION_AUTO option will append enough characters from the latest commit id to be unique in our kernel's version name. For me this was g1df0d8960499
LOCALVERSION will also add -eudyptula to that same version string

producing a kernel named something like this:

linux-image-5.8.0-rc4-eudyptula-00381-g1df0d8960499

Installing

Our last and final step to completing this challenge is to install the kernel along with its supporting modules into our /boot and /lib/modules folders. This will allow our boot loader, GRUB, to properly initialize our freshly compiled linux kernel the next time we reboot.

Each distribution of linux installs the kernel in their way. For example Arch Linux's documentation tells us to manually copy (with cp) and not to use make install to install the kernel in our /boot directory. All this to say "your mileage may vary".

For most linux distributions we can install our kernel with a simple make command:

$ sudo make modules_install install

Along with installing all the kernel's modules and the kernel itself, this command will also update our /boot/grub/grub.conf files, so we don't have to manually update GRUB ourselves.

Ubuntu/Debian Weirdness

If your linux distribution has trouble with the make install commands above, an alternative is to bundle our linux kernel into a package. This method is slower (a real pain when developing drivers) however, packages usually have more success installing on certain systems.

For Debian and Ubuntu based distributions we can use deb-pkg to generate the packages:

$ make deb-pkg

This command will generate up to 5 Debian packages:

linux-image-version which contains the kernel image and the associated modules.
linux-headers-version has the header files required to build external modules.
linux-firmware-image-version contains the firmware files needed by some drivers (this package could be missing when you build from the kernel sources provided by Debian).
linux-image-version-dbg which contains the debugging symbols for the kernel image and its modules
and linux-libc-dev holds headers relevant to some user-space libraries.

All of these packages can easily be installed with a simple dpkg command:

sudo dpkg -i ../*.deb

And that's it! Reboot you computer, and run uname -r and you should see something like:

$ uname -r
linux-image-5.8.0-rc4-eudyptula-00381-g1df0d8960499

This output will prove to The Little Penguin you've successfully compiled and installed your first (of many I'm sure) custom linux kernel. Great Job!

The Hello World Kernel Module

2020-07-15T00:00:00+00:00

As a brief introduction to this (very long) essay. What lies below are my notes while completing the 1^st in a set of 20 tasks from The Eudyptula Challenge. Each task, emailed one at a time, starting with building a "hello world" kernel module (this essay) and progressed in difficulty until we ultimately submit patches into the main tree of the Linux kernel. The ultimate goal of The Eudyptula Challenge is to get new developers comfortable with the somewhat unique world of kernel development by separating the "on-boarding" process into focused manageable tasks.

Sadly, The Eudyptula Challenge is no longer accepting new applicants. However, if you wish to work on the tasks yourself, I've published the 20 tasks I've managed to find along with the code I used to "complete" them in a git repo here.

Task No.1

Write a Linux kernel module, and stand-alone Makefile, that when loaded prints to the kernel debug log level, "Hello World!" Be sure to make the module be able to be unloaded as well.

The Makefile should build the kernel module against the source for the currently running kernel, or, use an environment variable to specify what kernel tree to build it against.

Please show proof of this module being built, and running, in your kernel. What this proof is is up to you, I'm sure you can come up with something. Also be sure to send the kernel module you wrote, along with the Makefile you created to build the module.

—Little Penguin

What Is A Module

A kernel module is piece of code designed to be loaded and unloaded on demand by our kernels. For example, the device drivers for your keyboard or a network card are a type of module. By separating the kernel into individual software components, we can keep the overall size of the kernel small, letting Linux fit into the smallest of embedded systems. Some kernel modules, like the one we'll be building, can even be installed without the need to recompile and reboot our kernel, making upgrades easy, and saving us a lot of time.

If you have access to a Linux machine, you can find the modules that are currently loaded into the kernel by using the lsmod command, which gets its information from /proc/modules.

Chiseling A Cave Module

Every kernel module must have at least two functions, one that will be called when we install the module and another function to remove it from the kernel. Back in the pre v2.3 era (early 2000s) this could only be done with a "start" function, called init_module() and an "end" function, called cleanup_module(). There are more modern (and preferred) methods available to us today, however some developers still use these, so it's a great starting point.

#include <linux/kernel.h>  /* for KERN_DEBUG */
#include <linux/module.h>  /* for all kernel modules */

int init_module(void)
{
        printk(KERN_DEBUG "Hello World.\n");
        return 0; /* init_module loaded successfully */
}

void cleanup_module(void)
{
        printk(KERN_DEBUG "oh, the rest is silence.\n");
}

Typically init_module() is used to register handlers or alter some other part of the kernel for a device or something. The cleanup_module() will then undo those changes, allowing the module to be removed safely from the kernel. Both of these functions (as of version 5.7) can be found on line 75, as well as everything else we need, in linux/module.h of the source code.

printk() != printf()

To print Hello World on "the kernel debug log level", we'll need to use another, very old, function called printk(). Unlike the printf() commonly used in userspace applications, printk() is not designed to communicate to the user (or say hello to worlds). It's a logging mechanism used to give warnings and to log messages. This is why each printk() statement also comes with a priority. There are currently 8 defined priorities we can use ranging from KERN_DEBUG to KERN_EMERG. You can see them all, and their definitions, currently (version 5.7) in linux/kern_levels.h in the source code.

Pay attention to the single argument passed to printk(). Looking into the source code shows that printk(const char *ftm, ...) accepts only one string, with space to pass extra arguments to format the string if needed, for example, our "Hello World" statement from above, which doesn't need formatting and therefore passes no extra arguments:

printk(KERN_DEBUG "Hello World.\n");

The KERN_DEBUG macro will expand to "\001" "7", turning our statement into:

printk("\001" "7" "Hello World.\n");

Our C lexer will then combine the adjacent string literals to produce our formatted string for the kernel to log:

printk("\0017Hello World.\n");

Even though printk() is falling out of style with modern Linux maintainers, as we will see in later sections, there is a lot more to read about how to work with printk() and format specifiers in the kernel in the documentation here if you're into that kind of stuff.

Making A Kernel Module

Much like how kernel modules are a little different than userspace application modules, the Makefiles that compile the kernel are also a bit different than Makefiles in userspace.

Originally, as the Linux code-base grew, so did its Makefiles. As they continued to grow in complexity, they eventually became a burden to maintain. Fortunately a solution, called the "kbuild system", was created and accepted into the kernel to help organize and simplify the kernel's building process. If you are interested, there is an entire section about the kbuild system in the documentation.

Kbuild Makefile

Just like Makefiles in userspace, we can start a Kbuild Makefile by creating a new file called …wait for it… Makefile in the same folder as our hello-world.c module we made in the sections above.

$ ls -l
total 8
-rw-rw-r-- 1 me us 903 Jul  5 00:00 hello-world.c
-rw-rw-r-- 1 me us 167 Jul  5 00:00 Makefile

We can alternatively use the name Kbuild (not preferred) to indicate to other developers that the Makefile is intended to run using the kbuild system. However, while the Kbuild name is not preferred, interestingly, if both Makefile and Kbuild files exist in the same directory the Kbuild file will be used. (source)

Goal Definitions

The "heart" of the kbuild system uses lines called "goal definitions" to define all the various target files, special compilation options, and any sub-directories to enter. When we compile the kernel (with its thousands of Makefiles) the goal definitions are collected and used to build all the various, documentation files, modules, and other files we need for our particular kernel.

The simplest Kbuild Makefile we can write for our module contains a single line:

obj-m += hello-world.o

obj-m tells kbuild that our hello-world.o object file is a loadable kernel module (LKM) that can be loaded and unloaded at any time without needing to reboot the kernel. This line will also tell the kbuild system to look for files in our directory named hello-world.c or hello-world.S to compile into the hello-world.o object file, before building the kernel object file hello-world.ko we'll use to load into our kernel.

Convenience Targets

For the pure convenience of it, we can add extra phony targets to our Kbuild Makefile to easily compile our module for the kernel currently running on our computer, simplifying the task of compiling our module down to just typing make into our terminals:

all:
    ${MAKE} -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

And make clean to clean up everything afterwards:

clean:
    ${MAKE} -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

Both of these phony targets use the -C option to move out of our current directory and into our kernel's source directory. There make can find and use the top most kbuild Makefile, which takes the M option to locate the folder we are current working in, and build the files defined using the obj-m goal definition we setup above.

Installing A Kernel Module

Just as with our userspace applications, kernel modules need to be compiled. Using the kbuild system, along with our convenience targets above, we can compile our kernel module by issuing the make command, and if all goes well, you should see an output similar to this:

$ make
make -C /lib/modules/4.15.0-108-generic/build M=/home/me/src/eudyptula ...
/tasks/01 modules
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-108-generic'
  CC [M]  /home/me/src/eudyptula/tasks/01/hello-world.o
  Building modules, stage 2.
  MODPOST 1 modules
WARNING: modpost: missing MODULE_LICENSE() in /home/me/src/eudyptula ...
see include/linux/module.h for more information
  CC      /home/me/src/eudyptula/tasks/01/hello-world.mod.o
  LD [M]  /home/me/src/eudyptula/tasks/01/hello-world.ko
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-108-generic'

The .ko extension was introduced around kernel version 2.6 to help differentiate between userspace object files and kernel object files, which contain a .modinfo section to hold extra metadata information about the module. We can use the modinfo command to see and interpret the contents of the section:

$ modinfo hello-world.ko
filename:       /home/me/src/eudyptula/tasks/01/hello-world.ko
srcversion:     18005133D4ECFCDD12928D8
depends:
retpoline:      Y
name:           hello_world
vermagic:       4.15.0-108-generic SMP mod_unload

Installing the Module

With our hello-world.c module freshly compiled, we can insert it into our kernel using the insmod command as root or another user with sudo privileges:

$ sudo insmod hello-world.ko

Congratulations!, you have created your first kernel module! A quick inspection of the kernel's diagnostic messages, using dmesg, should show our Hello World. message:

$ dmesg | tail -1
[241745.247591] Hello World.

Removing the Module

After the well deserved pat-on-the-back and when you are ready to continue, we can uninstall our module with the rmmod command as root or someone with sudo privileges:

$ sudo rmmod hello_world

The only indication we've uninstalled our module will be in dmesg from our printk() statement in the cleanup_module() function.

$ dmesg | tail -1
[241751.401232] oh, the rest is silence.

Kernel Taint

There are plenty of ways we can taint our kernel. Don't worry too much about this though, most of the time it is completely fine to run a tainted kernel. When something happens that could be important to an investigation later on, a kernel will mark itself as "tainted". Usually the event that caused the kernel to become tainted is the problem being investigated.

We can find our kernel's tainted state by reading our /proc/sys/kernel/tainted file. Every way we can taint our kernels is assigned one bit in a bit-field, meaning any value other than 0 indicates our kernel is tainted. To decode the bit-field values, we can use the tools/debugging/kernel-chktaint script found in the source code, to decode its meaning.

$ tools/debugging/kernel-chktaint
Kernel is "tainted" for the following reasons:
 * proprietary module was loaded (#0)
 * kernel issued warning (#9)
 * externally-built ('out-of-tree') module was loaded  (#12)
 * unsigned module was loaded (#13)
For a more detailed explanation of the various taint flags see
 Documentation/admin-guide/tainted-kernels.rst in the the Linux kernel sources
 or https://kernel.org/doc/html/latest/admin-guide/tainted-kernels.html
Raw taint value as int/string: 12801/'P        W  OE    '

Licensing & Documentation

One of the ways we can taint our kernels is by loading proprietary modules or modules that use licenses not compatible with the General Public License (GPL) (bit 0 in the tainting list). Modules that don't use the MODULE_LICENSE() macro will also be considered proprietary and taint our kernel, if loaded (this is why we saw the warning above).

There are many documentation macros, defined in linux/module.h, some of the basics I added are:

MODULE_LICENSE("MIT");
MODULE_AUTHOR("Bryan Brattlof <email@example.com>");
MODULE_DESCRIPTION("A Hello World Driver");
MODULE_SUPPORTED_DEVICE("testdevice");

Once we add our module's license, author and other information to the end of our hello-world.c module, when we compile our module again using make, the WARNING should be gone:

$ make
make -C /lib/modules/4.15.0-108-generic/build M=/home/me/src/eudyptula ...
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-108-generic'
  CC [M]  /home/me/src/eudyptula/tasks/01/hello-world.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/me/src/eudyptula/tasks/01/hello-world.mod.o
  LD [M]  /home/me/src/eudyptula/tasks/01/hello-world.ko
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-108-generic'

There where many reasons to add this system to the kernel. For example, it gives developers a way to easily find who maintains a module, describe what the module does, and what license the code is protected with. It also provides an easy method to inform users when they are using non open source software.

Updating the Module

Everything in my notes, to this point, was needed to complete the 1^st task assigned to us by the Little Penguin. However, just like with every software project, the Linux kernel is constantly adding new features and adopting new coding styles, ensuring that my notes will become obsolete as soon as I've writing them.

With that said, the sections below, while not technically needed to complete the task, are my notes on the macros and functions I saw in the drivers directory of the Linux source code that I found particularly interesting. These functions are mostly stylistic changes or they introduce functionality that improves efficiency and modularity of the Linux kernel in some way.

module_init() & module_exit()

Introduced in version 2.4 of the kernel, and defined in linux/init.h of the source code, we can now rename our "start" and "end" functions to whatever we wish. In this example, I've chosen to rename the "start" function to hello_world_init()

-int init_module(void)
+static int __init hello_world_init(void)
 {
         printk(KERN_DEBUG "Hello World.\n");
         return 0; /* init_module loaded successfully */
 }

And renamed the "exit" function hello_world_exit()

-void cleanup_module(void)
+static void __exit hello_world_exit(void)
 {
         printk(KERN_DEBUG "oh, the rest is silence.\n");
 }

The kernel will then use the module_init() macro to find the function to execute when the module is installed and module_exit() to find the function to cleanup before being removed.

module_init(hello_world_init);
module_exit(hello_world_exit);

To avoid compiling issues, both the module_init() and module_exit() macros must be defined below our newly named "start" and "end" functions.

init & exit

I also introduced two macros to our "start" and "end" functions above called __init and __exit. These macros, defined in linux/init.h of the source code, help reduce memory used by the kernel depending on how the module is installed.

For built-in modules, where our module cannot be removed from the kernel without recompiling and restarting, the __init keyword will tell our C lexer to place our module's "start" function into a special section inside the compiled kernel. After the module is loaded and our "start" function has finished, the kernel will never have to run the code again until reboot. So this special section can be freed, saving memory.

The same is true for the __exit macro. For built-in modules, the module cannot be removed from the kernel without recompiling and restarting. So the kernel will never need to run our module's "exit" function to safely remove it from the kernel. This means our C lexer can safely omit our "exit" function from the compiled kernel.

pr_debug()

In the beginning there was printk(), and the kernel's diagnostic messages structure was formless. The lack of any format for printk() messages is one of a number of reasons why developers are replacing printk() statements with their newer equivalents. Depending on what section of the kernel we are in, there are newer functions that have some benefits for us.

For example, the pr_debug() function, which has the benefit of being less syntactically verbose than printk(KERN_DEBUG ...) also allows us to take advantage of the dynamic debugging interface, which gives developers a uniform control interface for debugging kernel messages while avoiding cluttering the kernel.

 static int __init hello_world_init(void)
 {
-    printk(KERN_DEBUG "Hello World.\n");
+    pr_debug("Hello World.\n");
     return 0; /* means init_module loaded successfully */
 }

 static void __exit hello_world_exit(void)
 {
-    printk(KERN_DEBUG "oh, the rest is silence.\n");
+    pr_debug("oh, the rest is silence.\n");
 }

Wrapping Up

If you made it here, all I can say is you are a very brave person, and I'm glad my notes were able to help you in some way. If you see any issues or have a question, please feel free to contact me, or better yet subscribe to the kernel newbies mailing list.

For the next challenge, we will be building the Linux Kernel from scratch, as well as installing and booting from it. If you want to work on this challenge before you read my notes (recommended), I've published a copy of the challenges in a git repo here.

Next: My notes on How to build the Linux Kernel from scratch.

Humble Pi

2020-06-30T00:00:00+00:00

We all make mistakes … especially math mistakes. Either we forgot to carry the one, used "freedom units" instead of metric, lost the decimal point, or found some other creative way to make a mistake. Usually the consequences are small enough that we can chuckle when we find out Michael Bloomberg can't afford to give everyone $1 million dollars. However, for the engineers and software developers among us, these mistakes can have real consequences.

For the vast majority of us, math is not an easy subject, nor does it come naturally to us. One of my favorite examples of this (also in the book) happened in 1852. Radhanath Sikdar, a young mathematician at the time, was compiling measurements from multiple different observations for a mountian peak named "Peak XV" for the Great Trigonometrical Survey, a project to survey the entire Indian subcontinent.

While Sikdar was calculating the measurements, he came to the conclusion that Peak XV was the tallest mountian on record, standing 25,000ft (8,839.2m) tall. Excited, he gave his report to Andrew Scott Waugh, the director of the survey at that time, who spent a few years validating the data. However, when the time came to publish their findings, knowing the public was largely bad at math and would incorrectly assume 25,000ft as a rounded number and not the exact height they spent years calculating, they decided to add 2ft to the peak, giving it the more precise feeling, but completely wrong, height of 25,002ft (8,839.8m). This would also lead Waugh to being playfully credited with being the first person to put two feet on top of the mountian that would eventually be named Mount Everest.

Even though we know everyone makes mistakes, and math does not come naturally to us, we still have an extremely hard time admitting to our errors. Matt Parker and his book Humble Pi believe that, as our world grows in complexity, accidents will continue to happen. Instead of hiding them in hopes that no one will notice, we should build systems that encourage us to learn from our mistakes. Much like the United State's National Aeronautics and Space Administration (NASA) and National Transportation Safety Board (NTSB), both famously publicize their investigations into the failures and accidents we see all over the news. Matt Parker's book does a great job of showing us mistakes can be both funny and a great teaching tool for our next generation of engineers and everyone else who continues to explore the boundaries of this world.

Probably Approximately Correct

2020-06-17T00:00:00+00:00

It was Charles Darwin who taught us the foundational and the now widely accepted concept of evolution. Stating that all life has, over time, descended from a common ancestor through a process of natural selection, much like how we selectively bred wolves to produce hundreds of the domesticated dog breeds we see today. The struggle of existence has only allowed the best of us to produce offspring, resulting in a natural process of selection, where only the fittest among us will survive. This evolutionary process tells us why, over millennia, the Giraffes got taller, Sloths got slower, Alligators got scarier and we doubly wise men became (kinda) smarter.

As life continues to evolve, it also continues to grow in its complexity. If you want to succeed in an interview, or choose a life partner, or (if you're like my wife) decide on a restaurant, you can be sure there is no equation that could guarantee you success. Even if we could collect all the relevant information we needed to answer these questions (assuming we even knew what that relevant information was) there's still no sure-fire way to combine the information in a way to yield an answer for us. Yet, every day, billions of us answer these questions as we go about our lives. It's these "guesses," "hunches," and "gut instincts" that Dr. Leslie Valiant, a world-renowned theoretical computer scientist, and his book are interested in.

Historically, computers were only suited for solving the theory-full questions in our lives, like modeling fluid flows or calculating the orbits of satellites using Newton's laws. These theory-full questions, typically mathematical or scientific theories, like Einstein's famous equation E=mc², have clearly defined inputs and instructions telling us how they operate to accurately calculate a solution to our questions. More recently though, computers have been growing "softer" skills, ones that require them to answer more theory-less questions, which have no well-defined formula to calculate, like deciding on a movie we'd like to watch, or how to safely drive us home from the pub.

Computers today learn to answer these theory-less questions much like how we learned about our world in our infancy (or adolescence for some), by drawing general lessons from a particular experience to better answer the next question life throws. Unlike the theory-full algorithms, expertise is not given from the designer of the equation to the student or computer, but extracted from the experiences gained from experimentation or through evolution by the student. These are the equations for learning, rather than answer using E=mc² absolutes, these algorithms produce probably approximately correct answers.

Probably Approximately Correct aims to explain the current state and future goals of our attempts to find a mathematical definition of the learning algorithms that we have, through evolution, used to prosper in this (sometimes excessively) complex world. With the minuscule small hint of our current understanding of these learning algorithms, we have revolutionized and transformed our world (mostly) for the better. Through the power of learning algorithms that can organize the endless supply of websites, the least educated person today has immeasurably more knowledge at their fingertips than the most educated had just a few decades ago. With a more thorough understanding of these learning algorithms, we stand to reshape our civilization even more dramatically in the decades to come.

The Mosquito

2020-05-28T00:00:00+00:00

For thousands of years, we humans were terrorized by a deadly disease, without effective treatments or a clue what actually caused malaria. Hindu texts from the 6^th century BC, Mesopotamian tablets from 2000 BC, and Chinese documents from around 2700 BC all make references to something we are almost certain was malaria. It wasn't until the early Greeks, including Homer (850 BC) and Hippocrates (400 BC) that documents survived well enough about the enlarged spleens and malarial fevers from people living in marshy areas. This observation would help popularize miasma theory and how malaria would eventually receive its name, which literally translates into "bad air" in medieval Italian.

Even with remedies like chinhona bark, a genus of around 23 species of plants "discovered" in the early-1600s in tropical regions of the Andes Mountains, it would take another 200 years for scientists like Charles Louis and Alphonse Laveran in 1880 to discover the parasites causing the malarial fevers, and another 17 years for Ronald Ross in 1897 to finally incriminate the mosquito as the delivery system for malaria. Proving once and for all that, despite all the hollywood movies about snakes on planes, man-eating crocodiles, or shark-nadios, the mosquito is by far our deadliest predator.

Timothy Winegard's book The Mosquito focuses on our collective history as a species and what the mosquito has done to change it, which turns out to be quite a lot. Starting with King Tut (1324 BC) who (probably) died at the hands of a mosquito carrying malaria, to the invisible army of mosquitoes in the Pontine Marshes literally sucking the life out of Rome's enemies as they invaded, and includes modern events like the mind boggling death toll it took to colonize the Americas, and Dr. Seuss' creative advertisement campaigns needed to remind soldiers to protect themselves from mosquitoes in World War II.

A particularly interesting event (for me) covered in the book happened in 1698, when five ships began their journey in Scotland, planning to ride the Trade Winds to the Darién region of Panama. These ships where loaded with trading goods, like wigs, woolen socks and blankets, mother-of-pearl combs, Bibles, twenty-five thousand pairs of leather shoes, and even a printing press. Their goal was to establish a trading post that would help the fiercely independent kingdom of Scotland, struggling from years of famine, to secure a colony on the Isthmus of Panama on the Gulf of Darién.

This plan was insanely popular in the struggling kingdom, attracting all forms of investors from poor farmers to members of the national parliament, who collectively invested approximately 20% of all the money circulating in Scotland at that time with the Company of Scotland who was establishing the trading post in the new world. Mercantilism, the accepted economic theory at the time, meant that instead of simply competing with England or Spain who, thanks to their colonies in the new world, where literally moving boat loads of money, Scotland would need to take market share from these larger empires in order to stay relevant (and independent from England).

The Scot's plan did not go well...

To quote Winegard, "The words that are repeated to the point of nausea in the diaries, letters, and accounts of the Scottish settlers are mosquitoes, fever, ague, and death". What the 1200 colonists didn't anticipate was the deadly swarms of malaria and yellow fever carrying mosquitoes waiting to feast on the Scottish settlers when they arrived. Having never experienced malaria or yellow fever in the temperate climate of Scotland, the colonists had little resistance and quickly fell sick. After six months, with nearly half of the colonists dead, the survivors still able to move under their own power (six colonists where left on the beach) picked-up and fled.

To make matters worse, 300 additional colonists in a resupply mission and another 1000 colonists in a second expedition left in 1699, before news of the complete failure could reach Scotland. These later expeditions largely met the same fate as the first. Out of the combined 2500 colonists only a few hundred survived the expedition to the New World.

The monumental failure of the Darién scheme left Scotland, its landed aristocracy, mercantile elites, and nobles in overwhelming debt, which was cited as one of the motivations for signing the Acts of Union with England in 1707. The deadly mosquitoes of Darién, one blood-meal at a time, changed world history and forced a reluctant Scotland to forfeit their independence and join England to create Great Britain.

Even today with the advent of pesticides like DDT, modern medicines like antimalarial drugs, and other repellents like insecticide-treated nets, we are still not 100% protected from malaria. In 2018 alone, the WHO found somewhere between 206 to 258 MILLION malaria infections worldwide, with 93% (213 million) of these infections taking place in Africa. Of the 405,000 deaths, a horrifying 67% (272,000) where children under the age of five years old. While these terrifying numbers are still 260 million too big, they show how, even with the advances we've made from Hippocrates' time, our deadliest predator continues to be a 0.002 gram insect.

Removing Git LFS From A Project

2020-05-01T00:00:00+00:00

Git Large File Storage (LFS) is an extension to Git that separates large, frequently changing, binary files and saves them in a separate storage location outside of the normal Git project. LFS replaces these binary files with smaller text files, called pointers, that hold information about the original file and how to download them. These pointers look something like this:

version https://git-lfs.github.com/spec/v1
oid sha256:d7cbc07ce9b78a89764c0ac5e9c8e1b9dbdeb42c30d8396cba8f75aace5ba225
size 145930

Git-LFS uses git hooks to transparently replace and create these pointers whenever we make a commit and saves the binary data outside of Git until the next push where it will sync with your LFS server. When everything is working correctly, we should never see these pointer files.

When we say "remove LFS from a project" we really mean removing the hooks from our project, which, if not done correctly, will leave these pointer files all throughout our project's history with no way to download the actual files they pointed to, preventing us from ever building an older release of our project again.

To successfully uninstall LFS, we'll need to rewrite the project's history, replacing all the LFS pointer files with the actual file they pointed to, adding them back into our project's history. In the end, the project will look as if LFS was never installed in the project at all.

Step: 0 - Make A Backup

Before we begin, rewriting history is remarkably dangerous, with many opportunities to silently break something you'll only find out about after it's too late. Make a backup before you start and save it until you're absolutely sure the project's history is correct.

git clone --mirror $URL backup && cd backup
git lfs fetch --all

Replacing $URL with the location of the project, these two commands will:

Create a bare copy of the project and navigate into it.
Download all the files in LFS from the remote LFS server.

This will give you a complete backup of the project, including branches and other refs like tags and notes, as well as download all the files from LFS, ensuring any screw-ups with the next step are recoverable.

Step: 1 - Rewrite History

In the working copy of our project, filter-branch can easily replace all the LFS pointer files for us. However this step will also change all the names of the commits (git objects) in your project, so references to specific commits like Fixed in a4749f3 will break. If you have a lot of these types of commits you might want to use more advanced tools like filter-repo.

git filter-branch -f --prune-empty --tree-filter '
[ -f .gitattributes ] &&  git rm -f .gitattributes
find . -type f | while read FILE; do
   if head -2 "$FILE" | grep -q "^oid sha256:"; then
      POINTER=$(cat "$FILE")
      echo -n "$POINTER" | git lfs smudge > "$FILE"
      git add "$FILE"
   fi
done' --tag-name-filter cat -- --all

This uses git filter-branch --tree-filter to go through each commit for every branch, and use find and grep to list every LFS pointer file so we can use git lfs smudge to replace the pointers with the LFS data, before committing the changes and moving to the next commit in the history.

Step: 2 - Remove Hooks & Filters

If this is the only project that uses LFS, you can remove the --local option from the following command to also remove the filters from your global Git config file ~/.gitconfig

git lfs uninstall --local

Once the pointer files have been replaced, we can remove the hooks and filters LFS used to read the pointer files every time we fetched, pushed or committed changes to the project, with this simple built-in LFS command.

Step: 3 - Remove the LFS cache

rm -r .git/lfs

Finally, if this is your working copy of the project, you can save some disk space by removing the LFS cache folder inside the git directory of your project.

Upheaval

2020-01-27T00:00:00+00:00

Jared Diamond is one of those prolific authors with so much clout to his name that when he writes a new book, everyone knows about it. That's why when he published his new book called Upheaval all of the algorithms I trust easily shoved this straight to the top of my "recommended reading" list and they had good reason.

This book, inspired by his wife, Marie Cohn who works as a psychologist, uses crisis therapy techniques developed to help people through periods of crisis in their own lives (like the death of a loved one) and adapts them to gain understanding of how nations going through crisis can successfully navigate that critical time.

Diamond begins the first part of the book by taking the 12 things therapists have identified as steps that indicate whether someone will succeed in resolving a personal crisis (like acknowledging you're in a crisis), and adapts them into 12 success factors to apply to the case studies of countries going through a crisis at different times in history.

While all the case studies were interesting, my favorite case study (largely because I knew so little about the situation) was how Finland, which shares a 1,000 mile border with the Soviet Union, was able to successfully bridge the desires of Stalin, who at the time was invading all his neighboring countries, (Finland was the only Soviet neighboring country not occupied by the USSR) and the Allied Powers who largely abandoned Finland during the war.

After Finland refused Joseph Stalin's demands on November 30, 1939, the Soviets invaded and kicked off what would eventually become known as the Winter War that officially lasted until March 13, 1940 and unofficially continued throughout all of World War II.

The Finnish volunteers used creative tactics to fight the Soviets, taking advantage of the Finn's small population and their knowledge of the terrain, ensuring the Soviets would have to pay a high price to capture Finland (8 Soviets per Finn killed).

However, the size of the Soviet army meant that defeat was still a very real possibility and Finland would need outside help defending itself. So the Finns asked for help from the Allied Powers, and controversially, after the Allies refused (they were busy fighting Nazi Germany) they asked for and received help from Nazi Germany.

Finland would eventually find that Nazi Germany wasn't an ally they wanted and began to realize their main problem was something totally out of their control, their geography, (one of the success factors). So they began to refuse support to the Nazis (possibly the turning point in the Battle of Leningrad) and started to evaluate their situation.

The Soviet army was so large in comparison to Finland's that if the Soviets really wanted to control Finland, they could. And although Finland to this day remains a very proud nation, it is also realistic. Instead of ignoring the Soviets (like they've been doing up to World War II) they began negotiating with them, and persuaded the Soviets that they would gain nothing by occupying Finland.

Amazingly, this approach allowed Finland to maintain its independence, and eventually became an important source of western technology to the Soviets during the Cold War (making Finland a powerful friend to the Soviets and ensuring Finland's independence). However, true to all compromises, it also came with drawbacks.

On top of having to drive crappy Soviet cars during the Cold War, Finnish newspapers were expected to censor themselves from reporting on Soviet abuses (a large compromise for a liberal democratic nation like Finland) in order to keep from offending Soviet sensibilities.

During the Cold War, western diplomats would coin the term "Finlandization" to mean weaker countries (Finland) pandering to stronger ones (USSR), but Diamond reminds us that these diplomats were from countries that don't share the same geography as Finland and never offered help when Finland was desperately trying to remain independent during the Soviet invasion in World War II.

In the last sections of the book, Diamond switches from looking at historical examples of nations in crises, and begins focusing on the current and future crisis we're dealing with, everything from climate change to political polarization, and applies the 12 success factors so that we might come out better at the end (much like Finland was able to after World War II),

Over all, while Jared Diamond doesn't go as far as to predict if we'll be able to successfully navigate through our future challenges, he does do an amazing job showing how other nations have creatively solved their biggest issues and provides a 12 step process to successfully navigate ours if we choose to follow it.

A Gentleman In Moscow

2020-01-05T00:00:00+00:00

At this point in my reading career (amateur status), most of the books I read can easily be classified as non-fiction, and when I do read fiction, it's rarely beyond a comic strip like xkcd, Sarah's Scribbles or Calvin & Hobbes. However, after watching Jordan read Amor Towles' A Gentleman in Moscow, it was easy to add this great book to my reading list.

At one point Jordan burst into laughter (not too uncommon) envisioning the Count, our main character, witness guests at the hotel dropping different objects (including an egg) from the second floor balcony of the Metropol's ballroom to test the accuracy of Newton's law of universal gravitation they had learned about in school. Towles' ability to illustrate scenes like this and his quirky attention to detail brings the Count to life in a way that by the end of the book you really feel like friends.

This surprisingly upbeat book follows the life of Count Alexander Ilyich Rostov, an unrepentant (according to the Bolshevik tribunal) Russian aristocrat, sentenced to life under house arrest in Moscow’s Metropol Hotel in 1922, after the Great October Socialist Revolution when the Bolshevik Party, founded by Vadimir Lenin, took power of the newly formed Soviet Union currently under provisional governmental control after Grand Duke Michael declined to take power after Tsar Nicholas II abdicated in 1917.

And while the book is fictional (and the Count as far as I know), the events happening outside of the very real Metropol Hotel are true to the interesting (and volatile) Russian history in the 20^th century. The book stays fairly focused on the Count's life inside the hotel, who feels like a man stuck in time with the Metropol, watching life go on around them, which allows large historical events, like World War II, to get little mention outside of the footnotes (a great story by themselves). So you don't need to be a Russophile to understand this addicting and fun book.

Without spoiling too much of this rich story, my favorite scenes of the book were of the Count, who together with the cook and maître d’, have managed to secretly collect all of the ingredients needed for a decadent seafood stew, called bouillabaisse, to share amongst themselves. After they've prepared the dish, they spend the afternoon telling stories and reliving memories. This moment in the book, together with Towles' ability to bring these moments to life, really do make you feel like you've missed a wonderful afternoon with friends.

Overall A Gentleman in Moscow is such a fun read with such a rich and amazing story spanning the full spectrum of romance, fun, politics, and thrills, making this book easy to recommend to friends for their 2020 reading.

Scraping the MLB for Science & Glory

2019-09-14T00:00:00+00:00

If you’ve been following baseball long enough you might have noticed nobody likes baseball anymore, and everyone has an opinion or stance or view or insight into how to fix it.

As someone who likes baseball (and statistics), I want to analyze these trends myself and find out if I need a new “national pastime”, or if these charlatans are just talking out their …er… padding their word counts.

So for this essay, we'll begin a multi-step journey, analyzing baseball statistics collected from the MLB to analyze the cause of "baseball's declining fan base" in later essays.

The Plan

Just like landing on the moon, blind dates, or playing with data science, having a plan before you start is usually a good idea.

As for our plan, we'll need to define a scientific question from the internet's "opinions" about baseball's decline before we can find sources of information we'll use to answer it.

The Question

While there are literally TONS [1] of articles with opinions on "how to fix baseball", the main claim I want to answer is:

This question seems to have been asked every baseball season, or at least before I stopped searching the internet at a 1999 Chicago Tribune article interviewing Ted Williams about the rise in strikeouts with the "new wave" of power hitters like Mark McGuire and Sammy Sosa.

Which I get, if no one can hit the ball then baseball is really 9 dudes throwing the ball to themselves for 3 hours, which is boring.

So, with a solid question defined, let's start looking for sources of information that could answer it.

[1]	"literally TONS" measured by page count from a google search, not by weight which the internet is estimated to only weigh ~50 grams or 0.00005 metric tons.

What Data Do We Need

Our question is dealing with trends in pitching and hitting statistics. So we'll need to find a source of baseball statistics going back to the beginning of baseball (anno domini ~1876) or as far back as we can find.

While there are many websites to choose from, I feel the data from the MLB is a good choice, mainly so we can say "according to the official MLB data…" before we start arguing with strangers on the internet.

Unfortunately though, as is often the case, the MLB doesn't give us a link to download their statistics for further analysis.

So we'll need to build a "web scraper" to download the data directly from their website, essentially making our own "download" button.

Some Rules Before We Start

Before we begin collecting data from the MLB, I feel I should give a few words (warnings) about internet etiquette:

Respect the MLB's terms and conditions. The MLB are savages in the box and can defy the laws of physics to destroy you if you mess with their copyright.
Don't stress their servers. Our scrapers can easily make thousands of requests per second which their servers will struggle to fulfill. (See rule #1 for why)
The code will break. Websites change their layouts like I change my …er… socks. Make sure you save the data you've downloaded so you only have to do this once.

Now with all the etiquette out of the way, let's start collecting some data.

Collecting Some Data

Back "in the day" this step would require "web scraping", using python libraries like BeautifulSoup, to parse the raw HTML for the data we want.

Now-a-days, with the increasing popularity of tools like AngularJS and RESTful APIs, websites are using JavaScript to render the web-page on our browsers (client-side) instead of on their servers (server-side).

The MLB is one of these "now-a-days" websites.

With a client-side app, the static HTML our browsers download is a template (skin) for the data. The JavaScript will download the data using the RESTful API and convert it into something we can easily read by applying the template.

As complicated as that sounds, because the data returned by the API is in a form our computers can use, it's often easier than scraping the web-page directly.

We just have to find the API.

Finding The Data

Because our browsers are doing all the work of building the MLB's client-side web-page, we can use our browser's built-in developer tools to find where the data is coming from.

Depending on which browser you prefer, open your developer tools:

How to open Chrome's Developer Tools.
How to open Firefox's Developer Tools.

With the developer tools open:

Find and open the Network tab.

For Firefox 69.0 the Network tab is in Tools → Web Developer → Network or if you're into key bindings Ctrl+Shift+E.

Next, go to the MLB's stats page and refresh the page with the Network tab open.

You should see a list of all the requests your browser made while building the MLB's web-page. Something similar to this (I'm using Firefox 69.0):

If you're in the year 2019, our browsers made 128 individual requests to their servers (54 more than average) for information to build the web-page. Which is 127 more requests I'm willing to look through by hand, so let's start filtering these requests down:

Click on the XHR icon to filter out unrelated requests.

The requests we're interested in are XHR (XMLHttpRequest) objects. These are requests made by the web-page's JavaScript to get data (usually in XML or JSON form) from an API.

With the requests filtered, we can start exploring for the data we want. We know we're looking for data which usually comes in either XML or JSON format, so we can ignore the "js" types.

Click on each Response looking for the data.

I usually look at responses with the largest size first and work my way down. After a few attempts, we'll find a JSON blob with keys that match the data we want.

Huzzah! We've found the API endpoint with the data we need.

Before we move on, take note of the data's shape in the response. Notice the list rows, with all the baseball stats, is inside queryResults inside team_pitching_season_leader_master. We'll need this when we start building the web scraper later on.

Our next step is to find how we can recreate the request using python:

Switch to that request's Headers tab and copy the URL.

This will show us the URL and the Headers we sent to the server.

If everything went according to plan, when you

Copy the URL into a new browser tab,

you should see the same information we found in the Developer Tools just a moment ago telling us we've found the URL to the API we want.

Good Job!

The Code

With the API discovered, we can begin building the web scraper to download the pitching statistics that will help us answer our initial question.

To make things easier on ourselves, let's install some libraries to help us download the data.

We'll be using:

requests to create the HTTP requests to the MLB's API and return its content.
and pandas to help us manipulate and analyze (data wrangle) later on, after we've downloaded the data from the MLB.

Both can easily be installed via pip like this:

$ pip install requests pandas

Writing the Code

From our exploration above, we know the URL to our endpoint is this:

team_pitching = (
    'http://lookup-service-prod.mlb.com/'
    'json/named.team_pitching_season_leader_master.bam?'
    'season=2019&sort_order=%27asc%27&sort_column=%27whip%27'
    '&game_type=%27R%27&sport_code=%27mlb%27&recSP=1&recPP=50'
)

However, this will only download each team's stats for the 2019 season. This API uses the query string parameters as a kind of filter for the content it provides to us.

For example, changing season from 2019 to 2018 would yield us each team's pitching statistics for the 2018 season or changing game_type to "%27S%27" will give us statistics on spring training games.

So let's break team_pitching down into its component parts, allowing us to make requests for the different seasons the MLB has in their database.

We can start by separating the domain, path, and query string parameters into separate variables like this:

domain = 'https://lookup-service-prod.mlb.com'
path = '/json/named.team_pitching_season_leader_master.bam'
query_string = {
    'season': 2019,
    'sort_order': '%27asc%27',
    'sort_column': '%27whip%27',
    'game_type': '%27R%27',
    'sport_code': '%27mlb%27',
    'recSP': 1,
    'recPP': 50,
}

After looking up what the %27 means (they're URL encoded ' apostrophes), let's make our future selves happy by making a simple function to replace the %27 (with a note) and turn our query_string into this:

# words are quoted with apostrophes
quote = "'{}'".format

query_string = {
    'season': 2019,
    'sort_order': quote('asc'),
    'sort_column': quote('whip'),
    'game_type': quote('R'),
    'sport_code': quote('mlb'),
    'recSP': 1,
    'recPP': 50,
}

With our URL broken down, we can make a simple for loop to go through each year (1876 - 2019), making a request to the MLB to download the pitching statistics for that season and save the results into a list creatively called data.

import time

data = []  # we'll save all the stats here

# go through each season and make a request
for season in range(1876, 2019 + 1):
    time.sleep(1)  # rule #2 don't hammer the servers

    # update the query_string for the correct season
    query_string['season'] = season

    # make the request to the MLB
    response = requests.get(
        domain + path,
        params=query_string
    )

    # raise an error if the MLB gives
    # an invalid response back
    response.raise_for_status()

    # pull the list of stats out of the response
    stats = response.json()[
        'team_pitching_season_leader_master'
    ]['queryResults']['row']

    # add the season to each row, and
    # append it to our 'data' list
    for team in stats:
        team['season'] = season
        data.append(team)

And finally, after we've collected all the data from the MLB, we put the data into a pandas DataFrame to start the "data wrangling" phase of our journey.

import pandas as pd
df = pd.DataFrame(data)

For now though, remembering rule #3 above, we'll save our data to disk and wait until we have more time to answer the question "Are strikeouts making baseball boring".

df.to_pickle('mlb-team-pitching-statistics.pkl')

Good Job Everyone!

You, Me & Machines... Learning

2019-08-28T00:00:00+00:00

In this essay, we’ll introduce the terms and how each component of a neural network works together to tackle a classic computer vision problem: analyze thousands of MNIST images of handwritten numbers and sort them (with 97.88% accuracy) into the digits they represent.

The Problem

For us, our brains have no trouble looking at low quality (28 by 28 pixel) images and deciphering their meaning. We’re essentially hardwired to find patterns in everything we see.

But how do we program a computer to do the same thing? Assuming there is no mathematical function we can use and a “6” can come in so many different shapes and forms, we can't rely on a specific part of an image to tell us what the number is.

This is where neural networks come to the rescue.

The idea behind modern neural networks, and all machine learning applications, is to analyze data and try to discover general patterns in that data so it can make predictions about new data it hasn't seen before. [1]

Meaning, we can use a neural network to let our computer discover the patterns in our images, and then use that pattern to sort each image into the digit it's depicting.

The Plan

So to solve our goal of accurately sorting images into the digits they represent, we’ll:

Download and prepare thousands of images of hand written numbers.
Build a feed forward neural network to analyze the images and sort them.

Then after we've built and trained the neural network to sort images for us:

Test the neural network using another set of images it's never seen before to grade how well it classifies images.

First-up is preparing our computer to “learn."

Preparing The Environment

Before we get started, I should point out that I’ve published the jupyter notebooks I’ve used for writing this on a GitLab repository and Google Collab, so you can play with the code without installing anything on your computer.

If you want to set this up on your own computer, and you already have python installed, a simple pip command will install all the libraries we’ll need for this project.

$ pip install keras tensorflow numpy mnist

We’ll use TensorFlow (a powerful deep learning library published by Google) as a backend to Keras, a library that dramatically simplifies the programming required to build a neural network.

As per usual scientific and “big-math” related python projects, we’ll also need NumPy to give us fast multidimensional array support in python.

And finally we’ll utilize the MNIST dataset, which is a subset of a large database called the NIST Special Database. The MNIST dataset is maintained by Yann LeCun, contains the 70,000 images of handwritten numbers we’ll use to train and test our neural network.

Loading, & Exploring The Data

Now that we’ve built our working environment, we can download the images and begin preparing them for our neural network. Thankfully the MNIST library makes downloading thousands of images as simple as importing a module in a new python file.

import mnist

training_images = mnist.train_images()
training_labels = mnist.train_labels()

testing_images = mnist.test_images()
testing_labels = mnist.test_labels()

After the images have downloaded (could take a while on slower connections) we can see the MNIST library has helpfully put our images into a NumPy array for us.

print(type(training_images))  # <class 'numpy.ndarray'>
print(training_images.shape)  # (60000, 28, 28)
print(training_images.min())  # 0
print(training_images.max())  # 255

Printing out some statistics about our NumPy array tells us each image (60,000 images in the training set) is in a 28 by 28 pixel matrix, with each pixel having a value between 0 and 255 (an 8-bit number) to tell our computers how “on” that pixel should be. The higher the pixel’s value the more “on” that pixel will be.

And while this is great for our human eyes, this isn’t a great format for our neural network. Meaning, like all true data scientists, we have to massage some data.

Preparing The Data (Normalizing)

An important step in preparing our images for our neural network is to "normalize" them.

This (usually) speeds up the time it takes to train our neural network, as well as avoid any problems that arise, if we have multiple datasets that use different ranges.

For our images, we'll only need to scale their range from their current values [0 - 255] into a more standard range [0 - 1]. We can do this using a min-max feature scaling funciton:

And, because our minimum value is 0, we can simplify the formula to just dividing by our maximum value in our range (255).

# ensure our numpy arrays use floating point decimals
training_images = training_images.astype('float32')
testing_images = testing_images.astyle('float32')

# reduce range to [0 - 1] (normalize)
training_images = training_images / training_images.max()
testing_images = testing_images / testing_imaoges.max()

Reshaping our Data (Flattening)

Our neural network will expect each image to be in a long 1 dimensional list of pixels. This means we'll need to "flatten" our images by removing it's 2^nd dimension before we can start training our neural network.

To “flatten” our images , we simply need to call NumPy’s reshape() method and specify a list of dimensions we want our new matrix in.

# flatten our images into 1 dimension
training_images = training_images.reshape((-1, 784))
testing_images = testing_images.reshape((-1, 784))

And just like that we’ve transformed our images from human readable to neural network readable.

# stats of our normalized data
print(training_images.shape)  # (60000, 784)
print(training_images.max()) # 1
print(training_images.min()) # 0

One-Hot Encoding

Now that we’ve finished preparing our images (questions) for our neural network, we can begin working on the labels (answers).

In order to train our neural network, we need to convert our labels from their perfectly human readable base-10 digit into a 10 item list format called “one-hot” encoding.

Without getting into too much detail, our neural network will have 10 output neurons with each neuron representing the probability of an image representing any digit in our 0-9 range.

We need to transform our labels into the probabilities that those 10 output neurons should output, effectively telling our neural network that we’re 100% sure this image is a 6.

To convert our labels into one-hot format, we need to use another NumPy method called eye()

# convert labels to ‘one-hot’ encoding
training_labels = np.eye(10)[training_labels]
testing_labels = np.eye(10)[testing_labels]

And with that, we’ve finished preparing our data to be processed by our neural network. Now’s the time to actually start building it.

Building The Model

The term “model” in machine learning has taken on a few different meanings from its original definition a few years ago [2]. In Keras, a model is used to describe the overall structure of our neural network.

There are two types of models we can build with Keras:

We can define more complex networks using the Model API, which is more verbose (complex) when setting up, which could lead to errors if we’re not careful.

Or we can use the Sequential API, which can only define a simple linear stack of layers, but makes up for this by removing a lot of the complexities the Model API has.

The type of neural network we’ll be building is called a feed-forward neural network. It’s essentially a single stack of layers, where each layer of neurons feeds their information forward to the next layer, making the Sequential API a perfect fit for our project.

from keras.models import Sequential
model = Sequential()

After we’ve initialized our Sequential model, we can use the model.add() method to add individual layers to it later on.

Adding Layers

As you’ve probably gathered, a neural network is built by using many nodes called neurons.

Each input, in a neuron, is multiplied by a weight (importance factor) before being added together along with a bias. Then, to keep this number in a specific range, the number is then fed into an activation function (we'll learn about these later) to produce the neuron's output.

In a feed-forward neural network, neurons are organized into layers, where each neuron in a layer will only connect to the neurons from adjacent layers, forming a sequential stack of layers.

Typically each neuron in a layer will connect to every neuron from the adjacent layer, forming a fully interconnected (dense) layer of connections.

In Keras, we have many types of layers we can use in our model, however, for this project, we’ll only need to use the Dense layer.

from keras.layers import Dense

Layer No.1

For the first layer in our model, we'll be connecting the 784 neurons in our input layer (the pixels in our images) to the 512 neurons in our hidden layer.

To add a dense layer to our model, we simply need to initialize it with some options, and add it to our model’s stack using the add() method we talked about above.

model.add(Dense(512, activation='relu', input_shape=(784,)))

The first argument, called units in the documentation, describes the number of neurons our hidden layer will have. 512 is a somewhat arbitrary decision I landed on while testing (playing around).

Because this is the first layer in our stack, we need to tell Keras the input_shape our images will be in when we start training our neural network.
The activation function, like we discussed, is how the neurons in our hidden layer will "squash" their outputs before outputting their results.

We’ll be using the Rectified Linear Units activation function (ReLU) which leaves any positive number unchanged but transforms any negative number into a 0 (no learning).

Layer No.2

Now that we’ve set up our first layer, we’ll need to build the connections to our output layer, allowing us to get the predictions out of our neural network.

Note

We can add more hidden layers here, but I found that having more than one did nothing to improve the accuracy of our neural network and slowed the learning rate considerably. Feel free to play around with my jupyter notebook to see what you find.

Our last Dense layer will be the output of our neural network. This layer will need 10 output neurons, one for each digit our image could be.

For the activation function, we'll use the softmax function, which allows us to calculate the activation of the 10 output neurons relative to each other. (eg: "I'm 20% sure this is a 4")

Also, because this is the second layer we're adding to our model, we can omit the input_shape argument we added in the first layer we built.

model.add(Dense(10, activation='softmax'))

And with that, thanks to the simplicity of Keras, we’ve finished building the structure of our neural network.

Compiling The Model

Now that we're done building the structure of our neural network, we can work on the methods and algorithms it will use to “learn” as we compile our neural network.

In Keras, we can simply call the model.compile() method with a couple of options to prepare our neural network for service.

model.compile(
    loss='categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy'],
)

The loss argument is telling Keras what type of loss function we want to use. A loss function is essentially a function used to measure how “wrong” our neural network is during training.

While Keras supports many types of loss functions, typically a loss function is chosen for us based on the decisions we've made eairler. For example, our neural network is classifying images, and we've encoded our labels into a "one-hot" format, meaning we need to use a categorical crossentropy loss function.

The optimizer is a function used to adjust the weights and biases of our neurons in order to minimize the value of our loss function.

Much like with the loss function, Keras supports many types of optimizer functions. For our project, we’ll be using the ‘adam’ optimizer function, mainly because it’s usually a great optimizer.

The metric argument tells Keras how we wish to evaluate how well our neural network is doing. Usually this will be metrics=[‘accuracy’] but for more complex networks with multiple output layers this can be different.

How To Train Your Model

Now that we have built the structure and defined how our neural network will “learn”, we can finally start the training process.

Training our neural network is easily done by calling the fit() method as shown below, along with some extra arguments.

model.fit(
    training_images, # normalized images
    training_labels, # one-hot labels
    epochs=5,
    batch_size=64
)

The first 2 arguments are for our normalized and flattened images along with the one-hot encoded labels we’ll be using to train our neural network.

The epochs argument tells Keras the number of times we will go through the entire set of images. Think of this as how many times you want to take the test.
batch_size is the number of images our neural network should process before updating each neuron with the results from our optimizer function.

Note

Increasing the batch size is usually an acceptable tradeoff as it dramatically speeds up the learning process, with limited effects to overall accuracy. [3]

Running the model.fit() above will give us an output like this:

Epoch 1/5
60000/60000 [======] - 7s 118us/step - loss: 0.2309 - acc: 0.9330
Epoch 2/5
60000/60000 [======] - 7s 113us/step - loss: 0.0904 - acc: 0.9734
Epoch 3/5
60000/60000 [======] - 7s 114us/step - loss: 0.0588 - acc: 0.9824
Epoch 4/5
60000/60000 [======] - 7s 113us/step - loss: 0.0405 - acc: 0.9871
Epoch 5/5
60000/60000 [======] - 7s 114us/step - loss: 0.0302 - acc: 0.9908

Because we gave the metrics=[‘accuracy’] argument when we compiled the neural network, acc: is printed in this output showing us how accurate our neural network is after each epoch.

Looking at the last epoch, we can see our neural network achieved a 99.08% accuracy with our training data. Whoop!!!

But Wait! There's More!

Before we celebrate too hard though, keep in mind this score is from a test our neural network has already taken 5 times. Like aceing a test we’ve already taken, we can’t be sure if we just memorized the answer to the question, or really understood the material.

When our neural network is better at remembering answers than classifying images, it’s “overfitted,” [4] meaning it started using “patterns” in our data that doesn’t help it classify random handwritten numbers, but instead the specific handwritten numbers we gave it.

To determine if our neural network is overfitting, we’ll use the extra 10,000 images we prepared but we haven't given to our neural network yet.

The Ultimate Test

This time instead of calling fit() to train our neural network, we'll call the evaluate() method to evaluate our neural network’s accuracy. This time supplying the testing images and labels that our neural network hasn't seen before.

This way, we can get an accuracy score on how well our neural network can sort any image of handwritten number, instead of just our dataset of handwritten numbers.

model.evaluate(
    testing_images,
    testing_labels,
    verbose=0
)

Running this code will give us the following output:

[0.07108291857463774, 0.9788]

The first item in the array is the loss function value.
The second item is our accuracy metric we asked for when we compiled the neural network above.

Meaning we’ve achieved a 97.88% accuracy score on fresh data!

Not bad for our first neural network.