Bryan Brattlof - Notes

How to Proxy Git Connections

2022-04-02T00:00:00+00:00

One of the first things you will notice when you start developing open source software for a larger corporation is the company firewall or proxy blocking all outgoing traffic not routed through their network appliance for inspection and review first. This sadly includes all of our git fetch traffic.

Before we can do anything upstream we first need to configure Git to pull in updates through these corporate firewalls. This is where the multipurpose relay tool, socat, shines.

Proxy http(s) Remotes

Git uses the Smart HTTP protocol to download updates from a remote that starts with https://* URI. These type of connections have become, by far, the most popular way to use Git today and is the default protocol used by many popular forges like GitHub or GitLab.

Internally Git relies on the libcurl library to handle these HTTP connections which means Git comes with the ability to proxy these requests built-in.

It also means that Git will respect your system's http_proxy and https_proxy environment variables. Simply add something like this to your ~/.bashrc or execute these lines in your shell before you need to pull updates from your remote project and Git will proxy the requests through your corporate firewall appliance automatically.

$ grep 'https\?_proxy' ~/.bashrc
export http_proxy=http://mega.co.proxy.example.com:8080/
export https_proxy=http://mega.co.proxy.example.com:8080/

Alternatively, if you want to keep all your Git configurations in one place, you can add this to our ~/.gitconfig to configure Git's proxy settings:

[http]
    proxy = http://mega.co.proxy.example.com:8080

If you only need to proxy the repositories hosted outside of the company network, you can use a pattern like this to filter the domains that use the proxy configuration settings.

For example any repository hosted on kernel.org servers, use:

[http]
[http "https://git.kernel.org"]
    proxy = http://mega.co.proxy.example.com:8080

For more details on proxy-ing http based Git projects, check-out the --proxy option in the curl(1) man page, or the http.proxy entry in the git-config documentation.

Proxy Git Remotes

Another common way to fetch updates from a remote tree is by using their git://* URI which uses the Git Server Protocol. Often used to share projects that do not require any user authentication as it is the fastest of the three transfer protocols available.

To proxy these types of requests we will need Git to run a simple shell script, like the example below, any time we fetch a remote repository. This example script will proxy all repositories outside of the example.com domain.

$ cat ~/bin/gitproxy
#!/usr/bin/sh
if [ $1 == *example.com ]; then
    # the repository is outside the company network, proxy it
    socat tcp:mega.co.proxy.example.com:8080:$1:$2
else
    # no need to proxy internal network traffic
    socat tcp:$1:$2
fi

Then, to have Git run this script, we can set the gitProxy setting in our ~/.gitconfig like this:

[core]
    gitProxy = ~/bin/gitproxy

Because this is just a simple shell script, we can make this as complicated or as simple as we want. For example, you can add more logic if you use a laptop that may need work outside of the office.

Proxy SSH Remotes

Finally, if the remote starts with ssh://* or uses a typical ssh connection like git@git.sr.ht:* we are connecting via the SSH protocol. These connections are typically used when you maintain the remote repository or otherwise need to authenticate yourself before you can git push changes to the server.

In these situations, Git relies on SSH to handle the authentication and connection to the remote, which means we will need to edit our ~/.ssh/config file to proxy these types of connections.

Inside OpenSSH's configuration file, we can use the ProxyCommand option for all remote excluding any URI inside our company's network.

Host * !*.mega.co.example.com
    User git
    ProxyCommand socat tcp:mega.co.proxy.example.com:8080:%h:%p

Or we can manually add the remotes we want proxied. For example any remote at git.kernel.org:

Host git.kernel.org
    User git
    ProxyCommand socat tcp:mega.co.proxy.example.com:8080:%h:%p

For more information about OpenSSH's config file options, checkout the ssh_config(5) man page.

Wrapping Up

I hope these notes were of some use or helped you in some way. I recently accepted my first open source role working for a larger company (>10,000 people) and wanted to write these notes down in the hopes it would help others working in open source projects behind walled gardens.

As always if you see something that wasn't correct or I need to update or clarify, please feel free to send me a note any kind of way that is convenient to you.

~Bryan

My PGP Cheat Sheet

2021-11-02T00:00:00+00:00

I use gpg about as often as tar which is just enough time to forget how to use it.

What lies below are my (a Linux kernel contributor) notes on some of the less used commands to keep your gpg keys safe and your copy of the Linux Web of Trust up to date.

Please feel free to contact me if anything you read below is wrong, out of date, or find something I should expand on. :)

Helpful Jump To Points:

Why PGP?
Convert A PGP Key To An SSH Key
Locating A Public Key
Updating Your Keyring
Signing A Public Key
Extending An Expiration Date
Backing Up A GnuPG Directory

Why PGP?

Today, the Linux kernel development cycle is solely (for now) an email based work-flow involving pulling changes from over ~300 different repositories. [1] These changes will eventually make their way into Linus' main tree (the mainline) for the final approval before being distributed to the masses.

[1]	Not every repository needs to be pulled before each release - though the linux-next repository, the repository subsystem maintainers use to resolve merge conflicts before they reach Linus, tracks around 300.

With the existence of phishing, spoofing and other social-engineering attacks, any one of these pull requests could conceivably bring malicious code via a spoofed email, fooling either Linus or any one of the subsystem maintainers.

After the widespread credential stealing attack that compromised much of the core kernel.org infrastructure in 2011, the decision was made to develop a PGP based Web of Trust to give developers a way to independently verify other repositories without a central authority.

Now days, pull requests must have a signed tag from a PGP key the maintainer trusts not only to validate the pull request, but to attest to the changes being made. Like a chain of custody.

Convert A PGP Key To An SSH Key

Each PGP key can have one or more roles; Signing, Encryption, Authentication or Certification. One "neat" feature about authentication keys is they can be used as an OpenSSH key, allowing you to store your SSH and PGP keys together on the same NitroKey.

Any time you need to give out your public SSH key, just use this command:

$ gpg --export-ssh-key hello@bryanbrattlof.com
ssh-rsa AAAAB3NzaC1yc2E ...

Linode has a good writup on how to get started using your PGP key with SSH.

Locating A Public Key

Around June of 2019 [2], the public SKS Keyserver network used to publish public PGP keys was attacked in a way that could not be easily fixed. The attacker(s) attached thousands of valid but useless signatures to well known PGP keys to bloat the complexity of the Web of Trust graph and render gpg unusable for anyone unfortunate enough to download a "poisoned" key.

[2]	This was a known problem for more than a decade.

As a work-around, the Linux community published a repository of all the well known contributers to the kernel called pgpkeys.git, that we can use to send and receive updates to our PGP keys to the kernel community.

Before we can find a key, we need to clone the kernel's PGP keyring repository:

$ git clone https://git.kernel.org/pub/scm/docs/kernel/pgpkeys.git korg-keys

To find someone's PGP key, git grep for the key in the repository:

$ git grep -l torvalds *.asc
keys/79BE3E4300411886.asc

Then import the key into our keyring:

$ gpg --import keys/79BE3E4300411886.asc

Alternatively: we could import all keys currently in the repository:

$ gpg --import keys/*.asc

Done!

With your key imported, we will need a way to update these keys from the PGP repository. See the Updating Your Keyring section below.

Updating Your Keyring

After the public SKS keyserver network was abandoned and the Linux kernel developer's PGP keyring repository was created, any update to a kernel developer's PGP key should be uploaded to the repository so they can be shared with the community.

To submit changes to your keys, use this command to email your updates to the mailing list:

$ gpg -a --export [Email] | mail -s [Email] keys@linux.kernel.org

To download updates from the keyring repository, a helpful script was added, allowing us to download any new changes using cron, systemd, or invoking the script manually.

I prefer to let systemd run the script:

$ cat ~/.config/systemd/user/korg-refresh-keys.timer
[Timer]
OnCalendar=daily
Persistent=yes

[Install]
WantedBy=sockets.target

$ cat ~/.config/systemd/user/korg-refresh-keys.service
[Service]
ExecStart=%h/bin/korg-refresh-keys -q
Type=oneshot

So I can see how the script is doing at any time by running:

$ journalctl --user -fu korg-refresh-keys

Signing A Public Key

Once you feel confident signing someone's key or UID(s) and attesting to the validity of their PGP key, you can sign it with the following command:

$ gpg --ask-cert-level --ask-cert-expire --sign-key someone@example.com

Each situation is different, and everyone (and each community) has their own methods for signing someone's key. This is only the default arguments I use, and adapt to the situation.

`--ask-cert-level`
	Prompt for a certification level allowing you to specify how confident you are about this signature. Useful to signal the difference between the random person you met at a conference versus your workmate.
`--ask-cert-expire`
	Prompt for an expiration time, allowing your signature to expire after a set amount of time. I like to put a expiration date on my signatures when possible if only to stop an old email address having a valid signature from me.
`--sign-key <name>`
	Signs a public key with your secret certification key. This is a shortcut version of the subcommand "sign" from `--edit-key`.

If you're like me and store your certification key offline in an encrypted USB drive, (something I strongly encourage) this process will be a little more complicated.

Don't forget to update your backup after you finish.

Extending An Expiration Date

By default, master certification keys have an expiration date set two years from the date of their creation. This is for security reasons and to let obsolete keys disappear from the Web of Trust.

To add one year (from the current date) to the expiration, run:

$ gpg --quick-set expire hello@bryanbrattlof.com 1y

Make sure to backup your GnuPG directory as well as email the kernel developer's keyring mailing list to let everyone know about the changes you have made.

Backing Up A GnuPG Directory

Now that we live squarely in the hyper-connected information age, the more sensitive data we can store, encrypted and offline, the better. This includes keeping all the secret key material used in our PGP keys off our working computers.

How much effort you spend backing up your private keys is ultimately your decision. My goal is to have a good story to tell the mailing lists explaining how I lost control of my keys. I feel comfortable having two LUKS encrypted USB drives and a paperkey backup, storing all of my PGP keys, and allowing me to restore my Nitrokey if that device happens to die.

If you are like me and prefer to keep your main certification key safely stored in an encrypted drive and off our working computer, you will need to mount your encrypted device first:

$ mkdir /media/device/gnupg-backup
$ cryptsetup luksOpen [DEVICE] gnupg-backup-enc
$ mount /dev/mapper/gnupg-backup-enc /media/device/gnupg-backup

Then we can tell gpg to use it:

$ export GNUPGHOME=/media/device/gnupg-backup
$ gpg --list-secret-keys

You should now see that sec# has been replaced with sec indicating that both the public and the secret key material are available for your main certification key.

Now is a good time to make changes that require our main key like Extending An Expiration Date, Signing A Public Key or running commands like fsck on the decrypted volume.

Once you have finished making your changes, we need to import these changes back into our everyday working .gnupg directory:

$ gpg --export | gpg --homedir ~/.gnupg --import
$ unset GNUPGHOME
$ gpg --list-secret-keys

You should now see sec# and ssb> have returned.

Finally, unmount and close our encrypted volume, and return our USB drive to its safe place.

$ unmount /media/device/gnupg-backup
$ cryptsetup luksClose gnupg-backup-enc
$ rmdir /media/device/gnupg-backup

Wrapping Up

I will say gpg is not the easiest tool to use. After 30 years of development, the time and effort needed to maintain the Web of Trust seems to be more than many are willing to endure.

As difficult as it is, gpg is a great open source tool, helping developers from all around the world regardless timezone, language, or access to git-forges like Github the opportunity to work on one of the most widely used software projects in the world.

I hope these notes helped you in some way. If you read anything that needs to be updated or you feel like I should expand on something, please feel free to write me an email.

The Linux Kernel Coding Style

2021-05-28T00:00:00+00:00

Like the (very) old saying goes: "Beauty is in the eye of the beholder". This old (Greek maybe?) saying even holds true for the software we write. The Linux kernel, like any sufficiently large software project, with thousands of developers each having their own ideas of beautifully readable code, the issue of how the code looks eventually becomes a topic of discussion.

These coding style standards, while sometimes annoying, are essential to letting developers worry about developing rather than trying to decipher another developer's "beautifully" written code.

So for the 4^th challenge in The Eudyptula Challenge we'll return our focus on the Hello World Kernel Module we created in the 1^st challenge, bringing it up to the (charmingly unique) Linux kernel coding standard of what beautifully written C code looks like.

Enjoy!

Task No.4

Wonderful job in making it this far, I hope you have been having fun. Oh, you're getting bored, just booting and installing kernels? Well, time for some pedantic things to make you feel that those kernel builds are actually fun!

Part of the job of being a kernel developer is recognizing the proper Linux kernel coding style. The full description of this coding style can be found in the kernel itself, in the Documentation/CodingStyle file. I'd recommend going and reading that right now, it's pretty simple stuff, and something that you are going to need to know and understand. There is also a tool in the kernel source tree in the scripts/ directory called checkpatch.pl that can be used to test for adhering to the coding style rules, as kernel programmers are lazy and prefer to let scripts do their work for them...

And why a coding standard at all? That's because of your brain (yes, yours, not mine, remember, I'm just some dumb shell scripts). Once your brain learns the patterns, the information contained really starts to sink in better. So it's important that everyone follow the same standard so that the patterns become consistent. In other words, you want to make it really easy for other people to find the bugs in your code, and not be confused and distracted by the fact that you happen to prefer 5 spaces instead of tabs for indentation. Of course you would never prefer such a thing, I'd never accuse you of that, it was just an example, please forgive my impertinence!

Anyway, the tasks for this round all deal with the Linux kernel coding style. Attached to this message are two kernel modules that do not follow the proper Linux kernel coding style rules. Please fix both of them up, and send it back to me in such a way that does follow the rules.

What, you recognize one of these modules? Imagine that, perhaps I was right to accuse you of the using a "wrong" coding style :)

Yes, the logic in the second module is crazy, and probably wrong, but don't focus on that, just look at the patterns here, and fix up the coding style, do not remove lines of code.

Attachments:

hello_world.c - Note: This should be the module you submitted from the First Eudyptula Challenge. If you've been following along, this is the file we submitted.

coding_style.c

—Little Penguin

As the Little Penguin was saying, the Linux style guide is here not for your benefit, but to help other developers, maintainers, and reviewers understand and evaluate the code you write faster. The kernel works at a furious pace, receiving on average ~10 patches every hour, a pace that is only accelerating as more people and companies begin to depend on the project for the new device, radio, microwave or any other IoT gadget they're working on.

TLDR: The style guide helps everyone review your code which improves the chances of your patch being accepted.

Executive Summary of the Linux Kernel Coding Style

The style guide is a very straight forward document, the majority of it dealing with whitespace and naming things. (Something every project has to deal with regardless the number of developers it has) I'll list the "executive overview" here (for the SEO points 🙂) though you should really take a moment to skim the Linux coding style guide for the latest updates.

tabs are 8 characters, one statement per line. Historically tab keys were used to help tabulate charts on typewriters. Now they help maintainers keep track of your control blocks.

80 characters per line. An old throwback to when screens only displayed 80 characters per line. "Now-O-Days", with modern 48" 8k monitors, the 80 character hard limit is a bit softer. Today we can get away with a few lines under 100 characters if it improves readability. One possible exception to this is printk() statements, allowing us to easily grep for them when debugging.

brackets. Don't worry about brackets if you only need one line.

if (err)
        return -ENOMEM;

For multiple lines, place the bracket at the end of your if (switch, do, for, while, ...) statements.

if (err) {
        pr_dbg("Failed to allocate memory");
        return -ENOMEM;
}

Though function have brackets on the next new line.

int do_something_cool(void)
{
        return something_cool == DONE;
}

spaces. Most of the time, spaces go after each keyword. Some notable exceptions to this are sizeof, typeof, alignof, and __attribute__ which are treaded somewhat like functions in the kernel.
naming. Don't worry about descriptive variable names inside a function. tmp or err are perfectly acceptable. However global variables (to be used sparingly) should have descriptive names. Don't use MixedCase!
typedefs. Just Don't. Typedefs only hurt readability. You can't immediately tell if it's a struct or an integer of some specific size, or a pointer (to a pointer, to a pointer... to...).
commenting your code. If your explaining how the code works, you've written it wrong. Comments are for what your code does if the function's name didn't make this clear already.

These bullets take care of the majority of what a new kernel developer will deal with on a day-to-day basis, and if you're ever unsure about something you can always git grep <pattern> to see how others have coded similar things in the past.

scripts/checkpatch.pl

Fortunately, previous developers created a script to check for some of these style blunders and alleviate the burden put on maintainers to point out these minor coding style violations. This script is one of many helpful scripts inside the Linux source tree we downloaded in the second challenge. The one we're interested in and we will be using today is called scripts/checkpatch.pl.

At the root of the Linux source code, use the --help or --version options to see the usage instructions and get a complete list of all the options available.

$ ./scripts/checkpatch.pl --help

For this Eudyptula Challenge, we will remain focused on the --file or -f option to analyze the files the little penguin gave us (hello_world.c and coding_style.c) for style violations. I won't go into the specific changes we need to make (a task for you to complete). Instead I will go through how to use checkpatch.pl to find some of issues with the files the Little Penguin gave us.

We can check our file using the -f option followed by the file we want to check:

$ ./scripts/checkpatch.pl -f ~/eudyptupla/tasks/04/coding_style.c
...
total: 1 errors, 1 warnings, 1 checks, 35 lines checked

NOTE: For some of the reported defects, checkpatch may be able to
      mechanically convert to the typical style using --fix or --fix-inplace.

eudyptupla/tasks/04/coding_style.c has style problems, please review.

Then use you favorite editor to fix the issues checkpatch.pl printed out.

You can even use --strict or the --subjective option to enable more subjective tests and --codespell to check the file for misspelled words.

$ ./scripts/checkpatch.pl --strict --codespell -f ~/eudyptula ...

Once you've made your changes, and feel the code looks good (it's hard when you can remove any lines) use git format-patch HEAD~ and send in your changes.

Good Luck!

Wrapping Up

If you made it here, you may be surprised to see how little there was to this challenge. Essentially this "boils down" to reading the Linux style guide and practice using it. Though, so far in my software career, I have found that true for all coding standards.

PS: I made around 39 changes in all, though this may change depending on when you work on this. Hope that helps :)

Cgit, Nginx & Gitolite: A Personal Git Server

2021-01-12T00:00:00+00:00

As I (and my responsibilities) continue to grow older, I find myself having less and less time to do the things I used to (write and maintain servers just for fun). And while I continue to want to do these things, I have to be realistic and prioritize the things I wish to do with the ever decreasing time I have to do them.

All of that to say, I've "sold-out" and transitioned to paying someone else (sourcehut) to worry about maintaining my publicly accessable repositories for me and allowing me to prioritize other exciting and novel things.

I've been on a "own my online presence" kick for more than a year now. So for this (overly protracted) essay, I thought I'd publish my notes on how I created my own Git server.

There are many open source projects like GitTea or GitLab to make hosting your own git projects effortless; however I wanted a much more simple (read: old school) setup. I ended up with something that uses many of the same projects that the Linux Organization uses to publish the Linux Kernel

The server (as of this writing) uses Ubuntu's 20.04.1 LTS (Focal Fossa) running on Digital Ocean's hardware (referral-link). I wholeheartedly support and recommend you chose a different setup. Diversity in people and in tech stack is always and will always be a great thing.

What lies below can be broken into 3 main topics:

The Start prepares a newly minted server for git hosting duties. Creating a new admin user, locking down the OpenSSH daemon, and installing fail2ban.
Gitolite installs and configures the server to allow us (and colleagues) to have more fine-grained control over who has access to git push|fetch on the server.
And Cgit, fcgiwrap, and Nginx to create a web-server to view our published projects.

In the end, you'll have a server much like this one.

Enjoy!

The Start

I often find security "best practices" are a lot like driving down the highway. Some people speeding past you are "obviously" just moments away from a major data breach, while the others you're passing are "clearly" so worried about the entire data-center burning down, they couldn't possibly get anything else done. Everyone thinks everyone else has lost their marbles.

So with that in mind, here are a few steps I took to secure my newly minted server. Please feel free to use only the "best practices" you deem appropriate for your mission.

Or just skip to the "installing Gitolite" part directly.

Admin User

For whatever reason, be it for security or protecting the server from my stupidity, one of the first things I do when creating a new server is add a new user for my general admin tasks.

Adding a new user is remarkably easy to do on a Ubuntu system:

$ adduser limb

You'll be prompted to answer a few questions, including creating a new UNIX password. This will be the password you'll need to sudo -i and gain root permissions, so make it a good one, or use tools like Pass, or BitWarden to help you remember.

Then give our new limb user sudo permissions:

$ usermod -G sudo limb

I've also largely eliminated all password based authentication when signing into servers, relying on open source smart cards like NitroKey for authentication. If interested, this requires we setup .ssh/authorized_keys for our limb user:

Just replace key with your public ssh key:

$ mkdir /home/limb/.ssh
$ echo "key" > /home/limb/.ssh/authorized_keys

Next, set the .ssh directory's file permissions so the ssh daemon can read the files:

$ chown -R limb:limb /home/limb/.ssh
$ chmod 700 /home/limb/.ssh
$ chmod 644 /home/limb/.ssh/authorized_keys

If everything worked, after you restart the ssh daemon (service sshd restart) you will now be able to login as the administrator user:

$ ssh limb@host

OpenSSH

Git and Gitolite (installed in the next sections) will need us to keep port 22 open, allowing us to git push from anywhere on the internet. This open port will eventually attract "a lot" of attention from bots who endlessly scour the internet looking for vulnerable servers, mindlessly stuffing passwords, hoping one password will eventually let them in.

We can eliminate all worry about weak or compromised passwords by disabling all password based authentication, relying solely on asymmetric cryptography, or "ssh keys". Just use your favorite text editor to open /etc/ssh/sshd_config and ensure these lines exist somewhere in it:

PubkeyAuthentication yes
PasswordAuthentication no

While we're here, a large majority [1] of these bots are interested in logging in as the root user. If you created a new admin account in the previous section and ensured you can login using your public key, you can also disable root logins entirely with this line in the config:

PermitRootLogin no

[1]	Some simple "bash-fu" on my `/var/log/auth.log` shows ~93.58% of the roughly 15,000 login attempts since I started this server, tried to login as `root` Second place was the user `git` (including legitimate logins) at ~1.82%.

If you uploaded your public key to your VPS provider, most of these changes should have already been configured for you. But in the off chance you had to make some changes, restart the ssh service to load the new config changes in:

$ service ssh restart

Uncomplicated FireWall

Depending on your VPS provider, they may also have a firewall system built into their admin panel allowing you to apply rules simply by adding tags to a server. However, I enjoy keeping all my firewall rules inside each box, if only for the same reason I keep all my socks on the left hand drawer, so everything stays organized and in the same place.

You can install ufw using the Advanced Packaging Tool:

$ apt install ufw

Right now, the only thing we have enabled is ssh which uses port 22. To allow port 22 through ufw just use the following command:

$ ufw allow ssh

and then turn the firewall on:

$ ufw enable

and viola! You have a firewall.

fail2ban

Even though we've turned off password based authentication in a previous section, we will still receive a significant amount of bots wasting our compute cycles trying to login. And while the likelihood of this being successful is zero when rounded to any order of magnitude, the bots will nevertheless continue to pilfer a non-zero amount of CPU if given the opportunity.

To stop the most brazen of these bots, tools like Fail2Ban, which creates temporary firewall rules to block IP address who repeatedly fail to authenticate with ssh, are a great compromise between usefulness and annoyance.

The Advanced Packaging Tool can again help us install fail2ban.

$ apt install fail2ban

Once installed, the ssh "jail" will come pre-enabled for you. If you wish to make any changes, you will need to make a copy of the fail2ban config file:

$ cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Then add your changes to jail.local so they will persist after an upgrade.

fail2ban does a great job documenting what each option does in the config file. Some of the changes I made are:

because I use ufw to manage my firewall, I changed banaction = ufw.
enabled bantime.increment to increase the duration of a ban based on how many times the IP address has been banned previously.
enabled bantime.rndtime to "randomize" the length of a ban, preventing bots from knowing exactly when they can resume their assault.
enabled bantime.maxtime so I won't need to unban IP addresses (if you're unfortunate enough to share an IP with a bot).
lowered bantime, findtime and maxretry allowing me to issue small bans that increase in severity as the IP address continues to antagonize.

Once you're satisfied with your changes, start fail2ban using systemd.

$ systemctl enable fail2ban
$ systemctl start fail2ban

And Done!

Depending on how "popular" you are on the internet, you should start to see NOTICE lines in /var/log/fail2ban.log of misbehaving bots and the equivalent firewall rules in ufw.

$ cat /var/log/fail2ban.log | grep 'NOTICE' | tail -1
...  [125553]: NOTICE  [sshd] Ban 156.155.159.161

$ ufw status
Status: active

To                         Action      From
--                         ------      ----
Anywhere                   REJECT      156.155.159.161
22/tcp                     ALLOW       Anywhere
22/tcp (v6)                ALLOW       Anywhere (v6)

Gitolite

Installing Gitolite is amazingly simple, there are no binaries to compile or daemons to monitor.

At its core, Gitolite is just a collection of Perl scripts that run after someone signs into the server using the ssh daemon we configured in the previous sections. Once installed, Gitolite will give us more fine-grained-control over who has git push|fetch permissions to each repository. I encourage you to checkout Gitolite's amazing documentation if only to see how capable Gitolite can be.

Step: 1 - Create The Git User

Before we install Gitolite, we'll need to create a new user for everyone to log into and to run Gitolite's Perl scripts. I typically use the username git for this, feel free to replace git with the username that you feel is more appropriate.

$ adduser --system --group --disabled-password --home /var/lib/git git

This creates a new system-user on the server called git. Because this is not a "normal" user, there will be no aging information in /etc/shadow, which is convenient when nobody will be monitoring this account.

We also used the --home option to set the $HOME variable to /var/lib/git. This is where we will eventually put Gitolite's configuration files and our Git repositories. Feel free to adjust this to where you prefer, I've seen many use /home/git.

I included the --disabled-password to disable any password based access into our new user. In the previous sections, we've disabled all password based authentication into the server and Gitolite requires ssh keys for authentication, so disabling passwords for our user is a smart move.

Step: 2 - Install Gitolite

Because Gitolite is just a bunch of Perl scripts, I prefer to install Gitolite from the source. As we will see, installing Gitolite from source also has the benefit of making upgrades and adding custom patches in the future extremely easy.

This also means we'll need to install Gitolite's dependencies ourselves:

$ apt install perl git

When we (or a colleague) signs into the server, using the git user, we will automatically run Gitolite's Perl scripts, which means these scripts must be executable by our git user. So, to make managing file permissions easier, we'll use our git user for the rest of the installation process.

Log into our git user with the "substitute user" command: (assuming you're the root user)

$ su - git

Then clone Gitolite's source code into the $HOME directory: (this should be /var/lib/git unless you changed it when we setup the git user above)

$ git clone https://github.com/sitaramc/gitolite

Step: 3 - Setup Gitolite

To setup Gitolite on the server, we'll need to assign Gitolite an admin that will have full control over editing Gitolite's configuration repository. This will most likely be you.

Still as the git user, use your favorite text editor to create a new file with your desired username in the $HOME directory and copy your public ssh key into it. For example, my file would be called bryanbrattlof.pub and look like this:

$ cat $HOME/bryanbrattlof.pub
ssh-ed25519 AAAAC3NzaC1l ...

Then run Gitolite's sanity checks and setup script to finish the installation:

$ $HOME/gitolite setup -pk bryanbrattlof.pub

Färdig!

If everything went well, you should now be able to clone Gitolite's configuration repository:

$ git clone git@host:gitolite-admin

I recommend consulting Gitolite's incredible documentation to understand how to properly configure access and add hooks to all of your projects.

Step: 4 - Add ~/.profile

While not technically needed for Gitolite to function properly, I find adding gitolite to our git user's $PATH is a great quality of life improvement on the rare days I need to play system administrator.

First, as our git user, create a new $HOME/bin directory:

$ mkdir -p $HOME/bin

Next, create the $HOME/.profile file and add the $HOME/bin directory to our $PATH.

PATH=$HOME/bin:$PATH

The born again shell will automatically run $HOME/.profile when someone starts a new session.

Now, we can use Gitolite's install script to add a symbolic link inside our $HOME/bin folder:

$ gitolite/install -ln $HOME/bin

Done!

Logout and back in to the git user (or use source $HOME/.profile) to pick up the changes. If everything was done correctly, you won't need to type the full path to Gitolite anymore.

$ whereis gitolite
gitolite: /var/lib/git/bin/gitolite

Cgit

Cgit is a script (written in C) that uses the Common Gateway Interface (CGI) specification to give people a web view of our projects. Convenient when you don't have access to your terminal or just want to lookup (or showoff) some changes to a project.

It operates as a back-end (much like PHP) to a webserver (we'll install Nginx in the next sections) that will parse our repositories and return a web-page for our webserver to distribute.

To get an idea for what Cgit will look like, some of the more popular projects that use Cgit are the Linux and FreeBSD kernels, along with Wireguard and Cgit itself.

Step: 1 - Install Cgit

Just like with Gitolite, I prefer to install Cgit from source so I can add personal patches and quickly change what version is running on the server. This also means we'll need to install the dependencies ourselves:

$ apt install libc6 liblua5.1-0 zlib1g \
  python3-docutils python3-markdown python3-pygments

We'll also need the build-essential packages to install the gcc and make tools needed to compile Cgit after we've cloned the project:

$ apt install build-essential

Then, as the root user in the /root directory clone the Cgit project:

$ git clone https://git.zx2c4.com/cgit

Because Cgit uses parts of Git's source code, (included as a submodule) we'll need to use git submodule to download the remaining code from the Git project.

After you cd into cgit

$ git submodule init    # register the git submodule in .git/config
$ git submodule update  # clone/fetch and checkout correct git version

Step: 2 - Build Cgit

With a full copy of Cgit on the server, we can now create some patches to customize it for our use-case. We'll start with creating cgit.conf inside the cgit project we just cloned, to tell make where we want to install the Cgit binaries.

CGIT_SCRIPT_PATH = /var/www/html/cgit/cgi
CGIT_CONFIG = /var/www/html/cgit/cgitrc
CACHE_ROOT = /var/www/html/cgit/cache
prefix = /var/www/html/cgit
libdir = $(prefix)
filterdir = $(libdir)/filters

Because this is a version controlled project, we can commit our changes to save our work:

$ git add -f cgit.conf
$ git commit -m "installation path changes"

Some additional changes I made:

Updated the cgit.png and favicon.ico icons
Changed the pygments highlighting style to "algol_nu"
Removed the Data URI icons from the tab menu
Limited the max-width of readme pages to 95ch
Added padding: 1em; to code-blocks

When you're satisfied with your changes, use make to compile and install Cgit:

$ make && make install

If everything went well, when you execute Cgit from the terminal, a web-page should print out:

$ ./cgit
Content-Type: text/html; charset=UTF-8
Last-Modified: Tue, 12 Jan 2021 22:35:43 GMT
Expires: Tue, 12 Jan 2021 22:40:43 GMT

<!DOCTYPE html>

And Done!

We can further customize Cgit's behavior using the cgitrc file located at /var/www/html/cgit/cgitrc. Feel free to check out the man page for a complete description of what every option does.

Some of the options I used:

set scan-path to the location of our repositories /var/lib/git/repositories
set project-list to the location of the projects.list file Gitolite creates, adding descriptions and categories to the list of repositories on Cgit's index page

FastCGI Wrapper

Cgit, which uses code from Git, was designed to let users run a command (eg: git push) then exit, allowing our computers to reclaim the used resources between each call. Nginx uses a faster protocol (FastCGI) which calls the same program multiple times without exiting.

However because Cgit was designed to exit after every run, it will never give back its used resources and will continue to take more, quickly exhausting all of the computer's available resources. This is why we need fcgiwrap.

Thankfully this is easy to install. The Advanced Packaging Tool can, once again, help:

$ apt install fcgiwrap

Then just enable and start the service:

$ systemctl enable fcgiwrap
$ systemctl start fcgiwrap

And that's a wrap!

Nginx

With Cgit and the FastCGI Wrapper installed, we can now turn our attentions to Nginx, which can be installed using the Advanced Packaging Tool:

$ apt install nginx

Next, create a new configuration file in the /etc/nginx/sites-enabled directory, replacing git.bryanbrattlof.com with your domain. The minimum configuration file you'll need for Cgit to work will look something like this:

server {
    server_name  git.bryanbrattlof.com;

    listen [::]:80;
    listen 80;

    access_log  /var/log/nginx/cgit-access.log;
    error_log   /var/log/nginx/cgit-error.log;

    root /var/www/html/cgit/cgi;
    try_files $uri @cgit;

    location @cgit {
        include          fastcgi_params;
        fastcgi_param    SCRIPT_FILENAME /var/www/html/cgit/cgi/cgit.cgi;
        fastcgi_pass     unix:/run/fcgiwrap.socket;

        fastcgi_param    PATH_INFO    $uri;
        fastcgi_param    QUERY_STRING $args;
        fastcgi_param    HTTP_HOST    $server_name;
    }
}

Feel free to add more to this, I've added a custom 5xx page, caching headers, as well as recommendations from Mozilla Observatory.

Once satisfied, start the Nginx service and open port 80 in the firewall:

$ service nginx start
$ ufw allow http

If something went wrong, or if you ever change the configuration file, you can use nginx -t to check the configuration for errors and nginx -s reload to restart the Nginx server.

$ nginx -t && nginx -s reload

Wrapping Up

By now we should have a working git server. However like most creative things "90% done ... 90% left to go." There is truly an endless supply of things you can and should add or configure to make your server more secure and accessible. If you're the type that likes to learn, then you'll likely find this as fun and rewarding experience as I did.

Some of the extra things I added:

Installed certbot to install and manage a free SSL certificate from Let's Encrypt. This has largely been mandatory for any public server for around 5 years now
Created a Borg based backup script with a Borg specific subscription to rsync.net to backup my projects. Useful when that jackass we talked about finds a 0-day
Created a Git Daemon service to allow people to clone my projects using the git:// protocol, if they prefer
Placed a bunch of Healthchecks Pings in the scripts and service required to keep everything running. Fail2Ban, Borg Backup, Certbot, all will alert me when cron or systemd fall over

All of which should be given their own essay as they can be used in all your setups, not just in this fully open-source and free (as in libre) git server.

Modify The Linux Kernel

2020-08-15T00:00:00+00:00

For the 3^rd task (this task) in The Eudyptula Challenge, we continue our work from the last challenge, compiling the absolute latest Linux Kernel from the source code. This time we focus on modifying the Makefiles and .config files that help us compile the kernel.

Before we begin, if you wish to work on The Eudyptula Challenge yourself before you read my notes (recommended), you can use my git repository, which has all 20 tasks and the code I used to complete each one.

Task No.3

Now that you have your custom kernel up and running, it's time to modify it!

The tasks for this round is:

take the kernel git tree from Task 02 and modify the Makefile to and modify the EXTRAVERSION field. Do this in a way that the running kernel (after modifying the Makefile, rebuilding, and rebooting) has the characters "-eudyptula" in the version string.

show proof of booting this kernel. Extra cookies for you by providing creative examples, especially if done in interpretive dance at your local pub.

Send a patch that shows the Makefile modified. Do this in a manner that would be acceptable for merging in the kernel source tree. (Hint, read the file Documentation/SubmittingPatches and follow the steps there.)

—Little Penguin

Depending on how literally we interpret "modifying the Makefile", there are multiple ways we can accomplish this challenge. One of which we briefly talked about in the last challenge was to override variables in our make command when compiling the kernel.

Override Directives

Just like with a lot of userspace projects, the Linux kernel uses GNU make to compile the various files into its final form. This means we can use make's ability to pass variable assignments as command line arguments.

From Chapter 6 of make's documentation, if we use a command line argument to set the EXTRAVERSION variable, then all other assignments to EXTRAVERSION within the Makefile will be ignored. For example, we can override EXTRAVERSION by using a make command like this:

$ make -j `getconf _NPROCESSORS_ONLN` EXTRAVERSION=-eudyptula

This will force make to set EXTRAVERSION to -eudyptula and ignore the value set in the kernel's Makefile, accomplishing our task for The Little Penguin.

Modify the Makefile

Our second option, if you want to take "modifying the Makefile" literally, is to do exactly that.

Simply open the kernel's Makefile, located in the root directory in the source code we copied from Linus in the last challenge, with your favorite text editor. The first five lines will have EXTRAVERSION somewhere in it. Change the value to -eudyptula and save.

-EXTRAVERSION = -rc1
+EXTRAVERSION = -eudyptula

Now that the kernel's Makefile will append -eudyptula to the kernel's version string by default, we can simplify our make command to build the kernel into:

$ make -j `getconf _NPROCESSORS_ONLN`

Modify menuconfig

Our third option and arguably the least accurate of the three ways to interpret "modifying the Makefile" is to use the ncurses configuration tool that we briefly talked about in the last challenge. I say this is the least accurate because this method modifies the CONFIG_LOCALVERSION variable and not the EXTRAVERSION variable, which will add -eudyptula to our kernel's version string, however not technically in the correct place.

We can start the ncurses menu with the following terminal command:

$ make menuconfig

where you should see a screen that looks something like this:

From here, navigate to the General setup option and press ENTER to move into the next menu. Next, use the arrow keys to highlight the Local version option and press ENTER. A new menu will appear letting you enter a value.

Type in -eudyptula and press TAB to move our cursor to the <OK> option and press ENTER to return back to the General setup menu. If everything went according to plan you should see "-eudyptula" set in the Local version menu:

From here, press TAB a few more times to move our cursor to highlight the <SAVE> option at the bottom. Then press ENTER to save the changes we made to our .config file.

After we've saved our changes press TAB a couple more times to highlight the <EXIT> option and ENTER to exit the ncurses menu.

Finally, with the changes to our .config file saved, we can re-compile our kernel with the same simpified command from above:

$ make -j `getconf _NPROCESSORS_ONLN`

Building The Linux Kernel

2020-07-31T00:00:00+00:00

Previously, for the 1^st task in The Eudyptula Challenge, we built a Loadable Kernel Module (LKM) that adds "Hello World" to our kernel's message buffer any time the module is installed. You can read my notes all about that here if you want to learn more.

Or, if you wish to work on The Eudyptula Challenge yourself before you read my notes (recommended), you can use my git repository, which has all 20 tasks and the code I used to complete each one.

For this challenge, the 2^nd task of The Eudyptula Challenge, we will use the Kbuild system once again. This time we will use the Kbuild system to compile the latest version of the Linux Kernel. Then after compiling it (this takes some time), we install and boot from it.

Task No.2

Now that you have written your first kernel module, it's time to take off the training wheels and move on to building a custom kernel. No more distro kernels for you, for this task you must run your own kernel. And use git! Exciting isn't it! No, oh, ok...

The tasks for this round is:

download Linus's latest git tree from git.kernel.org (you have to figure out which one is his, it's not that hard, just remember what his last name is and you should be fine.)

build it, install it, and boot it. You can use whatever kernel configuration options you wish to use, but you must enable CONFIG_LOCALVERSION_AUTO=y.

show proof of booting this kernel. Bonus points for you if you do it on a "real" machine, and not a virtual machine (virtual machines are acceptable, but come on, real kernel developers don't mess around with virtual machines, they are too slow. Oh yeah, we aren't real kernel developers just yet. Well, I'm not anyway, I'm just a script...) Again, proof of running this kernel is up to you, I'm sure you can do well.

Hint, you should look into the 'make localmodconfig' option, and base your kernel configuration on a working distro kernel configuration. Don't sit there and answer all 1625 different kernel configuration options by hand, even I, a foolish script, know better than to do that!

After doing this, don't throw away that kernel and git tree and configuration file. You'll be using it for later tasks, a working kernel configuration file is a precious thing, all kernel developers have one they have grown and tended to over the years. This is the start of a long journey with yours, don't discard it like was a broken umbrella, it deserves better than that.

—Little Penguin

Build Tools

Just like mechanics, plumbers, and all the other skilled trades, having the right tools for the job makes a world of difference. This "tools maketh man" proverb rings true even for software developers, even though most of our tools are less tangible.

The tools we will need to properly build the kernel vary wildly depending on many factors, like your hardware, kernel version, how we plan to install the kernel, as well as the linux distribution on your computer. All the minimum requirements we will need to build the kernel are listed in the documentation here, chances are though you will need to use more.

The tools I needed to build version 5.8 of the kernel on my (charmingly retro) Lenovo T430, running Ubuntu's 18.04 LTS (Bionic Beaver) release, are easily installed with:

$ sudo apt install build-essential libncurses-dev bison flex libssl-dev libelf-dev

Cloning

When you first navigate to the kernel's git repositories, you will be greeted with hundreds of projects all having something to do with archiving or improving the Linux Kernel in some way. To help simplify the process of finding Linus' repository, which has the latest patches, we can search through the projects listed (using Ctrl+F) for "torvalds/".

Once you have found the URL, we can use git to download a copy of Linus' Linux repository using the typical git clone command. If you are unsure about how to clone a repository, the git documentation has a great article about getting started with git.

$ git clone https://git.kernel.org/pub/scm/linux ...

Configuring

With a copy of the kernel's source code in hand, our next task is to set the many thousands of configuration options needed for the Kbuild System to properly compile our kernel. Each configuration option, like the CONFIG_LOCALVERSION_AUTO the Little Penguin mentioned in the challenge message above, informs the Kbuild System of the drivers and features needed to be installed to function properly with our computer's hardware.

However, instead of setting roughly 8700 config options by hand, one-at-a-time (a very error prone, time consuming process), we can copy the configuration file from the linux distribution currently running on our computer as a sort of starting point.

Configuration Plagiarism

The first step in copying our distribution's configuration file is to find it in our /boot directory.

There are multiple configuration files inside our /boot directory. The one our computer is using depends on the kernel version our computer is using. For example my computer, using kernel version 4.15.0-112, will have a configuration file named config-4.15.0-112-generic in the /boot directory.

To copy your distribution's config file into the root directory of the kernel use the this command:

$ cp /boot/config-`uname -r` .config

Next, answer yes to any new configuration options added between the release of our distribution's kernel (they usually lag behind a few versions) and the bleeding edge we've downloaded from Linus.

$ yes "" | make oldconfig

Configuring By Hand

While not technically needed to complete this eudyptula challenge, this is a good place to talk about how we can customize our newly copied configuration file for our kernels. While there are a few, largely similar, commands that help us edit the .config file, the command I prefer is a ncurses based menu that can run on most systems or on remote servers if wanted.

$ make menuconfig

It produces a color menu, full of radio-lists and dialogues, that looks something like this:

Each menu option allows us to enable and disable certain features in our kernels, along with anything that depended on that option. For example, if we disabled Networking, all network-related options, like Amateur Radio, will also be disabled.

The full list of commands used to edit the kernel's configuration file is available to you in the kernel documentation, if you want to learn more. They largely use different GUIs to edit our .config file.

Building

With all the preparations complete, we can start to work on compiling the kernel into a compressed image ready for our computers. Just like in userspace applications, a simple make command is all that is needed to build our kernel:

$ make -j `getconf _NPROCESSORS_ONLN` CONFIG_LOCALVERSION_AUTO=y LOCALVERSION=-eudyptula

While a bare make command, without the added arguments, will also work to build our kernel, however, the kernel is such a large project, using the -j flag to compile multiple files at the same time will dramatically reduce the total time needed to compile.

We are also passing two extra configuration options with the make command above:

CONFIG_LOCALVERSION_AUTO=y LOCALVERSION=-eudyptula

Both of these configuration options will add information to our kernel's version string, helping us prove to The Little Penguin we installed the latest kernel published by Linus Torvalds.

The CONFIG_LOCALVERSION_AUTO option will append enough characters from the latest commit id to be unique in our kernel's version name. For me this was g1df0d8960499
LOCALVERSION will also add -eudyptula to that same version string

producing a kernel named something like this:

linux-image-5.8.0-rc4-eudyptula-00381-g1df0d8960499

Installing

Our last and final step to completing this challenge is to install the kernel along with its supporting modules into our /boot and /lib/modules folders. This will allow our boot loader, GRUB, to properly initialize our freshly compiled linux kernel the next time we reboot.

Each distribution of linux installs the kernel in their way. For example Arch Linux's documentation tells us to manually copy (with cp) and not to use make install to install the kernel in our /boot directory. All this to say "your mileage may vary".

For most linux distributions we can install our kernel with a simple make command:

$ sudo make modules_install install

Along with installing all the kernel's modules and the kernel itself, this command will also update our /boot/grub/grub.conf files, so we don't have to manually update GRUB ourselves.

Ubuntu/Debian Weirdness

If your linux distribution has trouble with the make install commands above, an alternative is to bundle our linux kernel into a package. This method is slower (a real pain when developing drivers) however, packages usually have more success installing on certain systems.

For Debian and Ubuntu based distributions we can use deb-pkg to generate the packages:

$ make deb-pkg

This command will generate up to 5 Debian packages:

linux-image-version which contains the kernel image and the associated modules.
linux-headers-version has the header files required to build external modules.
linux-firmware-image-version contains the firmware files needed by some drivers (this package could be missing when you build from the kernel sources provided by Debian).
linux-image-version-dbg which contains the debugging symbols for the kernel image and its modules
and linux-libc-dev holds headers relevant to some user-space libraries.

All of these packages can easily be installed with a simple dpkg command:

sudo dpkg -i ../*.deb

And that's it! Reboot you computer, and run uname -r and you should see something like:

$ uname -r
linux-image-5.8.0-rc4-eudyptula-00381-g1df0d8960499

This output will prove to The Little Penguin you've successfully compiled and installed your first (of many I'm sure) custom linux kernel. Great Job!

The Hello World Kernel Module

2020-07-15T00:00:00+00:00

As a brief introduction to this (very long) essay. What lies below are my notes while completing the 1^st in a set of 20 tasks from The Eudyptula Challenge. Each task, emailed one at a time, starting with building a "hello world" kernel module (this essay) and progressed in difficulty until we ultimately submit patches into the main tree of the Linux kernel. The ultimate goal of The Eudyptula Challenge is to get new developers comfortable with the somewhat unique world of kernel development by separating the "on-boarding" process into focused manageable tasks.

Sadly, The Eudyptula Challenge is no longer accepting new applicants. However, if you wish to work on the tasks yourself, I've published the 20 tasks I've managed to find along with the code I used to "complete" them in a git repo here.

Task No.1

Write a Linux kernel module, and stand-alone Makefile, that when loaded prints to the kernel debug log level, "Hello World!" Be sure to make the module be able to be unloaded as well.

The Makefile should build the kernel module against the source for the currently running kernel, or, use an environment variable to specify what kernel tree to build it against.

Please show proof of this module being built, and running, in your kernel. What this proof is is up to you, I'm sure you can come up with something. Also be sure to send the kernel module you wrote, along with the Makefile you created to build the module.

—Little Penguin

What Is A Module

A kernel module is piece of code designed to be loaded and unloaded on demand by our kernels. For example, the device drivers for your keyboard or a network card are a type of module. By separating the kernel into individual software components, we can keep the overall size of the kernel small, letting Linux fit into the smallest of embedded systems. Some kernel modules, like the one we'll be building, can even be installed without the need to recompile and reboot our kernel, making upgrades easy, and saving us a lot of time.

If you have access to a Linux machine, you can find the modules that are currently loaded into the kernel by using the lsmod command, which gets its information from /proc/modules.

Chiseling A Cave Module

Every kernel module must have at least two functions, one that will be called when we install the module and another function to remove it from the kernel. Back in the pre v2.3 era (early 2000s) this could only be done with a "start" function, called init_module() and an "end" function, called cleanup_module(). There are more modern (and preferred) methods available to us today, however some developers still use these, so it's a great starting point.

#include <linux/kernel.h>  /* for KERN_DEBUG */
#include <linux/module.h>  /* for all kernel modules */

int init_module(void)
{
        printk(KERN_DEBUG "Hello World.\n");
        return 0; /* init_module loaded successfully */
}

void cleanup_module(void)
{
        printk(KERN_DEBUG "oh, the rest is silence.\n");
}

Typically init_module() is used to register handlers or alter some other part of the kernel for a device or something. The cleanup_module() will then undo those changes, allowing the module to be removed safely from the kernel. Both of these functions (as of version 5.7) can be found on line 75, as well as everything else we need, in linux/module.h of the source code.

printk() != printf()

To print Hello World on "the kernel debug log level", we'll need to use another, very old, function called printk(). Unlike the printf() commonly used in userspace applications, printk() is not designed to communicate to the user (or say hello to worlds). It's a logging mechanism used to give warnings and to log messages. This is why each printk() statement also comes with a priority. There are currently 8 defined priorities we can use ranging from KERN_DEBUG to KERN_EMERG. You can see them all, and their definitions, currently (version 5.7) in linux/kern_levels.h in the source code.

Pay attention to the single argument passed to printk(). Looking into the source code shows that printk(const char *ftm, ...) accepts only one string, with space to pass extra arguments to format the string if needed, for example, our "Hello World" statement from above, which doesn't need formatting and therefore passes no extra arguments:

printk(KERN_DEBUG "Hello World.\n");

The KERN_DEBUG macro will expand to "\001" "7", turning our statement into:

printk("\001" "7" "Hello World.\n");

Our C lexer will then combine the adjacent string literals to produce our formatted string for the kernel to log:

printk("\0017Hello World.\n");

Even though printk() is falling out of style with modern Linux maintainers, as we will see in later sections, there is a lot more to read about how to work with printk() and format specifiers in the kernel in the documentation here if you're into that kind of stuff.

Making A Kernel Module

Much like how kernel modules are a little different than userspace application modules, the Makefiles that compile the kernel are also a bit different than Makefiles in userspace.

Originally, as the Linux code-base grew, so did its Makefiles. As they continued to grow in complexity, they eventually became a burden to maintain. Fortunately a solution, called the "kbuild system", was created and accepted into the kernel to help organize and simplify the kernel's building process. If you are interested, there is an entire section about the kbuild system in the documentation.

Kbuild Makefile

Just like Makefiles in userspace, we can start a Kbuild Makefile by creating a new file called …wait for it… Makefile in the same folder as our hello-world.c module we made in the sections above.

$ ls -l
total 8
-rw-rw-r-- 1 me us 903 Jul  5 00:00 hello-world.c
-rw-rw-r-- 1 me us 167 Jul  5 00:00 Makefile

We can alternatively use the name Kbuild (not preferred) to indicate to other developers that the Makefile is intended to run using the kbuild system. However, while the Kbuild name is not preferred, interestingly, if both Makefile and Kbuild files exist in the same directory the Kbuild file will be used. (source)

Goal Definitions

The "heart" of the kbuild system uses lines called "goal definitions" to define all the various target files, special compilation options, and any sub-directories to enter. When we compile the kernel (with its thousands of Makefiles) the goal definitions are collected and used to build all the various, documentation files, modules, and other files we need for our particular kernel.

The simplest Kbuild Makefile we can write for our module contains a single line:

obj-m += hello-world.o

obj-m tells kbuild that our hello-world.o object file is a loadable kernel module (LKM) that can be loaded and unloaded at any time without needing to reboot the kernel. This line will also tell the kbuild system to look for files in our directory named hello-world.c or hello-world.S to compile into the hello-world.o object file, before building the kernel object file hello-world.ko we'll use to load into our kernel.

Convenience Targets

For the pure convenience of it, we can add extra phony targets to our Kbuild Makefile to easily compile our module for the kernel currently running on our computer, simplifying the task of compiling our module down to just typing make into our terminals:

all:
    ${MAKE} -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

And make clean to clean up everything afterwards:

clean:
    ${MAKE} -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

Both of these phony targets use the -C option to move out of our current directory and into our kernel's source directory. There make can find and use the top most kbuild Makefile, which takes the M option to locate the folder we are current working in, and build the files defined using the obj-m goal definition we setup above.

Installing A Kernel Module

Just as with our userspace applications, kernel modules need to be compiled. Using the kbuild system, along with our convenience targets above, we can compile our kernel module by issuing the make command, and if all goes well, you should see an output similar to this:

$ make
make -C /lib/modules/4.15.0-108-generic/build M=/home/me/src/eudyptula ...
/tasks/01 modules
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-108-generic'
  CC [M]  /home/me/src/eudyptula/tasks/01/hello-world.o
  Building modules, stage 2.
  MODPOST 1 modules
WARNING: modpost: missing MODULE_LICENSE() in /home/me/src/eudyptula ...
see include/linux/module.h for more information
  CC      /home/me/src/eudyptula/tasks/01/hello-world.mod.o
  LD [M]  /home/me/src/eudyptula/tasks/01/hello-world.ko
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-108-generic'

The .ko extension was introduced around kernel version 2.6 to help differentiate between userspace object files and kernel object files, which contain a .modinfo section to hold extra metadata information about the module. We can use the modinfo command to see and interpret the contents of the section:

$ modinfo hello-world.ko
filename:       /home/me/src/eudyptula/tasks/01/hello-world.ko
srcversion:     18005133D4ECFCDD12928D8
depends:
retpoline:      Y
name:           hello_world
vermagic:       4.15.0-108-generic SMP mod_unload

Installing the Module

With our hello-world.c module freshly compiled, we can insert it into our kernel using the insmod command as root or another user with sudo privileges:

$ sudo insmod hello-world.ko

Congratulations!, you have created your first kernel module! A quick inspection of the kernel's diagnostic messages, using dmesg, should show our Hello World. message:

$ dmesg | tail -1
[241745.247591] Hello World.

Removing the Module

After the well deserved pat-on-the-back and when you are ready to continue, we can uninstall our module with the rmmod command as root or someone with sudo privileges:

$ sudo rmmod hello_world

The only indication we've uninstalled our module will be in dmesg from our printk() statement in the cleanup_module() function.

$ dmesg | tail -1
[241751.401232] oh, the rest is silence.

Kernel Taint

There are plenty of ways we can taint our kernel. Don't worry too much about this though, most of the time it is completely fine to run a tainted kernel. When something happens that could be important to an investigation later on, a kernel will mark itself as "tainted". Usually the event that caused the kernel to become tainted is the problem being investigated.

We can find our kernel's tainted state by reading our /proc/sys/kernel/tainted file. Every way we can taint our kernels is assigned one bit in a bit-field, meaning any value other than 0 indicates our kernel is tainted. To decode the bit-field values, we can use the tools/debugging/kernel-chktaint script found in the source code, to decode its meaning.

$ tools/debugging/kernel-chktaint
Kernel is "tainted" for the following reasons:
 * proprietary module was loaded (#0)
 * kernel issued warning (#9)
 * externally-built ('out-of-tree') module was loaded  (#12)
 * unsigned module was loaded (#13)
For a more detailed explanation of the various taint flags see
 Documentation/admin-guide/tainted-kernels.rst in the the Linux kernel sources
 or https://kernel.org/doc/html/latest/admin-guide/tainted-kernels.html
Raw taint value as int/string: 12801/'P        W  OE    '

Licensing & Documentation

One of the ways we can taint our kernels is by loading proprietary modules or modules that use licenses not compatible with the General Public License (GPL) (bit 0 in the tainting list). Modules that don't use the MODULE_LICENSE() macro will also be considered proprietary and taint our kernel, if loaded (this is why we saw the warning above).

There are many documentation macros, defined in linux/module.h, some of the basics I added are:

MODULE_LICENSE("MIT");
MODULE_AUTHOR("Bryan Brattlof <email@example.com>");
MODULE_DESCRIPTION("A Hello World Driver");
MODULE_SUPPORTED_DEVICE("testdevice");

Once we add our module's license, author and other information to the end of our hello-world.c module, when we compile our module again using make, the WARNING should be gone:

$ make
make -C /lib/modules/4.15.0-108-generic/build M=/home/me/src/eudyptula ...
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-108-generic'
  CC [M]  /home/me/src/eudyptula/tasks/01/hello-world.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/me/src/eudyptula/tasks/01/hello-world.mod.o
  LD [M]  /home/me/src/eudyptula/tasks/01/hello-world.ko
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-108-generic'

There where many reasons to add this system to the kernel. For example, it gives developers a way to easily find who maintains a module, describe what the module does, and what license the code is protected with. It also provides an easy method to inform users when they are using non open source software.

Updating the Module

Everything in my notes, to this point, was needed to complete the 1^st task assigned to us by the Little Penguin. However, just like with every software project, the Linux kernel is constantly adding new features and adopting new coding styles, ensuring that my notes will become obsolete as soon as I've writing them.

With that said, the sections below, while not technically needed to complete the task, are my notes on the macros and functions I saw in the drivers directory of the Linux source code that I found particularly interesting. These functions are mostly stylistic changes or they introduce functionality that improves efficiency and modularity of the Linux kernel in some way.

module_init() & module_exit()

Introduced in version 2.4 of the kernel, and defined in linux/init.h of the source code, we can now rename our "start" and "end" functions to whatever we wish. In this example, I've chosen to rename the "start" function to hello_world_init()

-int init_module(void)
+static int __init hello_world_init(void)
 {
         printk(KERN_DEBUG "Hello World.\n");
         return 0; /* init_module loaded successfully */
 }

And renamed the "exit" function hello_world_exit()

-void cleanup_module(void)
+static void __exit hello_world_exit(void)
 {
         printk(KERN_DEBUG "oh, the rest is silence.\n");
 }

The kernel will then use the module_init() macro to find the function to execute when the module is installed and module_exit() to find the function to cleanup before being removed.

module_init(hello_world_init);
module_exit(hello_world_exit);

To avoid compiling issues, both the module_init() and module_exit() macros must be defined below our newly named "start" and "end" functions.

init & exit

I also introduced two macros to our "start" and "end" functions above called __init and __exit. These macros, defined in linux/init.h of the source code, help reduce memory used by the kernel depending on how the module is installed.

For built-in modules, where our module cannot be removed from the kernel without recompiling and restarting, the __init keyword will tell our C lexer to place our module's "start" function into a special section inside the compiled kernel. After the module is loaded and our "start" function has finished, the kernel will never have to run the code again until reboot. So this special section can be freed, saving memory.

The same is true for the __exit macro. For built-in modules, the module cannot be removed from the kernel without recompiling and restarting. So the kernel will never need to run our module's "exit" function to safely remove it from the kernel. This means our C lexer can safely omit our "exit" function from the compiled kernel.

pr_debug()

In the beginning there was printk(), and the kernel's diagnostic messages structure was formless. The lack of any format for printk() messages is one of a number of reasons why developers are replacing printk() statements with their newer equivalents. Depending on what section of the kernel we are in, there are newer functions that have some benefits for us.

For example, the pr_debug() function, which has the benefit of being less syntactically verbose than printk(KERN_DEBUG ...) also allows us to take advantage of the dynamic debugging interface, which gives developers a uniform control interface for debugging kernel messages while avoiding cluttering the kernel.

 static int __init hello_world_init(void)
 {
-    printk(KERN_DEBUG "Hello World.\n");
+    pr_debug("Hello World.\n");
     return 0; /* means init_module loaded successfully */
 }

 static void __exit hello_world_exit(void)
 {
-    printk(KERN_DEBUG "oh, the rest is silence.\n");
+    pr_debug("oh, the rest is silence.\n");
 }

Wrapping Up

If you made it here, all I can say is you are a very brave person, and I'm glad my notes were able to help you in some way. If you see any issues or have a question, please feel free to contact me, or better yet subscribe to the kernel newbies mailing list.

For the next challenge, we will be building the Linux Kernel from scratch, as well as installing and booting from it. If you want to work on this challenge before you read my notes (recommended), I've published a copy of the challenges in a git repo here.

Next: My notes on How to build the Linux Kernel from scratch.

Bryan Brattlof - Notes

How to Proxy Git Connections

Proxy http(s) Remotes

Proxy Git Remotes

Proxy SSH Remotes

Wrapping Up

My PGP Cheat Sheet

Why PGP?

Convert A PGP Key To An SSH Key

Locating A Public Key

Updating Your Keyring

Signing A Public Key

Extending An Expiration Date

Backing Up A GnuPG Directory

Wrapping Up

The Linux Kernel Coding Style

Task No.4

Executive Summary of the Linux Kernel Coding Style

scripts/checkpatch.pl

Wrapping Up

Cgit, Nginx & Gitolite: A Personal Git Server

The Start

Admin User

OpenSSH

Uncomplicated FireWall

fail2ban

Gitolite

Step: 1 - Create The Git User

Step: 2 - Install Gitolite

Step: 3 - Setup Gitolite

Step: 4 - Add ~/.profile

Cgit

Step: 1 - Install Cgit

Step: 2 - Build Cgit

FastCGI Wrapper

Nginx

Wrapping Up

Modify The Linux Kernel

Task No.3

Override Directives

Modify the Makefile

Modify menuconfig

Building The Linux Kernel

Task No.2

Build Tools

Cloning

Configuring

Configuration Plagiarism

Configuring By Hand

Building

Installing

Ubuntu/Debian Weirdness

The Hello World Kernel Module

Task No.1

What Is A Module

Chiseling A Cave Module

printk() != printf()

Making A Kernel Module

Kbuild Makefile

Goal Definitions

Convenience Targets

Installing A Kernel Module

Installing the Module

Removing the Module

Kernel Taint

Licensing & Documentation

Updating the Module

module_init() & module_exit()

__init & __exit

pr_debug()

Wrapping Up

init & exit