From 1cdbbedecd504474f33e9286ac04d87f64501aed Mon Sep 17 00:00:00 2001 From: Matthias Kruk Date: Wed, 9 Feb 2022 20:45:26 +0900 Subject: [PATCH] Add README explaining how to use foundry The repository doesn't have a README file, making it hard for new users to understand what foundry is, how it works, and how to use it. This commit adds a README file explaining about the architecture of foundry, and how to install and start foundry. --- README.md | 298 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 298 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..f2eaed8 --- /dev/null +++ b/README.md @@ -0,0 +1,298 @@ +# foundry - a distributed package build system + +Foundry is a distributed package build system. It is made up from a number of workers, +called bots, each of which performs one step of the package build process. The bots +talk to each other asynchronously using pub-sub messaging, making foundry highly +scalable and fault-tolerant. Because of its message-based architecture, new bots can +be written and integrated into foundry within minutes. + +## Architecture + +Foundry consists of four bots that cooperate to build packages from a git repository +and place the resulting packages in a package repository. These are the four bots: + +| Bot | Performed task | +|----------|---------------------------------------------| +| watchbot | Watch git repositories for changes | +| buildbot | Build Debian packages from a git repository | +| signbot | Sign Debian packages | +| distbot | Add signed packages to a Debian repository | + +The bots communicate via pub-sub messaging, publishing their results (if successful) +under the following topics. + +| Topic | Message type | Description | +|---------|--------------|-------------------------------------------------------------------| +| commits | commit | A change was detected on a git repository | +| builds | build | A build has completed successfully | +| signs | sign | The packages from a build message have been successfully signed | +| dists | dist | The packages from a sign message have been successfully published | + +The following diagram gives an overview how the individual foundry components interact. For +understandability's sake each bot is shown only once, but the foundry architecture does not +impose a limit on the number of bots. The messaging system that foundry is based on (toolbox's +ipc module) performs automatic load-balancing, allowing bots to be added or removed on demand. + +#### How to read the architecture diagram + +``` + ( data store ) + [ process ] + < pub-sub topic > + + Relationships are expressed by arrows pointing from the subject to the object. + The relationship below means "AAA writes to BBB". + + AAA + | + | writes to + v + BBB +``` + +#### Architecture Diagram + +``` +( git repository ) + ^ + | watch for changes + | + [ watchbot ] + | + | announce commit + v + < commits > + ^ + | watch for commits + | + [ buildbot ] + | + | announce successful build + v + < builds > + ^ + | watch for builds + | + [ signbot ] + | + | announce signed packages + v + < signs > + ^ + | watch for signed packages + | + [ distbot ] ---------------. + | | + | publish packages | announce published packages + v v +( Debian repository ) < dists > +``` + +### Proof of Concept: Foundry for toolbox + +Foundry is used to automate the build process of toolbox packages. It detects when changes +have been pushed to the stable or unstable branches of any of the toolbox repositories, and +immediately builds, signs, and publishes the new packages. The diagram below shows how the +components relate to each other. + +``` +( https://github.com/m10k/toolbox ) + ^ + | watch stable and + | unstable branches + | +[ watchbot ] + | + .-' ( https://github.com/m10k/toolbox-goodies ) + | ^ + | | watch stable and + | | unstable branches + | | + | [ watchbot ] + | | + | .---------' ( https://github.com/m10k/toolbox-linux ) + | | ^ + | | | watch stable and + | | | unstable branches + | | | + | | [ watchbot ] + | | | + | | .-----------------' ( https://github.com/m10k/toolbox-restapis ) + | | | ^ + | | | | watch stable and + | | | | unstable branches + | | | | + | | | [ watchbot ] + | | | | + | | | .-------------------------' + | | | | + | | | | announce commits + | | | | + v v v v + < commits > + ^ ^ + | | watch for commits + | | + | '-----------. + | | +[ buildbot ] [ buildbot ] + | | + | .-----------' + | | + | | announce successful builds + v v + < builds > + ^ + | watch for builds + | + [ signbot ] + | + | announce successful signs + v + < signs > + ^ + | watch for signs + | + [ distbot ] -----------------------------. + | | | announce published packages + | '---------------------------. v + | | < dists > + | | + | publish packages built | publish packages built + | from stable branches | from unstable branches + v v +( deb.m10k.eu stable ) ( deb.m10k.eu unstable ) +``` + +### Installation + +There are no packages for foundry yet, so installation is still a manual (but +straightforward) process. This section explains how to get started. + + +#### Dependencies + +First of all, make sure you have [toolbox](https://github.com/m10k/toolbox) installed +and the user you're going to use for foundry is member of the *toolbox_ipc* group. +Next, create `/var/lib/foundry/contexts` exists and make sure your user can write to it. + + # mkdir -p /var/lib/foundry/contexts + # chown -R . /var/lib/foundry + # chmod -R 770 /var/lib/foundry + + +#### Getting the sources + +Foundry needs to be installed from the sources, so first you have to clone the git +repository. + + $ git clone https://github.com/m10k/foundry + +Then, change into the foundry directory. + + $ cd foundry + +Foundry comes with a number of toolbox modules, so we need to make sure toolbox can load +them. + + $ ln -s "$PWD/include" "$HOME/.toolbox/include/foundry" + + +### Starting foundry + +Now we can start the build system. + + +#### Starting watchbot + +First, we will start a watchbot to watch the toolbox repository's stable branch. This +can be achieved with the following command. + + $ ./watchbot.sh --repository https://github.com/m10k/toolbox#stable \ + --name "somename" + +A single watchbot can watch an arbitrary number of repositories. If you want to watch more +than one repository, pass `--repository` multiple times, as shown below. + + $ ./watchbot.sh --repository https://github.com/m10k/toolbox#stable \ + --repository https://github.com/m10k/toolbox#unstable \ + --name "somename" + +The purpose of the `--name` parameter was to make it easier to distinguish between running +instances. It's not used at all, so you can pass anything you like. This parameter will be +removed soon. + +Watchbot supports watching of remote repositories (via git's dumb and smart HTTP transports) +as well as local repositories. + +#### Starting buildbot + +This one is really simple. + + $ ./buildbot.sh + +By default, buildbot will use the IPC endpoint `pub/buildbot` to subscribe to *commits*. +This means, by default build jobs will be balanced over all buildbots. If you would prefer +multiple buildbots to build *the same* job in parallel, for example because you are +building for multiple architectures, you need to pass the `--endpoint` parameter. You can +think of the IPC endpoint as a load-balancing group. For example, consider you have started +four buildbots as shown below. + + $ ./buildbot.sh --endpoint pub/buildbot_i386 + $ ./buildbot.sh --endpoint pub/buildbot_i386 + $ ./buildbot.sh --endpoint pub/buildbot_amd64 + $ ./buildbot.sh --endpoint pub/buildbot_amd64 + +In this scenario, if a new commit is published, it is sent to either IPC endpoint. This +means, one of the buildbots listening on *pub/buildbot_i386* and one of the buildbots +listening on *pub/buildbot_amd64* will see the message. + +#### Starting signbot + +Signbots are started as shown below. + + $ ./signbot.sh --key + +The purpose of signbot is to sign Debian packages, so you need to tell it which key to +use using the `--key` option. The key must be a key-id from the default GPG key ring. +By default, signbots are load-balanced the same way as the buildbots. If you want to sign +packages going to one repository with a different key than packages going to a different +repository, you will have to start two signbots with differing `--endpoint` and +`--publish-to` options. + +#### Starting distbot + +Distbot has a few more options, but most of them are fairly self-explanatory. + + $ ./distbot.sh --name deb.example.org \ + --output /srv/www/deb \ + --arch amd64,i386 \ + --gpgkey \ + --description "My repository" + +Like signbot, distbot needs a GPG key id. Unlike the signbot key, which is used to sign +packages, this key is used to sign the metadata in the repository. You are strongly +encouraged *not* to use the same key for both. If the path passed via `--output` +contains an existing repository, distbot will reuse it. Otherwise, it will create a new +repository in that path. + +**Note:** If you have multiple signbots publishing messages to more than one topic, you +can change the topic that a distbot watches by passing `--watch` and the name of the +topic that your signbot is publishing messages on. + + +### Herding the bots + +If you want to see a list of the running bots, pass `--list` to the bot you're interested +in. For example, the following command will list all running watchbots. + + $ ./watchbot.sh --list + +If you want to stop a particular bot, use `--stop`. + + $ ./watchbot.sh --stop 12345 + +That's all you need to know to get started. If everything went well, you should be seeing +built packages in your Debian repository. If not, the logs within `$HOME/.toolbox/log` +might tell you what's going on. This is also a good opportunity to mention that you can +tell each of the bots to be more talkative by passing `--verbose` one or more times. -- 2.47.3