;;; guix-forge --- Guix software forge meta-service ;;; Copyright © 2022, 2023 Arun Isaac ;;; ;;; This file is part of guix-forge. ;;; ;;; guix-forge is free software: you can redistribute it and/or modify ;;; it under the terms of the GNU General Public License as published ;;; by the Free Software Foundation, either version 3 of the License, ;;; or (at your option) any later version. ;;; ;;; guix-forge is distributed in the hope that it will be useful, but ;;; WITHOUT ANY WARRANTY; without even the implied warranty of ;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU ;;; General Public License for more details. ;;; ;;; You should have received a copy of the GNU General Public License ;;; along with guix-forge. If not, see ;;; . (use-modules (skribilo source lisp) (doc skribilo)) (document :title [guix-forge] (toc :subsection #t) (chapter :title [Introduction] :ident "chapter-introduction" (p [guix-forge is a Guix service that lets you run a complete ,(ref :url "https://en.wikipedia.org/wiki/Forge_(software)" :text "software forge") in the manner of GitHub, GitLab, etc. Unlike other free software forges such as GitLab, Gitea, etc., guix-forge is not a monolith but is an assemblage of several pieces of server software wired up to function as one. In this sense, it is a ,(emph "meta-service"). guix-forge does for software forges what ,(ref :url "https://mailinabox.email/" :text "Mail-in-a-Box") does for email.]) (p [guix-forge integrates the following software components:] (itemize (item [,(ref :url "https://laminar.ohwg.net/" :text "laminar") for continuous integration]) (item [,(ref :url "https://github.com/jonashaag/klaus/" :text "klaus") to serve project git repositories on the web]))) (p [In the future, it will also provide:] (itemize (item [web server to serve static project sites]) (item [,(ref :url "https://git.zx2c4.com/cgit/" :text "cgit") to serve project git repositories on the web]) (item [,(ref :url "https://public-inbox.org/README.html" :text "public-inbox") for project discussions]))) (p [A choice of different software components may be offered provided it does not complicate the interface too much.]) (p [guix-forge is provided on a best effort basis. Its design is unstable, and open to change. We will try our best to not break your system configuration often, but it might happen.]) (section :title [Philosophy] (p [In order to empower ordinary users, software should not just be free (as in freedom), but also be simple and easy to deploy, especially for small-scale deployments. guix-forge is therefore minimalistic, and does not require running large database servers such as MariaDB and PostgreSQL.]) (p [While some state is inevitable, server software should strive to be as stateless as an old analog television set. You switch it on, and it works all the time. There are no pesky software updates, and complex hidden state. guix-forge tries to be as stateless as possible. Almost all of guix-forge's state can be version controlled, and the rest are simple files that can be backed up easily.]) (p [,(ref :url "https://drewdevault.com/2018/07/23/Git-is-already-distributed.html" :text "Git is already federated and decentralized") with email. guix-forge acknowledges this and prefers to support git's ,(ref :url "https://drewdevault.com/2018/07/02/Email-driven-git.html" :text "email driven workflow") with project discussion, bug reports and patches all happening over email.]) (p [guix-forge is opinionated and will not expose all features provided by the software components underlying it. Keeping configuration options to a minimum is necessary to help casual users deploy their own forge, and to reduce the likelihood of configuration bugs.]))) (chapter :title [Tutorial] :ident "chapter-tutorial" (p [In this tutorial, you will learn how to set up guix-forge to host continuous integration for a project. For the purposes of this tutorial, we will set up continuous integration for the ,(ref :url [https://github.com/aconchillo/guile-json] :text [guile-json]) project.]) (p [First, we clone the upstream guile-json repository into a local bare clone at ,(file [/srv/git/guile-json]).]) (prog [$ git clone --bare https://github.com/aconchillo/guile-json /srv/git/guile-json Cloning into bare repository '/srv/git/guile-json'... remote: Enumerating objects: 1216, done. remote: Counting objects: 100% (162/162), done. remote: Compressing objects: 100% (107/107), done. remote: Total 1216 (delta 96), reused 106 (delta 54), pack-reused 1054 Receiving objects: 100% (1216/1216), 276.10 KiB \| 3.89 MiB/s, done. Resolving deltas: 100% (742/742), done.] :line #f) (p [Now that we have a git repository to work with, we start writing our Guix system configuration. We begin with a bunch of ,(code [use-modules]) statements importing all required modules.]) (prog (source :language scheme :file "doc/snippets/tutorial.scm" :start 0 :stop 9) :line #f) (p [Then, we define the ,(ref :url "https://guix.gnu.org/en/manual/devel/en/html_node/G_002dExpressions.html" :text "G-expression") that will be run as a continuous integration job on every commit. This G-expression uses ,(code [invoke]) from ,(code [(guix build utils)]). Hence, we make it available to the G-expression using ,(code [with-imported-modules]). In addition, it needs a number of packages which we make available using ,(code [with-packages]). And finally, within the body of the G-expression, we have commands cloning the git repository, building the source and running the tests.]) (p [The attentive reader may notice what looks like ,(code [(guix build utils)]) being referenced twice—once with ,(code [with-imported-modules]) and again with ,(code [use-modules]). This is not a mistake. G-expressions are serialized into Guile scripts. ,(code [with-imported-modules]) ensures that code for ,(code [(guix build utils)]) is available and is in the ,(ref :url "https://www.gnu.org/software/guile/manual/html_node/Load-Paths.html" :text "load path"). ,(code [use-modules]) actually imports ,(code [(guix build utils)]) when the script runs. ,(code [with-imported-modules]) is like installing a library in your system, and ,(code [use-modules]) is like actually importing that library in a script. Both are necessary.]) (prog (source :language scheme :file "doc/snippets/tutorial.scm" :start 11 :stop 22) :line #f) (p [Now, we configure a ,(code []) record that holds metadata about the project and wires up the G-expression we just defined into a continuous integration job.]) (prog (source :language scheme :file "doc/snippets/tutorial.scm" :start 24 :stop 32) :line #f) (p [The ,(code [name]) and ,(code [description]) fields are hopefully self-explanatory. The ,(code [user]) field specifies the user who will own the git repository at the path specified by ,(code [repository]). That user will therefore be able to push into the repository through ssh or similar. git provides various ,(ref :url "https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks" :text "server-side hooks") that trigger on various events. Of these, the ,(file [post-receive]) hook triggers when pushed commits are received. guix-forge sets up a ,(source-ref "guix/forge/forge.scm" "\\(define\\* \\(ci-jobs-trigger-gexp" "post-receive hook script") in the repository to trigger a continuous integration run on every ,(command [git push]).]) (p [And finally, we put everything together in an ,(code [operating-system]) declaration. Notice the forge service configured with ,(code [guile-json-project]) and the laminar service configured with a port for the web interface to listen on.]) (prog (source :language scheme :file "doc/snippets/tutorial.scm" :start 34) :line #f) (p [Now that we have a complete ,(code [operating-system]) definition, let's use the following command to build a container. After a lot of building, a container script should pop out.]) (prog [$ guix system container --network --share=/srv/git/guile-json tutorial.scm /gnu/store/ilg7c2hpkxhwircxpz22qhjsqp3i9har-run-container] :line #f) (p [The ,(code [--network]) flag specifies that the container should share the network namespace of the host. To us, this means that all ports opened by the container will be visible on the host without any port forwarding or complicated configuration. The ,(code [--share=/srv/git/guile-json]) option shares the git repository we cloned earlier, with the container.]) (p [To start the container, simply run the container script as root.]) (prog [# /gnu/store/ilg7c2hpkxhwircxpz22qhjsqp3i9har-run-container] :line #f) (p [Now, you can see the status of laminar and running jobs through its web interface listening on ,(ref :url "http://localhost:8080"). You can list and queue jobs on the command-line like so:]) (prog [$ laminarc show-jobs guile-json $ laminarc queue guile-json guile-json:1] :line #f) (p [That's it! You just set up your own continuous integration system and took the first steps to owning your code!]) (p [You could easily use the same configuration to configure a Guix system instead of a container. To do so, you will have to take care of defining the bootloader, file systems and other settings as per your needs. The overall configuration used in this tutorial is repeated below for your reference.]) (prog (source :language scheme :file "doc/snippets/tutorial.scm"))) (chapter :title [How To] :ident "chapter-how-to" (section :title [How to set up cgit] :ident "section-how-to-set-up-cgit" (p [guix-forge comes with an end-to-end cgit solution that not only sets up cgit itself but also an nginx server complete with automatically renewed TLS certificates. cgit even runs in its own container for maximal security.]) (p [The cgit service uses the forge-nginx service as its web server. The forge-nginx service in turn uses the ACME service to fetch and renew TLS certificates. Here's a minimal working configuration.]) (prog (source :language scheme :file "doc/snippets/how-to-set-up-cgit.scm") :line #f) (p [The cgit service configuration specifies the domain ,(samp [git.example.org]) to serve cgit on and the ,(file "/srv/git") repository directory containing bare git repositories to publish. The forge nginx service configuration specifies the ports to serve HTTP and HTTPS on. The ACME service configuration specifies the email address to register an ACME account with. The sudoers file declaration is required to allow the ,(samp [acme]) user to restart the nginx server when a certificate is renewed. The configured machine will start out with self-signed certificates. Run ,(samp [/usr/bin/acme renew]) the first time to get CA-issued certificates. Thereafter, certificates will auto-renew via a cron job.]) (p [When testing your deployment, it might help to start with the Let's Encrypt staging server as shown below. This will give you dummy certificates, but will help you avoid running afoul of Let's Encrypt rate limits. Once you know everything works, delete the ACME state directory (,(file "/var/lib/acme") by default) and run ,(samp [/usr/bin/acme renew]) again to get real certificates.] (prog (source :language scheme :file "doc/snippets/acme-staging-url.scm") :line #f)))) (chapter :title [Services] :ident "chapter-services" (section :title [Git web viewers] :ident "section-git-web-viewers" (subsection :title [cgit service] :ident "subsection-cgit-service" (p [cgit is a web frontend to serve git repositories on the web. Our cgit service features] (itemize (item [clonable URLs via the smart HTTP protocol through ,(command [git-http-backend])]) (item [syntax highlighting for a wide variety of programming languages using ,(ref :url "https://pygments.org/" :text "Pygments")]) (item [rendering markdown, org mode, reStructuredText, man page, HTML or plain text README files in the ,(emph [About]) page]) (item [hiding full email addresses on cgit web pages])) [Note that this service is different from the cgit service of the same name in Guix upstream.]) (description (record-documentation "guix/forge/cgit.scm" ' (record-field "cgit" [,(code [cgit]) package to use]) (record-field "git" [,(code [git]) package to use. ,(code [git]) provides the smart HTTP protocol backend.]) (record-field "server-name" [Domain name to serve cgit on]) (record-field "repository-directory" [Directory containing git repositories to serve]) (record-field "socket" [Socket that the internal cgit fcgiwrap instance listens on]) (record-field "readme" [README file to serve as the ,(emph [About]) page of the repository. This field is a list of candidate README files looked up in the default branch of the repository. cgit will serve the first file that is found.]) (record-field "snapshots" [List of strings specifying snapshot formats that cgit generates links for. Valid strings are ,(code ["tar"]), ,(code ["tar.gz"]), ,(code ["tar.bz2"]), ,(code ["tar.lz"]), ,(code ["tar.xz"]), ,(code ["tar.xst"]) and ,(code ["zip"]).]) (record-field "about-filter" [Script invoked to format the content of about pages]) (record-field "commit-filter" [Script invoked to format commit messages]) (record-field "email-filter" [Script invoked to format email addresses]) (record-field "source-filter" [Script invoked to format plaintext blobs in the tree view]) (record-field "mimetype-file" [File to use for automatic mimetype lookup. This is used by the plain endpoint when serving blob content]) (record-field "repository-sort" [Order in which repositories are sorted on the index page. Valid values are ,(code ['name]) (sorting by repository name) and ,(code ['age]) (sorting most recently updated repository first).]) (record-field "plain-email?" [If ,(code [#true]), full email addresses will be shown. Else, they won't.]) (record-field "extra-options" [Association list of additional key-value option pairs to include in the generated ,(file [cgitrc]) configuration file]))))) (section :title [forge nginx service] :ident "section-forge-nginx-service" (p [The forge nginx service is a wrapper around the nginx web service in Guix upstream. It features] (itemize (item [automatic HTTPS for all sites through the ,(ref :ident "section-acme-service" :text "ACME service")]) (item [HTTP endpoint that redirects to HTTPS and responds to ACME HTTP-01 challenges]) (item [automatic provision of ,(ref :url "https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security" :text (abbr :short "HSTS" :long "HTTP Strict Transport Security")) headers]))) (p [When using this service, you must allow the ,(samp [acme]) user to restart nginx using ,(command [sudo]). This is so that newly obtained certificates can be deployed to nginx. You may achieve this with the following in the ,(code [sudoers-file]) field of your ,(code [operating-system]) definition.] (prog (source :language scheme :file "doc/snippets/acme-restart-nginx-sudoers.scm") :line #f)) (description (record-documentation "guix/forge/nginx.scm" ' (record-field "http-listen" [Socket to listen on for HTTP requests. Socket may be a ,(record-ref ""), ,(record-ref ""), or ,(record-ref "") object.]) (record-field "https-listen" [Socket to listen on for HTTPS requests. Socket may be a ,(record-ref ""), ,(record-ref ""), or ,(record-ref "") object.]) (record-field "acme-state-directory" [State directory of the ,(ref :ident "section-acme-service" :text "ACME service")]) (record-field "acme-challenge-directory" [Directory to serve on ,(samp [/.well-known/acme-challenge/]) in response to ACME HTTP-01 challenges]) (record-field "server-blocks" [List of ,(ref :url "https://guix.gnu.org/manual/en/html_node/Web-Services.html#index-nginx_002dserver_002dconfiguration" :text "") objects describing server blocks to add to the nginx configuration])))) (section :title [ACME service] :ident "section-acme-service" (p [,(abbr :short "ACME" :long "Automatic Certificate Management Environment") is a protocol popularized by the Let's Encrypt certificate authority for the automatic issue and renewal of TLS certificates. The guix-forge ACME service featuers] (itemize (item [a ,(samp [/usr/bin/acme]) script to help with certificate management tasks]) (item [cron job that runs at a random minute during the day and renews certificates older than 30 days]) (item [HTTP-01 challenge support. The DNS-01 challenge and wildcard certificates will be supported in the future.]) (item [completely rootless operation]))) (p [The first time the ACME service is set up or each time new certificates are configured, self-signed certificates are created so that processes (such as nginx) that depend on these certificates can start up successfully. You must replace these with certificate authority issued certificates by running ,(samp [/usr/bin/acme renew]). ,(samp [/usr/bin/acme renew]) automatically registers an ACME account unless one already exists and renews all configured certificates. It uses parameters that were configured in the ACME service and does not need any additional command-line arguments.]) (p [The ACME service does not use ,(ref :url "https://certbot.eff.org/" :text "certbot"), the official Let's Encrypt client. It instead uses ,(ref :url "https://github.com/ndilieto/uacme/" :text "uacme"). uacme is smaller, simpler, manages far less state, does no magic, and is better suited to automation. However, the choice of backend tool is an implementation detail. The ACME service is an ,(emph [abstract]) service that is largely independent of the backend tool that powers it.]) (p [By using the ACME service, you agree to the Terms of Service of your ACME server.]) (description (record-documentation "guix/forge/acme.scm" ' (record-field "uacme" [,(code [uacme]) package to use]) (record-field "email" [Email ID to register ACME account with]) (record-field "acme-url" [URL of the ACME server to use. This field can be set to ,(code [%letsencrypt-staging-url]) when testing your deployment.]) (record-field "state-directory" [State directory in which private keys and certificates are stored]) (record-field "http-01-challenge-directory" [Directory served by the web server at ,(samp "/.well-known/acme-challenge/") in response to ACME HTTP-01 challenges]) (record-field "http-01-authorization-hook" [Script invoked to complete a HTTP-01 challenge]) (record-field "http-01-cleanup-hook" [Script invoked after the completion of a HTTP-01 challenge]) (record-field "key" [,(record-ref "") or ,(record-ref "") object describing the ACME account and TLS certificate keys. Changing this field does not affect keys already generated and stored on disk.]) (record-field "certificates" [List of ,(record-ref "") objects describing certificates to configure]))) (p [The ,(code [http-01-authorization-hook]) and ,(code [http-01-cleanup-hook]) scripts are invoked with the following three command-line arguments.]) (description (item :key (code [identifier]) [Primary domain name on the certificate]) (item :key (code [token]) [Filename of the resource requested under ,(samp "/.well-known/acme-challenge/") during the HTTP-01 challenge]) (item :key (code [auth]) [Authorization string expected to be at the requested resource])) (description (record-documentation "guix/forge/acme.scm" ' (record-field "domains" [List of domain names that the certificate is valid for. Each domain name is a string.]) (record-field "deploy-hook" [Script invoked to deploy a new certificate after successful renewal. This script is invoked without any command-line arguments.])) (record-documentation "guix/forge/acme.scm" ' (record-field "length" [Length of the RSA key in number of bits. Must be a multiple of 8 between 2048 and 8192.])) (record-documentation "guix/forge/acme.scm" ' (record-field "length" [Length of the ECDSA key in number of bits. Must be either 256 or 384.])))) (section :title [Specialized application deployment services] :ident "section-specialized-application-deployment-services" (subsection :title [fcgiwrap service] :ident "fcgiwrap-service" (p [fcgiwrap is a specialized web server for ,(ref :url "https://en.wikipedia.org/wiki/Common_Gateway_Interface" :text "CGI") applications. It provides a ,(ref :url "https://en.wikipedia.org/wiki/FastCGI" :text "FastCGI") interface that web servers such as nginx can talk to. We run separate containerized instances of fcgiwrap for each application.]) (p [Note that this service is different from the fcgiwrap service of the same name in Guix upstream.]) (description (record-documentation "guix/forge/fcgiwrap.scm" ' (record-field "package" [,(code [fcgiwrap]) package to use]) (record-field "instances" [List of ,(record-ref "") objects describing fcgiwrap instances to run])) (record-documentation "guix/forge/fcgiwrap.scm" ' (record-field "name" [Name of the fcgiwrap instance]) (record-field "socket" [Socket the fcgiwrap instance listens on. Socket may be a ,(record-ref ""), ,(record-ref "") or ,(record-ref "") object.]) (record-field "user" [User the fcgiwrap instance should run as]) (record-field "group" [Group the fcgiwrap instance should run as]) (record-field "processes" [Number of fcgiwrap worker processes]) (record-field "environment-variables" [Association list mapping environment variables that should be set in the execution environment to their values]) (record-field "mappings" [List of ,(code []) objects describing additional directories that should be shared with the container fcgiwrap is run in])))) (subsection :title [gunicorn service] (p [gunicorn is a specialized web server for Python ,(ref :url "https://en.wikipedia.org/wiki/Web_Server_Gateway_Interface" :text "WSGI") applications. We run separate containerized instances of gunicorn for each application.]) (description (record-documentation "guix/forge/gunicorn.scm" ' (record-field "package" [,(code [gunicorn]) package to use]) (record-field "apps" [List of ,(record-ref "") objects describing gunicorn apps to run])) (record-documentation "guix/forge/gunicorn.scm" ' (record-field "name" [Name of the app]) (record-field "package" [Package of the app]) (record-field "wsgi-app-module" [WSGI app module passed to gunicorn]) (record-field "sockets" [List of ,(record-ref ""), ,(record-ref "") or ,(record-ref "") objects describing sockets to listen on]) (record-field "workers" [Number of worker processes]) (record-field "environment-variables" [Association list mapping environment variables that should be set in the execution environment to their values]) (record-field "mappings" [List of ,(code []) objects describing additional directories that should be shared with the container gunicorn is run in])))))) (chapter :title [Reference] :ident "chapter-reference" (description (record-documentation "guix/forge/forge.scm" ' (record-field "projects" [List of ,(record-ref "") objects describing projects managed by guix-forge])) (record-documentation "guix/forge/forge.scm" ' (record-field "name" [Name of the project]) (record-field "repository" [Path to a local git repository, or URI to a remote git repository]) (record-field "user" [User who owns the repository if it is local. This field is disregarded if the repository is remote.]) (record-field "description" [Short one-line description of the project. It is used to set the ,(file "description") file in the repository and will appear in the cgit web interface.]) (record-field "website-directory" [Path to the document root of the project website. The ownership of its parent directory is granted to the ,(code "laminar") user. The idea is that the website is built by a Guix derivation as a store item and a symbolic link to that store item is created in the parent directory.]) (record-field "ci-jobs" [List of ,(record-ref "") objects describing ,(abbr :short "CI" :long "continuous integration") jobs to configure]) (record-field "ci-jobs-trigger" [One of ,(code ['post-receive-hook]), ,(code ['webhook]), or ,(code ['cron]) representing the type of trigger for continuous integration jobs. ,(description (item :key (code ['post-receive-hook]) [If ,(code ['post-receive-hook]) is specified, the ,(file "post-receive") hook of the repository is configured to trigger CI jobs. This is possible only for local repositories. Note that any pre-existing ,(file "post-receive") hook is overwritten.]) (item :key (code ['webhook]) [If ,(code ['webhook]) is specified, a webhook server is configured to trigger CI jobs when a request is received on ,(samp "http://hostname:port/hooks/") \.]) (item :key (code ['cron]) [If ,(code ['cron]) is specified, a cron job triggers the CI jobs once a day.]))] :default [,(code ['post-receive-hook]) for local repositories and ,(code ['cron]) for remote repositories]) (record-field "repository-branch" [Main branch of the repository. This field is currently unused unused, and may be deprecated in the future.])) (record-documentation "guix/forge/laminar.scm" ' (record-field "name" [Name of the job]) (record-field "run" [G-expression to be run]) (record-field "after" [G-expression to be run after the main job script]) (record-field "trigger?" [If ,(code [#t]), this job is run on every commit. Else, it must be manually set up to run some other way.])) (record-documentation "guix/forge/socket.scm" ' (record-field "hostname" [Name of the host]) (record-field "port" [Port number])) (record-documentation "guix/forge/socket.scm" ' (record-field "ip" [IP address, either IPv4 or IPv6, as a string. The loopback address is ,(code ["127.0.0.1"]) and ,(code ["::1"]) for IPv4 and IPv6 respectively. The any address is ,(code ["0.0.0.0"]) and ,(code ["::"]) for IPv4 and IPv6 respectively.]) (record-field "port" [Port number to listen on.])) (record-documentation "guix/forge/socket.scm" ' (record-field "path" [Path to socket file.])) (record-documentation "guix/forge/webhook.scm" ' (record-field "package" [,(code [webhook]) package to use]) (record-field "socket" [Socket, a ,(record-ref "") object, to listen on.]) (record-field "log-directory" [Directory to write log files to]) (record-field "hooks" [List of ,(record-ref "") objects describing hooks to configure])) (record-documentation "guix/forge/webhook.scm" ' (record-field "id" [Identifier of the webhook. This hook is triggered at ,(ref :url [http://host:port/hooks/]).]) (record-field "run" [G-expression to run when the webhook is triggered])) (docstring-function-documentation "guix/forge/klaus.scm" 'klaus-gunicorn-app))))