path: root/doc/forge.skb

;;; guix-forge --- Guix software forge meta-service
;;; Copyright © 2022, 2023 Arun Isaac <arunisaac@systemreboot.net>
;;;
;;; This file is part of guix-forge.
;;;
;;; guix-forge is free software: you can redistribute it and/or modify
;;; it under the terms of the GNU General Public License as published
;;; by the Free Software Foundation, either version 3 of the License,
;;; or (at your option) any later version.
;;;
;;; guix-forge is distributed in the hope that it will be useful, but
;;; WITHOUT ANY WARRANTY; without even the implied warranty of
;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
;;; General Public License for more details.
;;;
;;; You should have received a copy of the GNU General Public License
;;; along with guix-forge.  If not, see
;;; <https://www.gnu.org/licenses/>.

(use-modules (skribilo source lisp)
             (doc skribilo))

(document :title [guix-forge]
  (toc)
  (chapter :title [Introduction]
           :ident "chapter-introduction"
    (p [guix-forge is a Guix service that lets you run a complete
,(ref :url "https://en.wikipedia.org/wiki/Forge_(software)"
:text "software forge") in the manner of GitHub, GitLab, etc. Unlike
other free software forges such as GitLab, Gitea, etc., guix-forge is
not a monolith but is an assemblage of several pieces of server
software wired up to function as one. In this sense, it is a
,(emph "meta-service"). guix-forge does for software forges what ,(ref
:url "https://mailinabox.email/" :text "Mail-in-a-Box") does for
email.])
    (p [guix-forge integrates the following software components:]
       (itemize
        (item [,(ref :url "https://laminar.ohwg.net/" :text "laminar")
for continuous integration])
        (item [,(ref :url "https://github.com/jonashaag/klaus/" :text
"klaus") to serve project git repositories on the web])))
    (p [In the future, it will also provide:]
       (itemize
        (item [web server to serve static project sites])
        (item [,(ref :url "https://git.zx2c4.com/cgit/" :text "cgit")
to serve project git repositories on the web])
        (item [,(ref :url "https://public-inbox.org/README.html"
:text "public-inbox") for project discussions])))
    (p [A choice of different software components may be offered
provided it does not complicate the interface too much.])
    (p [guix-forge is provided on a best effort basis. Its
design is unstable, and open to change. We will try our best to not
break your system configuration often, but it might happen.])
    (section :title [Philosophy]
      (p [In order to empower ordinary users, software should not just
be free (as in freedom), but also be simple and easy to deploy,
especially for small-scale deployments. guix-forge is therefore
minimalistic, and does not require running large database servers such
as MariaDB and PostgreSQL.])
      (p [While some state is inevitable, server software should
strive to be as stateless as an old analog television set. You switch
it on, and it works all the time. There are no pesky software updates,
and complex hidden state. guix-forge tries to be as stateless as
possible. Almost all of guix-forge's state can be version controlled,
and the rest are simple files that can be backed up easily.])
      (p [,(ref
:url "https://drewdevault.com/2018/07/23/Git-is-already-distributed.html"
:text "Git is already federated and decentralized") with
email. guix-forge acknowledges this and prefers to support git's ,(ref
:url "https://drewdevault.com/2018/07/02/Email-driven-git.html"
:text "email driven workflow") with project discussion, bug reports
and patches all happening over email.])
      (p [guix-forge is opinionated and will not expose all features
provided by the software components underlying it. Keeping
configuration options to a minimum is necessary to help casual users
deploy their own forge, and to reduce the likelihood of configuration
bugs.])))
  (chapter :title [Tutorial]
           :ident "chapter-tutorial"
    (p [In this tutorial, you will learn how to set up guix-forge to
host continuous integration for a project. For the purposes of this
tutorial, we will set up continuous integration for the ,(ref :url
[https://github.com/aconchillo/guile-json] :text [guile-json])
project.])
    (p [First, we clone the upstream guile-json repository into a
local bare clone at ,(file [/srv/git/guile-json]).])
    (prog [$ git clone --bare https://github.com/aconchillo/guile-json /srv/git/guile-json
Cloning into bare repository '/srv/git/guile-json'...
remote: Enumerating objects: 1216, done.
remote: Counting objects: 100% (162/162), done.
remote: Compressing objects: 100% (107/107), done.
remote: Total 1216 (delta 96), reused 106 (delta 54), pack-reused 1054
Receiving objects: 100% (1216/1216), 276.10 KiB \| 3.89 MiB/s, done.
Resolving deltas: 100% (742/742), done.]
          :line #f)
    (p [Now that we have a git repository to work with, we start
writing our Guix system configuration. We begin with a bunch of ,(code
[use-modules]) statements importing all required modules.])
    (prog (source :language scheme
                  :file "doc/snippets/tutorial.scm"
                  :start 0 :stop 9)
          :line #f)
    (p [Then, we define the ,(ref :url
"https://guix.gnu.org/en/manual/devel/en/html_node/G_002dExpressions.html"
:text "G-expression") that will be run as a continuous integration job
on every commit. This G-expression uses ,(code [invoke]) from ,(code
[(guix build utils)]). Hence, we make it available to the G-expression
using ,(code [with-imported-modules]). In addition, it needs a number
of packages which we make available using ,(code [with-packages]). And
finally, within the body of the G-expression, we have commands cloning
the git repository, building the source and running the tests.])
    (p [The attentive reader may notice what looks like ,(code [(guix
build utils)]) being referenced twice—once with ,(code
[with-imported-modules]) and again with ,(code [use-modules]). This is
not a mistake. G-expressions are serialized into Guile scripts. ,(code
[with-imported-modules]) ensures that code for ,(code [(guix build
utils)]) is available and is in the ,(ref :url
"https://www.gnu.org/software/guile/manual/html_node/Load-Paths.html"
:text "load path"). ,(code [use-modules]) actually imports ,(code
[(guix build utils)]) when the script runs. ,(code
[with-imported-modules]) is like installing a library in your system,
and ,(code [use-modules]) is like actually importing that library in a
script. Both are necessary.])
    (prog (source :language scheme
                  :file "doc/snippets/tutorial.scm"
                  :start 11 :stop 22)
          :line #f)
    (p [Now, we configure a ,(code [<forge-project>]) record that
holds metadata about the project and wires up the G-expression we just
defined into a continuous integration job.])
    (prog (source :language scheme
                  :file "doc/snippets/tutorial.scm"
                  :start 24 :stop 32)
          :line #f)
    (p [The ,(code [name]) and ,(code [description]) fields are
hopefully self-explanatory. The ,(code [user]) field specifies the
user who will own the git repository at the path specified by ,(code
[repository]). That user will therefore be able to push into the
repository through ssh or similar. git provides various ,(ref :url
"https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks" :text
"server-side hooks") that trigger on various events. Of these, the
,(file [post-receive]) hook triggers when pushed commits are
received. guix-forge sets up a ,(source-ref "guix/forge/forge.scm"
"\\(define\\* \\(ci-jobs-trigger-gexp" "post-receive hook script") in
the repository to trigger a continuous integration run on every
,(command [git push]).])
    (p [And finally, we put everything together in an ,(code
[operating-system]) declaration. Notice the forge service configured
with ,(code [guile-json-project]) and the laminar service configured
with a port for the web interface to listen on.])
    (prog (source :language scheme
                  :file "doc/snippets/tutorial.scm"
                  :start 34)
          :line #f)
    (p [Now that we have a complete ,(code [operating-system])
definition, let's use the following command to build a
container. After a lot of building, a container script should pop
out.])
    (prog [$ guix system container --network --share=/srv/git/guile-json tutorial.scm
/gnu/store/ilg7c2hpkxhwircxpz22qhjsqp3i9har-run-container]
          :line #f)
    (p [The ,(code [--network]) flag specifies that the container
should share the network namespace of the host. To us, this means that
all ports opened by the container will be visible on the host without
any port forwarding or complicated configuration. The ,(code
[--share=/srv/git/guile-json]) option shares the git repository we
cloned earlier, with the container.])
    (p [To start the container, simply run the container script as
root.])
    (prog [# /gnu/store/ilg7c2hpkxhwircxpz22qhjsqp3i9har-run-container]
          :line #f)
    (p [Now, you can see the status of laminar and running jobs
through its web interface listening on ,(ref
:url "http://localhost:8080"). You can list and queue jobs on the
command-line like so:])
    (prog [$ laminarc show-jobs
guile-json
$ laminarc queue guile-json
guile-json:1]
          :line #f)
    (p [That's it! You just set up your own continuous integration
system and took the first steps to owning your code!])
    (p [You could easily use the same configuration to configure a
Guix system instead of a container. To do so, you will have to take
care of defining the bootloader, file systems and other settings as
per your needs. The overall configuration used in this tutorial is
repeated below for your reference.])
    (prog (source :language scheme
                  :file "doc/snippets/tutorial.scm")))
  (chapter :title [Reference]
           :ident "chapter-reference"
    (description
     (record-documentation "guix/forge/forge.scm" '<forge-configuration>
       (record-field "projects"
         [List of ,(ref :mark "<forge-project>" :text
(code [<forge-project>])) objects describing projects managed by
guix-forge]))
     (record-documentation "guix/forge/forge.scm" '<forge-project>
       (record-field "name"
         [Name of the project])
       (record-field "repository"
         [Path to a local git repository, or URI to a remote git
repository])
       (record-field "user"
         [User who owns the repository if it is local. This field is
disregarded if the repository is remote.])
       (record-field "description"
         [Short one-line description of the project. It is used to set
the ,(file "description") file in the repository and will appear in
the cgit web interface.])
       (record-field "website-directory"
         [Path to the document root of the project website. The
ownership of its parent directory is granted to the ,(code "laminar")
user. The idea is that the website is built by a Guix derivation as a
store item and a symbolic link to that store item is created in the
parent directory.])
       (record-field "ci-jobs"
         [List of ,(ref :mark "<forge-laminar-job>") objects
describing ,(abbr :short "CI" :long "continuous integration") jobs to
configure])
       (record-field "ci-jobs-trigger"
         [One of ,(code ['post-receive-hook]), ,(code ['webhook]), or
,(code ['cron]) representing the type of trigger for continuous
integration jobs.
,(description
  (item :key (code ['post-receive-hook])
        [If ,(code ['post-receive-hook]) is specified, the ,(file
"post-receive") hook of the repository is configured to trigger CI
jobs. This is possible only for local repositories. Note that any
pre-existing ,(file "post-receive") hook is overwritten.])
  (item :key (code ['webhook]) [If ,(code
['webhook]) is specified, a webhook server is configured to trigger CI
jobs when a request is received on ,(samp "http://hostname:port/hooks/<name>") \.])
  (item :key (code ['cron]) [If ,(code ['cron]) is
specified, a cron job triggers the CI jobs once a day.]))]
         :default [,(code ['post-receive-hook]) for local repositories
and ,(code ['cron]) for remote repositories])
       (record-field "repository-branch"
         [Main branch of the repository. This field is currently
unused unused, and may be deprecated in the future.]))
     (record-documentation "guix/forge/laminar.scm" '<forge-laminar-job>
       (record-field "name"
         [Name of the job])
       (record-field "run"
         [G-expression to be run])
       (record-field "after"
         [G-expression to be run after the main job script])
       (record-field "trigger?"
         [If ,(code [#t]), this job is run on every commit. Else, it
must be manually set up to run some other way.]))
     (record-documentation "guix/forge/socket.scm" '<forge-host-socket>
       (record-field "hostname"
         [Name of the host])
       (record-field "port"
         [Port number]))
     (record-documentation "guix/forge/socket.scm" '<forge-ip-socket>
       (record-field "ip"
         [IP address, either IPv4 or IPv6, as a string. The loopback
address is ,(code ["127.0.0.1"]) and ,(code ["::1"]) for IPv4 and IPv6
respectively. The any address is ,(code ["0.0.0.0"]) and ,(code
["::"]) for IPv4 and IPv6 respectively.])
       (record-field "port"
         [Port number to listen on.]))
     (record-documentation "guix/forge/socket.scm" '<forge-unix-socket>
       (record-field "path"
         [Path to socket file.]))
     (record-documentation "guix/forge/gunicorn.scm" '<gunicorn-configuration>
       (record-field "package"
         [,(code [gunicorn]) package to use])
       (record-field "apps"
         [List of ,(ref :mark "<gunicorn-app>" :text (code
[<gunicorn-app>])) objects describing gunicorn apps to run]))
     (record-documentation "guix/forge/gunicorn.scm" '<gunicorn-app>
       (record-field "name"
         [Name of the app])
       (record-field "package"
         [Package of the app])
       (record-field "wsgi-app-module"
         [WSGI app module passed to gunicorn])
       (record-field "sockets"
         [List of ,(ref :mark "<forge-ip-socket>" :text (code
[<forge-ip-socket>])) or ,(ref :mark "<forge-unix-socket>" :text (code
[<forge-unix-socket>])) objects describing sockets to listen on])
       (record-field "workers"
         [Number of worker processes])
       (record-field "environment-variables"
         [Association list mapping environment variables that should
be set in the execution environment to their values])
       (record-field "mappings"
         [List of ,(code [<file-system-mapping>]) objects describing
additional directories that should be shared with the container
gunicorn is run in.]))
     (record-documentation "guix/forge/webhook.scm" '<webhook-configuration>
       (record-field "package"
         [,(code [webhook]) package to use])
       (record-field "socket"
         [Socket, a ,(ref :mark "<forge-ip-socket>" :text (code
[<forge-ip-socket>])) object, to listen on.])
       (record-field "log-directory"
         [Directory to write log files to])
       (record-field "hooks"
         [List of ,(ref :mark "<webhook-hook>" :text (code
[<webhook-hook>])) objects describing hooks to configure]))
     (record-documentation "guix/forge/webhook.scm" '<webhook-hook>
       (record-field "id"
         [Identifier of the webhook. This hook is triggered at ,(ref
:url [http://host:port/hooks/<id>]).])
       (record-field "run"
         [G-expression to run when the webhook is triggered]))
     (docstring-function-documentation "guix/forge/klaus.scm" 'klaus-gunicorn-app))))