This Banner is For Sale !!
Get your ad here for a week in 20$ only and get upto 15k traffic Daily!!!

Dagger 101: How to Get Started with Containerized CI Workflows


Steady Integration and Steady Supply are the key sauces of delivery new options persistently and reliably to your software program. Nevertheless, the effectiveness of this course of is carefully tied to the tooling that orchestrates it. Among the ache factors of CI/CD techniques are gradual suggestions loops, vendor lock-in, lack of abstraction, restricted composability, or YAML itself. That is the place Dagger comes into the highlight, promising a extra unified and accelerated path.




Introduction

The event and deployment course of at dyrector.io has already develop into a lot quicker annually as we undertake and combine higher instruments and strategies. Nevertheless, we purpose to additional unify and speed up this. Dagger philosophy aligns with what we contemplate essential for a really speedy and seamless course of:

  • Native testing: Allow builders to check their code immediately, domestically
  • Programmable CI: Change messy YAML-based, advanced CI with code
  • Compatibility: If it runs in a container, you’ll be able to add it to your pipeline
  • Portability: The identical pipeline can run in your native machine, a CI runner, a devoted server, or any container internet hosting service
  • Common caching: Each operation is cached by default, and caching works the identical in every single place

At the moment, we’ve the choice to make use of our personal dyrector.io (we’ll consult with it as dyo many instances on this weblog put up) go CLI with our instructions or Docker Compose with its YAML to spin up our stack for native testing, whereas we additionally preserve a GitHub Actions workflow for working end-to-end exams on GitHub. This setup lacks coherence, as we can’t make use of the specialised GitHub Actions workflow YAML in an area setting or with a distinct CI/CD surroundings.

We need to get nearer to with the ability to ship each single day, and even a number of instances a day, as shortly as we presumably can, utilizing the identical instrument working domestically and in CI. Dagger looks like an precise innovation in CI/CD, and it appears it can allow us to do this. There may be additionally a robust give attention to getting suggestions from the neighborhood and using it after we’re designing and constructing one thing that folks really want.



Organising Dagger CI/CD

We wish to use Dagger domestically with the dyo Go CLI, and for this we want the Dagger Go SDK for integration (there are lots of Dagger SDKs) and the Dagger Engine, which can run our pipelines. We developed a small proof of idea (POC) to guage if we may use our complete stack domestically with Dagger. If this POC will likely be profitable, we plan to make use of the identical setup in our GitHub workflow, primarily utilizing GitHub Actions simply to set off the Dagger pipeline.

Steps to arrange Dagger for our challenge:

  1. Set up the Dagger Go SDK
    (once more, you should use every other Dagger SDK to your challenge, however we use Go)
    Go to your present challenge – in our case it’s dyrectorio.
$ go get dagger.io/dagger
$ go mod tidy
Enter fullscreen mode

Exit fullscreen mode

  1. Add native Dagger take a look at to our Makefile
    It’s for easy and quick “make take a look at” (equally to our different instructions).
# Shortcut for native testing
.PHONY: take a look at
take a look at:
    go run golang/cmd/dagger/important.go
Enter fullscreen mode

Exit fullscreen mode

  1. Create Dagger important.go
    We have already got dyo, dagent and crane in our golang/cmd, so put dagger right here too.

  2. Import Dagger SDK

Create a Dagger shopper utilizing the SDK
This may let you work together with the Dagger Engine and create pipelines.

  1. Create Dagger pipelines

Extra word:
We are able to additionally set up the Dagger CLI if we need to, however that is an non-obligatory instrument to work together with the Dagger Engine from the command-line – it has a pleasant terminal UI although, with parallel progress bars which might be visually spectacular if you’re into that form of factor.

Set up the Dagger CLI

$ cd /usr/native
$ curl -L https://dl.dagger.io/dagger/set up.sh | sh
Enter fullscreen mode

Exit fullscreen mode



Workflow Integration

As you will note, the “Dagger method” is a really “Docker-ish” method – no shock, one of many co-founders of Dagger is Solomon Hykes, earlier founder and technical director of Docker.

To indicate you concrete code examples from our POC:

Import Dagger SDK
In our important.go:

import (
    "context"
    "dagger.io/dagger"
    …)
Enter fullscreen mode

Exit fullscreen mode

Create a Dagger shopper utilizing the SDK

func initDaggerClient(ctx context.Context) *dagger.Shopper {
    shopper, err := dagger.Join(ctx, dagger.WithLogOutput(os.Stdout))
    if err != nil {
        panic(err)
    }
    return shopper
}
Enter fullscreen mode

Exit fullscreen mode

And we are able to name this initDaggerClient() operate in our important() like this:

    ctx := context.Background()
    shopper := initDaggerClient(ctx)
    defer shopper.Shut()
Enter fullscreen mode

Exit fullscreen mode

Run unit exams on our NestJS-based Crux backend:

func runCruxUnitTestPipeline(ctx context.Context, shopper *dagger.Shopper) {
    log.Information().Msg("Run crux unit take a look at pipeline...")

    _, err := shopper.Container().From("node:20-alpine").
        WithDirectory("/src", shopper.Host().Listing("net/crux/"), dagger.ContainerWithDirectoryOpts{
            Exclude: []string{"node_modules"},
        }).
        WithWorkdir("/src").
        WithExec([]string{"npm", "ci"}).
        WithExec([]string{"npm", "run", "take a look at"}).
        Stdout(ctx)
    if err != nil {
        panic(err)
    }

    log.Information().Msg("Crux unit take a look at pipeline accomplished.")
}
Enter fullscreen mode

Exit fullscreen mode

We are able to name this runCruxUnitTestPipeline() operate in our important():
runCruxUnitTestPipeline(ctx, shopper)

Run unit exams on our Subsequent.js-based Crux UI frontend is similar to the above code, we solely want to alter the host listing to “net/crux-ui/” and an extra “.subsequent” exclusion, all the things else stays the identical:

    WithDirectory("/src", shopper.Host().Listing("net/crux-ui/"), dagger.ContainerWithDirectoryOpts{
        Exclude: []string{"node_modules", ".subsequent"},
    }).
Enter fullscreen mode

Exit fullscreen mode

A barely extra superior instance after we run our Crux backend in manufacturing mode (as we do for e2e take a look at) with a linked PostgreSQL DB service container:

func getEnv(envPath string) map[string]string {
    cruxEnv, err := godotenv.Learn(envPath)
    if err != nil {
        panic(err)
    }
    return cruxEnv
}

func getCruxPostgres(shopper *dagger.Shopper, cruxEnv map[string]string) *dagger.Container {
    databaseURL := cruxEnv["DATABASE_URL"]
    parsedURL, err := url.Parse(databaseURL)
    if err != nil {
        panic(err)
    }
    postgresUsername := parsedURL.Person.Username()
    postgresPassword, _ := parsedURL.Person.Password()
    postgresDB := strings.TrimPrefix(parsedURL.Path, "https://style-tricks.com/")

    dataCache := shopper.CacheVolume("knowledge")

    cruxPostgres := shopper.Pipeline("crux-postgres").Container().From("postgres:14.2-alpine").
        WithMountedCache("/knowledge", dataCache).
        WithEnvVariable("POSTGRES_USER", postgresUsername).
        WithEnvVariable("POSTGRES_PASSWORD", postgresPassword).
        WithEnvVariable("POSTGRES_DB", postgresDB).
        WithEnvVariable("PGDATA", "/knowledge/postgres").
        WithExposedPort(5432)

    return cruxPostgres
}

func runCruxProd(ctx context.Context, shopper *dagger.Shopper, cruxPostgres *dagger.Container) *dagger.Container {
    crux := shopper.Pipeline("crux").Container().From("node:20-alpine")
    crux = crux.
        WithDirectory("/src", shopper.Host().Listing("net/crux/"), dagger.ContainerWithDirectoryOpts{
            Exclude: []string{"node_modules"},
        }).
        WithWorkdir("/src").
        WithServiceBinding("localhost", cruxPostgres).
        // WithEnvVariable("NOCACHE", time.Now().String()).
        WithExec([]string{"npm", "ci"}).
        WithExec([]string{"npm", "run", "construct"}).
        WithExec([]string{"npm", "run", "prisma:migrate"}).
        WithExec([]string{"npm", "run", "begin:prod"})

    _, err := crux.Stdout(ctx)
    if err != nil {
        panic(err)
    }

    return crux
}
Enter fullscreen mode

Exit fullscreen mode

We are able to run the above code in our important() like this:

    cruxEnv := getEnv("net/crux/.env") 
    cruxPostgres := getCruxPostgres(shopper, cruxEnv) 
    runCruxProd(ctx, shopper, cruxPostgres) 
Enter fullscreen mode

Exit fullscreen mode

We wish to word that we made our POC with Dagger 0.8.x throughout September, so the code snippets above will present that. However even then the brand new API improvement of Dagger Companies v2 (which we are going to want for our advanced e2e pipeline) was in progress at Dagger in a separate characteristic department and so they promised on their Discord discussion board again then that this new API with some breaking adjustments will likely be included in Dagger 0.9. It wasn’t simply us displaying demand for parallel lengthy working service containers – and so they stored their phrase and it’s certainly included in Dagger 0.9.0 launched on the finish of October. Shouts to Crew Dagger!

We put our POC on maintain in October, however we’ve been keeping track of Service v2 developments and information. We are going to check out Service v2 within the close to future and dedicate one other weblog put up as to if we managed to unravel our complete e2e pipeline with Dagger.

Dagger effectively caches every step of the pipelines, mechanically dealing with the caching of supply code copies, containers and builds, and when builders configure it programmatically, it additionally caches mounted volumes equivalent to database knowledge, node_modules, and Go build-cache. Our logs present clear examples of this on reruns with out code modifications.

    copy net/crux/ CACHED
    > in host.listing net/crux/
    …
    pull docker.io/library/postgres:14.2-alpine CACHED
    > in crux-postgres > from postgres:14.2-alpine
    > in crux > service bvqf991cmob5i.97ul8ph8qf1qc.dagger.native
    …
    exec docker-entrypoint.sh postgres
    > in crux > service bvqf991cmob5i.97ul8ph8qf1qc.dagger.native
    [0.15s] PostgreSQL Database listing seems to comprise a database; Skipping initialization
    …
    [0.30s] 2023-11-08 10::11.131 UTC [15] LOG:  database system is able to settle for connections
    ...
    exec docker-entrypoint.sh npm run construct CACHED
    > in crux
    exec docker-entrypoint.sh npm run prisma:migrate CACHED
    > in crux
    exec docker-entrypoint.sh npm ci CACHED
    > in crux
    copy / /src CACHED
    > in crux
    exec docker-entrypoint.sh npm run begin:prod
    > in crux
    [0.57s] > crux@0.7.0 begin:prod
    [0.57s] > node dist/important
    [2.31s] [Nest] 33  - 11/07/2023, 14:24:13.142 AM     LOG [NestFactory] Beginning Nest utility...
    ...
Enter fullscreen mode

Exit fullscreen mode



Challenges and Classes Realized

We had been in a position to run most of our stack with Dagger 0.8.x, the Crux backend and the Crux-UI frontend individually, however our complete e2e take a look at would require Dagger 0.9.x with the Companies v2 API that we are able to run Crux, Crux-ui, Traefik and Kratos as lengthy working service containers for the Playwright e2e container.

If you wish to know extra in regards to the Companies v2, Dagger wrote a weblog put up about it right here:
https://dagger.io/blog/dagger-0-9



Finest Practices for Dagger CI/CD

The truth that we are able to write the CI/CD code in Go and in a docker-like type had a refreshing impact on us. Listed below are some common ideas:

  • Iterate small: Begin with a small POC to grasp how Dagger matches into your workflow earlier than scaling up
  • Neighborhood engagement: Keep lively in Dagger’s neighborhood boards or Discord channels for help and to maintain up with the newest developments
  • Documentation: Preserve your Dagger configurations well-documented to ease onboarding and upkeep
  • Monitor and optimize: Repeatedly evaluate the efficiency of your pipelines and optimize caching methods for higher effectivity



Conclusion

We now have seen firsthand the transformative nature of Dagger and the flexibleness of its programmable pipelines. It stands out as a forward-thinking answer, addressing typical CI/CD bottlenecks with a developer-centric method. Since Dagger is comparatively new and evolving, keeping track of updates and neighborhood suggestions may help in adopting greatest practices as they emerge.



Dagger Sources

There’s nonetheless lot to find out about Dagger, so it is perhaps definitely worth the time to take a look at the next assets to find out about this instrument:


This blogpost was written by the staff of dyrector.io. dyrector.io is an open-source steady supply & deployment platform with model administration.

Assist us with a star on GitHub.

The Article was Inspired from tech community site.
Contact us if this is inspired from your article and we will give you credit for it for serving the community.

This Banner is For Sale !!
Get your ad here for a week in 20$ only and get upto 10k Tech related traffic daily !!!

Leave a Reply

Your email address will not be published. Required fields are marked *

Want to Contribute to us or want to have 15k+ Audience read your Article ? Or Just want to make a strong Backlink?