Want to Contribute to us or want to have 15k+ Audience read your Article ? Or Just want to make a strong Backlink?

Building a fully Type-Safe Event-Driven Backend in Go



TL;DR

This information exhibits you how one can construct a totally Sort-Protected event-driven backend in Go, implementing an Uptime Monitoring system for instance.

We’ll be utilizing Encore to construct our backend, because it supplies end-to-end type-safety together with infrastructure(!).

💡Having type-safety in infrastructure is nice as a result of it means fewer bugs attributable to issues like message queues. You’ll be able to simply determine points throughout growth and keep away from post-deployment points that have an effect on customers. Extra on this later!

🚀 What’s on deck:

  • Set up Encore
  • Create your app from a starter department
  • Run regionally to attempt the frontend
  • Construct the backend
  • Deploy to Encore’s free growth cloud

✨ Ultimate consequence:

Uptime Monitor

Demo app: Try the app

Once we’re accomplished, we’ll have a backend with this type-safe event-driven structure:

Uptime Monitor Architecture
On this diagram (routinely generated by Encore) you’ll be able to see particular person companies as white bins, and Pub/Sub matters as black bins.



🏁 Let’s go!

To make it simpler to comply with alongside, we have laid out a path of croissants to information your method.

Everytime you see a 🥐 it means there’s one thing so that you can do!



💽 Set up Encore

Set up the Encore CLI to run your native setting:

  • macOS: brew set up encoredev/faucet/encore
  • Linux: curl -L https://encore.dev/set up.sh | bash
  • Home windows: iwr https://encore.dev/set up.ps1 | iex



Create your Encore software

🥐 Create your new app from this starter department with a ready-to-go frontend to make use of:

encore app create uptime --example=github.com/encoredev/example-app-uptime/tree/starting-point
Enter fullscreen mode

Exit fullscreen mode



💻 Run your app regionally

🥐 Verify that your frontend works by working your app regionally.

cd uptime
encore run
Enter fullscreen mode

Exit fullscreen mode

It is best to see this:
Encore Run This implies Encore has began your native setting and created native infrastructure for Pub/Sub and Databases.

Then go to http://localhost:4000/frontend/ to see the frontend.
The performance will not work but, since we have not but constructed the backend but.

– Let’s try this now!



🔨 Create the monitor service

Let’s begin by creating the performance to examine if a web site is at the moment up or down.

Later we’ll retailer this lead to a database so we will detect when the standing adjustments and
ship alerts.

🥐 Create a service named monitor containing a file named ping.go. With Encore, you do that by making a Go bundle:

mkdir monitor
contact monitor/ping.go
Enter fullscreen mode

Exit fullscreen mode

🥐 Add an API endpoint named Ping that takes a URL as enter and returns a response indicating whether or not the positioning is up or down.

With Encore you do that by making a operate and including the //encore:api annotation to it.

Paste this into the ping.go file:

bundle monitor

import (
    "context"
    "internet/http"
    "strings"
)

// PingResponse is the response from the Ping endpoint.
sort PingResponse struct {
    Up bool `json:"up"`
}

// Ping pings a particular website and determines whether or not it is up or down proper now.
//
//encore:api public path=/ping/*url
func Ping(ctx context.Context, url string) (*PingResponse, error) {
    // If the url doesn't begin with "http:" or "https:", default to "https:".
    if !strings.HasPrefix(url, "http:") && !strings.HasPrefix(url, "https:") {
        url = "https://" + url
    }

    // Make an HTTP request to examine if it is up.
    req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
    if err != nil {
        return nil, err
    }
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return &PingResponse{Up: false}, nil
    }
    resp.Physique.Shut()

    // 2xx and 3xx standing codes are thought of up
    up := resp.StatusCode < 400
    return &PingResponse{Up: up}, nil
}
Enter fullscreen mode

Exit fullscreen mode

🥐 Let’s attempt it! Be sure you have Docker put in and working, then run encore run in your terminal and it’s best to see the service begin up.

🥐 Now open up the Native Dev Dashboard working at http://localhost:9400 and take a look at calling
the monitor.Ping endpoint, passing in google.com because the URL.

Should you favor to make use of the terminal as an alternative run curl http://localhost:4000/ping/google.com in a brand new terminal as an alternative. Both method, it’s best to see the response:

{"up": true}
Enter fullscreen mode

Exit fullscreen mode

You too can attempt with httpstat.us/400 and some-non-existing-url.com and it ought to reply with {"up": false}.
(It is all the time a good suggestion to check the unfavorable case as properly.)



🧪 Add a take a look at

🥐 Let’s write an automatic take a look at so we do not break this endpoint over time. Create the file monitor/ping_test.go and add this code:

bundle monitor

import (
    "context"
    "testing"
)

func TestPing(t *testing.T) {
    ctx := context.Background()
    checks := []struct {
        URL string
        Up  bool
    }{
        {"encore.dev", true},
        {"google.com", true},
        // Check each with and with out "https://"
        {"httpbin.org/standing/200", true},
        {"https://httpbin.org/standing/200", true},

        // 4xx and 5xx ought to thought of down.
        {"httpbin.org/standing/400", false},
        {"https://httpbin.org/standing/500", false},
        // Invalid URLs needs to be thought of down.
        {"invalid://scheme", false},
    }

    for _, take a look at := vary checks {
        resp, err := Ping(ctx, take a look at.URL)
        if err != nil {
            t.Errorf("url %s: sudden error: %v", take a look at.URL, err)
        } else if resp.Up != take a look at.Up {
            t.Errorf("url %s: obtained up=%v, need %v", take a look at.URL, resp.Up, take a look at.Up)
        }
    }
}
Enter fullscreen mode

Exit fullscreen mode

🥐 Run encore take a look at ./... to examine that all of it works as anticipated. It is best to see one thing like this:

$ encore take a look at ./...
9:38AM INF beginning request endpoint=Ping service=monitor take a look at=TestPing
9:38AM INF request accomplished code=okay length=71.861792 endpoint=Ping http_code=200 service=monitor take a look at=TestPing
[... lots more lines ...]
PASS
okay      encore.app/monitor      1.660
Enter fullscreen mode

Exit fullscreen mode

🎉 It really works. Nicely accomplished!



🔨 Create website service

Subsequent, we wish to maintain monitor of a listing of internet sites to observe.

Since most of those APIs will likely be easy CRUD (Create/Learn/Replace/Delete) endpoints, let’s construct this service utilizing GORM, an ORM library that makes constructing CRUD endpoints actually easy.

🥐 Create a brand new service named website with a SQL database. To take action, create a brand new listing website within the software root with migrations folder inside that folder:

mkdir website
mkdir website/migrations
Enter fullscreen mode

Exit fullscreen mode

🥐 Add a database migration file inside that folder, named 1_create_tables.up.sql. The file title is necessary (it should look one thing like 1_<title>.up.sql) as Encore makes use of the file title to routinely run migrations.

Add the next contents:

CREATE TABLE websites (
    id BIGSERIAL PRIMARY KEY,
    url TEXT NOT NULL
);
Enter fullscreen mode

Exit fullscreen mode

🥐 Subsequent, set up the GORM library and PostgreSQL driver:

go get -u gorm.io/gorm gorm.io/driver/postgres
Enter fullscreen mode

Exit fullscreen mode

Now let’s create the website service itself. To do that we’ll use Encore’s help for dependency injection to inject the GORM database connection.

🥐 Create website/service.go and add this code:

bundle website

import (
    "encore.dev/storage/sqldb"
    "gorm.io/driver/postgres"
    "gorm.io/gorm"
)

//encore:service
sort Service struct {
    db *gorm.DB
}

var siteDB = sqldb.Named("website").Stdlib()

// initService initializes the positioning service.
// It's routinely known as by Encore on service startup.
func initService() (*Service, error) {
    db, err := gorm.Open(postgres.New(postgres.Config{
        Conn: siteDB,
    }))
    if err != nil {
        return nil, err
    }
    return &Service{db: db}, nil
}
Enter fullscreen mode

Exit fullscreen mode

🥐 With that, we’re now able to create our CRUD endpoints.
Create the next information:

website/get.go:

bundle website

import "context"

// Web site describes a monitored website.
sort Web site struct {
    // ID is a singular ID for the positioning.
    ID int `json:"id"`
    // URL is the positioning's URL.
    URL string `json:"url"`
}

// Get will get a website by id.
//
//encore:api public methodology=GET path=/website/:siteID
func (s *Service) Get(ctx context.Context, siteID int) (*Web site, error) {
    var website Web site
    if err := s.db.The place("id = $1", siteID).First(&website).Error; err != nil {
        return nil, err
    }
    return &website, nil
}
Enter fullscreen mode

Exit fullscreen mode

website/add.go:

bundle website

import "context"

// AddParams are the parameters for including a website to be monitored.
sort AddParams struct {
    // URL is the URL of the positioning. If it would not include a scheme
    // (like "http:" or "https:") it defaults to "https:".
    URL string `json:"url"`
}

// Add provides a brand new website to the record of monitored web sites.
//
//encore:api public methodology=POST path=/website
func (s *Service) Add(ctx context.Context, p *AddParams) (*Web site, error) {
    website := &Web site{URL: p.URL}
    if err := s.db.Create(website).Error; err != nil {
        return nil, err
    }
    return website, nil
}
Enter fullscreen mode

Exit fullscreen mode

website/record.go:

bundle website

import "context"

sort ListResponse struct {
    // Websites is the record of monitored websites.
    Websites []*Web site `json:"websites"`
}

// Checklist lists the monitored web sites.
//
//encore:api public methodology=GET path=/website
func (s *Service) Checklist(ctx context.Context) (*ListResponse, error) {
    var websites []*Web site
    if err := s.db.Discover(&websites).Error; err != nil {
        return nil, err
    }
    return &ListResponse{Websites: websites}, nil
}
Enter fullscreen mode

Exit fullscreen mode

website/delete.go:

bundle website

import "context"

// Delete deletes a website by id.
//
//encore:api public methodology=DELETE path=/website/:siteID
func (s *Service) Delete(ctx context.Context, siteID int) error {
    return s.db.Delete(&Web site{ID: siteID}).Error
}
Enter fullscreen mode

Exit fullscreen mode

🥐 Restart encore run to trigger the website database to be created, after which name the website.Add endpoint:

curl -X POST 'http://localhost:4000/website' -d '{"url": "https://encore.dev"}'
{
  "id": 1,
  "url": "https://encore.dev"
}
Enter fullscreen mode

Exit fullscreen mode



📝 Report uptime checks

To be able to notify when a web site goes down or comes again up, we have to monitor the earlier state it was in.

To take action, let’s add a database to the monitor service as properly.

🥐 Create the listing monitor/migrations and the file monitor/migrations/1_create_tables.up.sql:

CREATE TABLE checks (
    id BIGSERIAL PRIMARY KEY,
    site_id BIGINT NOT NULL,
    up BOOLEAN NOT NULL,
    checked_at TIMESTAMP WITH TIME ZONE NOT NULL
);
Enter fullscreen mode

Exit fullscreen mode

We’ll insert a database row each time we examine if a website is up.

🥐 Add a brand new endpoint Verify to the monitor service, that takes in a Web site ID, pings the positioning, and inserts a database row within the checks desk.

For this service we’ll use Encore’s sqldb package as an alternative of GORM (with the intention to showcase each approaches).

Add this to monitor/examine.go:


bundle monitor

import (
    "context"

    "encore.app/website"
    "encore.dev/storage/sqldb"
)

// Verify checks a single website.
//
//encore:api public methodology=POST path=/examine/:siteID
func Verify(ctx context.Context, siteID int) error {
    website, err := website.Get(ctx, siteID)
    if err != nil {
        return err
    }
    consequence, err := Ping(ctx, website.URL)
    if err != nil {
        return err
    }
    _, err = sqldb.Exec(ctx, `
        INSERT INTO checks (site_id, up, checked_at)
        VALUES ($1, $2, NOW())
    `, website.ID, consequence.Up)
    return err
}
Enter fullscreen mode

Exit fullscreen mode

🥐 Restart encore run to trigger the monitor database to be created, after which name the brand new monitor.Verify endpoint:

curl -X POST 'http://localhost:4000/examine/1'
Enter fullscreen mode

Exit fullscreen mode

🥐 Examine the database to verify the whole lot labored:

encore db shell monitor
Enter fullscreen mode

Exit fullscreen mode

It is best to see this:


psql (14.4, server 14.2)
Sort "assist" for assist.

monitor=> SELECT * FROM checks;
 id | site_id | up |          checked_at
----+---------+----+-------------------------------
  1 |       1 | t  | 2022-10-21 09:58:30.674265+00
Enter fullscreen mode

Exit fullscreen mode

If that is what you see, the whole lot’s working nice!🎉



⏰ Add a cron job to examine all websites

We now wish to repeatedly examine all of the tracked websites so we will
instantly reply in case any of them go down.

We’ll create a brand new CheckAll API endpoint within the monitor service that may record all of the tracked websites and examine all of them.

🥐 Let’s extract a few of the performance we wrote for the
Verify endpoint right into a separate operate.

In monitor/examine.go it ought to appear to be so:

// Verify checks a single website.
//
//encore:api public methodology=POST path=/examine/:siteID
func Verify(ctx context.Context, siteID int) error {
    website, err := website.Get(ctx, siteID)
    if err != nil {
        return err
    }
    return examine(ctx, website)
}

func examine(ctx context.Context, website *website.Web site) error {
    consequence, err := Ping(ctx, website.URL)
    if err != nil {
        return err
    }
    _, err = sqldb.Exec(ctx, `
        INSERT INTO checks (site_id, up, checked_at)
        VALUES ($1, $2, NOW())
    `, website.ID, consequence.Up)
    return err
}
Enter fullscreen mode

Exit fullscreen mode

Now we’re able to create our new CheckAll endpoint.

🥐 Create the brand new CheckAll endpoint inside monitor/examine.go:

import "golang.org/x/sync/errgroup"

// CheckAll checks all websites.
//
//encore:api public methodology=POST path=/checkall
func CheckAll(ctx context.Context) error {
    // Get all of the tracked websites.
    resp, err := website.Checklist(ctx)
    if err != nil {
        return err
    }

    // Verify as much as 8 websites concurrently.
    g, ctx := errgroup.WithContext(ctx)
    g.SetLimit(8)
    for _, website := vary resp.Websites {
        website := website // seize for closure
        g.Go(func() error {
            return examine(ctx, website)
        })
    }
    return g.Wait()
}
Enter fullscreen mode

Exit fullscreen mode

This makes use of an errgroup to examine as much as 8 websites concurrently, aborting early if we encounter any error. (Notice {that a} web site being down is just not handled as an error.)

🥐 Run go get golang.org/x/sync/errgroup to put in that dependency.

🥐 Now that now we have a CheckAll endpoint, outline a cron job to routinely name it each 5 minutes.

Add this to monitor/examine.go:

import "encore.dev/cron"

// Verify all tracked websites each 5 minutes.
var _ = cron.NewJob("check-all", cron.JobConfig{
    Title:    "Verify all websites",
    Endpoint: CheckAll,
    Each:    5 * cron.Minute,
})
Enter fullscreen mode

Exit fullscreen mode

Notice: For ease of growth, cron jobs aren’t triggered when working the appliance regionally, however work when deploying the appliance to your cloud.



🚀 Deploy to Encore’s free growth cloud

To check out your uptime monitor for actual, let’s deploy it to Encore’s growth cloud.

Encore comes with built-in CI/CD, and the deployment course of is so simple as a git push encore.

(You too can combine with GitHub to activate per Pull Request Preview Environments, study extra within the CI/CD docs.)

🥐 Deploy your app by working:

git add -A .
git commit -m 'Preliminary commit'
git push encore
Enter fullscreen mode

Exit fullscreen mode

Encore will now construct and take a look at your app, provision the wanted infrastructure, and deploy your software to the cloud.

After triggering the deployment, you will note a URL the place you’ll be able to view its progress in Encore’s Cloud Dashboard. It would look one thing like: https://app.encore.dev/$APP_ID/deploys/...

From there you may as well see metrics, traces, hyperlink your app to a GitHub repo to get automated deploys on new commits, and join your personal AWS or GCP account to make use of for manufacturing deployment.

🥐 When the deploy has completed, you’ll be able to check out your uptime monitor by going to:
https://staging-$APP_ID.encr.app/frontend.

You now have an Uptime Monitor working within the cloud, properly accomplished!✨



Publish Pub/Sub occasions when a website goes down

An uptime monitoring system is not very helpful if it would not
really notify you when a website goes down.

To take action let’s add a Pub/Sub topic
on which we’ll publish a message each time a website transitions from being as much as being down, or vice versa.



🔬 Sort-Protected Infrastructure: Sensible instance

Usually, Pub/Sub mechanisms are blind to the information constructions of the messages they deal with. It is a widespread supply of hard-to-catch errors that may be a nightmare to debug.

Nonetheless, due to Encore’s Infrastructure SDK, you get absolutely type-safe infrastructure! Now you can obtain end-to-end type-safety from the second of publishing a message, proper via to supply. This not solely eliminates these annoying hard-to-debug errors but in addition interprets to main time financial savings for us builders.

— Now let’s really implement it!👇

🥐 Outline the subject utilizing Encore’s Pub/Sub bundle in a brand new file, monitor/alerts.go:

bundle monitor

import "encore.dev/pubsub"

// TransitionEvent describes a transition of a monitored website
// from up->down or from down->up.
sort TransitionEvent struct {
    // Web site is the monitored website in query.
    Web site *website.Web site `json:"website"`
    // Up specifies whether or not the positioning is now up or down (the brand new worth).
    Up bool `json:"up"`
}

// TransitionTopic is a pubsub matter with transition occasions for when a monitored website
// transitions from up->down or from down->up.
var TransitionTopic = pubsub.NewTopic[*TransitionEvent]("uptime-transition", pubsub.TopicConfig{
    DeliveryGuarantee: pubsub.AtLeastOnce,
})
Enter fullscreen mode

Exit fullscreen mode

Now let’s publish a message on the TransitionTopic if a website’s up/down state differs from the earlier measurement.

🥐 Create a getPreviousMeasurement operate in alerts.go to report the final up/down state:

import "encore.dev/storage/sqldb"

// getPreviousMeasurement stories whether or not the given website was
// up or down within the earlier measurement.
func getPreviousMeasurement(ctx context.Context, siteID int) (up bool, err error) {
    err = sqldb.QueryRow(ctx, `
        SELECT up FROM checks
        WHERE site_id = $1
        ORDER BY checked_at DESC
        LIMIT 1
    `, siteID).Scan(&up)

    if errors.Is(err, sqldb.ErrNoRows) {
        // There was no earlier ping; deal with this as if the positioning was up earlier than
        return true, nil
    } else if err != nil {
        return false, err
    }
    return up, nil
}
Enter fullscreen mode

Exit fullscreen mode

🥐 Now add a operate in alerts.go to conditionally publish a message if the up/down state differs:

import "encore.app/website"

func publishOnTransition(ctx context.Context, website *website.Web site, isUp bool) error {
    wasUp, err := getPreviousMeasurement(ctx, website.ID)
    if err != nil {
        return err
    }
    if isUp == wasUp {
        // Nothing to do
        return nil
    }
    _, err = TransitionTopic.Publish(ctx, &TransitionEvent{
        Web site: website,
        Up:   isUp,
    })
    return err
}
Enter fullscreen mode

Exit fullscreen mode

🥐 Lastly modify the examine operate in examine.go to name the publishOnTransition operate:

func examine(ctx context.Context, website *website.Web site) error {
    consequence, err := Ping(ctx, website.URL)
    if err != nil {
        return err
    }

    // Publish a Pub/Sub message if the positioning transitions
    // from up->down or from down->up.
    if err := publishOnTransition(ctx, website, consequence.Up); err != nil {
        return err
    }

    _, err = sqldb.Exec(ctx, `
        INSERT INTO checks (site_id, up, checked_at)
        VALUES ($1, $2, NOW())
    `, website.ID, consequence.Up)
    return err
}
Enter fullscreen mode

Exit fullscreen mode

Now the monitoring system will publish messages on the TransitionTopic each time a monitored website transitions from up->down or from down->up.

Nonetheless, it would not know or care who really listens to those messages. The reality is true now no one does. So let’s repair that by including a Pub/Sub subscriber that posts these occasions to Slack.



Ship Slack notifications when a website goes down

🥐 Begin by making a Slack service slack/slack.go containing the next:

bundle slack

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "internet/http"
)

sort NotifyParams struct {
    // Textual content is the Slack message textual content to ship.
    Textual content string `json:"textual content"`
}

// Notify sends a Slack message to a pre-configured channel utilizing a
// Slack Incoming Webhook (see https://api.slack.com/messaging/webhooks).
//
//encore:api personal
func Notify(ctx context.Context, p *NotifyParams) error {
    reqBody, err := json.Marshal(p)
    if err != nil {
        return err
    }
    req, err := http.NewRequestWithContext(ctx, "POST", secrets and techniques.SlackWebhookURL, bytes.NewReader(reqBody))
    if err != nil {
        return err
    }
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return err
    }
    defer resp.Physique.Shut()

    if resp.StatusCode >= 400 {
        physique, _ := io.ReadAll(resp.Physique)
        return fmt.Errorf("notify slack: %s: %s", resp.Standing, physique)
    }
    return nil
}

var secrets and techniques struct {
    // SlackWebhookURL defines the Slack webhook URL to ship
    // uptime notifications to.
    SlackWebhookURL string
}
Enter fullscreen mode

Exit fullscreen mode

🥐 Now go to a Slack group of your alternative (the place you will have permission to create a brand new Incoming Webhook). If you have no, be a part of the Encore Slack and ask in #assist and we’re glad to assist out.

🥐 After getting the Webhook URL, put it aside as a secret utilizing Encore’s built-in secrets manager:

encore secret set --local,dev,prod SlackWebhookURL
Enter fullscreen mode

Exit fullscreen mode

🥐 Check the slack.Notify endpoint by calling it through cURL:

curl 'http://localhost:4000/slack.Notify' -d '{"Textual content": "Testing Slack webhook"}'
Enter fullscreen mode

Exit fullscreen mode

It is best to see the Testing Slack webhook message seem within the Slack channel you designated for the webhook.

🥐 It is now time so as to add a Pub/Sub subscriber to routinely notify Slack when a monitored website goes up or down. Add the next to slack/slack.go:

import (
    "encore.dev/pubsub"
    "encore.app/monitor"
)

var _ = pubsub.NewSubscription(monitor.TransitionTopic, "slack-notification", pubsub.SubscriptionConfig[*monitor.TransitionEvent]{
    Handler: func(ctx context.Context, occasion *monitor.TransitionEvent) error {
        // Compose our message.
        msg := fmt.Sprintf("*%s is down!*", occasion.Web site.URL)
        if occasion.Up {
            msg = fmt.Sprintf("*%s is again up.*", occasion.Web site.URL)
        }

        // Ship the Slack notification.
        return Notify(ctx, &NotifyParams{Textual content: msg})
    },
})
Enter fullscreen mode

Exit fullscreen mode



🎉 Deploy your completed Uptime Monitor

You are now able to deploy your completed Uptime Monitor, full with a Slack integration!

🥐 As earlier than, deploying your app to the cloud is so simple as working:

git add -A .
git commit -m 'Add slack integration'
git push encore
Enter fullscreen mode

Exit fullscreen mode

You now have a totally featured, production-ready, Uptime Monitoring system working within the cloud. Nicely accomplished! ✨



🤯 Wrapping up: All of this got here in at simply over 300 traces of code

You’ve got now constructed a totally functioning uptime monitoring system, conducting a exceptional quantity with little or no code:

  • You’ve got constructed three completely different companies (website, monitor, and slack)
  • You’ve got added two databases (to the website and monitor companies) for monitoring monitored websites and the monitoring outcomes
  • You’ve got added a cron job for routinely checking the websites each 5 minutes
  • You’ve got arrange a totally type-safe Pub/Sub implementation to decouple the monitoring system from the Slack notifications
  • You’ve got added a Slack integration, utilizing secrets and techniques to securely retailer the webhook URL, listening to a Pub/Sub subscription for up/down transition occasions

All of this in only a bit over 300 traces of code!🤯



🎉 Nice job – you are accomplished!

Preserve constructing with these Open Supply App Templates.👈

When you have questions or wish to share your work, be a part of the builders hangout in Encore’s community Slack.👈

Add a Comment

Your email address will not be published. Required fields are marked *

Want to Contribute to us or want to have 15k+ Audience read your Article ? Or Just want to make a strong Backlink?