TL;DR
This information exhibits you how one can construct a totally Sort-Protected event-driven backend in Go, implementing an Uptime Monitoring system for instance.
We’ll be utilizing Encore to construct our backend, because it supplies end-to-end type-safety together with infrastructure(!).
💡Having type-safety in infrastructure is nice as a result of it means fewer bugs attributable to issues like message queues. You’ll be able to simply determine points throughout growth and keep away from post-deployment points that have an effect on customers. Extra on this later!
🚀 What’s on deck:
- Set up Encore
- Create your app from a starter department
- Run regionally to attempt the frontend
- Construct the backend
- Deploy to Encore’s free growth cloud
✨ Ultimate consequence:
Demo app: Try the app
Once we’re accomplished, we’ll have a backend with this type-safe event-driven structure:
On this diagram (routinely generated by Encore) you’ll be able to see particular person companies as white bins, and Pub/Sub matters as black bins.
🏁 Let’s go!
To make it simpler to comply with alongside, we have laid out a path of croissants to information your method.
Everytime you see a 🥐 it means there’s one thing so that you can do!
💽 Set up Encore
Set up the Encore CLI to run your native setting:
-
macOS:
brew set up encoredev/faucet/encore
-
Linux:
curl -L https://encore.dev/set up.sh | bash
-
Home windows:
iwr https://encore.dev/set up.ps1 | iex
Create your Encore software
🥐 Create your new app from this starter department with a ready-to-go frontend to make use of:
encore app create uptime --example=github.com/encoredev/example-app-uptime/tree/starting-point
💻 Run your app regionally
🥐 Verify that your frontend works by working your app regionally.
cd uptime
encore run
It is best to see this: This implies Encore has began your native setting and created native infrastructure for Pub/Sub and Databases.
Then go to http://localhost:4000/frontend/ to see the frontend.
The performance will not work but, since we have not but constructed the backend but.
– Let’s try this now!
🔨 Create the monitor service
Let’s begin by creating the performance to examine if a web site is at the moment up or down.
Later we’ll retailer this lead to a database so we will detect when the standing adjustments and
ship alerts.
🥐 Create a service named monitor
containing a file named ping.go
. With Encore, you do that by making a Go bundle:
mkdir monitor
contact monitor/ping.go
🥐 Add an API endpoint named Ping
that takes a URL as enter and returns a response indicating whether or not the positioning is up or down.
With Encore you do that by making a operate and including the //encore:api
annotation to it.
Paste this into the ping.go
file:
bundle monitor
import (
"context"
"internet/http"
"strings"
)
// PingResponse is the response from the Ping endpoint.
sort PingResponse struct {
Up bool `json:"up"`
}
// Ping pings a particular website and determines whether or not it is up or down proper now.
//
//encore:api public path=/ping/*url
func Ping(ctx context.Context, url string) (*PingResponse, error) {
// If the url doesn't begin with "http:" or "https:", default to "https:".
if !strings.HasPrefix(url, "http:") && !strings.HasPrefix(url, "https:") {
url = "https://" + url
}
// Make an HTTP request to examine if it is up.
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return nil, err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return &PingResponse{Up: false}, nil
}
resp.Physique.Shut()
// 2xx and 3xx standing codes are thought of up
up := resp.StatusCode < 400
return &PingResponse{Up: up}, nil
}
🥐 Let’s attempt it! Be sure you have Docker put in and working, then run encore run
in your terminal and it’s best to see the service begin up.
🥐 Now open up the Native Dev Dashboard working at http://localhost:9400 and take a look at calling
the monitor.Ping
endpoint, passing in google.com
because the URL.
Should you favor to make use of the terminal as an alternative run curl http://localhost:4000/ping/google.com
in a brand new terminal as an alternative. Both method, it’s best to see the response:
{"up": true}
You too can attempt with httpstat.us/400
and some-non-existing-url.com
and it ought to reply with {"up": false}
.
(It is all the time a good suggestion to check the unfavorable case as properly.)
🧪 Add a take a look at
🥐 Let’s write an automatic take a look at so we do not break this endpoint over time. Create the file monitor/ping_test.go
and add this code:
bundle monitor
import (
"context"
"testing"
)
func TestPing(t *testing.T) {
ctx := context.Background()
checks := []struct {
URL string
Up bool
}{
{"encore.dev", true},
{"google.com", true},
// Check each with and with out "https://"
{"httpbin.org/standing/200", true},
{"https://httpbin.org/standing/200", true},
// 4xx and 5xx ought to thought of down.
{"httpbin.org/standing/400", false},
{"https://httpbin.org/standing/500", false},
// Invalid URLs needs to be thought of down.
{"invalid://scheme", false},
}
for _, take a look at := vary checks {
resp, err := Ping(ctx, take a look at.URL)
if err != nil {
t.Errorf("url %s: sudden error: %v", take a look at.URL, err)
} else if resp.Up != take a look at.Up {
t.Errorf("url %s: obtained up=%v, need %v", take a look at.URL, resp.Up, take a look at.Up)
}
}
}
🥐 Run encore take a look at ./...
to examine that all of it works as anticipated. It is best to see one thing like this:
$ encore take a look at ./...
9:38AM INF beginning request endpoint=Ping service=monitor take a look at=TestPing
9:38AM INF request accomplished code=okay length=71.861792 endpoint=Ping http_code=200 service=monitor take a look at=TestPing
[... lots more lines ...]
PASS
okay encore.app/monitor 1.660
🎉 It really works. Nicely accomplished!
🔨 Create website service
Subsequent, we wish to maintain monitor of a listing of internet sites to observe.
Since most of those APIs will likely be easy CRUD (Create/Learn/Replace/Delete) endpoints, let’s construct this service utilizing GORM, an ORM library that makes constructing CRUD endpoints actually easy.
🥐 Create a brand new service named website
with a SQL database. To take action, create a brand new listing website
within the software root with migrations
folder inside that folder:
mkdir website
mkdir website/migrations
🥐 Add a database migration file inside that folder, named 1_create_tables.up.sql
. The file title is necessary (it should look one thing like 1_<title>.up.sql
) as Encore makes use of the file title to routinely run migrations.
Add the next contents:
CREATE TABLE websites (
id BIGSERIAL PRIMARY KEY,
url TEXT NOT NULL
);
🥐 Subsequent, set up the GORM library and PostgreSQL driver:
go get -u gorm.io/gorm gorm.io/driver/postgres
Now let’s create the website
service itself. To do that we’ll use Encore’s help for dependency injection to inject the GORM database connection.
🥐 Create website/service.go
and add this code:
bundle website
import (
"encore.dev/storage/sqldb"
"gorm.io/driver/postgres"
"gorm.io/gorm"
)
//encore:service
sort Service struct {
db *gorm.DB
}
var siteDB = sqldb.Named("website").Stdlib()
// initService initializes the positioning service.
// It's routinely known as by Encore on service startup.
func initService() (*Service, error) {
db, err := gorm.Open(postgres.New(postgres.Config{
Conn: siteDB,
}))
if err != nil {
return nil, err
}
return &Service{db: db}, nil
}
🥐 With that, we’re now able to create our CRUD endpoints.
Create the next information:
website/get.go
:
bundle website
import "context"
// Web site describes a monitored website.
sort Web site struct {
// ID is a singular ID for the positioning.
ID int `json:"id"`
// URL is the positioning's URL.
URL string `json:"url"`
}
// Get will get a website by id.
//
//encore:api public methodology=GET path=/website/:siteID
func (s *Service) Get(ctx context.Context, siteID int) (*Web site, error) {
var website Web site
if err := s.db.The place("id = $1", siteID).First(&website).Error; err != nil {
return nil, err
}
return &website, nil
}
website/add.go
:
bundle website
import "context"
// AddParams are the parameters for including a website to be monitored.
sort AddParams struct {
// URL is the URL of the positioning. If it would not include a scheme
// (like "http:" or "https:") it defaults to "https:".
URL string `json:"url"`
}
// Add provides a brand new website to the record of monitored web sites.
//
//encore:api public methodology=POST path=/website
func (s *Service) Add(ctx context.Context, p *AddParams) (*Web site, error) {
website := &Web site{URL: p.URL}
if err := s.db.Create(website).Error; err != nil {
return nil, err
}
return website, nil
}
website/record.go
:
bundle website
import "context"
sort ListResponse struct {
// Websites is the record of monitored websites.
Websites []*Web site `json:"websites"`
}
// Checklist lists the monitored web sites.
//
//encore:api public methodology=GET path=/website
func (s *Service) Checklist(ctx context.Context) (*ListResponse, error) {
var websites []*Web site
if err := s.db.Discover(&websites).Error; err != nil {
return nil, err
}
return &ListResponse{Websites: websites}, nil
}
website/delete.go
:
bundle website
import "context"
// Delete deletes a website by id.
//
//encore:api public methodology=DELETE path=/website/:siteID
func (s *Service) Delete(ctx context.Context, siteID int) error {
return s.db.Delete(&Web site{ID: siteID}).Error
}
🥐 Restart encore run
to trigger the website
database to be created, after which name the website.Add
endpoint:
curl -X POST 'http://localhost:4000/website' -d '{"url": "https://encore.dev"}'
{
"id": 1,
"url": "https://encore.dev"
}
📝 Report uptime checks
To be able to notify when a web site goes down or comes again up, we have to monitor the earlier state it was in.
To take action, let’s add a database to the monitor
service as properly.
🥐 Create the listing monitor/migrations
and the file monitor/migrations/1_create_tables.up.sql
:
CREATE TABLE checks (
id BIGSERIAL PRIMARY KEY,
site_id BIGINT NOT NULL,
up BOOLEAN NOT NULL,
checked_at TIMESTAMP WITH TIME ZONE NOT NULL
);
We’ll insert a database row each time we examine if a website is up.
🥐 Add a brand new endpoint Verify
to the monitor
service, that takes in a Web site ID, pings the positioning, and inserts a database row within the checks
desk.
For this service we’ll use Encore’s sqldb
package as an alternative of GORM (with the intention to showcase each approaches).
Add this to monitor/examine.go
:
bundle monitor
import (
"context"
"encore.app/website"
"encore.dev/storage/sqldb"
)
// Verify checks a single website.
//
//encore:api public methodology=POST path=/examine/:siteID
func Verify(ctx context.Context, siteID int) error {
website, err := website.Get(ctx, siteID)
if err != nil {
return err
}
consequence, err := Ping(ctx, website.URL)
if err != nil {
return err
}
_, err = sqldb.Exec(ctx, `
INSERT INTO checks (site_id, up, checked_at)
VALUES ($1, $2, NOW())
`, website.ID, consequence.Up)
return err
}
🥐 Restart encore run
to trigger the monitor
database to be created, after which name the brand new monitor.Verify
endpoint:
curl -X POST 'http://localhost:4000/examine/1'
🥐 Examine the database to verify the whole lot labored:
encore db shell monitor
It is best to see this:
psql (14.4, server 14.2)
Sort "assist" for assist.
monitor=> SELECT * FROM checks;
id | site_id | up | checked_at
----+---------+----+-------------------------------
1 | 1 | t | 2022-10-21 09:58:30.674265+00
If that is what you see, the whole lot’s working nice!🎉
⏰ Add a cron job to examine all websites
We now wish to repeatedly examine all of the tracked websites so we will
instantly reply in case any of them go down.
We’ll create a brand new CheckAll
API endpoint within the monitor
service that may record all of the tracked websites and examine all of them.
🥐 Let’s extract a few of the performance we wrote for theVerify
endpoint right into a separate operate.
In monitor/examine.go
it ought to appear to be so:
// Verify checks a single website.
//
//encore:api public methodology=POST path=/examine/:siteID
func Verify(ctx context.Context, siteID int) error {
website, err := website.Get(ctx, siteID)
if err != nil {
return err
}
return examine(ctx, website)
}
func examine(ctx context.Context, website *website.Web site) error {
consequence, err := Ping(ctx, website.URL)
if err != nil {
return err
}
_, err = sqldb.Exec(ctx, `
INSERT INTO checks (site_id, up, checked_at)
VALUES ($1, $2, NOW())
`, website.ID, consequence.Up)
return err
}
Now we’re able to create our new CheckAll
endpoint.
🥐 Create the brand new CheckAll
endpoint inside monitor/examine.go
:
import "golang.org/x/sync/errgroup"
// CheckAll checks all websites.
//
//encore:api public methodology=POST path=/checkall
func CheckAll(ctx context.Context) error {
// Get all of the tracked websites.
resp, err := website.Checklist(ctx)
if err != nil {
return err
}
// Verify as much as 8 websites concurrently.
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(8)
for _, website := vary resp.Websites {
website := website // seize for closure
g.Go(func() error {
return examine(ctx, website)
})
}
return g.Wait()
}
This makes use of an errgroup to examine as much as 8 websites concurrently, aborting early if we encounter any error. (Notice {that a} web site being down is just not handled as an error.)
🥐 Run go get golang.org/x/sync/errgroup
to put in that dependency.
🥐 Now that now we have a CheckAll
endpoint, outline a cron job to routinely name it each 5 minutes.
Add this to monitor/examine.go
:
import "encore.dev/cron"
// Verify all tracked websites each 5 minutes.
var _ = cron.NewJob("check-all", cron.JobConfig{
Title: "Verify all websites",
Endpoint: CheckAll,
Each: 5 * cron.Minute,
})
Notice: For ease of growth, cron jobs aren’t triggered when working the appliance regionally, however work when deploying the appliance to your cloud.
🚀 Deploy to Encore’s free growth cloud
To check out your uptime monitor for actual, let’s deploy it to Encore’s growth cloud.
Encore comes with built-in CI/CD, and the deployment course of is so simple as a git push encore
.
(You too can combine with GitHub to activate per Pull Request Preview Environments, study extra within the CI/CD docs.)
🥐 Deploy your app by working:
git add -A .
git commit -m 'Preliminary commit'
git push encore
Encore will now construct and take a look at your app, provision the wanted infrastructure, and deploy your software to the cloud.
After triggering the deployment, you will note a URL the place you’ll be able to view its progress in Encore’s Cloud Dashboard. It would look one thing like: https://app.encore.dev/$APP_ID/deploys/...
From there you may as well see metrics, traces, hyperlink your app to a GitHub repo to get automated deploys on new commits, and join your personal AWS or GCP account to make use of for manufacturing deployment.
🥐 When the deploy has completed, you’ll be able to check out your uptime monitor by going to:https://staging-$APP_ID.encr.app/frontend
.
You now have an Uptime Monitor working within the cloud, properly accomplished!✨
Publish Pub/Sub occasions when a website goes down
An uptime monitoring system is not very helpful if it would not
really notify you when a website goes down.
To take action let’s add a Pub/Sub topic
on which we’ll publish a message each time a website transitions from being as much as being down, or vice versa.
🔬 Sort-Protected Infrastructure: Sensible instance
Usually, Pub/Sub mechanisms are blind to the information constructions of the messages they deal with. It is a widespread supply of hard-to-catch errors that may be a nightmare to debug.
Nonetheless, due to Encore’s Infrastructure SDK, you get absolutely type-safe infrastructure! Now you can obtain end-to-end type-safety from the second of publishing a message, proper via to supply. This not solely eliminates these annoying hard-to-debug errors but in addition interprets to main time financial savings for us builders.
— Now let’s really implement it!👇
🥐 Outline the subject utilizing Encore’s Pub/Sub bundle in a brand new file, monitor/alerts.go
:
bundle monitor
import "encore.dev/pubsub"
// TransitionEvent describes a transition of a monitored website
// from up->down or from down->up.
sort TransitionEvent struct {
// Web site is the monitored website in query.
Web site *website.Web site `json:"website"`
// Up specifies whether or not the positioning is now up or down (the brand new worth).
Up bool `json:"up"`
}
// TransitionTopic is a pubsub matter with transition occasions for when a monitored website
// transitions from up->down or from down->up.
var TransitionTopic = pubsub.NewTopic[*TransitionEvent]("uptime-transition", pubsub.TopicConfig{
DeliveryGuarantee: pubsub.AtLeastOnce,
})
Now let’s publish a message on the TransitionTopic
if a website’s up/down state differs from the earlier measurement.
🥐 Create a getPreviousMeasurement
operate in alerts.go
to report the final up/down state:
import "encore.dev/storage/sqldb"
// getPreviousMeasurement stories whether or not the given website was
// up or down within the earlier measurement.
func getPreviousMeasurement(ctx context.Context, siteID int) (up bool, err error) {
err = sqldb.QueryRow(ctx, `
SELECT up FROM checks
WHERE site_id = $1
ORDER BY checked_at DESC
LIMIT 1
`, siteID).Scan(&up)
if errors.Is(err, sqldb.ErrNoRows) {
// There was no earlier ping; deal with this as if the positioning was up earlier than
return true, nil
} else if err != nil {
return false, err
}
return up, nil
}
🥐 Now add a operate in alerts.go
to conditionally publish a message if the up/down state differs:
import "encore.app/website"
func publishOnTransition(ctx context.Context, website *website.Web site, isUp bool) error {
wasUp, err := getPreviousMeasurement(ctx, website.ID)
if err != nil {
return err
}
if isUp == wasUp {
// Nothing to do
return nil
}
_, err = TransitionTopic.Publish(ctx, &TransitionEvent{
Web site: website,
Up: isUp,
})
return err
}
🥐 Lastly modify the examine
operate in examine.go
to name the publishOnTransition
operate:
func examine(ctx context.Context, website *website.Web site) error {
consequence, err := Ping(ctx, website.URL)
if err != nil {
return err
}
// Publish a Pub/Sub message if the positioning transitions
// from up->down or from down->up.
if err := publishOnTransition(ctx, website, consequence.Up); err != nil {
return err
}
_, err = sqldb.Exec(ctx, `
INSERT INTO checks (site_id, up, checked_at)
VALUES ($1, $2, NOW())
`, website.ID, consequence.Up)
return err
}
Now the monitoring system will publish messages on the TransitionTopic
each time a monitored website transitions from up->down or from down->up.
Nonetheless, it would not know or care who really listens to those messages. The reality is true now no one does. So let’s repair that by including a Pub/Sub subscriber that posts these occasions to Slack.
Ship Slack notifications when a website goes down
🥐 Begin by making a Slack service slack/slack.go
containing the next:
bundle slack
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"internet/http"
)
sort NotifyParams struct {
// Textual content is the Slack message textual content to ship.
Textual content string `json:"textual content"`
}
// Notify sends a Slack message to a pre-configured channel utilizing a
// Slack Incoming Webhook (see https://api.slack.com/messaging/webhooks).
//
//encore:api personal
func Notify(ctx context.Context, p *NotifyParams) error {
reqBody, err := json.Marshal(p)
if err != nil {
return err
}
req, err := http.NewRequestWithContext(ctx, "POST", secrets and techniques.SlackWebhookURL, bytes.NewReader(reqBody))
if err != nil {
return err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Physique.Shut()
if resp.StatusCode >= 400 {
physique, _ := io.ReadAll(resp.Physique)
return fmt.Errorf("notify slack: %s: %s", resp.Standing, physique)
}
return nil
}
var secrets and techniques struct {
// SlackWebhookURL defines the Slack webhook URL to ship
// uptime notifications to.
SlackWebhookURL string
}
🥐 Now go to a Slack group of your alternative (the place you will have permission to create a brand new Incoming Webhook). If you have no, be a part of the Encore Slack and ask in #assist
and we’re glad to assist out.
🥐 After getting the Webhook URL, put it aside as a secret utilizing Encore’s built-in secrets manager:
encore secret set --local,dev,prod SlackWebhookURL
🥐 Check the slack.Notify
endpoint by calling it through cURL:
curl 'http://localhost:4000/slack.Notify' -d '{"Textual content": "Testing Slack webhook"}'
It is best to see the Testing Slack webhook message seem within the Slack channel you designated for the webhook.
🥐 It is now time so as to add a Pub/Sub subscriber to routinely notify Slack when a monitored website goes up or down. Add the next to slack/slack.go
:
import (
"encore.dev/pubsub"
"encore.app/monitor"
)
var _ = pubsub.NewSubscription(monitor.TransitionTopic, "slack-notification", pubsub.SubscriptionConfig[*monitor.TransitionEvent]{
Handler: func(ctx context.Context, occasion *monitor.TransitionEvent) error {
// Compose our message.
msg := fmt.Sprintf("*%s is down!*", occasion.Web site.URL)
if occasion.Up {
msg = fmt.Sprintf("*%s is again up.*", occasion.Web site.URL)
}
// Ship the Slack notification.
return Notify(ctx, &NotifyParams{Textual content: msg})
},
})
🎉 Deploy your completed Uptime Monitor
You are now able to deploy your completed Uptime Monitor, full with a Slack integration!
🥐 As earlier than, deploying your app to the cloud is so simple as working:
git add -A .
git commit -m 'Add slack integration'
git push encore
You now have a totally featured, production-ready, Uptime Monitoring system working within the cloud. Nicely accomplished! ✨
🤯 Wrapping up: All of this got here in at simply over 300 traces of code
You’ve got now constructed a totally functioning uptime monitoring system, conducting a exceptional quantity with little or no code:
- You’ve got constructed three completely different companies (
website
,monitor
, andslack
) - You’ve got added two databases (to the
website
andmonitor
companies) for monitoring monitored websites and the monitoring outcomes - You’ve got added a cron job for routinely checking the websites each 5 minutes
- You’ve got arrange a totally type-safe Pub/Sub implementation to decouple the monitoring system from the Slack notifications
- You’ve got added a Slack integration, utilizing secrets and techniques to securely retailer the webhook URL, listening to a Pub/Sub subscription for up/down transition occasions
All of this in only a bit over 300 traces of code!🤯
🎉 Nice job – you are accomplished!
Preserve constructing with these Open Supply App Templates.👈
When you have questions or wish to share your work, be a part of the builders hangout in Encore’s community Slack.👈