HTTP API in Go – Part II

Level: Beginner

In the first part of these tutorials, we covered the basics of building an HTTP API using the standard library including attaching HTTP handlers to the routes, setting HTTP headers, working with status codes and sending response back in JSON format.

In this part, we are going to learn: 

  • How to serve different HTTP methods (GET, POST)
  • How to put HTTP method constraints on the routes
  • How to pass client data to the server using request body and query string parameters
  • What are middle-wares and how to chain them together
  • and a new technique for implementing handlers to make the components more modular and extendable. 

As a bonus, I’m also going to introduce you to our little friend Badger, a great local embeddable key-value store which we are going to use as the persistent layer in our little project today. 

Mr. Bookworm’s

Last summer, Robbin, the owner of our local bookstore decided to take his small business, Mr Bookworm’s, to the next level by building a website and making his amazing books available to the local community over the internet. It seems that everything worked out extremely well for him and this year he has decided to franchise Mr. Bookworm’s  to the new resellers in the neighbourhood and expand his little empire.

To make that happen, Robbin needs to give the franchisees access to his catalogue by providing APIs for listing and ordering books. Here is the list of the endpoint he’s asked us to build:

  • GET /list?category=fiction: To list all the available books in the specified category
  • POST /order : To order a title before it runs out of stock

Currently, Mr. Bookworm’s listing is kept in a Badger key-value store, keyed by each title’s ISBN.

Models

Our domain definition consists of two simple types to model the main entities, the books and the orders:

type Book struct {
	ISBN string `json:"-"`
	Title string `json:"title"`
	Category string `json:"category"`
	Author string `json:"author"`
	Year int `json:"year"`
	Catalogued time.Time `json:"catalogued"`
	Stock int `json:"stock"`
}

type Order struct {
	ID string `json:"-"`
	Count int `json:"count"`
	ISBN string `json:"isbn"`
	Date time.Time `json:"date"`
}

As you might have noticed, we excluded the book’s ISBN and the order’s ID fields from serialisation (by setting json tag to -), because our database is keyed by those fields and we don’t need to store them in the payload as well.

DAL (Data Access Layer)

The data access layer, must be able to list the books, filtered by a category, and put a new order of N copies of a specific title. Keeping those two requirements in mind, here is how our DAL interface will look like:

 
type Database interface {
	ReadCategory(category string) ([]models.Book, error)
	PlaceOrder(order models.Order) (string, error)
	io.Closer
}

To keep the tutorial relevant to its title and also give you an idea of how Badger works, I only mention the implementation of  PlaceOrder method here. Please check out the source code to see the full implementation of the Database interface.

func (l *LocalKeyValueDB) PlaceOrder(order models.Order) (string, error) {
	id := uuid.New()
	order.Date = time.Now().UTC()
	err := l.orders.Update(func(txn *badger.Txn) error {
		b, err := json.Marshal(order)
		if err != nil {
			return errors.Wrap(err, "failed to marshal the order into Json")
		}
		return txn.Set([]byte(id.String()), b)
	})
	if err != nil {
		return "", err
	}
	return id.String(), nil
}

The API

We are all set and ready to jump on the fun part, the API. But this time we are going to do it a little bit differently. Let’s start by creating a new type and call it bookStore which is going to be our central hub to serve the two routes we mentioned earlier.

type bookStore struct {
	db storage.Database
}

func newBookStore(db storage.Database) *bookStore {
	return &bookStore{
		db: db,
	}
}
func (b *bookStore) putOrder(allowedMethod string) http.HandlerFunc {
	return func(w http.ResponseWriter, req *http.Request) {
		panic("implement me")
	}
}

func (b *bookStore) listBooks(defaultCategory, allowedMethod string) http.HandlerFunc {
	return func(w http.ResponseWriter, req *http.Request) {
		panic("implement me")
	}
}

As you can see, bookStore has an injectable dependency to the Database interface which makes it super easy for us to switch to a different implementation, in case Robbin has decided that Badger is not the way to go in the future.

Also, look closer to the signature of the handler methods. It’s a bit different to what we did in the first part of this tutorial. The handlers above are not actually handlers. They are more like handler factories which return http.HandleFunc instead. There are a few advantages in using this pattern instead of implementing HandleFunc directly. 

Firstly, we can use closures to parameterise each handler independently. For example, see how we used allowedMethod argument in the both handlers to put a constraint on the supported verbs (more on this shortly), or how we parameterised listBooks to fall back to a default category if it’s not been defined by the user.

Secondly, this pattern gives us the ability of adding one-off initialisation to the handlers of our choice. For example, we could use this approach to cache the top ten best sellers in memory to achieve better performance:

func (b *bookStore) listBooks(defaultCategory, allowedMethod string)
http.HandlerFunc {
	b.cacheTop10Categories()
 	return func(w http.ResponseWriter, req *http.Request) {
		panic("implement me")
	}
}

Notice the caching code will only be executed once and not with every single execution of the handler.

And finally, it makes it possible to chain handlers together and build the middle-wares (more on this later) easily.

Now that we learned the new trick, let’s go ahead and implement the handlers together. First thing first. Let’s see how we can put a constraint on the HTTP method for each route.

Unfortunately the standard library does not give us a built-in mechanism to define route specific constraints. To achieve this, we need to manually reject the unsupported verbs:

func (b *bookStore) putOrder(allowedMethod string) http.HandlerFunc {
	return func(w http.ResponseWriter, req *http.Request) {
		if req.Method != allowedMethod {
			http.Error(w, "unsupported method", http.StatusMethodNotAllowed)
			return
		}
     }
}

func (b *bookStore) listBooks(defaultCategory, allowedMethod string) http.HandlerFunc {
	return func(w http.ResponseWriter, req *http.Request) {
		if req.Method != allowedMethod {
			http.Error(w, "unsupported method", http.StatusMethodNotAllowed)
			return
		}
	}
}

Simple as that! Sending request the API using any HTTP method except the allowed one, will get rejected.

Reading Query String Parameters

The /list route needs a way to figure out which category to load. One way to achieve this is to use the well-know HTTP URL parameters a.k.a query string parameters as the input. In Go, we can access the URL parameters through http.Request‘s URL method:

func (b *bookStore) listBooks(defaultCategory, allowedMethod string) http.HandlerFunc {
	return func(w http.ResponseWriter, req *http.Request) {
		cat := req.URL.Query().Get("category")
		if len(cat) == 0 {
			cat = defaultCategory
		}
	}
}

The code will fall back to defaultCategory if no specific category is requested by the user. Now, let’s put everything together and finish the listBooks handler with the database calls:

func (b *bookStore) listBooks(defaultCategory, allowedMethod string) http.HandlerFunc {
	return func(w http.ResponseWriter, req *http.Request) {
		if req.Method != allowedMethod {
			http.Error(w, "unsupported method", http.StatusMethodNotAllowed)
			return
		}
		cat := req.URL.Query().Get("category")
		if len(cat) == 0 {
			cat = defaultCategory
		}
		books, err := b.db.ReadCategory(cat)
		if err != nil {
			writeError(w, "failed to fetch the books", err, http.StatusInternalServerError)
			return
		}
		w.Header().Set("Content-Type", "application/json")
		enc := json.NewEncoder(w)
		err = enc.Encode(books)
		if err != nil {
			writeError(w, "failed to serialise the book list", err, http.StatusInternalServerError)
		}
	}
}

func writeError(w http.ResponseWriter, msg string, err error, code int) {
	log.Printf("%s: %s", msg, err)
	http.Error(w, msg, code)
}

Next stop, putOrder handler.

Reading Json from the Request

Using URL parameters is a simple method for passing user data to the server, as long as you are dealing with a limited number of parameters. Putting the restrictions on the URL size aside (imposed differently by different web browsers), things will quickly get messy if we put everything in the URL. I am not sure about you, but my brain hurts every time I see something like this: 
https://example.com?mode=base&action=this&from=c30a4638-a6ca-4c5b-8876-c388fcec0735&to=a2d71a48-ba54-4754-9874-4e21b35e89ab&token=aHR0cHM6Ly94aXRvbml4LmlvLzIwMTgvMTAvMDEvaHR0cC1hcGktaW4tZ28tcGFydC1paS8=

When serving a new /order request, we will be going to add a new resource to our database. Therefore, POST is going to be the perfect HTTP method to choose (click here if you are not sure about which method to choose), and not surprisingly, reading JSON formatted payload from a POST request’s body has been a text book approach since Adam and Eve. Let’s stick to the same method because I am sure no one wants to upset our Gopherpas:

func (b *bookStore) putOrder(allowedMethod string) http.HandlerFunc {
	return func(w http.ResponseWriter, req *http.Request) {
		body, err := ioutil.ReadAll(req.Body)
		if err != nil {
			writeError(w, "invalid order data", err, http.StatusBadRequest)
			return
		}
		// DO WE NEED THIS?
		defer req.Body.Close()
	}
}

No need to elaborate as the code is self explanatory! Only one small point which I think is worth mentioning here. The need for closing the Body stream is something that I’ve seen many many times, in pretty much every single article or forum. While closing Body is absolutely necessary at the client side to avoid memory leak (as of today), looking at the documentation makes it super clear that this is not the case for server side code:

// Body is the request's body.
//
// For client requests a nil body means the request has no
// body, such as a GET request. The HTTP Client's Transport
// is responsible for calling the Close method.
//
// For server requests the Request Body is always non-nil
// but will return EOF immediately when no body is present.
// The Server will close the request body. The ServeHTTP
// Handler does not need to.
Body io.ReadCloser

It’s clearly documented that  “the Server will close the request body” and “the ServeHTTP Handler does not need to“. So the answer to the question in the previous code block is a No. We don’t need to close the body at the server side.

Once again, let’s put everything together and see how the final version of our putOrder handler looks like:

func (b *bookStore) putOrder(allowedMethod string) http.HandlerFunc {
	return func(w http.ResponseWriter, req *http.Request) {
		if req.Method != allowedMethod {
			http.Error(w, "unsupported method", http.StatusMethodNotAllowed)
			return
		}
		body, err := ioutil.ReadAll(req.Body)
		if err != nil {
			writeError(w, "invalid order data", err, http.StatusBadRequest)
			return
		}
		order := models.Order{}
		err = json.Unmarshal(body, &order)
		if err != nil {
			writeError(w, "failed to read the order", err, http.StatusInternalServerError)
			return
		}
		id, err := b.db.PlaceOrder(order)
		if err != nil {
			writeError(w, "failed to place a new order", err, http.StatusInternalServerError)
			return
		}
		fmt.Fprint(w, id)
	}
}

Middle-wares

It’s a common requirement of every web application to share functionalities between different routes. For example, you might need to log every single request for debugging purposes in the future, or add authentication to some routes. This is where middle-wares come to rescue.

Even though the name “middle-ware” might seem a bit intimidating but they are simply handlers which call other handlers, that’s all! Same signature, same everything. Let’s go with a simple example to see how they work in action. 

As I mentioned earlier, logging the incoming requests is a very common practice in building we applications and this is where we are going to use a middle-ware for.

func logger(next http.HandlerFunc) http.HandlerFunc {
	return func(w http.ResponseWriter, r *http.Request) {
		log.Printf("Servinge %s %s %s", r.Method, r.URL.Path, r.URL.Query())
		next(w, r)
	}
}

The logger takes a handler in, logs the route data and hands the control over to the input handler (next). We can use the same piping technique to chain together as many handlers as we need. Now that we have got our middle-ware ready, all we need to do is to pipe it into listBooks and putOrder handlers:

func (b *bookStore) listBooks(defaultCategory, allowedMethod string) http.HandlerFunc {
	handler := func(w http.ResponseWriter, req *http.Request) {...}
	return logger(handler)
}

Run the HTTP server

We are almost there! We got all the moving parts in place and ready to be glued together. To make it even sexier, we are going to wrap everything up into a new type called httpServer

type httpServer struct {
	bs *bookStore
	db storage.Database
}

func newHTTPServer() (*httpServer, error) {
	db, err := storage.NewLocalKVStore()
	if err != nil {
		return nil, errors.Wrap(err, "failed to initialise the database")
	}

	return &httpServer{
		db: db,
		bs: newBookStore(db),
	}, nil
}

func (s *httpServer) initRoutes() {
	http.HandleFunc("/order", s.bs.putOrder(http.MethodPost))
	http.HandleFunc("/list", s.bs.listBooks("all", http.MethodGet))
}

func (s *httpServer) start(port uint32) error {
	if port == 0 {
		port = 80
	}
	defer func() {
		err := s.db.Close()
		if err != nil {
			log.Printf("Failed to close the database connections: %s", err)
		}
	}()
	return http.ListenAndServe(fmt.Sprintf(":%d", port), nil)
}

And finally, run the server:

func main() {
	server, err := newHTTPServer()
	if err != nil {
		log.Fatal(err)
	}
	server.initRoutes()
	if err = server.start(8080); err != nil {
		log.Fatal(err)
	}
}

Summary

We covered the basics of implementing an HTTP server using the Go’s standard library in this series. Even though the standard library is very feature rich and more than enough in pretty much all the use cases, but there are a lot of well-known third party packages out there which help you reduce the boilerplate code and implement you web server in a more modular fashion.

I have used GinEcho and Beego in the past and I would say they are all awesome frameworks to use, each optimised for a specific use-case.   You can also find an exhaustive list of other frameworks here. Give them a shot, benchmark them, and see which one you would like to use.

Have fun and enjoy the beauty of Go people. 

Lastly, you can find the code for this tutorial along with a sample Books database in my github.

Leave a Reply

Your email address will not be published. Required fields are marked *