# Book Cover ![Book Cover](/files/kvarDHj5koyRl7wvIpI7) ## Acknowledgment The book cover was designed by [Cedric Mosdell](https://www.linkedin.com/in/cedricmosdell), a gifted and talented Designer. # About the Author Yichen Zhu has been working as a full-stack developer in the software development industry since 2012. He has worked for small- and medium-sized businesses as well as large enterprises, and he has experience in a variety of sectors including (but not limited to) consulting, finance, and telecommunications. Yichen also holds an Honours Bachelor of Science degree in Computer Science from the University of Toronto. When he's not coding, Yichen likes to teach and educate others about programming and software development. ## Links Personal website: LinkedIn: [https://www.linkedin.com/in/yichenzhu1337](https://www.linkedin.com/in/yichenzhu1337/) GitHub: # Links and Resources Official website: Book (*online version*): ~~Book (*physical version*):~~ [~~https://www.amazon.com~~](https://www.amazon.com) Project source code: # Foreword Work in progress. # Preface The fall of 2014 was when I got my first major breakthrough as a software developer; although I was still in university at the time I managed to land a high-paying contract gig at a local digital agency. I was hired to build a large property management web application. Prior to being hired for this contract my work experience mostly consisted of freelancing and a couple of short-term jobs at startups. At the time I was hired by the digital agency I had just over two years of experience under my belt building real-world professional software. Education and experience aside, I had successfully secured this year-long contract job and I was tasked to lead the development of the entire backend system. The technology stack for this project was PHP with Laravel and Node.js with Express in the backend along with jQuery, Backbone.js and AngularJS in the frontend. I spent an entire year working on the backend for this project from beginning to end. I initially started building the entire RESTful API on the PHP and Laravel side, but later transitioned to working more on the Node.js side. By the end of the project, I had created a staggering number of API endpoints: over 150. These endpoints included modules for user authentication, permission management, messaging, payment integration, and many more. ![This is one of the web pages of the property management software I worked on.](/files/3h1UWu2qyJhk5ftNO512) Let's take a step back for a moment and summarize the situation: I was a (relatively) new software developer hired to design and develop a complex backend system for a widely-used property management application. At this point, you may be wondering: "So Yichen, how did everything turn out?" To summarize what that year was like, all I can say is that it was… ***hell.*** I was expected to deliver high-quality software, but instead, I managed to give my boss, my CTO and my client all near-death heart attacks. I was sloppy with my code and I had no idea how to design maintainable software that could scale. Many of my APIs would break unexpectedly, and without proper tests to back them up, I often did lots of manual regression testing to keep my anxiety in check. When it came time to clean up and refactor my code - just kidding, I never cleaned up or refactored code during this period of my career. Don't even ask me about documentation - I didn’t have any. The chaos and hellishness of that year of work taught me a lot of valuable lessons about backend software development, and it is my hope that this book can help you avoid learning things the hard way and making the same mistakes that I did. ![March 15, 2015. That was the day I decided to start writing this book.](/files/tfaeCfb4C7Si4caPYzwY) I often take notes at work and in my day-to-day life so that I can revisit them later on, and this book actually began as a collection of items that I wrote in my [Evernote](https://evernote.com). By the time the property management software was launched I had hundreds of pages of notes written in my notebooks over the course of that year. These notes were actually filled with more questions and uncertainties than anything else. In fact, the initial commit to my GitHub repository was called "things-to-learn" and was full of random notes and questions that I had transferred from my Evernote notebook stacks. Some of these questions included: 1. What is REST (actually)? 2. What is HATEOAS (really)? 3. What are the "correct" ways to create REST APIs? 4. What is the difference between HTTP and REST? 5. How do you structure a REST API in Node.js? 6. Where are all the Node.js architecture books? 7. And many, many more... And because I was curious, I wanted answers... My curiosity and desire for answers didn't go away after the conclusion of my difficult project, and I did a lot of research with the intention of trying to answer all of these questions. I was able to compile an exhaustive list of all the things to consider before creating any RESTful APIs, and that list has evolved into the book you’re reading right now. I wrote this book with the goal of answering the questions listed above and clearing up any doubts for Node.js developers as they conceive, build and maintain REST APIs throughout their careers. # Chapter 1: Introduction In this chapter, we will talk about the rise of REST in the last 15+ years and the problems that came with it. Then, we'll move on to the emergence of Node.js in the last 10+ years and the problems that it came with. Finally, we will explore the (possible) solution(s) to those 2 problems. # The Rise of REST and Distributed Systems > The concept of an API predates even the advent of personal computing, let alone the Web, by a very long time! The principle of a well-documented set of publicly addressable "entry points" that allow an application to interact with another system has been an essential part of software development since the earliest days of utility data processing. However, the advent of distributed systems, and then the web itself, has seen the importance and utility of these same basic concepts increase dramatically. > > *-Martin Bartlett* According to [apievangelist.com](https://github.com/restful-node/restful-nodejs-book/tree/f9c75b4d6e2573d479aaa31c28fd9b9d1630e2c9/srcGitBook/Chapter%201:%20Introduction/apievangelist.com), the first web API was introduced by [*Salesforce*](https://en.wikipedia.org/wiki/Salesforce) on February 7th, 2000. However, that API was not to be made publicly available to the world at the time. Back then, Salesforce launched its web-based sales force automation as an "Internet as a service", and XML APIs became part of salesforce.com from day one. The main role of these APIs was for customers to share data across their different business platforms and applications. ![The salesforce.com website back in 2000.](https://lh5.googleusercontent.com/RyoMISz805tldyg_gOGVlOxZ1HwAM0SsHmn3BbpK9_jbTaJGYi46YwBJ5jknctI0d21x594mnoDgkEIcC823rHvIOyPXwFwTbAKssj3Lk8HwhyeEK4gTbQRB-qaT8G26vURvZMyE) Meanwhile, during the same year, about 9 months later, [*eBay*](https://en.wikipedia.org/wiki/EBay) launched the eBay Application Program Interface (API), along with the eBay Developers Program, making eBay’s APIs openly available for anyone to use. Therefore, both *Salesforce* and *eBay* are often regarded as the pioneering companies who introduced the first ever public APIs to the world. ![The ebay.com website back in 2001.](https://lh4.googleusercontent.com/RxTUINafVNm1U-hh5OO953KQHOnXnvJgB2sfc1BdMgDFzKYIfH1ShevRnTzLArB63MpnqzLdkLJgg44R6HfIvwBVVlmTQOXq9CV92foGUREP5U2KU1_uNl6UgCjW9X16UMcGdiBa) Then on July 16, 2002, *Amazon* launched Amazon.com Web Services allowing developers to incorporate Amazon.com's content and features into their own web sites. Amazon.com Web Services allowed third party sites to search and display products from Amazon.com in an XML format. Since then, companies such as *Flickr*, *Facebook*, and *Twitter* followed suit and have all released their own publicly available web APIs. ![](https://lh4.googleusercontent.com/xnKI2B3YON_U7xkE46WaxIwwgQ9BwsgzETfjXkHWzeTb_foT3KHDgExpxmxOxdT7Jm0EyXuJh3RAalunW3B2e2u-i_LoDnutoA8GoRrmx3LppXt4SzH8COERbaQ6-7uHKgquqjgQ) According to [ProgrammableWeb.com](https://www.programmableweb.com), there are over 25,000 RESTful APIs open to the public. As you can imagine, RESTful Web APIs have become extremely popular and are nowadays the defacto standard for communication across different systems and transportation of data over the web. # Problem #1: Structureless Design, Structureless REST In the last decade or so, we can clearly see that [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) has become so ubiquitous and common that many of the newer generations of developers have never even heard of other architectural styles like SOAP and XML-RPC. The term REST API has become synonymous with terms like HTTP and AJAX. In fact, there is a huge confusion when it comes to understanding the difference between REST and HTTP. We will explore more on this topic in the later chapters of this book. Although many people create RESTful services and API endpoints, there are still **no real formal guidelines and/or principles** when it comes to designing them. ![Even the first answer from google tells us that there are no "official" standards for building REST APIs.](/files/SYFH1Muyw1jyI6SR0qFj) This has become a huge issue because if there does not exist a set of standard guidelines, then everyone would do things a little bit differently. That is exactly how things have turned out in the past 10 - 15 years. Take the simple act of designing a [URI](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier) for a resource for example. If we take a look at even some of the most popular tech companies like *GitHub*, *Slack*, and *Twitter* - you can see that even they kind of just wing it... They all have their own little way of building their own APIs. Take one of GitHub's endpoints for example: `https://api.github.com/repos/octocat/hello-world/code-scanning/alerts`. Notice the `api` in the subdomain and all of the letters in the URI are strictly lowercase. ![https://docs.github.com/en/rest/reference/code-scanning](https://lh3.googleusercontent.com/_5Aqt-86QgDAPv0C1oHlJN8NQnRUcdXnZE2lC8QWekxtRy4lZj93vOi5p8g6lqzzkCY3mF_-6VYvp_plFS1aV7ZkljIHtwZsoG8s_5GRZE_tntDEZNct4hdN1yfCAJjwkYu8Rf_Q) Now take a look at one of *Slack*'s API endpoints: `https://slack.com/api/chat.getPermalink`. The `api` is prefixed just after the top-level domain, it basically acts like a global namespace. What's more is that the `chat.getPermalink` almost let us wonder why they didn't just do something like `/api/chat/permalink` instead. Could it be that internally *Slack* has a chat module and `getPermalink` is an action that's part of it? ![https://api.slack.com/methods/chat.getPermalink](https://lh3.googleusercontent.com/kvzuMRyDZdSjkA1LJ2tdOe8o2T6M0chn_Bk6Wg9HwZkv60-bXn_7O3ujmM1K3D1C1McpT3bTY2sk68om0x52XxxpZF2HfrwriwVqVkSos1MWutBnH-WU9NVsVw_8KfgXcTCvpL_t) And then there's *Twitter*... Take a look at this example: `https://api.twitter.com/1.1/statuses/user_timeline.json`. Everything seems to be fine… until you realize they use underscores ( \_ ) instead of hyphens ( - ). Also, notice that you can specify the output response by adding a ".json" at the very end of the endpoint. ![https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-user\_timeline](https://lh6.googleusercontent.com/CilJ0ER1cfrrF9E1rDm2UMpSNnIXbf47iBmzuXUUTXWbmYWzbHaihGCwNe9vFs3prqXWH-W25NORfUt1QK0VXJ5NLJYnBK7uS_85j1I8pW2nRk9faFlqcsl1GUJKbxoFwePOcXOh) The question would then be, if there are no formal best practices and even the leading tech companies seem to be just winging it, then what set of guidelines can we follow when building our own APIs? # The Emergence of JavaScript and Node.js Although REST APIs have been getting extremely popular over the last 15 years, so are Node.js and JavaScript. At the annual [JSConf EU 2009](https://www.jsconf.eu/2009/speaker/speakers_selected.html) in Berlin, Germany, young 28 year old Ryan Dauh would introduce Node.js to the world, a run-time environment that would enable developers to write JavaScript code on the server. This meant that the language was no longer bound to just the browser, but would enable developers to create command line tools, backend server-side driven web applications, and even programmable robots. Little did we know just how much of an impact **'this'** (Pun intended hehe...) would have on the future generations of developers, companies, and the rest of the world. Then in January 2010, Isaac Z. Schlueter, Rebecca Turner, Kat Marchán, and a couple of other developers released npm, a package manager that would resemble tools like Maven and Ant in the Java community and RubyGems in the Ruby community. This gave JavaScript developers the ability to not only create modules more easily, but the ability to distribute them to the open source world for other developers to use. In only 10 years after the initial inception, we would see the big enterprises like *Netflix*, *Paypal*, and *Uber* all adopt Node.js into their technology stacks. We would also see the emergence of Node.js in the startup world where it would be every startup CEO’s wish list is to hire a "Fullstack JavaScript Developer” who could work in both the frontend and backend. At his point, Node.js has (almost) become a replacement of server-side languages like PHP, Java, C#, and many more. Many young and aspiring developers all flock to JavaScript due to its "learn once, write everywhere" mentality. Almost all coding bootcamps have taken languages like Ruby out of their fullstack curriculum and replaced them with JavaScript and Node.js. Many college and university students no longer want to learn languages like Java and C++ because the job market is no longer saturated with only those types of jobs. Most jobs nowadays are web-based, and even for pure backend developers, one has to eventually touch JavaScript at one point or another. Furthermore, with the rise of frameworks like Electron and React Native, even desktop and mobile have been conquered by JavaScript. Nowadays, it would be entirely possible (and efficient) to write a full backend API, a web frontend, a mobile app, and a desktop application all in JavaScript. > Any application that *can* be written in JavaScript, *will* eventually be written in JavaScript. > > *-Jeff Atwood, Co-founder of Stack Overflow* But with such a fast and evolving language came a massive amount of responsibility with which JavaScript developers needed to face in the community. It came to a point where we were no longer simply bound to using simple jQuery to perform animations in the browser. It has come to a point that JavaScript the language has caught up with the likes of Java and C# in the enterprise, but the developers who originally came from that non-traditional background have been lagging behind in learning about design patterns, software architecture, and traditional object-oriented concepts and methodologies. # Problem #2: Structureless JavaScript, Structureless Node.js Unlike traditional object-oriented oriented languages like Java and C# that contain structure, JavaScript has always been the wild west. Whereas Java and C# would have *Oracle* and *Microsoft* give "best practices" and guidelines to their developers, JavaScript’s version of “best practices” has always been the community itself. > Because JavaScript was malleable, because there were so many web designers, you would see different schools of thought on how to use it… I think that's actually an advantage… we're not telling you there's the one way to write it, here's the one true object-oriented paradigm, here's the only way you should ever make a reusable abstraction. it's not unmixed… it's hard for beginners, and people reinvent certain wheels... If I had done something more rigid, I think the odds are, it pretty much would have failed. > > *-Brendan Eich, Creator of the JavaScript Language* Just like how Brendan Eich envisioned JavaScript to be malleable, Node.js followed suit. The developers (and designers) who came from that originally malleable background began to write code on the server side. The backend and the server - what was once a playground for only the Java, C#, and C programmer became the playground for the average JavaScript developer. What was once a very object-oriented playground became a "not-so-structured jungle". What would have been built at the enterprise or commercial level with a robust language like Java or C#, would now be built entirely in Node.js. This produced many problems as the habits of the old-school web developer who came from that jQuery and pre-2010 JavaScript background were now working in large scale applications written in JavaScript. Because everywhere you go, you would see JavaScript on the server side - you would see messy codebases that were not maintainable, not scalable, and not robust enough to work with on a long term basis. # Behold, the Solution: A Structured Approach So what must we, as JavaScript Developers do to overcome this "structurelessness"? How can we possibly design proper RESTful Web APIs when there are no such formal guidelines or best practices? How can we, as JavaScript developers, in which the majority did not formally study computer science and software development possibly structure our code to be scalable and maintainable? We will first apply the best practices that I have personally learned over the years from designing dozens of RESTful APIs. I will show you what we can take away from companies like *Microsoft* and *Google* who created and followed their own set of guidelines. I will show you some of the best practices I implemented from books such as Mark Masse's [*REST API Design Rulebook* ](https://www.amazon.com/REST-Design-Rulebook-Mark-Masse/dp/1449310508)and Subbu Allamaraju's [*RESTful Web Services Cookbook*](https://www.amazon.com/RESTful-Web-Services-Cookbook-Scalability/dp/0596801688). We will then look at [Domain-Driven Design](https://en.wikipedia.org/wiki/Domain-driven_design) and the patterns that other communities use from a more traditional background. We will learn about layered architectures and how to not reinvent the wheel, but to apply proven software architectural styles to our node.js application. Finally, we'll learn how to apply both the best practices of designing good consumable RESTful web APIs and how to structure good JavaScript code into a simple ***8 Step Recipe*** that you can follow every time you decide to design a RESTful Web API. We will then build a real world RESTful Web API using this approach and discover how this approach can be used and applied to building any real-life applications. We will write code based on this ***8 Step Recipe*** in a methodical manner, write tests to back them up, and document our APIs for long-term use. You will go home every night feeling more fulfilled and less anxious. You will become a better developer and take your career and skills to the next level! # Summary REST and distributed systems have become very popular in the last 10 - 15 years and that JavaScript has become very popular due to the introduction of Node.js. The problem is that there are no REST best practices or guidelines, and JavaScript does not have a good reputation for structuring larger applications. In order to combat against these 2 problems, we must evolve as developers and use modern practices to solve them. # Chapter 2: REST Origins In order to understand REST, we must first go back in time. We will begin by covering a brief history of the internet and the world wide web, which will then enable us to understand how and why REST was created. We will then clear up any confusion and misunderstanding between REST and HTTP. Once that is done, we will then dive into more detail into REST's architecture and its constraints. After that, we will take a look at HTTP, the implementation of REST and exactly how HTTP follows the REST architecture. Then, we will be ready to answer the question of what it really means for an API to be truly "RESTful" and take a look at the Richardson Maturity Model for measuring "RESTfulness". Finally, we'll touch briefly on the subject of pragmatism over dogmatism when it comes to building and designing RESTful APIs. # A Brief History of the Web and the Birth of REST ## History of the Internet The Internet is a network of networks of computers. It was created with the purpose of facilitating communication and sharing of resources between different machines. The birth of the Internet could be traced back during the cold war between the U.S and Soviet Union when the two began competing for the [Space Race](https://en.wikipedia.org/wiki/Space_Race). On October 4, 1957, Sputnik 1 was launched into space by the Soviet Union, the world’s first satellite. Caught off-guard by the Soviet's rapid technological advancements, the U.S. government created ARPA (Advanced Research Projects Agency) which was responsible for the both technological and scientific advancement of the country. Despite the success of ARPA, computers were still not able to communicate with one other. Thus, began the development of a network called [ARPANET (Advanced Research Projects Agency Network)](https://en.wikipedia.org/wiki/ARPANET). During September of 1969, four nodes (stations) located at University of California - Los Angeles, Stanford University, Massachusetts Institute of Technology, and the University of Utah were established, giving birth to the ARPANET and what we now know it as the Internet. In month of December 1974, the word "Internet" would officially be introduced to the world. Throughout the 1970's and 1980's, various different nodes would be added to the internet. A variety of different Internet protocols like TCP/IP, UDP, and FTP were introduced. > An RFC or ***Request For Comments*** is a document that contains proposals for new protocols or systems. Today, RFCs are submitted to and handled by the Internet Society which has a sub-body called the Internet Engineering Task Force (IETF). This sub-body works on the standardization of Internet protocols and systems. An RFC is then deliberated on by experts, revised and then hopefully, eventually adopted as a standard. > > *-*[*RFC*](https://en.wikipedia.org/wiki/Request_for_Comments) ## The Birth of the World Wide Web and REST During the 1990’s, the researchers at the [European Council for Nuclear Research (CERN)](https://en.wikipedia.org/wiki/CERN) felt a need for automated sharing of their findings between their machines. CERN had documents that cross-referenced other documents, these are what we now would call (hyper) links. On November 12, 1990, Tim Berners-Lee introduced his *World Wide Web* project in order to store and retrieve these inter-connected documents. Concepts such as URI, HTML, and HTTP were first introduced at this time. On August 6, 1991, Tim Berners-Lee published the first-ever [website](http://info.cern.ch/hypertext/WWW/TheProject.html) () in the world to the public. After this, the "browser wars" began, browsers such as Netscape Navigator and Internet Explorer competed for the number one spot. Between the years 1991 - 1997, the number of users on the web reached over 50 million. The Internet's infrastructure was not developed enough at the time, having the system lack in both stability and availability. Thus, [Roy Thomas Fielding](https://en.wikipedia.org/wiki/Roy_Fielding) and [Sir Tim Berners-Lee](https://en.wikipedia.org/wiki/Tim_Berners-Lee) began their work on HTTP/1.1 in order to solve this massive scaling issue that the Internet had. The resulting work was a standardized specification that was released in January 1997, [RFCs 2068, 2616, 7230-7235](https://tools.ietf.org/html/rfc2616). By this time, over 70% of all browsers were HTTP/1.1 compliant. After that date, the Internet's problems of scalability began to slowly fade away. ![https://tools.ietf.org/html/rfc2068](/files/T3NuJtoK2OdX4digygEE) In completing and releasing HTTP/1.1, Fielding decided to generalize HTTP into a more abstract architectural style, called REST. > I finished the HTTP standard in 1997, and it wasn't until I had done the work that a colleague of mine, Larry Masinter came and was talking to me about a related subject. and I was telling him about how I've done all this work and that I don't know what to do for my dissertation. He just looked at me, 'well you're the only one who can describe HTTP and why it's there and what is there for, why don't you just do that'. > > *-Roy Fielding, Creator of REST and HTTP 1.1* Fielding would then go on to write his doctoral dissertation titled [**Architectural Styles and the Design of Network-based Software Architectures**](https://www.ics.uci.edu/~fielding/pubs/dissertation/fielding_dissertation.pdf), where he would describe the REST architectural style in detail. # REST vs. HTTP Two of the most confusing and misunderstood concepts used by everyday web developers are HTTP and REST, because there are developers who have worked for decades in this industry and still do not know the differences between those two. We are going to clear that up before moving onto the details of what REST and HTTP actually are. **REST** is an **architectural style guideline** which enforces the correct use of the HTTP transport protocol. **HTTP** is a communications **protocol** that transports messages over a network. The HTTP 1.1 protocol was built to be the ideal protocol to follow the principles and constraints of REST. One way to look at the relationship between HTTP and REST is that **REST is the design**, and **HTTP 1.1 is an implementation** of that design. In reality the two were designed simultaneously. So are REST and HTTP the same? No, they are not. HTTP stands for HyperText Transfer Protocol and is a way to transfer files. This protocol is used to link pages of hypertext on the world-wide-web. However, there are other transfer protocols available like FTP and [Gopher](https://en.wikipedia.org/wiki/Gopher_\(protocol\)). Representational State Transfer, or REST, is a set of constraints that ensure a scalable, fault-tolerant and easily extensible system. One thing that confuses people, is that REST and HTTP seem to be used hand-in-hand. After all, the world-wide-web itself runs on HTTP, and it makes sense. However, there is nothing in the REST constraints that make the usage of HTTP as a mandatory transfer protocol. It's perfectly possible to use other transfer protocols like SMTP, SMS, or even Bluetooth. An API could use any protocol and could still very well be a RESTful API. In practice, most - if not all - RESTful APIs currently use HTTP as a transport layer, since the infrastructure, servers, and client libraries for HTTP are widely available already. # REST - The Abstract Web Architecture REST is a pseudo-acronym for **RE**presentational **S**tate **T**ransfer. The idea here is to have the transfer of data or state be representational of the kinds of messages that you want to use. In his dissertation,[ **Architectural Styles and the Design of Network-based Software Architectures**](https://www.ics.uci.edu/~fielding/pubs/dissertation/fielding_dissertation.pdf), Fielding came up with ***six categories of constraints*** of the web, they are: 1. Client-Server 2. Layered system 3. Cache 4. Stateless 5. Code-on-demand 6. Uniform interface 1. Identification of resources 2. Manipulation of resources through representations 3. Self-descriptive messages 4. Hypermedia as the engine of application state (HATEOAS) Let's go through each of them briefly. ## Client-Server > Separation of concerns is the principle behind the client-server constraints. By separating the user interface concerns from the data storage concerns, we improve the portability of the user interface across multiple platforms and improve scalability by simplifying the server components. Perhaps most significant to the Web, however, is that the separation allows the components to evolve independently, thus supporting the Internet-scale requirement of multiple organizational domains. This essentially means that the client and the server should be separate and can both be deployed independently of each other. ## Layered System > In order to further improve behavior for Internet-scale requirements, we add layered system constraints... \[t]he layered system style allows an architecture to be composed of hierarchical layers by constraining component behavior such that each component cannot "see" beyond the immediate layer with which they are interacting. By restricting knowledge of the system to a single layer, we place a bound on the overall system complexity and promote substrate independence. Layers can be used to encapsulate legacy services and to protect new services from legacy clients, simplifying components by moving infrequently used functionality to a shared intermediary. Intermediaries can also be used to improve system scalability by enabling load balancing of services across multiple networks and processors. This is a handful to digest, but what it essentially means is that the layered architecture allows us to sandwich in different layers in between the client and server. Examples of these layers might be a *cache proxy layer*, an *API gateway layer*, or even a *load balancer*. ## Cache > Cache constraints require that the data within a response to a request be implicitly or explicitly labeled as cacheable or non-cacheable. If a response is cacheable, then a client cache is given the right to reuse that response data for later, equivalent requests. This constraint suggests that when a server sends a response to a client, it should indicate the response can be cached or not. A cache may exist anywhere along the network path between the client and server. For instance, inside an organization’s web server network, within specialized content delivery networks (CDNs), or inside a client itself. ## Stateless > Each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client. This means that the communication between the client and server should not maintain any sort of state. ## Code-On-Demand (Optional) > The final addition to our constraint set for REST comes from the code-on-demand. REST allows client functionality to be extended by downloading and executing code in the form of applets or scripts. This simplifies clients by reducing the number of features required to be pre-implemented. Allowing features to be downloaded after deployment improves system extensibility. The language here is quite old, but if you are a JavaScript developer then this is quite easy to understand. This just means that a server is able to provide executable code to the client. The most ubiquitous example is having including an `app.js` file in a script tag in your `.html` file. ## Uniform Interface > The central feature that distinguishes the REST architectural style from other network-based styles is its emphasis on a uniform interface between components. By applying the software engineering principle of generality to the component interface, the overall system architecture is simplified and the visibility of interactions is improved. Implementations are decoupled from the services they provide, which encourages independent evolvability. The trade-off, though, is that a uniform interface degrades efficiency, since information is transferred in a standardized form rather than one which is specific to an application's needs. The REST interface is designed to be efficient for large-grain hypermedia data transfer, optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction. The interactions between the Web’s components (clients, servers, and network-based intermediaries) all depend on the uniformity of their interfaces. These *Web components* communicate within the uniform interface’s ***four constraints***, which Fielding identified as: 1. Identification of resources 2. Manipulation of resources through representations 3. Self-descriptive messages, and 4. Hypermedia as the engine of application state (HATEOAS) There **4 interface constraints** and can be can summarized in the following subsections. ### Identification of resources Identification of resources refers to the ability for one to address a unique resource via a unique identifier, such as a URI. For example, , uniquely identifies the concept of a specific resource - a specific web page in this case. ### Manipulation of resources through representations Manipulation of resources through representations refers to the way in which the same exact resource can be represented to different clients in different ways. For example, a document might be represented as HTML to a web browser, and as JSON to an automated program. The client is also able to view the resource in different ways and formats without ever changing its identifier. For example, one can call a URI and 'Accept' it as `application/json` or `application/xml`. ### Self-descriptive messages Self-descriptive messages refer to the details within the request/response itself. Relating this to HTTP, this is where the headers comes into play along with its various types of metadata that can be sent with the request/response. ### Hypermedia as the engine of application state (HATEOAS) Probably one of the most confusing concepts that all web developers have struggle to understand. I will try to explain this in very simple terms. Resources themselves should have links in them. Imagine you are on a website and you are clicking on different links that take you to different web pages. **HATEOAS** refers to the links you see on the navigation bar, the home page button, the about page, the profiles of your friends, and the plethora of user-generated content that have their own unique URIs. The difference in this case is that there is no UI, but only representations that contain those links. ![The "Hypermedia APIs" presentation at the 2010 oredev.org conference by Jon Moore.](/files/bP8YKDIPdI4gVOVxyT9x) In the next section, we will look at how HTTP is in fact an implementation of the REST architecture. # HTTP - A Peak at REST's Concrete Implementation HTTP is the protocol that allows for sending documents back and forth on the web. A protocol is a set of rules that allow for messages can be exchanged between two or more systems. A common protocol for example, is POP3, which is used to retrieve email from mail servers. In HTTP, there are two different roles: the client and the server. In general, the client always initiates with a request and the server replies with a response. HTTP is text based; that is, messages are essentially bits of text, although the message body can also contain other media. Text usage makes it easy to monitor an HTTP exchange. The HTTP ***request*** messages made by the client are made of a **verb**, a **header**, and a **body**. You may already be familiar with the verbs: `GET, POST, PUT, DELETE, PATCH`, and so on. The header contains metadata that describes the entire message. The body is where all the contents are stored. For the HTTP ***response*** messages, they are made up of a **status code**, a **header**, and a **body**. Status codes range anywhere from the 100s to the 500s representing success to failures, they are an extra layer of metadata essentially. ![HTTP's Request - Response Cycle.](https://lh6.googleusercontent.com/8UYW_qRLmOhnyN9f76_i4nSX0LsI1GomMvM8g_vfQywS8IybIHMmVIk5VnwAvY1k--LvTlCOoze0Le4QO-NGUglAYNdEOrJFb5XyoGljtjfQ7uOKRkWPMRRszrlFKTTsSgPoMjY1) ## Relating HTTP to REST In the previous section of the book, we went over the 6 categories of constraints of REST. Essentially, any protocol that adheres to those constraints would be considered to having a REST-based architecture. We will go over each of those constraints and see just how much HTTP innately adheres to them. ### **1. Client-server** This is a simple one, we just looked at how HTTP has this request-response type of behavior, therefore the first constraint has been met. ### **2. Layered System** There are many different layers that are in between HTTP. As stated previously: concrete examples are a cache layer, an API gateway layer, and a load balancer. This constraint has now been met too. ### **3. Cache** In HTTP/1.1, we have mechanisms like 'ETag' and 'Cache-Control' which are built into HTTP. This means this constraint has been met also. ### **4. Stateless** HTTP is stateless because each request is executed independently, without any knowledge of the requests that were executed before it, which means once the transaction ends, the connection between the browser and the server is also lost. So this would also mean that the stateless property is met as well. ### **5. Code-On-Demand (Optional)** Nothing to say here, this was optional, so let's leave it at that XD. ### **6. Uniform Interface** And now\... we move on to the most interesting constraint, uniform interface. In order to satisfy this constraint, we must first break it down to its four sub-constraints. #### **6.1. Identification of resources** How do we identify resources in HTTP? We use URIs. #### **6.2. Manipulation of resources through representations** Are we able to manipulative resources? Yes, we have metadata in the header to change to different formats like JSON or XML. #### **6.3. Self-descriptive messages** Same as above, we can use the metadata to manipulative information that can describe the message in different ways. #### **6.4. Hypermedia as the engine of application state (HATEOAS)** Now comes the tricky part, hypermedia. This essentially is where we, the developers, have to put in our effort to make our API truly RESTful. We will look more into this in the next section, but do note that this is the only (sub)constraint that HTTP ***does not provide*** with us out of the box. # What does it mean for an API to be RESTful? The most asked question with the most ambiguous answer(s). "What does it mean for a web API to be RESTful?". In short, as we saw in the previous chapter, because HTTP has fulfilled almost every requirement of the REST constraints, the only thing missing is the HATEOAS requirement. ![Depiction of how HTTP (the protocol) is an implementation of REST (the architectural style).](/files/QBMclPIK5o2sj2ndQUX2) > A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API). From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user’s manipulation of those representations. The transitions may be determined (or limited by) the client’s knowledge of media types and resource communication mechanisms, both of which may be improved on-the-fly (e.g., code-on-demand). \[Failure here implies that out-of-band information is driving interaction instead of hypertext.] > > *-Roy Fielding* To put simply, every response returned from the server to the client must provide appropriate links to the other resources in the application. ## A Not-So-RESTful Case Study: PivotalTracker > I am getting frustrated by the number of people calling any HTTP-based interface a REST API. > > *-Roy Fielding* Let's examine a "REST API". For those we are not familiar with [*PivotalTracker*](https://www.pivotaltracker.com), it's essential a project management tracking tool similar to Trello and Jira. When I first used this API, I thought to myself, where did the word "REST" come from in terms of its title on the documentation page? ![PivotalTracker's web interface.](/files/Cxpphih1Xs7PBt0mSCml) I dug a little deeper to just make sure, and have confirmed that this API is truly by no means RESTful. Let's explore why is not a REST API. ![PivotalTracker's "REST" API v5](/files/ZQ6nPsMGIOa6dp0A0of7) The official documentation can be found here: . First, let's take a look at the `GET /projects/{project_id}` endpoint. The response: ```json { "account_id": 100, "atom_enabled": false, "automatic_planning": true, "bugs_and_chores_are_estimatable": false, "created_at": "2021-01-12T12:00:05Z", "current_iteration_number": 1, "enable_following": true, "enable_incoming_emails": true, "enable_tasks": true, "has_google_domain": false, "id": 1900, "initial_velocity": 10, "iteration_length": 1, "kind": "project", "name": "Executioner", "number_of_done_iterations_to_show": 4, "point_scale": "0,1,2,3", "point_scale_is_custom": false, "project_type": "private", "public": false, "start_time": "2021-01-12T12:00:00Z", "time_zone": { "kind": "time_zone", "olson_name": "America/Los_Angeles", "offset": "-08:00" }, "updated_at": "2021-01-12T12:00:05Z", "velocity_averaged_over": 3, "version": 1, "week_start_day": "Monday" } ``` There is no sign of any links. What about the `POST /projects` endpoint? The response: ```json { "account_id": 100, "atom_enabled": true, "automatic_planning": true, "bugs_and_chores_are_estimatable": false, "created_at": "2021-01-12T12:00:05Z", "current_iteration_number": 15, "description": "Expeditionary Battle Planetoid", "enable_following": true, "enable_incoming_emails": true, "enable_tasks": true, "has_google_domain": false, "id": 99, "initial_velocity": 10, "iteration_length": 1, "kind": "project", "name": "Death Star", "number_of_done_iterations_to_show": 4, "point_scale": "0,1,2,3", "point_scale_is_custom": false, "profile_content": "This is a machine of war such as the universe has never known. It's colossal, the size of a class-four moon. And it possesses firepower unequaled in the history of warfare.", "project_type": "private", "public": false, "start_date": "2020-09-28", "start_time": "2021-01-12T12:00:10Z", "time_zone": { "kind": "time_zone", "olson_name": "America/Los_Angeles", "offset": "-08:00" }, "updated_at": "2021-01-12T12:00:10Z", "velocity_averaged_over": 3, "version": 66, "week_start_day": "Monday" } ``` Again, it gives the same general structure as the GET request previously, there are no links what's so ever. So there you have it, PivotalTracker, although being a very well documented and easy to use API, is indeed ***not RESTful***. ## A Truly RESTful Case Study: Twilio But hold on there... let's not end this section on a bad note, because the truth is that there are plenty of good public APIs that are truly RESTful. One of the more well known APIs (by developers) is the Twilio API. I got to say, Twilio has done a great job documenting all of their APIs and have definitely been able to adhere (almost 100%) to the hypermedia constraints by providing as many links within their responses as possible. ![The official Twilio API Documentation in 2020.](/files/u11c3i09MkYoj5GSVmZR) ![https://www.twilio.com/docs](/files/D8vVHTOWk6Bc04P9d2s3) The official documentation that we are going to be exploring can be found here: Let's begin by first examining the `POST https://api.twilio.com/2010-04-01/Accounts.json` create account endpoint. The response: ```json { "auth_token": "auth_token", "date_created": "Thu, 30 Jul 2015 20:00:00 +0000", "date_updated": "Thu, 30 Jul 2015 20:00:00 +0000", "friendly_name": "friendly_name", "owner_account_sid": "ACX", "sid": "ACX", "status": "active", "subresource_uris": { "available_phone_numbers": "/2010-04-01/Accounts/ACX/AvailablePhoneNumbers.json", "calls": "/2010-04-01/Accounts/ACX/Calls.json", "conferences": "/2010-04-01/Accounts/ACX/Conferences.json", "incoming_phone_numbers": "/2010-04-01/Accounts/ACX/IncomingPhoneNumbers.json", "notifications": "/2010-04-01/Accounts/ACX/Notifications.json", "outgoing_caller_ids": "/2010-04-01/Accounts/ACX/OutgoingCallerIds.json", "recordings": "/2010-04-01/Accounts/ACX/Recordings.json", "transcriptions": "/2010-04-01/Accounts/ACX/Transcriptions.json", "addresses": "/2010-04-01/Accounts/ACX/Addresses.json", "signing_keys": "/2010-04-01/Accounts/ACX/SigningKeys.json", "connect_apps": "/2010-04-01/Accounts/ACX/ConnectApps.json", "sip": "/2010-04-01/Accounts/ACX/SIP.json", "authorized_connect_apps": "/2010-04-01/Accounts/ACX/AuthorizedConnectApps.json", "usage": "/2010-04-01/Accounts/ACX/Usage.json", "keys": "/2010-04-01/Accounts/ACX/Keys.json", "applications": "/2010-04-01/Accounts/ACX/Applications.json", "short_codes": "/2010-04-01/Accounts/ACX/SMS/ShortCodes.json", "queues": "/2010-04-01/Accounts/ACX/Queues.json", "messages": "/2010-04-01/Accounts/ACX/Messages.json", "balance": "/2010-04-01/Accounts/ACX/Balance.json" }, "type": "Full", "uri": "/2010-04-01/Accounts/ACX.json" } ``` As you can see not only does it contain the `"uri"` property of the account that has just been created, it also provides a bunch of `"subresource_uris"`. Next, let's checkout the `GET https://api.twilio.com/2010-04-01/Accounts/ACXXXXX.json` fetch account endpoint. The response: ```json { "auth_token": "auth_token", "date_created": "Thu, 30 Jul 2015 20:00:00 +0000", "date_updated": "Thu, 30 Jul 2015 20:00:00 +0000", "friendly_name": "friendly_name", "owner_account_sid": "ACX", "sid": "ACX", "status": "active", "subresource_uris": { "available_phone_numbers": "/2010-04-01/Accounts/ACX/AvailablePhoneNumbers.json", "calls": "/2010-04-01/Accounts/ACX/Calls.json", "conferences": "/2010-04-01/Accounts/ACX/Conferences.json", "incoming_phone_numbers": "/2010-04-01/Accounts/ACX/IncomingPhoneNumbers.json", "notifications": "/2010-04-01/Accounts/ACX/Notifications.json", "outgoing_caller_ids": "/2010-04-01/Accounts/ACX/OutgoingCallerIds.json", "recordings": "/2010-04-01/Accounts/ACX/Recordings.json", "transcriptions": "/2010-04-01/Accounts/ACX/Transcriptions.json", "addresses": "/2010-04-01/Accounts/ACX/Addresses.json", "signing_keys": "/2010-04-01/Accounts/ACX/SigningKeys.json", "connect_apps": "/2010-04-01/Accounts/ACX/ConnectApps.json", "sip": "/2010-04-01/Accounts/ACX/SIP.json", "authorized_connect_apps": "/2010-04-01/Accounts/ACX/AuthorizedConnectApps.json", "usage": "/2010-04-01/Accounts/ACX/Usage.json", "keys": "/2010-04-01/Accounts/ACX/Keys.json", "applications": "/2010-04-01/Accounts/ACX/Applications.json", "short_codes": "/2010-04-01/Accounts/ACX/SMS/ShortCodes.json", "queues": "/2010-04-01/Accounts/ACX/Queues.json", "messages": "/2010-04-01/Accounts/ACX/Messages.json", "balance": "/2010-04-01/Accounts/ACX/Balance.json" }, "type": "Full", "uri": "/2010-04-01/Accounts/ACX.json" } ``` Again, same general format. This is great because it gives you everything you need. Imagine as a consumer of this API when I make a fetch to a specified account ID, I can more or less access all necessary endpoints of that particular user without ever consulting the documentation. Do I need to know what the usage for that particular account will look like? We have the information right here `"usage": "/2010-04-01/Accounts/ACXX/Usage.json"`. What about all the calls or notifications we want to see for that particular account? They are all under the `"subresource_uris"` property of this response. # Measuring "RESTfulness" with Richardson Maturity Model When it comes to determining whether an API is RESTful or not, it's not an all or nothing matter. We can measure the "RESTfulness" using an already existing model or framework. In 2008, Leonard Richardson proposed a system that was able to classify the maturity of a web service. This model is called the **Richardson Maturity Model** (**RMM**). Leonard’s model describes three levels of service maturity based on a service’s support for URIs, HTTP verbs, and hypermedia. The model suggests that the higher the level of the service, the closer it is to being truly RESTful as each level inherits the one below it. ![https://martinfowler.com/articles/richardsonMaturityModel.html](/files/iWxsVTJZuYXwp8GdI0sB) ## Level Zero Services: Plain Old XML Level zero, also known as Plain Old XML (POX) is the most basic level of maturity. It states that the service will only have a single URI and that only a single HTTP method (usually POST) is used. For example. `POST https://www.website.com/api/users` will be used for both creating a user and retrieving a list of users. This seems quite primitive don't you agree? Luckily, most modern web APIs I've seen are not at level zero. A note here, the service does not have to be in XML, Plain old XML is just the name because JSON was not as popular in the 2000's as opposed to today. ## Level One Services: URIs The next level of service maturity states that many URIs are used as opposed to just one, but with only a single HTTP verb is still being used. For example. `POST https://www.website.com/api/users` will still be used for both creating a user and retrieving a list of users. But we will have other URIs that map to other resources such as this. `POST https://www.website.com/api/users/1` will be used for retrieving a single user with the ID of 1. ## Level Two Services: HTTP verbs Now we are on to level two services, this is where most people are at. This is where multiple URIs are used to address different types of resources. Also, multiple different HTTP verbs (GET, POST, PUT, DELETE) are used on those exposed resource. For example. `GET https://www.website.com/api/users` will be used for retrieving a list of users. `GET https://www.website.com/api/users/1` will be used for retrieving a single user with the ID of 1. `POST https://www.website.com/api/users` will be used for creating a user . ## Level Three Services: Hypermedia Now we have arrived at the final level, the level 3 service, where the web service contains hypermedia. If a service reaches this level, it has achieved all the other levels prior and has all inner resources being linked to one another in a coherent manner. At this level, the web service is considered "truly RESTFul". We have already discussed this topic, so have a look and [revisit the previous section to learn more about what hypermedia is](https://yichenzhu1337.gitbook.io/restful-node/part-2/chapter-2/2_0_5_what_does_restful_mean). # Pragmatic REST vs Dogmatic REST > ***Pragmatism*** is a philosophical tradition that considers words and thought as tools and instruments for prediction, problem solving, and action, and rejects the idea that the function of thought is to describe, represent, or mirror reality. Pragmatists contend that most philosophical topics—such as the nature of knowledge, language, concepts, meaning, belief, and science—are all best viewed in terms of their **practical uses** and successes. > > *-Wikipedia* > ***Dogmatism*** in the broad sense is any **belief held unquestioningly and with undefended certainty**. It may be in the form of an official system of principles or doctrines of a religion, such as Roman Catholicism or Protestantism, as well as the positions of a philosopher or of a philosophical school such as Stoicism. > > *-Wikipedia* Building RESTful APIs is challenging by nature, we've discovered in the prior section what it really means for an API to be truly "RESTful". This is why it is important to consider the realities of pragmatism over dogmatism when it comes to building and designing APIs, because there is no definite right answer when it comes to that. An example of pragmatism is [Zalando's public API guidelines](https://opensource.zalando.com/restful-api-guidelines/#hypermedia) where every API endpoint created must have a "REST maturity level 2", but a "REST maturity level 3" is optional. Notice the fluidity of such guidelines is what makes this so practical in real world uses. ![https://opensource.zalando.com/restful-api-guidelines/#hypermedia](/files/yar26Wb4yBXH3lDNv7Xj) In fact, there are plenty of companies that do not have a truly RESTful API. Examples include APIs from Amazon, Ebay, Instagram, and even Google. Perhaps REST may just not be for you, there plenty of other architectural styles out there. There's GraphQL, gRPC, SOAP, and many more. My suggestion is pick one that suits [your needs](https://apisyouwonthate.com/blog/picking-the-right-api-paradigm). > I'm still not on board with doing this (adding hypermedia links) in each and every project, but there are projects, especially ones that may be generatively created, that can benefit from using hypermedia. I think it's one of those things that the complexity it adds isn't always worth the cost of actually developing it. It depends, again, on your users and the requirements. Ultimately I think pragmatism needs to win here and that most projects that I've dealt with don't need hypermedia, but every once in a while it comes and it's super useful, so understanding it as something you can optionally include in your APIs is important. I just don't want to get stuck in that if it doesn't include hypermedia or links to other parts that are self-describing, that you're purposely not RESTful. > > *-Shawn Wildermuth, Microsoft MVP* With all that being said, we will not be designing and building a fully hypermedia-friendly API in the main parts of the book. However, we will be refactoring our final project and make it hypermedia-friendly with links in a [final bonus chapter](/bonus/refactoring-to-hateoas). # Summary In this chapter, we looked at the history of the web and how REST was introduced. We clarified certain confusions around the topic of knowing the differences between HTTP and REST. We have taken a look at Roy Fielding's dissertation and have broken down its components and how each of them are related to HTTP. We then looked at real world examples and examined what and why certain APIs were RESTful and others not. We then delve into the topic of the Richardson Maturity Model and explored why being RESTful is not a black or white topic, but rather a complex subject in which many levels exist. That being said, knowing what is truly RESTful is useless if put into practice the wrong way, that is why we must be willing to follow pragmatism over dogmatism in our everyday jobs as developers. # Chapter 3: RESTful API Design Guidelines and "Best Practices" In this chapter, we will introduce how to properly design a RESTful web API. We'll be going over the best practices when designing URIs, representations, metadata, versioning, and more. The focus of this chapter is not to enforce the absolute "best practices", but rather show pragmatic examples that the community has learned in the past decade. The set of guidelines and best practices come from notable books such as ***REST API Design Rulebook** by Mark Masse***, *****RESTful Web Services Cookbook** by Subbu Allamaraju***,** and ***The REST API Design Handbook** by George Reese*; 3 books by incredible authors with a wide array of experiences in building APIs. Other resources come from large tech companies such as *Microsoft*, *Google*, and *Zalando* who have publicly shared their own REST API guidelines. And of course, the remaining best practices and guidelines come from me, my own personal experiences and what I learned in the my career. The point of this chapter is to depict what has been considered "best practices" deemed by various different sources. This is so that we may use what others have discovered from their own real world experiences and apply them to our API design. # Theories vs. Principles vs. Guidelines Before moving on to the best practices and guidelines, I want to emphasize a bit on the word *"guideline"*. In my final year of university, I took a class called [*CSCC10H3 Human-Computer Interaction*](https://utsc.calendar.utoronto.ca/course/cscc10h3) taught by [Professor Naureen Nizam](https://www.linkedin.com/in/naureen-nizam-phd-7643011) at the University of Toronto. To be really honest, I didn't really pay much attention in that class, I only took this course in order to fulfill one of my upper-year computer science requirements. There was one lecture where I actually did take notes, and that lecture was about understanding the differences between **theories**, **principles**, and **guidelines**. Although that class that I took many years ago was on usability and interaction design, I will never forget what the differences between the three are. Here are the key differences between theories, principles, and guidelines: **Theories**: High-level widely applicable and predictive frameworks to draw on during design and evaluation. **Principles**: Mid-level strategies or rules to analyze and compare design alternatives. **Guidelines**: Low-level focused advice about good practices and cautions against dangers. > Guidelines provide a shared language to promote consistency among multiple people in terminology usage. It is simply a set of recommendations that are there to create reliability and consistencies among a group of individuals. > > *-Naureen Nizam, Computer Science Professor at the University of Toronto* It is important to realize that guidelines and principles are rule of thumbs, not widely adopted standards used by the masses. Therefore, the purpose of this chapter is to **propose a set of rules and recommendations from the significant trends in API design that we have seen emerge in the past couple of years**, and leverage them when it comes to designing and developing our own RESTful APIs. # URI Design When it comes to designing URIs, it's important keep everything clean and consistent. Before we take a look at the guidelines, we need to first look at the actual structure of the URI. According to [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986), the generic URI syntax is as follows: URI = **Protocol** "://" **Host** ":" **Port** "/" **Path** \[ "?" **Query** ] \[ "#" **Fragment** ] Example: [`https://www.domain.com:8080/user/profiles?lastname=Smith&firstname=John#topheader`](#uri-formatting-and-styling) * **Protocol**; e.g. `http` or `https` * **Host**; e.g. `www.domain.com` * **Port**; e.g. 80 (this will be hidden), 8080, 3000, etc.. * **Path** (consists of fragments); e.g. `/user/profiles` * **Query Parameters (Optional)**; e.g. `?lastname=smith&firstname=john` * **Fragments (Optional)**; e.g. `#topheader` ## URI Formatting and Styling Very similar to following a set of coding conventions, URIs should follow a set of formatting guidelines. Here are 5 guidelines that I have personally followed in the past couple of years when formatting the URIs of my API endpoints. ### Guideline #1: URIs should contain only lower case characters This is because URIs are case-sensitive (except for the protocol and host) according to [RFC 3986](https://tools.ietf.org/html/rfc3986). Therefore, no *camelCase* or other use of capitals. ### Guideline #2: URIs should not contain a trailing slash ("/") This is because every character within a URI counts toward a resource’s unique identity, the following two URIs are not equal. `http://website.com/api/cars/1` !== `http://website.com/api/cars/1/` ### Guideline #3: Use the hyphen ("-") for word separation Many times, people will use underscores ("\_") for word separation, but this has drawbacks since the default behavior of many tools (browsers for example) will indicate an underscore ("\_") when there are clickable hyperlinks. Using only hyphens ("-") to make the visual formatting a lot more clear and evident. ### Guideline #4: Do not include file extensions in URIs Do not use any file extensions like `.json` or `.xml` in your URI. Instead, make use of media types though the `Content-Type` header to determine how to process the body’s content. ### Guideline #5: Avoid white spaces in URIs Do not use blanks spaces as this will confuse whichever client you are using when calling your service endpoint. Whatever client you are using, may it be `cURL`, `node-fetch`, `axios`, or `jQuery Ajax`, not all clients will support the encoding and decoding of the blank white spaces. It is best to just stick to hyphens ("-"). ## URI Path Design When it comes to designing resources and their respective pathings, I generally like to focus on 3 aspects: The **first** is to always think of your endpoints as resources, so anything you return should be a resource representation. There are mainly 2 types of resources, ***a singular resource*** and a ***collection of resources***. The **second** is to make use of ***hierarchies*** properly and to nest them accordingly. The **third** is to ***avoid treating your URIs as actions or functions*** and to only create functional endpoints as a last resort, because as mentioned, you should always think of your URIs or endpoints as resources. Here are some guidelines that I have personally stuck to. ### Guideline #1: Resource paths should be plural Resources should be nouns and pluralized. This is very important as the other guidelines build on top of this concept. Here are some good examples: `https://example.com/api/students` `https://example.com/api/courses` `https://example.com/api/teams` `https://example.com/api/people` `https://example.com/api/comments` The reason why we want to design URIs as resources is because the HTTP method verbs already describe a corresponding CRUD-based style operation. ### Guideline #2: Use identity-based values for fetching singular resources Building on top of the previous guideline, giving a URI the ability to dynamically fetch any single resourced based off of an ID value is quite common. Here are some good examples: `https://example.com/api/students/john` Here, "john" is the identifier. `https://example.com/api/teams/raptors` Here, "raptors" is the identifier. `https://example.com/api/users/id-123456789` Here, "id-123456789" is the identifier. ### Guideline #3: Separate resources via hierarchies This means, separate the many different relations of nouns with the use of '/' in order to create some sort of hierarchical structure. This creates an inheritance pattern of one-to-one, one-to-many, and many-to-many relationships between singular resources and collections of resources. Here is one good example: `https://example.com/api/schools/harvard/students/yichen/courses` As you can see we are using the ID based approach for fetching singular resources while going down the chain. In this case, *schools*, *students*, and *courses*, are the different levels of hierarchy. The IDs are *harvard*, and *yichen*. This is quite easy to understand and can give us many flexibilities. For example, if we wanted to only fetch all the students, then we would intuitively believe this is the URI to call. `https://example.com/api/schools/harvard/students` Here are some other good examples: `https://example.com/api/teams/raptors/players/derozan` `https://example.com/api/videos/v-id-1234567890/comments` `https://example.com/api/companies/facebook/departments/accounting/people/john` ### Guideline #4: Avoid actions and verbs in the URI Do not use actions and/or verbs to describe the URIs, that's what the HTTP method verbs are for. Only create functional endpoints, meaning URIs with actions/verbs if you have to, try to avoid it as much as possible. Do this: 1. POST `https://example.com/api/users` 2. POST `https://example.com/api/students/yichen/courses` 3. PUT `https://example.com/api/videos/v-id-1234567890/comments/c-id-1234567890` Don't do this: 1. POST `https://example.com/api/register-user` 2. POST `https://example.com/api/add-in-list-of-courses-for-student/s-id-1234567890` 3. PUT `https://example.com/api/update-comment-by-id/c-id-1234567890` As you can see from example 1, the HTTP verb POST already implies a user will be created, adding any sort of verbs like "create" or "register" is redundant. Example 2 is a good depiction as well as the `POST` indicates that we are creating a collection of courses for the student with the ID of "yichen". There is no need to put any verbs like "add-courses" or "assign-courses" in the URI. Example 3 uses the HTTP method verb `PUT`, same as `POST`, the example above is quite clear, "update the comment with that particular ID". A rule of thumb is to use an action verb when a URI **does not adhere** to a CRUD operation. Some examples of this include an API endpoint used to log a user in or sending a one time email. ## URI Query Design Now comes the optional part of the URI component, the query strings. A very common question is when should one use a query string versus a URI parameter? The rule of thumb that I always tell myself is that query string parameters should be used the most with optional fields, this means any fields that are not required. The other rule of thumb is that there are really only 4 patterns/scenarios of using query strings, and they are: 1. Pagination 2. Limits and Offsets 3. Filtering 4. Sorting ### Guideline #1: Optional fields Take these 2 endpoints: 1. `https://example.com/api/users/{user-id}` 2. `https://example.com/api/users?user-id={user-id}` What is the difference between them? The difference between 1 and 2 is that the URI for the first endpoint is `https://example.com/api/users/{user-id}` whereas the URI for the second endpoint is `https://example.com/api/users`. Do you see the difference? For 1, `https://example.com/api/users/{user-id}` means that "user-id" is a required dynamic field that is part of this URI. Just because this particular endpoint exists does not mean `https://example.com/api/users` exists as an endpoint. This means although you can get a user with a specified "user-id", there is no endpoint for getting all the users. For 2, the endpoint is `https://example.com/api/users` and the "user-id" query string parameter is optional, so the original endpoint will get a list of users, and if the optional parameter being user-id is present and specified, then it will get a specified user. ### Guideline #2: *Pagination*, *limits*, *filtering*, and *sorting* should all be optional If you read the preface of this book, you will remember I mentioned something about writing over 150+ endpoints at digital agency job. Want to know why I ended up writing over 100 endpoints? It's because I didn't know how to handle situations that involved paginating and filtering. Take this endpoint for example: `https://example.com/api/students?page=2&limit=30&gpa=>3.0&sort=desc` This says I want to get a list of students who's gpas are above a 3.0 sorted from highest to lowest. I only want 30 students per page and I want the 2nd page. What if I didn't set a page number or limit? Then it would look something like this: `https://example.com/api/students?gpa=>3.0&sort=desc` What about if I just wanted all the students? Then it would look something like this: `https://example.com/api/students` Great, no problems here. Now let's see how this would play out if I converted those 4 fields into required (or non-optional) fields as part of the URI. Our URI definition would look something like this. `https://example.com/api/students/page/{page-number}/limit/{limit-amount-per-page}/gpa/{>|=|<(number)}/sort/{asc|desc}` If we wanted to the same results as above, then we would call the endpoint as follows. `https://example.com/api/students/page/2/limit/30/gpa/>3.0/sort/desc` What happens now when I don't want to set a page number or a limit? I can't do this, because then this would be a new URI endpoint. `https://example.com/api/students/gpa/>3.0/sort/desc` What about if I just wanted all the students, so I may do something like this. `https://example.com/api/students` Oh, but wait, we technically never created an endpoint for that. This is what we call a "telescoping characteristic", where because the fields are not optional and are fixed in a certain order, we are unable to dynamically insert values with any level of flexibility. Hence, the most commonly used fields when designing URIs such as *pagination, limits, filtering, and sorting* **should all be optional**. # Method Verbs *GET, POST, PUT, PATCH, DELETE,* these are the most commonly used HTTP verbs. We will first take a look at the idea of **safety** and **idempotency** and then move on to some of the guidelines on when to use each verb and certain things to avoid. ## Understanding Safety and Idempotency > Some methods (for example, HEAD, GET, OPTIONS and TRACE) are defined as safe, which means they are intended only for information retrieval and should not change the state of the server. In other words, they should not have side effects, beyond relatively harmless effects such as logging, caching, the serving of banner advertisements or incrementing a web counter. Making arbitrary GET requests without regard to the context of the application's state should therefore be considered safe. > > *-*[*MDN*](https://developer.mozilla.org/en-US/docs/Glossary/Safe/HTTP) The first important definition we need to introduce is what's called a **safe operation**. A safe operation is one that doesn't modify the resource on the server side. Only the GET method is considered a safe operation, there are other less commonly used methods such as HEAD and TRACE which are also considered to be safe. > In computer science, the term idempotent is used more comprehensively to describe an operation that will produce the same results if executed once or multiple times. This may have a different meaning depending on the context in which it is applied. In the case of methods or subroutine calls with side effects, for instance, it means that the modified state remains the same after the first call. > > -[*Wikipedia*](https://en.wikipedia.org/wiki/Idempotence)\_\_ The second important definition is the **idempotent operation**. An operation is considered to be idempotent when it can be applied multiple times without changing the result. All safe operations are idempotent, but not all idempotent methods are safe. The most confusing ones to understand are PUT and DELETE. In order to understand PUT, let's take an example. Imagine we had an API endpoint *PUT* `https://example.com/api/users/{user-id}` whereby we send in the following request body. ```json { "first_name": "John", "last_name": "Doe", "height": "1.8m" } ``` Do note that the user entity for the response is the same. if we were to hit that endpoint and pass in that request and update the entity, we would get the same result back. Do it a second, third, fourth, fifth time, and we would still get back the exact same response. This is the reason why the PUT method is idempotent. For *DELETE*, it's a little bit trickier to understand. if we were to hit the endpoint DELETE `https://example.com/api/users/{user-id}` the first time and we delete the entity, we might get a response like the following with a 200 status code. ```json { "message": "User with {user-id} has been deleted" } ``` But the second time, we could potentially get a 404. ```json { "message": "User with {user-id} is not found" } ``` What's going on here? Does this mean DELETE is not idempotent? Not necessarily. Idempotence means that making one request has the same end-result as making the same request multiple times, the end-result we are interested in is not the HTTP response or status code. The end-result is the server's state. ![Safety and idempotency chart for GET, POST, PUT, PATCH, DELETE.](https://lh3.googleusercontent.com/qK1iVcYsicq05UcbQMRnJ7kKaoc7puqPPSvex5nxuJl5TK3wktrPWUw8oUPem_ymdDf_Hxcu8boxtf0-KgSXtj7c_uutQqMV9TRRAs6ivoN418Dj_-0s8RTnd_Vaej-UvnVvw8a1) ## GET Guidelines * Only use GET for fetching a singular resource or a collection of resources. * Do not try to pass data (especially sensitive ones) along the URI of GET requests, use a POST or PUT instead. * GET must be both safe and idempotent, do not make anything that would lead to a side effect on the server. ## POST Guidelines * Only use POST for creating a new singular resource, nothing else. * Remember that POST is neither safe nor idempotent, specify if new duplicate resources are created when possible. ## PUT Guidelines * Use PUT to replace an existing resource or a collection of resources. * Use PUT in favor of POST for resource creation if you know what you want the newly created resource identifier to be. * Remember that PUT is idempotent, so there should not be any server-related side effects. ## PATCH Guidelines * Use PATCH when updating only a property of a resource. Unlike PUT where a complete representation is specified in the HTTP's request body, PATCH implements a partial update. ## DELETE Guidelines * Only use DELETE for deleting a single resource or a collection of resources. * Remember that DELETE is idempotent, so there should not be any server-related side effects. ![A summary of the GET, POST, PUT, PATCH, DELETE verbs.](https://lh3.googleusercontent.com/WkeY6J5idzsuhzSVrinEPn6GPiGGjD3qzlBIrXFX8FP7CtGRgQvkkb5HGKxOHb9nys6oEDk4HScrJz8aDfMbe3sLQt3Z9NFAhiL5OZyLtGNJRvctExKUlzlYyWtLVj38acozIa1z) # Status Codes If you've ever consumed and/or created any sort of RESTful service, then you will have encountered various HTTP status codes in the response. Many developers do not have very good knowledge in such a foundational area, probably because there are just way too many that it becomes overwhelming at times. The official standards and list of HTTP codes can be found here ([Standard - IETF RFC 7231](https://tools.ietf.org/html/rfc7231) and [Additional - IETF RFC 6585](https://datatracker.ietf.org/doc/html/rfc7231#section-6)). In this section we are going to go over what the big categories of HTTP status codes are. More importantly, we will cover only the ***most important and commonly used status codes*** and some guidelines on when/how to use them appropriately. ## Brief Overview of the Categories There are essentially 5 categories of HTTP status codes, and each can be summarized in one sentence. * 1xx (Informational): The request was received, continue processing. * 2xx (Successful): The request was successfully received, understood, and accepted. * 3xx (Redirection): Further action needs to be taken in order to complete the request. * 4xx (Client Error): The request from the client cannot be fulfilled due to malformed inputs. * 5xx (Server Error): The server failed to fulfill a valid request from the client. For the full list, head over to . ## 100 – 199 (Informational) Guidelines In the context of creating APIs, this can be ignored. To learn more, head over to . ## 200 - 299 (Successful) Guidelines The 200s status codes generally tell the client that the request was successful. This is usually used with either a POST or PUT request, or any time a user is creating something. There are a ton of rarely used 200 status codes, below is a list of the most used ones. | Code | Name | Guidelines | | ---- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 200 | OK |
  • Should be used to indicate generic success.
  • Unlike the 204 status code, a 200 response should include a response body.
  • Should not be used to communicate any types of errors in the response body.
| | 201 | Created |
  • Must be used to indicate successful resource creation; usually via a POST request.
| | 204 | No Content |
  • Should be used when the response body is intentionally kept empty.
  • If you are using any of the other 200 status codes besides this one, it is good to always provide some sort of non-empty body in order to avoid confusion.
  • The 204 status code is useful when using it with a DELETE request, since when you delete something there is nothing to return.
| ## 300 - 399 (Redirection) Guidelines The 300s are not particularly important in the context of creating RESTful APIs, because their main importance are in the field of SEO. However, below are a couple that are quite important to know and understand. | Code | Name | Guidelines | | ---- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 301 | Moved Permanently |
  • Should be used to relocate resources.
  • The 301 status code indicates that the current URI has been significantly redesigned and a new permanent URI has been assigned to the client’s requested resource. The REST API should specify the new URI in the response’s Location header.
| | 304 | Not Modified |
  • Should be used to cache information.
  • This is commonly used to cache GET requests, if a REST call is made from the client in which it yields the same unmodified result every time, then you should use the 304 status code.
| ## 400 - 499 (Client error) Guidelines The 400s are probably the most important out of all the other categories. These are the ones that the client calling your API has to interact with. Imagine these status codes as a way of providing form validation and signaling to the client what is right or wrong. The 400 category contains the most out of any of the other categories, there are over 30 different 400 status codes. Below are the most commonly used ones that you must know. | Code | Name | Guidelines | | ---- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 400 | Bad Request |
  • May be used to indicate a generic failure from the client.
  • 400 is the default generic client side error status, used when no other 4xx error code is appropriate.
  • Be as specific as possible by using the other 4xx error codes and don't overuse this.
| | 401 | Unauthorized |
  • Must be used when there is a problem with the client’s credentials.
  • A 401 status code indicates that the client tried to access a resource without proper authentication.
| | 403 | Forbidden |
  • Should be used to forbid access regardless of authorization state.
  • A 403 status code indicates that the client’s request lacks sufficient permissions; this is different from a 401 status code because with a 403 status code, the user can provide correct credentials but not have the necessary access levels.
| | 404 | Not Found |
  • Must be used when a client’s URI cannot be mapped to a resource.
  • The 404 status code is the most common, it is a catch-all-exception that gets defaulted when a URI cannot be found.
  • Use the 404 status code for dynamic URIs with entity Ids (ex. ).
| | 409 | Conflict |
  • Should be used to indicate a violation of resource state.
  • The 409 status code can be used in instances to indicate to the client that perhaps a user with the same email cannot be created more than once, or an update to a resource is not possible due to the application's business logic.
| | 422 | Unprocessable Entity |
  • Should be used indicate malformed inputs.
  • This is usual most commonly used when there are form validation errors.
| ## 500 - 599 (Server error) Guidelines Finally, the 500 status codes. Most of the time, a general 500 error is enough to indicate to the client what is going on. I am not going to stress too much about the 500 errors, it is beyond the scope of this book. You can read more about it on MDN: . | Code | Name | Guidelines | | ---- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | 500 | Internal Server Error |
  • Should be used to indicate to the client that they did nothing wrong, but rather there was an error on the server.
| # Representational Design The output of your API is extremely important. The goal of representational design is to create some level of consistency in order to facilitate the client's consumption of your service. This section goes over thoughts and considerations that should be made when designing the output response of your RESTful API. ## Choosing a Format: JSON vs XML JSON has definitely taken over XML in popularity when it comes to building internal RESTful APIs. Most newer systems by companies will most likely only support JSON for the client to consume. It isn't uncommon now for newer developers to have never even worked with XML in their life. Larger companies who provide a public RESTful API will generally support both JSON and XML, Twitter is a great example of this, but that is quickly changing day by day. At the end of the day, supporting JSON and/or XML is a decision that only you can make. My recommendation is that if you are building an internal RESTful API, pick one format and stick with it, it will most likely be JSON. If you are creating a public RESTful API, prioritize JSON over XML. ![Google Trends for JSON and XML between 2016 - 2021.](/files/DXp2O7l9LuVXwGYnuNmr) ## **Using a Consistently Wrapped Response** Consistency is key when building a RESTful API. A consistent output response makes the lives of both the consumer and other developers building the API much easier, because a consistently wrapped response is self-documenting for both the client and server. There are really only 2 types of responses, a success response and an error response. Below are examples of each. ### Success Response I personally use this format for all of my RESTful APIs I build. 1. **Code** – This can be the HTTP status code or some sort of internal status code. 2. **Status** – This can be either "success" or "error" depending on the state of the call. 3. **Message** – This is for both the client consuming the API and the developers creating the API so that it may be use for documenting and debugging purposes. 4. **Data** – That contains the meat and potatoes of the API, usually it is the specified resource or a collection of resources that is being asked from the client. A successful response in a consistently wrapped style may look something like this: ```json { "status": "success", "code": 200, "message": "Here is the user with the specified id.", "data": { "id": 12345, "first_name": "John", "last_name": "Doe", "height": "1.8m" } "errors": null } ``` ### Error Response Here is an example of an error response in a consistently wrapped style output: ```json { "status": "error", "code": 404, "message": "Unable to fetch entitiy with the specified id.", "data": null, "errors": ["The user with the id {user_id} is was not found."] } ``` Whatever style of output response you choose to have, the key thing here is to always keep it consistent. ## **Data Masking, Transformers and DTOs** Security is a big aspect when creating RESTful APIs. A common mistake I've seen from developers is when an output response mirrors the internals of a database schema. This is extremely common due to the increasingly popular powerful ORMs (Object Relational Mapper) that developers have access to today. Instead of exposing every single possible field from the database, it is important to *mask the data* via transformers and DTOs. Do note that we will talk more about DTOs, also known as Data Transfer Objects, in the [Input Request section of chapter 5](/part-2/chapter-5/2-input-request#using-data-transfer-objects). Example, consider the following *User* model created in mongoDB. ```javascript const mongoose = require('mongoose') const Schema = mongoose.Schema const UserSchema = new Schema({ first_name: { type: String, required: true }, last_name: { type: String, required: false }, email: { type: String, required: true, unique: true }, password: { type: String, required: true }, phone_number: { type: String, required: true } }) ``` If we were to simply use the ORM as follows. ```javascript const UserModel = require('models/UserModel') const UsersController = async (req, res) => { const users = await User.find() // this fetches all the users in our database. res.status(200).json([...users]) } ``` We would expose the *password* field in out output. ```json [ { "first_name": "John", "last_name": "Doe", "email": "john@doe.com", "password": "exposed_password_here_even_though_it_may_be_hashed", "phone_number": "123456780" } ... ] ``` ## Hypermedia and Links When following HATEOAS and creating links in our RESTful API, a consistent and intuitive representational form should be used to depict the resource's links and relations. Take for example the following resource: ```json { "user_id": 23, "first_name": "Michael", "lat_name": "Jordan", "_links": { "self": { "rel": "self", "href": "http://nba.com/api/teams/chicago_bulls/players/23" }: "parent": { "rel": "team", "href": "http://nba.com/api/teams/chicago_bulls" }, "fire_player": { "rel": "self", "href": "http://nba.com/api/teams/chicago_bulls/players/23", "method": "DELETE", "description": "Removes this player from the team's roster." } } } ``` A **"\_links"** key-value pair is provided. Each key in **"\_links"** provides some sort operation on the resource. In this case, "**self**", "**parent**", and "**fire\_player**" are operations of the player. The "**self**" represents the resource itself whereas "**rel**" and "**href**" represent the relationship of the link relative to the current resource and the link it goes to. Other optional fields like the HTTP verb "method" and "description" can also be included to provided further self-documentation. Later in this book, we will go deeper in the topic of HATEOAS and linking. Whatever format or types of representation you choose for links in your RESTful API, as long as it is easy to understand and self-explanatory for the client, then everything should be fine. # Metadata Design Metadata is data that further describes an already existing piece of information. In our case, metadata refers to the properties that are inside the headers of our HTTP calls. There are quite a number of HTTP header properties, to see the full list, visit: . HTTP headers can be categorized into 4 broad categories, they are *general headers*, *request headers*, *entity headers*, and *response headers*. We will be interested in mainly the **response headers**. Response headers can also be further broken down into *content headers*, *cache headers,* *media headers*, and many more. We'll take a look at with some of the best practices and guidelines regarding HTTP headers and how they should be applied when building out our RESTful APIs. ## Content Header Guidelines ### Guideline #1: "Content-Type" Is Required The Content-Type header reveals a specified media type in the response. It allows the client to know how to process the body of the response that is being sent over. Here are some common content types you may have seen. * `application/json` * `application/octet-stream` * `image/png` * `audi/mpeg` * `text/html` ### Guideline #2: "Content-Length" Should Be Provided The `Content-Length` header specifies the size of the response's body in bytes. This allows the client to know whether it has read the correct number of bytes from the connection and also allows the a client to make HEAD requests about the size of the body without downloading it completely. ## Cache Header Guidelines ### Guideline #1: "Cache-Control" Should Be Used for Caching The `Cache-Control` header is the most widely used client-side caching mechanism included since HTTP 1.1 (1997). Any application that wants to take advantage of client-side caching should considering including it in their response. Here are some of the most common directives used by `Cache-Control`. 1. **public** - Indicates that the response may be cached by any cache, either in the client's or any intermediary proxies between the client and the server. 2. **private** - Indicates that the response message is intended for a single user and must not be cached by a shared cache. 3. **no-cache** - A cache must not use the response to satisfy a subsequent request without successful re-validation with the origin server. 4. **no-store** - The cache should not store anything about the client request or server response. 5. **must-revalidate** - The cache must verify the status of the stale documents before using it and expired ones should not be used. 6. **max-age = seconds** - Indicates that the client is willing to accept a response whose age is not greater than the specified time in seconds. Example, consider the following: `Cache-Control: max-age=100, private`. This indicates that the cache can be cached for 100 seconds, but only at the requested user's client. The HTTP `Cache-Control` feature is quite an extensive topic, visit to get a complete list of directives and more information on it. ### Guideline #2: "Pragma" and "Expires" Should Still Be Used To Support Caching `Pragma` and `Expires` were the caching mechanisms that were used pre-HTTP 1.1, when HTTP 1.0 was still in use between 1991 and 1997. Although Cache-Control is most widely used today, and both Pragma and Expires are now deprecated since the release of HTTP 1.1 in 1997, it is still a good idea to add them to support backwards compatibility for the client. The `Expires` header takes in a date as its value, that date indicates how long the cache will last in the client. Here is an example. `Expires: Mon, 13 Mar 2017 12:22:00 GMT` The `Pragma` header has only one possible value, and that is simply `no-cache`, which basically tells the client to not cache anything. You would only need to use the `Pragma` header when you do not want to provide client side caching. Here is how it would look like. `Pragma: no-cache` ### Guideline #3: "ETag" Should Be Provided The `ETag` header is a generated hash that identifies a specific version of a resource. It is used for the client to make conditional requests in order to validate the cache from the server. For a complete list of HTTP conditional headers and what they do, see for more information. The two the conditional request headers that are used with ETags are `If-Match` and `If-None-Match`. There are also two different types of ETag hashes. The strong ETag hash indicates that resource content is same for response body and the response headers. The weak ETag hash indicates that the two representations are semantically equivalent. It compares only the response body. The weak ETag hash is prefixed with a "W/" whereas the strong ETag is not. The `ETag` header should always be provided as it gives the client the ability make conditional requests with the `If-Match` and `If-None-Match` headers. Take the following example. Suppose we make a request to our server and we get the following output with cURL as our client. `$ curl -v localhost:3000/ETag-test` ``` * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 3000 (#0) > GET /ETag-test HTTP/1.1 > Host: localhost:3000 > User-Agent: curl/7.64.1 > Accept: */* > < HTTP/1.1 200 OK < X-Powered-By: Express < Cache-Control: max-age=100, private < Content-Type: application/json; charset=utf-8 < Content-Length: 37 < ETag: "25-+vIrmGA7FcSjzeJueoK/J+jWGd4" < Date: Sat, 25 Dec 2021 19:32:13 GMT < Connection: keep-alive < Keep-Alive: timeout=5 < * Connection #0 to host localhost left intact {"message":"Hello world! ETag Test!"}* Closing connection 0 ``` The response we get back has the following headers. `Cache-Control: max-age=100, private` `ETag: "25-+vIrmGA7FcSjzeJueoK/J+jWGd4"` And the following body. ```json { "message": "Hello world! ETag Test!" } ``` This header tells us that we, the client will keep using this response from the cache for the next 100 seconds. Suppose we were to make a call to the server with the `If-None-Match` header, the server will then match the ETag of the resource with the newly available resource. If it doesn't match, server will respond with the new ETag and the new resource which will then be used to replace the old one. If it does match the existing resource, the server will respond with the status code of 304 which means "Not modified", and the client will renew the cache for another 3600 seconds. Let's see it here in action. `$ curl --header 'If-None-Match: "25-+vIrmGA7FcSjzeJueoK/J+jWGd4"' -v localhost:3000/ETag-test` ``` * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 3000 (#0) > GET /ETag-test HTTP/1.1 > Host: localhost:3000 > User-Agent: curl/7.64.1 > Accept: */* > If-None-Match: "25-+vIrmGA7FcSjzeJueoK/J+jWGd4" > < HTTP/1.1 304 Not Modified < X-Powered-By: Express < Cache-Control: max-age=100, private < ETag: "25-+vIrmGA7FcSjzeJueoK/J+jWGd4" < Date: Sat, 25 Dec 2021 19:33:23 GMT < Connection: keep-alive < Keep-Alive: timeout=5 < * Connection #0 to host localhost left intact * Closing connection 0 ``` Notice the `HTTP/1.1 304 Not Modified` in the response, whereas before, we had a `HTTP/1.1 200 Ok`. That's because there was a match with the ETag we sent over. If there was no match, then it would have been a status code of 200 being returned along with a new ETag hash in the response. ### Guideline #4: "Last-Modified" Should Be Provided The `Last-Modified` header serves a very similar purpose to the `ETag` header. It is used to indicate the date and time of when the content was last modified. The client can make a conditional request with the `Last-Modified` date along with conditional headers `If-Modified-Since` or `If-Unmodified-Since`. This will have similar very effect in which if the resource had not been modified, it will return a 304 — Not modified, or if the resource has been modified, it will return the brand new response. It is therefore important to provide the `Last-Modified` header if you want the client to have an alternative caching method so that they may make conditional requests with `If-Modified-Since` or `If-Unmodified-Since`. ### Guideline #5: Add Optional Caching to 3XX and 4XX Responses Although it is most common to cache mainly GET requests that return a 200 response. It can sometimes be beneficial to include caching in 300 or even 400 responses, this would reduce the load on the amount of redirection and errors on a RESTful API. ## Additional Header Guidelines ### Guideline #1: Consider Using An Application-Specific Media Type You may have come across the most commonly used content types such as these: * `application/json` * `application/octet-stream` * `image/png` * `audi/mpeg` * `text/html` But, there are actually many custom media types that available as well. Here are some examples: * `application/vnd.ms-excel` * `application/vnd.lotus-notes` * `text/vnd.sun.j2me.app-descriptor` Notice the "vnd" prefix, that's because these are vendor specific custom media types that are official media types similar to your everyday `application/json` or `text/html`. In fact anyone can register a custom media type with the Internet Assigned Numbers Authority (IANA). Here is a list of officially registered media types: . When building a RESTful API, it is important consider creating a custom application-specific media type. Although this is quite uncommon for most projects, this additional layer of structure can be extremely beneficial for larger APIs if you want better predefined structure and types. ### Guideline #2: Prefix Custom Headers If you've been a web developer for some time, you may have encountered several custom headers provided by different companies or software during your career. By convention, a customer header will always start with an `X-`. This is done so you won't accidentally override any existing predefined HTTP headers. Here are some examples of custom headers by software that you are probably already familiar with. ``` X-Powered-By: Express X-Cache: Miss from cloudfront X-Drupal-Cache: HIT ``` If you are in any situation where you want to add a custom header of your own in a RESTful API, do prefix it with an `X-`. ### Guideline #3: Use The Location Header Specify The URI Of A Newly Created Resource The `Location` header can contain a URI which identifies a resource that can be helpful to the client. When a resource is created, usually via a POST request, it can be very useful to include the new generated resource in the Location header. Below is a good example of a common use case involving the creation of new users in our application. If we make a cURL call with `curl -v localhost:3000/api/v1/users -X POST`. We get the following response. ``` * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 3000 (#0) > POST /api/v1/users HTTP/1.1 > Host: localhost:3000 > User-Agent: curl/7.64.1 > Accept: */* > < HTTP/1.1 200 OK < X-Powered-By: Express < Location: /api/v1/users/181de1eb98aa7ffe6a127232178793d7 < Content-Type: application/json; charset=utf-8 < Content-Length: 79 < ETag: "4f-GFjDt03GZ9sYOxv4+1mObRaXK5I" < Date: Sun, 26 Dec 2021 05:01:47 GMT < Connection: keep-alive < Keep-Alive: timeout=5 < * Connection #0 to host localhost left intact {"message":"New user has been created! Find the link the \"Location\" header!"} * Closing connection 0 ``` Notice the `Location: /api/v1/users/181de1eb98aa7ffe6a127232178793d7`. This is a great way for the client to know where the newly created user can be accessed. We could have also added some sort of link in the body of our response, which if we did, then we would have satisfied the "hypermedia" aspect of our RESTful API. Nonetheless, doing this with a header is still a good and clean convention to follow. # Versioning Strategies Any RESTful API (internal or public-facing) that has been around the block for a long period of time will eventually experience some sort of version change. API versioning essentially boils down to supporting forwards and backwards compatibility as well as supporting multiple different API versions. API versioning was never an issue for me since I had only worked for small to medium sized businesses in the beginning of my career. However, as I progressed further in my career and gained more experience working with older systems, the concept of versioning became more apparent. The concept of being able to label versions so that the client consuming your RESTful API can easily comprehend it is an important feature of your API. In this section we will talk about the ***4 main versioning strategies*** that can be applied when building your RESTful API. There is no real "correct" or "best" approach, but rather pros and cons to each of them. ## URI This is by far the most common and most popular method of versioning. Not only do the majority of developers already do this in their code bases, most of the large tech companies use this method as well. It is extremely easy for both developers developing the API and for clients consuming the API. It is a very visible, obvious, direct, and pragmatic. However, the only problem with this approach is... well... it isn't considered RESTful (by the man himself). ![Roy Fielding's "official" comment on using the "v1" versioning approach.](/files/VdXq7ztO7Fx0JacThwnX) **Examples** `https://api.website.com/v1/resources` `https://www.website.com/api/v1/resources` **Real World Examples** Stripe: PayPal: Twitter: ![https://developer.twitter.com/en/docs/twitter-api/versioning](/files/biGpw6SDssQr7joBZw0Y) ## Hostname The usage of hostnames as versioning is quite similar to using the URI. It's easy to understand from the client's perspective. There's quite a clear separation of different versions via the subdomains. The difficult part might be that one has to use DNS in order to split and manage versions over multiple servers, that could be both good and bad depending on the situation. **Examples** `https://api-v1.website.com/resources` `https://api-v2.website.com/resources` **Real World Example** League of legends: ![https://developer.riotgames.com/docs/lol](/files/fe6hw22eJboPvOmJjhsb) I've also seen a lot of the internal APIs built by companies that I worked at use this approach. Typically, different versions are used within the subdomains as well as a way to split up development, staging, and production environments. ## Query Parameter Using the optional query parameter is yet another popular method of versioning. It technically does follow REST since it is an optional parameter, which means it does not interfere with any concrete resources. However, I find that if not specified explicitly in the documentation, it's quite easy for developers to forget or to even know that there is a custom parameter required for a different version. **Examples** `https://www.website.com/api/resources?version=1.0` `https://www.website.com/api/resources?v=10-01-2049` **Real World Example** Google: ![https://developers.google.com/gdata/docs/developers-guide?csw=1#updating-a-raw-http-client](/files/RfXAjy3Kzp0hWwnq8X6a) ## Header An API that expects a consumer to communicate its version preference through HTTP headers normally defines a custom HTTP header. If you take this approach, your custom header should be "x-SOMETHING". I actually quite like this method, but the only problem with it is that some developers (mainly junior ones) can get confused if they do not know about headers very well. **Example** Making a GET request with a custom `x-version` header. ``` GET /resources HTTP/1.1 Host: api.website.com x-version: 1.0 ``` **Real World Example** Microsoft Azure: ![https://docs.microsoft.com/en-us/rest/api/storageservices/Versioning-for-the-Azure-Storage-Services?redirectedfrom=MSDN#specifying-service-versions-in-requests](/files/n0wHDZhmgDGeSvY2KcFB) # Security Considerations This is not a book on computer security, security is a very broad and complicated topic. Computer security is so broad that it can be separated into each of its specialized topic. Therefore, the scope in which we will cover security is in the context of the web. More specifically, we will be focusing on the application level of our RESTful web API. ![](https://lh6.googleusercontent.com/xZ6BJests8LT5Zy0kaBl47SILpzsWPMN4hHeihfd1FX8qv56TZuHAoibWAMOIVmGmmBfOx5Ujio9L5_qowR_pwK8suaR3NbDZDa_xXuII0LtR0-m0vYIqAbiPfV5-Kom6SqKWFu5) When I was building one of my first large scale commercial applications, I really wished someone would have told me, "these are the 80% of security vulnerabilities that you need to protect against, here's a list of them". This would have saved me a bunch of stress and anxiety, and would have definitely made me stop asking the question, "Am I doing this correctly? And will I get hacked?". In this section, that's exactly what I will be doing. I will be pointing out the **top 10 security considerations** **and guidelines** when building your RESTful web API and what you should do to go about protecting against them. The point is to not go in detail about every single possible security concern, but to rather give a general idea as to what you might need to look more into beyond the scope of this book. The main kinds of attacks we are interested in the context of the web are (No)SQL injection, Man-in-the-middle attack (MITM), Cross Site Scripting (XSS), Cross-site request forgery (CSRF), and Denial-of-service attacks (DDoS). ## 1. Ensure Proper Access Controls (Authentication & Authorization) **Authentication** determines whether someone accessing an API is really who they say they are, whereas **authorization** determines the level of access and the permissions a particular user has. Whatever method of authentication you decide to use, just make sure that from the end user's point of view that proper user authentication and privileges are granted accordingly. Although this topic is vast and broad, the general rule of thumb to keep in mind here is to use common sense and to give this security consideration its due diligence. Take the following example. ```javascript // allows the request to go through if the user is authenticated const isAuthenticated = (req, res, next) => { // if the user is authenticated, then next() // if not then access is denied } app.delete('/api/users/:id', isAuthenticated, async (req, res) => { // delete the user with id }) ``` It may seem as if the `isAuthenticated` middleware is doing its job, but the developer who wrote this has thought about the aspect of only allowing authenticated users to delete their accounts. The problem though is that **any authenticated user may delete any account they choose**. The solution is to add another middleware after the `isAuthenticated` middleware, let's call it the`hasPermissionsToDelete` middleware. ```javascript // allows the request to go through if the user is authenticated const isAuthenticated = (req, res, next) => { // if the user is authenticated, then next() // if not then access is denied } // allows the request to go through if and only if the current auth user // is equal to the request user id const hasPermissionsToDelete = (req, res, next) => { const theUserWeWantToDeleteId = UserModel.findById(req.params.id) const theCurrentAuthUserId = req.user.id if (theUserWeWantToDeleteId === theCurrentAuthUserId) { // we delete the user } else { // access is denied, you can't delete some one else's user } } app.delete( '/api/users/:id', isAuthenticated, hasPermissionsToDelete, async (req, res) => { // delete the user with :id } ) ``` ## 2. SQL Injection SQL Injection is extremely common and should be a top consideration when building any RESTful backend systems. Here is a typical example of SQL injection, where the hacker takes advantage of our sloppy insecure select statement and tries to delete our users table. ```javascript const mysql = require('mysql') const pool = mysql.createPool({ multipleStatements: true // ... }) // Hacker's malicious input const post_id = "'; DROP TABLE users; --" // Our SQL query statement here is vulnerable, can you see why? const statement = `SELECT * FROM posts WHERE id='${post_id}'` pool.getConnection(function (err, connection) { if (err) throw err connection.query(statement, function (error, results, fields) { if (error) throw error }) }) ``` If look more closely and replace the `post_id` variable, you will see the following statement being executed. ```sql SELECT * FROM posts WHERE id=''; DROP TABLE users; --' ``` There are plenty of ways to defend against this type of attack, such as using prepared statements, escaping the input, or black listing input. You can find additional details in this awesome [SQL Injection Prevention Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html) which gives you a bunch of options for defending against SQL injection attacks. For our example, we can do a simple raw check on the `post_id` variable, like this. ```javascript const mysql = require('mysql') const pool = mysql.createPool({ multipleStatements: true ///... }) // Hacker's malicious input const post_id = "'; DROP TABLE users; --" // Our SQL query statement here is vulnerable, can you see why? let statement = `SELECT * FROM posts WHERE id='${post_id}'` // Our simple and raw protection against SQL injection if (!Number.isInteger(post_id)) { throw Error('Post ID is not the correct type!') } pool.getConnection(function (err, connection) { if (err) throw err connection.query(statement, function (error, results, fields) { if (error) throw error }) }) ``` *Note*: there's also NoSQL Injection. Since the NoSQL movement has been quite new in the recent years, there are actually less protections against NoSQL injection versus just plain old SQL injection attacks. However, the most tiresome aspect of NoSQL databases and protecting against NoSQL injection attacks is that because there isn't a common language among NoSQL databases, protecting and testing requires knowledge specific to the database, syntax, data model, and their APIs. In fact, according to , there are over more than 225 different types of NoSQL databases on the market. With that being said, it is therefore important to also consider protecting against NoSQL injection attacks depending on the database you choose to use in your application. ## 3. Encrypt Sensitive Information in the Database Sensitive information that you do not want malicious hackers to know about should be at the very least encrypted before they hit your database. Sensitive information can be anything from passwords, credit card numbers, and social security numbers. Whatever you choose to store in your database, it's always best to have some level of encryption of sensitive information. Here is a great example of how most people store hashed passwords, we'll be using the built-in `crypto` library in node as well. This may not be the best way of doing things, but it is a way. ```javascript const crypto = require('crypto') /* Helpers */ function generateSalt() { return crypto.randomBytes(16).toString('base64') } function generateHash(password, salt) { var hash = crypto.createHmac('sha512', salt) hash.update(password) return hash.digest('base64') } /** These are our auth routes **/ app.post('/register', async (req, res, next) => { const username = req.body.username const password = req.body.password const salt = generateSalt() const hashedPassword = generateHash(password, salt) // When the user registers, we store both the randomly // generated salt as well as the hashed password await UserModel.update( { _id: username }, { _id: username, password: hashedPassword, salt }, { upsert: true } ) }) app.post('/login', async (req, res, next) => { const username = req.body.username const password = req.body.password // Retrieve user from the database const user = await UserModel.findOne({ _id: username }) // When the user tries to login, we compare the generated hash // which is produced from the user's input password along with // the generated salt if (user.password === generateHash(password, user.salt)) { // correct credentials, logs the user in... } else { // incorrect credentials, does not log the user in... } }) ``` ## 4. Encryption Sensitive Data in the URI On the topic of encryption, in order to prevent [man-in-the-middle attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) to our API, it's important to encrypt sensitive data in the URI. Better yet, to not send any sensitive data in the URI in the first place. If you do need to leave in sensitive data in the URI, make sure to encrypt it, because [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) will not prevent hackers from sniffing and intercepting HTTP, or rather HTTPS calls, if the data is in the URI. Recall that HTTPS will only protect and encrypt information in the header, not the URI. Below is an example of what **NOT** to do. `GET https://www.website.com/api/social-security-number/078-05-1120` Here is a way better version of the above example. `GET https://www.website.com/api/social-security-number/$6$FP1fYsh4CiH_rest_of_the_hash` ## 5. Prevent Cross-Site Request Forgery with CSRF Tokens CSRF (Cross Site Request Forgery ) requires quite an extensive amount of explanation and a little beyond the scope of this book. To learn more in depth, below is the best article I have ever read on the topic of CSRF written by [auth0.com](https://auth0.com). This is especially important if you are using any sort of session based authentication in your application. {% embed url="" %} Best article on how CSRF works and how to protect against it. {% endembed %} ## 6. Protect Your Cookies! ![These columns here in the cookies tab are not just for sure you know XD.](/files/XiLi9bC4NHTLIOyo8fzE) I've always been skeptical about the security of using cookies, but the main big 3 concepts about cookies in the context web security can be broken down to these 3 flags, ***HttpOnly***, ***Secure***, and ***SameSite***. Below is a brief summary of these 3 flags and what their purposes are. My suggestion is to look more in depth into these 3 security mechanisms and to play around to see how they work, because simply reading about it is not enough to truly understand them. **HttpOnly flag against XSS** The `HttpOnly` flag makes sure that our cookie cannot be read by arbitrary JavaScript code. If you were to go into the chrome console to any website and type into the console `document.cookie`, you will notice that the ones with a check mark in the secure column will not show up. This is to protect against the ability for any remote code from executing, you can see that this is especially useful to protect cookies such as session ids. **Secure flag against MITM** The `Secure` flag is quite straight forward. Essentially, it instructs the browser to re-attach cookies with HTTPS requests only (and not plain HTTP ones). This can be used to any potential mixed-content vulnerability that would leak a cookie value. **SameSite flag against CSRF** Cookies can sent to third parties with cross-origin requests. This can be abused by CSRF attacks. One way to avoid specific cookies to be sent with cross-origin requests is to set a special flag called SameSite. This will make it so that cookies can only be forwarded from the same domain origin. ## 7. Take Advantage of Cross Origin Resource Sharing Have you ever tried calling your API from the client side and you got a Cross Origin Resource Sharing (CORS) issue? ![Every Web Developer's favorite error when making an AJAX call.](/files/cQUCQFDvPcR70ZON1eMt) I'm talking about doing this. ```javascript app.get('/resource', (req, res) => { // Don't do this if you don't want every other // application in the world to call you API res.set('Access-Control-Allow-Origin', '*') res.json({ message: 'Hey there!' }) }) ``` Instead, a more "white-listing" approach is what you should be doing. ```javascript app.get('/resource', (req, res) => { // This is better! res.set('Access-Control-Allow-Origin', 'https://my-website-client.com') res.json({ message: 'Hey there!' }) }) ``` For better management of these headers, use the [cors](https://www.npmjs.com/package/cors) package. Another key point to remember is that CORS is a browser mechanism that prevents one domain origin from access another domain origin's response output. It does not actually prevent anything happening in the background when an API is called. So if you an API endpoint that is not safe, such as deleting a user in the database, don't expect CORS to help you prevent that. ## 8. Sanitizing All Input Data It is important to always remember to never trust the client's input. Always be on the look out to sanitize all input data given by the consumer of our API. Escaping and filtering input are the most common ways to defend against malicious user inputs. One of the most popular libraries in the node community is the [validator package](https://www.npmjs.com/package/validator). ![https://www.npmjs.com/package/validator](/files/XiLi9bC4NHTLIOyo8fzE) Although general and raw techniques of sanitizing data are good, I recommend using a package such as this one to further the defensive mechanism of your RESTful API, especially when it comes to escaping data. ## 9. Use the Content Security Policy (CSP) Header To Prevent XSS We just talked about how to never trust input, but you also can never trust the output. Imagine you have a list of comments on your website that you want to display to the public, those comments are of course made by the users themselves. You can decide to both sanitize the input and also escape the output when rendering the data to the screen. There is actually a more sophisticated way of doing this "output escape" from your API, and that is using the CSP header. [You can read more about it on MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP). Here's an example, when we request a comment by an id and we want to output it as a JSON response for which the client wants to consume, we send the following header to the client `Content-Security-Policy: script-src 'self'`. This header tells the browser to protect any malicious JavaScript from executing in the browser. ```javascript app.get('/comments/:id', (req, res) => { res.setHeader('Content-Security-Policy', "script-src 'self'") const comment = CommentModel.findOne({ id: req.params.id }) res.json({ data: comment.body }) }) ``` For more in depth look at how CSP works, I recommend this [YouTube video: Content Security Policy by Kyle Robinson Young](https://www.youtube.com/watch?v=JbfNWg6JS4U). ## 10. Prevent Denial-of-service Attacks by Rate Limiting Requests If you have a very popular and public application, you might want to think about protecting against DDoS attacks in order to maintain high availability of your API. The most common way to do so (at least for developers) is to add a rate limiter in your RESTful API. Ideally, we prevent [Denial-of-service attacks (DDoS)](https://en.wikipedia.org/wiki/Denial-of-service_attack) from outside the application level, so the attacker can never even reach our application. However, since this is not a book on systems administration and server configuration management, we'll just have to put a rate limiter directly in the code of our application. For this I recommend the [express-rate-limit](https://www.npmjs.com/package/express-rate-limit) package. Here is a simple sample code on how it works. ```javascript const rateLimit = require('express-rate-limit') // This rate limiter will only allow 100 requests per 1 minute from // the same IP address. const apiLimiter = rateLimit({ windowMs: 1 * 60 * 1000, // 1 minute intervals max: 100 }) // Only apply to requests that begin with /api/ app.use('/api/', apiLimiter) ``` # Documentation There are many specifications as well as tools that follow those specifications for documenting APIs. The 2 most common [API Description Languages](https://api-university.com/blog/what-are-api-description-languages) are [OpenAPI ](https://swagger.io/specification)(previously known as the Swagger Specifications) and [RAML ](https://raml.org/developers/raml-100-tutorial)(RESTful API Modeling Language). The OpenAPI specification created by [Swagger](https://swagger.io) is the most common and most popular among the API Description Languages. We will be using the OpenAPI specification in our application to document our RESTful API later in this book. Both the OpenAPI and RAML official specifications and documentations are available online. Take some time to read through some of it to decide which would suit your documentation style. {% embed url="" %} {% endembed %} {% embed url="" %} {% endembed %} # Case Study: GitHub Now that you have seen some of the guidelines proposed in this book. It is now time to observe how these guidelines have been implemented and followed in the real world. What better organization to observe over the years than every developer's favorite website, [GitHub](https://github.com). Both the documentation and the company have gone through enormous amounts of changes in the last decade. ![GitHub first release its public API on March 12, 2008.](/files/ugkjrgUYgctbxF0fYinl) The GitHub documentation website went from this funky looking thing... ![https://web.archive.org/web/20090325014302/http://develop.github.com](/files/pRYeGQJbHZOPbhoG0vAW) To this somewhat less funky looking thing... ![https://web.archive.org/web/20110711095759/http://develop.github.com](/files/gv2kkdXTwgfiBnyujwu3) To now to this. Yay! Much better don't you think? ![The official GitHub Docs website today: https://docs.github.com](/files/dlNUMIjWCMSOPdSNnCAs) Fun fact: V1 and V2 were both deprecated on March 28, 2012. V3 has been in effect ever since. In this section, we'll be exploring 3 key changes that GitHub has made since versions 1 and 2 all the way till now in V3, and how they were able to better their APIs by applying some of the guidelines written in this book. ## Application-Specific Media Type **Before** ![https://web.archive.org/web/20090421122659/http://develop.github.com/p/general.html](/files/qr0NbUUGO6S87EpOtPfV) If you dig around the old GitHub documentation, you will see that GitHub did not support any custom vendor media types in their API. They did however, support XML and JSON from their responses. This was quite primitive, but it did work. **After** ![https://docs.github.com/en/rest/overview/media-types](/files/rpNZLrZnqTmRx7im2up0) Nowadays, GitHub's public API is quite popular and is used by the masses. GitHub has added their own custom media type as `application/vnd.github.v3+json`. Not only that, GitHub has also dropped support for XML in favor of JSON. ![](/files/hgmiRTAIxOmFZFcc96Sr) ## URI Actions **Before** ![https://web.archive.org/web/20110711095603/http://develop.github.com/p/commits.html](/files/AWlNRquXFYCfKwJ9qLz3) Back in the day, GitHub would create their endpoints with an entity type like `http://github.com/api/commits` or `http://github.com/api/repos`, but would make the mistake of tacking on an extra verb like `list`, `show`, or even `create` in order to describe what action those endpoints were trying to perform. Here are some examples of what GitHub did back in V2 of their `commit` endpoints. ``` GET http://github.com/api/v2/json/commits/list/:user_id/:repository/:branch GET http://github.com/api/v2/json/commits/list/:user_id/:repository/:branch/*path GET http://github.com/api/v2/json/commits/show/:user_id/:repository/:sha ``` If you recall guideline #4 of the URI path design section of the book: "Avoid Actions/Verbs in the URI". `Do not use actions and/or verbs to describe the URIs, that's what the HTTP method verbs are for. Only create functional endpoints, meaning URIs with actions/verbs if you have to, try to avoid it as much as possible.` **After** ![https://docs.github.com/en/rest/reference/repos#commits](/files/YM6dvAUJGIbduhkeGPtJ) In V3 of the GitHub API, they followed the guideline for their `commit` API, and have since removed their action-oriented verbs in the URI and relied more on the implicitness of their HTTP verbs themselves. ![https://docs.github.com/en/rest/reference/repos#create-a-commit-comment](/files/zwcsEkKaNS0vkUsKDQ1c) ![https://docs.github.com/en/rest/reference/projects#get-a-project](/files/mCfinE5UDRUMl4TUfwJk) ![https://docs.github.com/en/rest/reference/orgs#update-an-organization](/files/l7gQDugLjcrSUEtpCsjV) Notice the lack of action verbs in the URI in the `commit comment`. `project`, and `organization` endpoints, and the proper usage of the GET, POST, and PATCH verbs. ## Pluralization of Nouns in the URI **Before** ![https://web.archive.org/web/20090325003132/http://develop.github.com/p/users.html](/files/x3KsnnFLBNiBH9UGr9KF) Before the release of V2 and V3, GitHub had a very inconsistent naming scheme, mixing in both namespaces like `user` vs `users` or `project` vs `projects` in their URIs. Although this might have been just a semantic issue, and not really a big deal, one can potentially see the issue when a person gets `api/user` confused with an endpoint for getting the current auth user or vs getting a list of users. Recall guideline #1 from the URI path design section: "Resource paths should be Plural". `Resources should be nouns and pluralized. This is very important as the other guidelines build on top of this concept.` **After** Perhaps I am nitpicking here, but it seems like GitHub did indeed change every single one of their endpoints in V3 to pluralized nouns. The most evident being the `user` in V2 vs `users` in V3 shown above and below. ![https://docs.github.com/en/rest/reference/users#get-a-user](/files/0ENMfYqttXdcMIbwcWgI) # Summary In this chapter, we focused primarily on what some of the "best practices" are when designing a RESTful API. We explored what it meant to properly design the URI portion by following a set of styling, pathing, and optional query guidelines. We also took a look at the different method verbs such as GET, POST, DELETE; the characteristics of what each of them contain and when to use them. On top of that, we took a peak into the most used response codes and when to properly use them in order effectively communicate with our clients and consumers. We then moved on to designing the response of our service and how we should implement consistency and safety in our representational output to the client. Then, we moved on to metadata and looked at the some of the most common used headers related to content, caching, and media types. Next, we saw how to version our API and what different types of strategies to employ to ensure that our RESTful service can withstand the test of time. Moving forward, we dove deep into the security aspect of our API and what are the most common security vulnerabilities we should be aware of. After that, we saw different types of documentation styles and how even a simple concept like documentation has official specifications that we can follow. Lastly, we put together all that we've learned in this chapter and examined how GitHub as a company has applied these very simple guidelines into their API over years. # Chapter 4: Structured JavaScript Architecture The 2010's have been an extraordinary decade for JavaScript. What was once a "toy language" used for interactivity on small websites, is now the most popular and inescapable programming language of the world. Even at the beginning of the last decade, most large websites would sprinkle on some JavaScript and jQuery here and there. Adding JavaScript was merely an afterthought to large server-side rendered applications. If you were to have told me in 2010 that an entire application, or better yet, an entire business could be built on just JavaScript, I would have said, "use something more mature and proven like Java!". But this is the world that we live in now, a world where 100% of the stack could be built just on JavaScript and JavaScript alone. This new and improved way of writing applications has introduced many problems which were previously unknown in the JavaScript community. There are now hundreds of large JavaScript-based applications that are unmaintainable, unscalable, and unfathomably horrific to work with. The good news is that we, as JavaScript developers do not have to reinvent the wheel, but rather take what has already worked in other communities and apply them to ours. In this chapter, we will explore some of the problems that JavaScript developers face when building large applications. We will see why building a monolithic application does not scale a project properly in the long term and why the community has had such a hard time structuring larger applications. We then take a page from **domain-driven design (DDD)** and learn about the **layered architecture**. We'll dive into the classic MVC pattern and then deeper into how we can extend MVC into repositories and services. We will also take a look at the microservices architecture and why it has become the poster child of the past decade and why buzzwords like *services oriented architecture* and *distributed systems* are not what we, as JavaScript Developers, should focus on. # The Monstrous Monolith and Its Downfall > In software engineering, a monolithic application describes a single-tiered software application in which the user interface and data access code are combined into a single program from a single platform. > > *-*[*Wikipedia*](https://en.wikipedia.org/wiki/Monolithic_application) Below is a Monolithic application in which most of the application is neither horizontally distributed nor vertically layered. There is all but one application and one layer. There are few levels of abstractions, and most of the time, it could just be one single file. ![https://www.youtube.com/watch?v=\_07NtoK-Kns](https://lh3.googleusercontent.com/8VH82iYoCwJl7EeHsNVINB8tE7tLncWTWLt0BT-9YanpG6g4Vrzz3biQaa5QN72ICyS3FxIMHPPiLl0vkSmz6cWEuK58wTEBt5QDuRaLDS1_QlGlPXCiWSDJpLn8d8zpwwrNbdVg) The JavaScript community is notorious for not having structure in their code and not following any design patterns or principles. We are known for getting things out the door quickly and for not having to think too much about scaling to more than just a 1 or 2 man team. The question is, why? ## The JavaScript Community and the Hackathon Mentality Unlike the developers in the Java community or even the Ruby community, we as JavaScript developers tend to want to get things done very quickly. And due to the easiness and the nature of being able to spin up a Node.js application with express in a matter of seconds, most projects start off as a monolithic application. However, as a project matures, so does the complexity of the code. From my experience, these are the three most common things that happen to most node.js projects: 1. Everything is put into just one file, usually named `app.js`. 2. No abstraction layers and encapsulation of any sorts; I mean we just have one file called `app.js` anyway, so why bother. 3. People don’t break things down, they magically forget to use built-in features like express' middleware feature and forget to create re-usable helper functions. ## The Lack of Central Authority When was the last time someone asked, "Which website do you use to learn JavaScript?", and your answer being, "Google, duh!". The sad fact is that there is no real central authority for learning JavaScript and Node.js. Most of the material out there are by third party developers and open source contributors. Rarely do we see a set of standards that are reinforced by one central source, but rather, a set of guidelines and standards become adopted through individuals in the community over time. Take Microsoft for example, they have official documentation on the C# language. ![https://docs.microsoft.com/en-us/dotnet/csharp](https://lh6.googleusercontent.com/PWr_bTwGPEnSzJw56aVsq0OGzRKNsHK9A0D3CiErWxkO6rxOs5DwqAyDVOlu4lLsm8kD7L6P0D26tT_aWXtTpObsgMuQjec7qDhNoB9ka4xw0X2RVzdbVbr5XuLOEqUGyZSTNJ3y) They also have official guides on how to structure code in the .NET ecosystem. ![https://bit.ly/3n7a3FS](https://lh6.googleusercontent.com/aoj9VbLgk32uTJjvriW7EOw21GizMyhto-a61uIVFAAYF5wwEF-t2mrxGbb0b7b_9GaEA4PbW8pw9mg-R41_-puw2S7YRF9OApW6YJmQ_tFAn6slqdiRQ3im45IFkiPql2g8LuQY) Not only that, they have a bunch of certifications and courses you can take as well. ![https://docs.microsoft.com/en-us/learn/certifications](https://lh6.googleusercontent.com/itiCNbBKp-ae13K6rJ7kXRSf3ANSva9PIkXlYK8lQZ_F49aFyw4jI3S3JXWNxjpZfQ1UteFt70ti9FPPHfr8FmTBwA4RDpqflrklf8XETHIH1q1Lb--MyZy5PlEQdFX6j7qaNX2L) The same goes for Oracle when it comes to Java. Oracle provides official training for their products through their certification and training courses, they are all under the umbrella of Oracle University. They have the ever so popular Java certifications like the OCA (Oracle Certified Associate) and OCP (Oracle Certified Professional). They also provide very well documented tutorials on learning Java if you wish to learn it from an official source. ![https://education.oracle.com](https://lh4.googleusercontent.com/5M0uqLm84fofJ-yoHzbhOzseE3XiLlJcHuUqA1cn32mAz2vLw1xxzB_8g1EqEtlcb4cbMTcRfa-4WSjrHwsGvaLIJn3vlD0FvHBzCccimaf_U36sSHNGyYqbBFQnBkucoKRSfCZ_) ![https://docs.oracle.com/javase/tutorial/java](https://lh3.googleusercontent.com/7WdAhlFn2QGd7f7hDAZDBrSXRTgDkXv9wOEvjB5p_Wy7pjrv3EK4TuYn8DiFV3ZQpB6ubLsbJuqMzw1Mx-UnIqri0_qnw3r1-r3vCR348I0u-XXuCHgAwWSZvgT6d_gylhy7xH_H) Even the annual conferences like [JavaOne](https://en.wikipedia.org/wiki/JavaOne) and [//build/](https://en.wikipedia.org/wiki/Build_\(developer_conference\)) are held officially by each of the respective companies. This is not to take away anything from [JSConf](https://jsconf.com), but having a central authority that tells the developers what is "right" or "best practice" is what the JavaScript community lacks in comparison to the other communities. ## The Lack of Literature Here are a list of (mostly well known) books: * *Domain-Driven Design: Tackling Complexity in the Heart of Software*, by Eric Evans * *Clean Code: A Handbook of Agile Software Craftsmanship*, by Robert C. Martin * *Design Patterns: Elements of Reusable Object-Oriented Software*, by Erich Gamma, Richard Helm, et al. * *Head First Design Patterns*, by Elisabeth Freeman, Kathy Sierra, et al. * *Cracking the Coding Interview*, by Gayle McDowell Do you see a pattern? If it is not so obvious, all these books are written in either Java, C#, or C++. To put it in more general terms, all these "classics" are written from the object-oriented paradigm. This isn't to say that there are no good JavaScript books, but the lack of literature when it comes to structuring JavaScript code over the years has been a big issue in the community. Imagine as a Java Developer trying to learn how to write "clean code". There are a plethora of books and resources that teach you how to write better code. Now imagine a JavaScript Developer who just graduated from a coding bootcamp. He would have to scour the web trying to find the correct book only to find that most books that talk about structuring code are written with a traditional OOP language. # Layered/N-Tier Architecture: The Unpopular Proven Way ## Layered Architecture ![Software Architecture Patterns by Mark Richards, 2015](https://lh6.googleusercontent.com/_7a8A4DR9dwoVppEPuNbqQ9QVH0sRr-r7DMSBg7zvvMUqmyz62J0v-0v0nzt_r6LFE86F0gV11X0lu6pVS4PrscMvI7i6qnDu61uvl7k5gGWjhtk2iRnomDpmwCzGHj1keUtgBpE) > The Layers Architecture pattern is considered by many to be the granddaddy of all. It supports N-tier systems and is, thus, commonly used in Web, enterprise, and desktop applications.... we rigorously separate the various concerns of our application or system into well-defined layers. > > *-Frank Buschmann, Author of Pattern-Oriented Software Architecture, A System of Patterns* > Isolate the expression of the domain model and the business logic, and eliminate any dependency on infrastructure, user interface, or even application logic that is not business logic. Partition a complex program into layers. Develop a design within each layer that is cohesive and that depends only on the layers below. > > *-Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software* The layered architecture was a core fundamental concept that has been written in books such as Eric Evans' **Domain-Driven Design: Tackling Complexity in the Heart of Software** (the blue book) and Vernon Vaughn's **Implementing Domain-Driven Design** (the red book). It is also one of the most popular concepts taught in traditional education. The layered architecture has been around for decades and has been the go-to proven default for many real world frameworks. Ever heard of frameworks like *Ruby on Rails*? Java's *Spring Framework*? Or even *Microsoft's ASP.net MVC*? And of course, how can we forget the plethora of PHP MVC Frameworks like *CodeIgniter*, *Zend*, and *Yii* that were extremely popular in the late 2000's and early 2010's. ## Old School MVC and Its Origins If you've been doing web development at all in the last 2 decades, then you would have heard of this design pattern. This has been an extremely popular pattern adopted by the object-oriented world. Most frameworks will use some variation of this type of pattern. Let's go dive a bit deeper into each of the components. Instead of me giving you the boring definition of what the Model-View-Controller pattern (MVC) is, I'd like to explain MVC from its origins and relate it to our present day's understanding of this pattern. This origins of MVC can actually be traced by one of Martin Fowler's boos in the early 2000s, *Patterns of Enterprise Application Architecture*. In his book, he centered his discussion around an architecture of "three primary layers: **presentation**, **domain**, and **data source**". ![Patterns of Enterprise Application Architecture, Martin Fowler, p. 19.](https://lh4.googleusercontent.com/cO9-JmsynnvzeoT-dT-o4nn84FBjEVYwgXO_rO8R6nlKYrI6YiaPJAT9T3PuOTJpIiU9M0Zq7icnyUvk9535gIXv4UuNoIpxjOfbrO83vfBJHJ3clDbH-hjvYMaAcI-ggidTYqq6) ### Views Fowler starts off with the presentation layer, as known as the view layer. > **Presentation logic** is about how to handle the interaction between the user and the software. This can be as simple as a command-line or text-based menu system, but these days it’s more likely to be a rich-client graphics UI or an HTML-based browser UI. The primary responsibilities of the presentation layer are to display information to the user and to interpret commands from the user into actions upon the domain and data source. > > *-Martin Fowler, Patterns of Enterprise Application Architecture, p. 19* As you can see, this is pretty spot on as what we would refer to in the last decade or so. In our case, the view layer is simply just a JSON output, the interaction portion of his definition can be seen as the hypermedia links in our output response. ### Models Fowler then moves on to the data source logic, which for us would be the persistence or model layer of our application. > **Data source logic** is about communicating with other systems that carry out tasks on behalf of the application. These can be transaction monitors, other applications, messaging systems, and so forth. For most enterprise applications the biggest piece of data source logic is a database that is primarily responsible for storing persistent data. > > *-Martin Fowler, Patterns of Enterprise Application Architecture, p. 20* As mentioned before, this is simply a layer with which we can communicate with the database through our own application code. The most common way of doing this today would be with an ORM (Object Relational Mapper). ### Controllers Finally, the domain logic, which can be a combination of the custom business services you write in your own application. if you are old school, the controller is essential where you would write most of your own business logic. > The remaining piece is the **domain logic**, also referred to as business logic. This is the work that this application needs to do for the domain you’re working with. It involves calculations based on inputs and stored data, validation of any data that comes in from the presentation, and figuring out exactly what data source logic to dispatch, depending on commands received from the presentation > > *-Martin Fowler, Patterns of Enterprise Application Architecture, p. 20* The way I understand the controller is that it simply directs the input and output of the request-response life cycle. It can contain hard logic of your business, but should be kept to a minimal length. ## "New" School MVC (Services and Repositories) Although the traditional MVC style is quite "good", there have been many additions to the old school way of thinking. If you have ever worked with other languages and frameworks such as Java or C#, you will have noticed that those 2 communities have adopted the "services and repository pattern". This means extending the traditional 3 layers by adding a repository layer and a services layer. Let's find out what those 2 layers are and how they can extend the layers in our domain logic. ![Graphical illustration of what it means for us to "extend" our domain logic](/files/6h6lcTfhNF3dNi5b97Ks) ### Repositories > Repositories are classes or components that encapsulate the logic required to access data sources. They centralize common data access functionality, providing better maintainability and decoupling the infrastructure or technology used to access databases from the domain model layer. > > *-*[*https://docs.microsoft.com/en-us/dotnet/architecture/microservices/microservice-ddd-cqrs-patterns/infrastructure-persistence-layer-design*](https://docs.microsoft.com/en-us/dotnet/architecture/microservices/microservice-ddd-cqrs-patterns/infrastructure-persistence-layer-design) Repositories solve 2 problems. The abstraction of (complex) database queries and the inversion of control of the persistence layer. I like to think of repositories as utilities for querying the database and an adapter layer that let's swap different implementations of databases. Understanding the abstraction of database queries is quite simple. Imagine having to write a [very long database query](https://ahmedfattah.com/2020/01/19/the-longest-sql-statement-i-have-ever-seen/). Now imagine having to rewrite that database query out every time you want to use it. Well, most people would just encapsulate a long database query in some sort of re-usable function. And that's just it, that is one of the purposes of the repository pattern, to create re-usable, abstracted, and encapsulated database utilities functions. The inversion of control of the databases is a little bit harder to explain, but here is my take on it. Imagine we had to use MongoDB as our database. Now imagine, you decide you want to use MySQL because you realized non-relational databases are just not for you. But hold up, now you realize you want to use a different SQL driver, you went from writing raw SQL with a native MySQL driver to using an ORM like Sequelize. Wouldn't it be great if you were able to swap out those different implementations whenever you wanted to? That is inversion of control, the ability to loosely couple our data layer in order to which make them testable, maintainable, and extensible. ### Services > A Service Layer defines an application’s boundary and its set of available operations from the perspective of interfacing client layers. It encapsulates the application’s business logic, controlling transactions and coordinating responses in the implementation of its operations. > > *-*[*https://martinfowler.com/eaaCatalog/serviceLayer.html*](https://martinfowler.com/eaaCatalog/serviceLayer.html) Services are what make up the business logic of your application. To best explain what a "service" is would be to give examples. Consider that an e-commerce website may have a: * `CustomerService` for fetching customer information. * `PaymentService` for making and processing payments from the purchases made by customers. * `OrderService` for fetching order information from the purchases customers have made. Services can access different data layers (repositories) and may call another different service from within. ## Node.js' Layered Architecture Adoption Layered architecture has actually made its way into the JavaScript community quite nicely. In the past decade, there have been various Layered-based Node.js frameworks that have been released. They have all taken a page from the frameworks of other traditional object-oriented programming languages. It first started off in 2012 with [Sails.js](https://sailsjs.com), an alternative or a more structured approach than just the vanilla express router. Sails.js was great, as it emulated many of the features of Ruby on Rails and was probably the first ever "real" Node.js MVC Framework. But it wasn't until 2017, when TypeScript's explosion in popularity began to bring a new level of innovation to bringing the traditional architectural frameworks into the JavaScript community. In 2018, we would see a brand new framework, [Nest.js](https://nestjs.com), take the enterprise world by storm. This is taken directly from the philosophy of Nest.js. > Nest provides an out-of-the-box application architecture which allows developers and teams to create highly testable, scalable, loosely coupled, and easily maintainable applications. The architecture is heavily inspired by Angular. > > *-*[*https://docs.nestjs.com*](https://docs.nestjs.com) In the same year, we would see another competing framework, [Adonis.js](https://adonisjs.com). A framework that very much reminds me of PHP's Laravel framework. To sum up what Adonis.js is really about. This is taken directly from their official website. > **How is AdonisJS different to Express or Koa?** > > Express and Koa are routing libraries with a thin layer of middleware on top. They are great for several use cases but fall apart when projects start to grow. > > Since your projects have their own standards and conventions, it may become harder to hire developers to work on them. As AdonisJS follows a set of standardized conventions, it should be easier to hire someone to work on existing AdonisJS apps. > > *-*[*https://adonisjs.com/docs/4.1/about#\_faqs*](https://adonisjs.com/docs/4.1/about#_faqs) # Microservices and Distributed Computing: A Popular Misdirection When you go to a tech conference, you'll likely come across topics related to [distributed systems](https://en.wikipedia.org/wiki/Distributed_computing). In fact if you were to type "distributed architecture" into YouTube, would find a plethora of conference video recordings with hundreds and thousands of views. ![](https://lh4.googleusercontent.com/PvkfQqu8XtCyCejBNxYEeUMEVMXUPdJfbiDO5Znkmwu8jMkNYgZoDOSy2BPHCTolcheNE6kvhuK3KPfjRR1EdTPghb5PdXIJZl0xljiVsZY3SLAh09zuWoSflIzo2ffjBAhwScRv) What about if you tried typing in "layered architecture"? First, there would be less people talking about it in live conferences. Second, you would see a significantly less number of views on those videos. Third, nobody would type that in since it's so old and boring. ![](https://lh6.googleusercontent.com/ZYmAlEnJPUUxXxr-3273KWECoz2Cie1rHv8oZmarcwO2gAe4D72vg1W4xMM_K1DxQbPi-6lAe4fdXko9b_X9wgcsfB1jt8QnUiYOiawwBJQhrA8AquXOms8O_sfPu1sQBwjbSuIy) But wait… why is this the case, even though layered/n-tier architecture is indeed the more common architecture. In fact, we can agree that layered architecture is most likely more "popular" than any other architectural pattern. Almost all projects use the layered architecture in their application structure. As mentioned before, the most popular frameworks like Rails, Spring, ASP.net MVC, Laravel are all MVC and layered-based. The problem is that microservices has been the most talked about topic of the previous decade. It is *the* hot topic, *the* shiny new object, and *the* buzzword in the past 10 or so years. You want to know why? Here's why. Big tech companies like Amazon, Google, Netflix, Uber, Facebook, and Snapchat host large conferences are all talking about it. Amazon is talking about how they created their own cloud solution to scale their own internal systems to millions of transactions per day. Netflix is talking about how they scale their microservices architecture so that millions of people can stream video without delay also sounds really interesting. And who can forget Google, the granddaddy of them all. Google is talking about things such as [MapReduce](https://en.wikipedia.org/wiki/MapReduce) and [Big Data](https://en.wikipedia.org/wiki/Big_data) and how they scale to billions of users with 0 downtime. All those mentioned are really exciting and enticing to the public. It also is interesting that these companies, who talk about these topics, also expect their future employees to know about them as well. In fact, all of these FAANG companies and their interview processes' involve a "Systems Design" portion. What is this "Systems Design" part of the interview about? It's about concepts such as: load balancers, caching, data partitioning indexes, redundancy, and replication. Notice how all these concepts are all about distributed systems and how to scale a system up. On the other hand, If you hear someone's talk about MVC or layered architecture at a conference, people are going to fall asleep, even though it is clear that layered architecture is what is important for structuring scalable and maintainable code. Distributed systems and microservices often get mixed up with RESTful services, which in turn misdirects people's attention to something that is utterly unrelated to actual code architecture. Therefore as JavaScript developers, we must not focus on the hot topic of the week and the trendy new technologies (I know it's hard). We have to go back to the "old school" ways of doing things and read some of the more classic books from the object-oriented world and bring those ideas into our own community. # Summary JavaScript developers just wing it when it comes to structuring their code. We can learn from other communities and use a proven way of building software, which is the **layered architectural approach**. Distributed Architecture is just all talk, so don't pay attention to it. # Chapter 5: The 8 Step Recipe In this chapter, we will take what we learned in chapters 3 and 4, and combine them into what I call a 7 (+3) step recipe to creating any API endpoint. We'll be extending the layered architecture as discussed previously by going over the middleware layer, the validation layer, and the event layer in this chapter. But wait… what is this extra +3 step in the brackets? This is an extra 3 sub-steps that do not pertain to building the actual endpoint itself, but rather help solidify and strengthen your endpoint if you so choose to add it. The 3 sub-steps are *Test*, *Refactor*, and *Document*. For simplicity sake, let's just group those last 3 sub-steps in 1 step, and call this ***"The 8 Step Recipe"*** from this point forward. This 8 step recipe is essentially a ***formal design document***, its purpose is to help you plan out your API endpoints before writing any code. It's a ***checklist of considerations when designing each layer of your endpoint***. Think of this as a nice starter template for helping you to design and structure your endpoint and codebase. This is the 7 (+3) step process that I have been using for the last 5+ years of my career and is something that has not only benefited me during solo projects, but it has also benefited many of the teams I have worked on when using this approach. Here are some key benefits from using my 8 step recipe. 1. It takes away the anxiety and pressure of feeling like you might be missing something because it gives you a list of common considerations to think about beforehand. 2. Having a structured approach keeps you level-headed and clear about the design and structure of your API and code. 3. Having a methodical way of planning your endpoint let's you break the problem down and see where and what you need to work on. 4. You remember what you were thinking because you wrote it down; you can then reference it later. 5. Not only do you have a better understanding of the API endpoint, your team members will understand your thought process as well. 6. It will save you a ton of time (trust me), which means less tech debt in the future. Here… is the 8 step recipe to creating any API endpoint. 1. Route Name (URI) 2. Input Request 3. Middleware 4. Validation 5. Business logic 6. Events 7. Output Response 8. Test, Refactor, Document We will be going over each concept carefully in this chapter, and I will be explaining briefly what each component is and go over some real world applications. In the next chapter, we will begin to apply these concepts in building our web API. This model or concept can be applied to any framework or language. This is supposed to be a framework and language agnostic blueprint so that you may apply this to any project you work on in the future. # Route Name (URI) The **route URI** is the entry point in which the client will use to initiate the request for a desired resource. There are 4 considerations when it comes to planning for the URI path. 1. API Versioning 2. URI Path 3. Query String Parameters 4. HTTP Verb ![The anatomy of an HTTP request.](https://lh5.googleusercontent.com/Wjteb5dSTv6RsHs_yYoj0ZyJ4sZr-j848sIv8nH6HOJouSus8SFgEFTAPcbaqj3--MM8uweE6XIcmZ9rEfw0ckkgfc-kMGa75JaAHLisOpUuT6BxbtMTM_BaS-yOooLqU9NWF5eY) ## API Versioning We talked about API versioning strategies under chapter 3, [Versioning Strategies](/part-2/chapter-3/7-versioning-strategies). Whichever strategy(ies) you decide to implore, make to specify it clearly in the design doc. For me personally, I have always used the URI prefixing approach like this: `https://example.com/api/v1`. ## URI Path Next is the URI path itself. Make sure to design the URIs by following the best practices and guidelines mentioned in chapter 3, [URI Design](/part-2/chapter-3/2-uri-design). Also, make sure to make them as RESTful as possible whilst being as pragmatic as possible. ## Query String Parameters Similar to the above, try following the best practices mentioned in the [URI Query Design](/part-2/chapter-3/2-uri-design#uri-query-design) section of chapter 3. Make sure to remember that query strings are used mainly as optional parameters. The final result of your decision should be something like this: `GET /api/v1/users?page=1&limit=10` Which should be very intuitive as it translates to "retrieve a list of the first 10 users in the application". ## HTTP Verb Similar to the above, try following the best practices mentioned in chapter 3, [Method Verbs](/part-2/chapter-3/3-method-verbs). Make sure to always utilize the verbs as much as possible without naming the URIs as actions or functional APIs. # Input Request The **input request** is any input we'd expect from the client to send to our API. They can be in the form of URI parameters, headers, body content, and more. A good way to know if something is an input request would be to just look at the Express documentation of all the `req` properties. ![https://expressjs.com/en/api.html#req](/files/XcipAXIBq9oKkAitLnSF) The goal of this step is to know what exactly we'd expect from the client and to create some sort of contract between us, the server, and the client calling our RESTful API. To do so, we will use what are called **Data Transfer Objects**. ## Using Data Transfer Objects Data Transfer Objects (DTOs) are objects that carry and transform data in between processes. They are extremely useful when you want to customize properties of data that you want be exposed or hidden. They are objects that can create adapter-like layers in order to smoothly control the input and output flow of data between different function calls within your application. In many other typed languages such as Java or C#, the built-in verbosity of those languages require you to type hint your input and output based on custom classes you would declare, making it easy to declare some sort of contract between the caller and callee. In JavaScript land however, we do not have that, at least not in raw native JavaScript. This is why it is important to create some sort of DTO, so that we may not only create those contracts, but also create a **self-documenting source of truth within our codebase**. DTOs will be used heavily in our project in the upcoming chapters and will be illustrated as we get closer to looking at the code. # Middleware Middleware is one of the most confusing and ambiguous terms used in the web development industry. The concept of middleware itself is different on every platform, what is referred to as a middleware in one framework can have a completely different meaning in another. In fact, I’ve taken this quote directly from *Wikipedia*, it states that "*middleware is sometimes used in a similar sense to a software driver, an abstraction layer that hides detail about hardware devices or other software from an application*". But if you have been working in JavaScript and Node.js for a while, you will notice that definition doesn't seem to fit our definition of what a middleware might be. Here are two more quotes from *Microsoft* and *Red Hat*. > Middleware is software that lies between an operating system and the applications running on it. > > *-*[*https://azure.microsoft.com/en-ca/overview/what-is-middleware*](https://azure.microsoft.com/en-ca/overview/what-is-middleware/) > Middleware is software that provides common services and capabilities to applications outside of what’s offered by the operating system > > *-*[*https://www.redhat.com/en/topics/middleware/what-is-middleware*](https://www.redhat.com/en/topics/middleware/what-is-middleware) As you can see, middleware is a term used in the operating systems world. The traditional definition of a middleware is more related to lower levels of programming that is closer to the computer's hardware. For us Web Developers, this might be a better definition. > Middleware is a (loosely defined) term for any software or service that enables the parts of a system to communicate and manage data. It is the software that handles communication between components and input/output, so developers can focus on the specific purpose of their application. > > In server-side web application frameworks, the term is often more specifically used to refer to prebuilt software components that can be added to the framework's request/response processing pipeline, to handle tasks such as database access. > > *-*[*https://developer.mozilla.org/en-US/docs/Glossary/Middleware*](https://developer.mozilla.org/en-US/docs/Glossary/Middleware) ## The Modern Middleware of the Web Middleware in the context of the Internet and Web is different from traditional middleware. Here are some concrete examples of modern frameworks that have middleware in it. **Express** > Middleware functions are functions that have access to the request object (req), the response object (res), and the next function in the application’s request-response cycle. The next function is a function in the Express router which, when invoked, executes the middleware succeeding the current middleware. > > *-*[*https://expressjs.com/en/guide/writing-middleware.html*](https://expressjs.com/en/guide/writing-middleware.html) **Laravel** > Middleware provides a convenient mechanism for filtering HTTP requests entering your application. For example, Laravel includes a middleware that verifies the user of your application is authenticated. If the user is not authenticated, the middleware will redirect the user to the login screen. However, if the user is authenticated, the middleware will allow the request to proceed further into the application. > > Additional middleware can be written to perform a variety of tasks besides authentication. A CORS middleware might be responsible for adding the proper headers to all responses leaving your application. A logging middleware might log all incoming requests to your application. > > There are several middleware included in the Laravel framework, including middleware for authentication and CSRF protection. All of these middleware are located in the app/Http/Middleware directory. > > *-*[*https://laravel.com/docs/6.x/middleware*](https://laravel.com/docs/6.x/middleware) **ASP.NET** > Middleware are software components that are assembled into an application pipeline to handle requests and responses. Each component chooses whether to pass the request on to the next component in the pipeline, and can perform certain actions before and after the next component is invoked in the pipeline. Request delegates are used to build the request pipeline. The request delegates handle each HTTP request. > > *-*[*https://jakeydocs.readthedocs.io/en/latest/fundamentals/middleware.html*](https://jakeydocs.readthedocs.io/en/latest/fundamentals/middleware.html) ## Most Common Real World Use Cases of Middleware Here is a list of common use cases I have seen for middleware. 1. Object transformation 2. Filtering and sanitization of data 3. Validation 4. Logging 5. API rate limiting and throttling 6. API Security (authentication and authorization) 7. Changes in meta-data like headers 8. Separation of concerns through layers by adding additional layers by design 9. Making reusable middleware layers that can be used in different routes 10. Decryption of parameters in the URI and headers 11. Calling an external service and passing those parameters down These are just some examples of real world use cases I have seen, this is such a broad topic because in reality, your entire application can just use middleware and you would be just fine. Whatever the case, make sure to use middleware to your advantage as it provides a very nice way to abstract even more layers out in your application if needed. # Validation **Validation** is the act of determining if the input data is in proper form. There are two types of validation, **form validation** and **service validation**. ## Form Validation **Form validation** is simply how one would validate the data if it were a form from the frontend. Below are some common examples of what would be considered form validation of a user registration endpoint you might build. * Email * Required. * Correct email format. * Password * At least 6 characters in length. * Must contain 1 number. * Must contain 1 alphabetical character. * Invite\_Code * A 10 digit number. Notice how these are quite generic and simple, because these do not pertain to any specific business rules. ## Service Validation **Service validation** is any type of validation that involves the business logic of your application. Suppose we use the same user registration endpoint example from above, except this time we have specific business logic that needs validation. * Email * Must not be taken already. * Invite\_Code * Must be a valid 10 digit number, where only users who have been sent email with that code can have a truly valid one. Notice these constraints are put in the specification of the application and are not generic at all. ## Sanitization In addition to validation, there is also the sanitization of data. **Sanitization** is the act of removing and/or replacing any illegal or unwanted characters from the data. It goes hand in hand with validation because sanitizing data before and/or after the validation of data allows us to pass that data with more confidence throughout our application. Here are some common examples: 1. Trimming out spaces. 2. Converting characters to HTML entities. 3. Escape strings. 4. Getting rid of special characters. 5. Converting strings to only lower cases. # Domain Our domain layer would mainly consist of 3 items: ***services***, ***models***, and ***repositories***. We went in depth in the last chapter, [Layered/N-Tier Architecture: The Unpopular Proven Way](/part-2/chapter-4/2-layered-n-tier-architecture-the-unpopular-proven-way#new-school-mvc-services-and-repositories), so make sure to read up on those concepts if you are not sure about what those 3 items are. ## Services We talked about services and what they are, services will usually use other services and repositories to perform some sort of business logic operation. For example, a UserService will have a method called `registerUser`. UserService would potentially use a UserRepository to insert the user into the database and an EmailService to send an email to the user if they successfully registered. ## Models (Entities) Models usually represent something in the business and are generally going to be some sort of Object Relational Mapper (ORM). They could be SQL based or NoSQL based. Models can also be just plain old objects. For us, we will be using a MongoDB, and [Mongoose ](https://mongoosejs.com)as our NoSQL database adapter. Examples of models would be a *User*, a *Car*, a *Book*, a *House*, an *Animal*, and more. ## Repositories If you can recall, repositories are essentially an abstraction layer of models. In general, repositories will have CRUD operations as their methods wrapped around the native models. An example would be: *BookRepository* * `getAllBooks()` * `createABook(book)` * `deleteABook(bookId)` # Events Events serve as a great way to ***decouple*** various aspects of your application, since a single event can have multiple listeners that do not depend on each other. For example, you may wish to send a Slack notification to your user each time an order has shipped. Instead of coupling your order processing code to your Slack notification code, you can raise an `OrderShipped` event, which a listener can receive and transform into a Slack notification. You can fire one event that can cause 3 different actions to go off or have 1 event be used multiple different times. Below is an example: Our Events List * `sendEmailNotification` * `sendSmsNotification` * `sendSlackNotification` Our Listeners List * event.listen(*'userHasCreatedANewListing''*, ***sendEmailNotification***) * event.listen(*'userHasRegistered'*, ***sendEmailNotification***) * event.listen(*'userHasRegistered'*, sendSmsNotification) * event.listen(*'userHasRegistered'*, sendSlackNotification) Notice how *userHasRegistered* triggers 3 different events and how ***sendEmailNotification*** can be used in more than 1 event listeners. ## The Node.js Gift: EventEmitter As JavaScript Developers, we are lucky enough to have the [EventEmiter](https://nodejs.org/api/events.html) utility which is a built-in module that functions much like the traditional observer pattern. We'll see how to use this later on in the next chapter. ## Examples of Events Here are some examples of when to use events in your application 1. Send an email after a user has registered. 2. Logging user activity data. 3. Sending an email and SMS to the user after he has deleted his account. It can be hard at times to know what is considered an event. The difficulty comes when deciding whether to put something in the event layer or to just put it directly in the service layer. The two heuristics that I follow when deciding whether something should be put in the service layer or the event layer are: 1. Do I see a certain function that gets called come up over and over again at the end of each service layer, if I do, then maybe it's time to move it to the event layer for more re-usability. 2. Do I see myself constantly adding new events inside the same service layer? If I do, then I would move those events to the event layer and try to chain the events through a single listener that can fire multiple events for that service. # Output Response The final step of our 8 step recipe is the output response. We will be following the guidelines suggested in chapter 3, [Representational Design](/part-2/chapter-3/5-representational-design). As mentioned in a previous section of this chapter, we'll be using [Data Transfer Objects](/part-2/chapter-5/2-input-request#using-data-transfer-objects) in our application to help with following those guidelines. Just in case you are not sure why we will be using DTOs for our output response, here are 3 main reasons for using them in our project. The **first** is that we can use them for creating a global wrapper in our code, that way we can have a consistent format being spat out as the response. The **second** is that it can transform the output of our response. This means adding in newer properties and in some cases taking away properties that we do not want to expose. The **third** is that it creates a contract. It creates a contract that must be obeyed because it let's the client consuming it know what should be returned. # Test, Refactor, Document Now comes the (+3) step. This step assumes that your API is already working and that you want to make sure your API is more bullet-proof. Note that although I consider this optional, I personally almost always will test, refactor, and document my code if I have the time. ## Test > If you write tests, it's like little robots that you build that do things for you instead of clicking > > *-Jeffrey Way, Creator of laracasts.com* There are many levels to testing, here are a couple: 1. Unit Tests 2. Integration Tests 3. End-to-end Tests 4. User Acceptance Tests 5. Load/Performance Tests There are many different ways to test our code, but I believe the two most important types of tests that we should write are unit and integration tests. **Unit tests** are low level tests that test functions in the code. One of the main advantages for using the layered architecture is for us to have the ability to intricately test each component of each layer. You will see in the later chapters just how much layered architecture shines and how easy it is to write isolated tests. The second type of tests that we'll be writing are high level **integration tests**. These types of tests are tests that simply call the API to check the response output and the database (if necessary). These are my favorite types of tests to write because these tests are essentially a simulation of how a developer would do to call your API. ### The AAA Framework If you have not written many tests before, here is the framework that most developers follow. It's called the ***Arrange, Act, and Assert Framework***. Here's a quick example of how we might want to plan our test cases. 1. *Arrange* 1. Setup the world; database 2. Setup mocks 2. *Act* 1. Call the function, or 2. Call the API 3. *Assert* 1. Validation checks 2. Database has been updated 3. Output response of our endpoint You will see more of this later as we go in depth into testing an endpoint from starting to finish in both unit and integration testing. ## Refactor Unfortunately refactoring is way beyond the scope of this book, but it is always something to consider when writing code. For those that wants to look more into refactoring, I recommend the following two pieces of literature: 1. , and 2. We will be refactoring our code, make sure to read the bonus chapter of this book, [*Refactoring to HATEOAS*](/bonus/refactoring-to-hateoas), as we refactor our code to make it hypermedia-friendly. ## Document We will be following the OpenAPI specification, and we'll be using Swagger as our tool for documenting all of our API endpoints. Needless to say, documentation is extremely important, especially if you are working in a team. # Summary There you have it, the 8 step recipe for creating any API endpoint. Follow this design document and you will be prepared to take on building any RESTful API endpoint. We will be using it for rest of the upcoming chapters in *Part 3: The Code,* so make sure to go back and review this chapter if you need to. # Chapter 6: Introduction to the Bookstore API In this chapter, we'll be covering the specifications on what exactly we'll to be building throughout the rest of the book. We'll be going over the API design and architectural decisions before we begin building our API. Lastly, we will be setting up our project locally so that you may follow along with the book. # The Bookstore API Endpoint Specifications Our Product Manager has just announced to everyone in a zoom meeting that we will be starting a new project. This new product is going to be a ***classified listings website focused on book exchange*** whereby people can post used books for sale and have the ability to view public book listings if they want to buy. Let's call this new application: ***trademebooks*** (***tmb*** for short). Our Product Manager has already had the Designer spin up the finalized wireframes for the project. The frontend and backend Tech Leads have already aligned, and have finalized the details of the backend RESTful API endpoints that we will be building. Below are all the RESTful endpoints we will be building for our new ***Bookstore API***. ## API Endpoint #1: User Registration We'll need a way for users to register for an account with our API. We'll need to collect a public user's *first name, last name, email, password, and phone number* and store them in the database of our application. We'll need to provide some level of form validation for all those user inputs. All the fields are required, of course the email and phone number should all be in the correct format. In additional, we'll need the user to type his password a second time to make sure there is some sort of password confirmation. Remember, there can only be one email registered to one user account, so there must also be some sort of check on our backend to make sure an email is not already taken. Last but not least, if the registration does go through, we want to send the user an email letting them know that they have successfully registered on our website. ## API Endpoint #2: User Login A user can login to our application with an email and password. Like most authentication system, there will be some sort of validation on both the frontend backend side. For the frontend side, it's quite simple, both fields are required. For the backend side, a user must exist in the database along with the correctly entered password. If both the frontend and backend validation both pass, then we will log the user into the application and persist a login session. ## ~~API Endpoint #3: User Logout~~ ~~Logging a user out of a session is quite simple since there are no inputs. If there is a currently authenticated user, then we will destroy that session. If not, then we don't do anything.~~ Update: our Product has realized that we are in too much of a rush to implement the logout feature. Maybe we'll have more bandwidth in the future :smile:. ## API Endpoint #4: Get Authenticated User We'll need some sort of way for the frontend team to get the currently authenticated user's profile. We'll need an endpoint to output their *first name, last name,* and *email.* If there is no currently authenticated user, then we won't return a user. ## API Endpoint #5: Retrieve all Book Listings Now comes the book listings part. The frontend team and design team have decided that they want the front page to display all the books on our website from the most recently added listings to the oldest listings. Each book listing will have an *ID, userId, title, description, price, author, published date, and date added* field. Note that the *ID* is for the book and the *userId* is used to identify the owner who posted that book. ## API Endpoint #6: Retrieve a Book Listing Retrieving a single book will be very similar to the retrieval of all books, except we will need to provide a specified book ID when calling our API endpoint. If the book listing is not found, then we will let the user know. Like before, the specified book listing will have an *ID, userId, title, description, price, author, published date, and date added* field. ## API Endpoint #7: Create a Book Listing For a user to create a listing, they must first be logged into the application, otherwise they will not be able to access this endpoint. In order to create a book listing, the *title, description, price, author, published date* fields are all required and must be in alphanumeric format, except for the *price* field which will be any number up to 2 decimal places and the *published date* field which will be of any valid date format. ## API Endpoint #8: Update a Book Listing For a user to update an existing book, the user must first know the ID of the book. The user also must be logged in as well be the owner or the creator of that book. If a book with a specified ID does not exist, then we'll let the user know that we can't update it since it is not in our database. If a user tries to update a book in which he is not the creator of, then we'll let them know that they do not have sufficient permissions. ## API Endpoint #9: Delete a Book Listing For a user to delete an existing book, it is very similar to updating a book. The user must know the ID of the book, the user must be logged in as well as be the owner or the creator of that book. If a book with a specified ID does not exist, then we'll let the user know that we can't delete it since it's not in our database. If a user tries to update a book in which he is not the creator of, then we'll let them know that they do not have sufficient permissions. # API Design and Code Structure ## Use JSON as the Main Format By this point, if you've read chapter 3 of this book, it should be obvious that we will only be supporting JSON in our API. Not only does GitHub and various other companies only support JSON, it would also create additional and unnecessary work for us if we were to support another format like XML ourselves. Hence, we will no JSON, yay! ## Deciding on the Versioning Strategy As stated previously, we will be picking the most simple API versioning strategy and we will be going with the URI prefixing method of putting `api/v1` in our routes. ## Global Response Message (Success and Errors) As mentioned in chapter 3 of Representational Design, we should be following a ["Using a Consistently Wrapped Response"](/part-2/chapter-3/5-representational-design#using-a-consistently-wrapped-response). ## Being Pragmatic and Breaking (some) Rules ### HATEOAS In chapter 2 of [Pragmatic REST vs Dogmatic REST](/part-2/chapter-2/7-pragmatic-rest-vs-dogmatic-rest), we stated that we will pragmatic with our approach as opposed to dogmatic. This means that we will not be purists and adhere to all of the guidelines that make our API RESTful. This means we will ***NOT be creating a hypermedia-friendly REST API***, but if you want to do so, you may check out the [bonus chapter of the book](/bonus/refactoring-to-hateoas). ### Cookies-based Authentication Fielding stated in his dissertation that using cookies actually "violates REST". Again, we are trying to be pragmatic, so we will be using cookie-based authentication in our application, you may use other forms of authentication, but for the purpose of this book, we will be using cookies. ![https://www.ics.uci.edu/\~fielding/pubs/dissertation/evaluation.htm](/files/wNt3d2l5qMVuYI6JsfxZ) ### Keeping in mind of Guidelines over Principles and Theories As a general precaution, we will be applying as much of the guidelines from chapter 3, but we will not be enforcing every single guideline. Keep in mind that: > Guidelines provide a shared language to promote consistency among multiple people in terminology usage. It is simply a set of ***recommendations*** that are there to create reliability and consistencies among a group of individuals. > > *-Professor Naureen Nizam at the University of Toronto* A good example of this is [Guideline #1: Consider Using An Application-Specific Media Type](/part-2/chapter-3/6-metadata-design#guideline-1-consider-using-an-application-specific-media-type) from chapter 3, [Metadata Design](/part-2/chapter-3/6-metadata-design). Why? Because it would be an overkill to submit your custom vendor data type to , because even a company such as GitHub has yet to be approved of their custom vendor data type. ## Overall Code Structure If you recall, in JavaScript/Node.js land, structure has become a big problem in our community. However, there are a ton of pre-made boilerplate starter kits available for our needs. You can find a list of them here in this article: {% embed url="" %} {% endembed %} These "Express Boilerplates" provide a nice foundation for starting a project in the right way. The best boilerplate start that I have seen is . It provides a bunch of pre-configured out-of-the-box features already baked into the repository. Features such as a global error handler, a validation library, a good authentication and authorization starter kit, and many more. We will be using my own custom setup, but feel free to change it up to your own accord, or pull in anything else as you see fit. # Project Setup The entire source code is available at the following GitHub repository: . Make sure to follow the instructions clearly as stated in the README. # Summary We have gathered all the requirements and specifications of the RESTful API we'll be building, planned out our API design and code structure, and setup up our project. It's time for us to dive straight in to the actual code! # Chapter 7: Retrieving Books from our API It is now time to finally put everything we've just learned in the past couple of chapters to practical use. We will be using our 8 step recipe in this chapter. We will first use the design doc to plan out the first endpoint and retrieve all the books in our application. After that, we will proceed to creating the endpoint for retrieving a single book by its id. The rest of the chapters will follow a similar format until we finish implementing of all our RESTful API endpoints. Let's get started with our first API endpoint! # Retrieving All Books - Planning Refer back to [*Chapter 6, The Bookstore API Endpoint Specifications, API Endpoint #5: Retrieve All Book Listings*](/part-3/chapter-6/1-the-bookstore-api-endpoint-specifications#api-endpoint-5-retrieve-all-book-listings) section for the API endpoint specifications. ## 1 - Route Name From the specification, it seems like this will be a `GET` request that will fetch a collection of books. We can refer back to the sections under chapter 3 for guidelines to follow, specifically the sections [*Method Verbs*](/part-2/chapter-3/3-method-verbs) and [*URI Design*](/part-2/chapter-3/2-uri-design). The following URI should suffice. `GET /api/v1/books` ## 2 - Input Request Since this is a `GET` request, there won't be any input request or payload from the client. ## 3 - Middleware This is a public URI, so there really is no need for any middleware. ## 4 - Validation Again, since this is just a simple `GET` request, there won't be any validation needed. ## 5 - Domain From the specifications, it looks like we will need the following book entity in our domain layer. **bookModel** * `title` * `description` * `price` * `author` * `datePublished` We will also need some sort of way to query the database to be able to retrieve all the books, `getAllBooks()` will be the method we'd call. **bookRepository** * `getAll()` On top of that, our controller will call a service, in this case, it will be the `bookService` which will retrieve all of the books in our database. **bookService** * `getAllBooks()` This is a rough outline of how we are going to be implementing these functions, we'll take a deeper dive at the implementations in the next section of this chapter. ## 6 - Events Judging from the specifications, there doesn't seem to be any events, so we'll leave this blank as well. ## 7 - Response As mentioned previous, we will be ["Using a Consistently Wrapped Response"](/part-2/chapter-3/5-representational-design#using-a-consistently-wrapped-response). We'll be adding in these fields: * `title` * `description` * `price` * `author` * `datePublished` for each of our book listings. ```json { "status": "success", "code": 200, "message": "List of all books in the database." "data": [ { "id": "61f88350745d83158f3c746d", "title": "Harry Potter and the Goblet of Fire", "description": "Mint condition, but will negotiate", "price": 99, "author": "J.K. Rowling", "datePublished": "Sun Oct 10 2021 23:56:34 GMT-0400" }, ... ], "errors": null } ``` # Retrieving All Books - Implementation ## 1 - Route Name `GET /api/v1/books` For the route version prefix, we can do the following. *File: src/server.js* ```javascript const getRouter = require('./routes') const router = getRouter() app.use('/api/v1', router) // add the prefix '/api/v1' to all our URIs ``` We will then proceed to include all of the book routes. *File: src/routes/index.js* ```javascript const express = require('express') const router = express.Router() const bookRoutes = require('./book.route') // all our book routes function getRouter() { router.use('/books', bookRoutes) // we prefix it here as 'books' return router } module.exports = getRouter ``` Here, we specify and create a controller method, let's call it `getAllBooks`. *File: src/routes/book.route.js* ```javascript const express = require('express') const router = express.Router() const { getAllBooks } = require('../controllers/book') router.get('/', getAllBooks) module.exports = router ``` This is where we will put the service layer and the response output. *File: src/controllers/book/getAllBooks.js* ```javascript const catchException = require('../utils/catchExceptions') const getAllBooks = catchException(async (req, res, next) => { // our code goes here... }) module.exports = getAllBooks ``` Now that all the setup is done, we can proceed with the other layers in our application. ## 2 - Input Request None. ## 3 - Middleware None. ## 4 - Validation None. ## 5 - Domain The domain layer is quite simple, we want to first create a model that satisfies all our business requirements from the user stories we have in the previous chapter. We are going to create the book model as follows. *File: src/domain/models/book.model.js* ```javascript const mongoose = require('mongoose') const Schema = mongoose.Schema const BookSchema = new mongoose.Schema({ userId: { type: Schema.Types.ObjectId, ref: 'user' }, title: { type: String, required: true }, description: { type: String }, price: { type: Number, required: true }, author: { type: String }, datePublished: { type: String }, createdAt: { type: Date, default: Date.now } }) module.exports = mongoose.model('book', BookSchema) ``` Then, we will wrap it in a repository. Remember why we are using repositories, we want to be able to create an abstraction layer between the concrete implementation of the model and our service layer so that we may swap out database types any time we want. In our case, we are using MongoDB. If for some reason in the future we decide to use something like MySQL, PostgresQL, or OracleDB, it would be much easier to swap out. *File: src/domain/repositories/book.repository.js* ```javascript const Model = require('../models/book.model') // Retrieve - all const getAll = async () => { return await Model.find({}) } module.exports = { getAll } ``` And finally, our service layer. *File src/domain/services/book.service.js* ```javascript const bookRepository = require('../repositories/book.repository') const ApiException = require('../../utils/ApiException') const mongoose = require('mongoose') // Retrieve - all books const getAllBooks = async () => { return bookRepository.getAll() } module.exports = { getAllBooks } ``` ## 6 - Events None. ## 7 - Response Here is what we will do to our controller in order to return a list of books *File: src/controllers/book.controller.js* ```javascript const globalResponseDto = require('../../responses/globalResponseDto') const booksResponseDto = require('../../responses/booksResponseDto') const catchException = require('../../utils/catchExceptions') const bookService = require('../../domain/services/book.service') /** * Gets all book listings from the database. */ const getAllBooks = catchException(async (req, res) => { const books = await bookService.getAllBooks() res.json( globalResponseDto({ status: 'success', code: 200, message: `List of all books in the database.`, data: booksResponseDto(books), errors: null }) ) }) module.exports = getAllBooks ``` *File: src/responses/bookResponseDTO.js* ```javascript function bookResponseDTO(book) { return { id: book['id'], title: book['title'], description: book['description'], price: book['price'], author: book['author'], datePublished: book['datePublished'] } } module.exports = bookResponseDTO ``` *File: src/responses/booksResponseDTO.js* ```javascript const bookResponseDTO = require('./bookResponseDTO') function booksResponseDTO(books) { return books.map((book) => bookResponseDTO(book)) } module.exports = booksResponseDTO ``` # Retrieving A Book By ID - Planning Refer back to [*Chapter 6, The Bookstore API Endpoint Specifications, API Endpoint #6: Retrieve a Book Listing*](/part-3/chapter-6/1-the-bookstore-api-endpoint-specifications#api-endpoint-5-retrieve-all-book-listings) section for the API endpoint specifications. ## 1 - Route Name Much like how we planned out the URI for retrieving all books, retrieving a single book is not too far off from that. From the specifications, it seems like this will be a `GET` request that will fetch a single book by its ID. We can refer back to the sections under chapter 3 for guidelines, specifically the sections [*Method Verbs*](/part-2/chapter-3/3-method-verbs) and [*URI Design*](/part-2/chapter-3/2-uri-design). The following URI should suffice. `GET /api/v1/books/:id` ## 2 - Input Request Similar to retrieving all books, there are none. ## 3 - Middleware Similar to retrieving all books, there are none. ## 4 - Validation Similar to retrieving all books, there are none. ## 5 - Domain Similar to the retrieval of all books, we will be using the same 3 entities to orchestrate our domain logic. * `bookModel` * `bookRepository` * `bookService` We will definitely use the `bookRepository` and get it a method something like `getById()`. In the `bookService`, we would want to have a method as well, perhaps, `getBookById()`. Also, remember that we want to do some sort of validation to check if the book we are trying to retrieve exists. Recall that we must return a 404 in our response to indicate that a book with the specified ID by the client does not exist. ## 6 - Events Similar to retrieving all books, there are none. ## 7 - Response(s) There are 2 types of responses that could possibly return from this endpoint. The first is the book with the specified ID if it does exist in the database. ```json { "status": "success", "code": 200, "message": "Here is the book with the specified ID." "data": { "id": "61f88350745d83158f3c746d", "title": "Harry Potter and the Goblet of Fire", "description": "Mint condition, but will negotiate", "price": 99, "author": "J.K. Rowling", "datePublished": "Sun Oct 10 2021 23:56:34 GMT-0400" }, "errors": null } ``` The second is a message indicating to the client that the book with the specified ID does not exist. ```json { "status": "error", "code": 404, "message": "That book with the specified ID does not exist." "data": null, "errors": [ "Book listing not found." ] } ``` # Retrieving A Book By ID - Implementation ## 1 - Route Name We will first add a route of `GET /api/v1/books/:id`*.* *File: src/routes/book.route.js* ```javascript const express = require('express') const router = express.Router() const { getAllBooks, getBookById } = require('../controllers/book') router.get('/', getAllBooks) router.get('/:id', getBookById) // This is our newly added route module.exports = router ``` Followed by the controller setup. *File: src/controllers/books/getABookById.js* ```javascript const catchException = require('../utils/catchExceptions') const getBookById = catchException(async (req, res, next) => { // our code goes here... }) ``` ## 2 - Input Request None. ## 3 - Middleware None. ## 4 - Validation None. ## 5 - Domain The `getById()` method will look this, nice and simple. *File: src/domain/services/bookRepository.js* ```javascript // Retrieve - one const getById = async (id) => { return await Model.findById(id) } ``` Now for us to actually use the `bookRepository` in our `bookService`. *File: src/domain/services/bookService.js* ```javascript // Retrieve - one const getBookById = async (bookId) => { const book = await bookRepository.getById(bookId) return book } ``` ## 6 - Events None. ## 7 - Response But wait! let's not forget the 404 validation we promised. *File: src/domain/services/bookService.js* ```javascript // Retrieve - one const getBookById = async (bookId) => { if (!mongoose.Types.ObjectId.isValid(bookId)) { // the id is invalid throw new ApiException({ message: `the book with that id: ${bookId} does not exist.`, status: 'failed', code: 404, data: null, errors: [`the book with that id: ${bookId} does not exist.`] }) } const book = await bookRepository.getById(bookId) if (!book) { throw new ApiException({ message: `the book with that id: ${bookId} does not exist.`, status: 'failed', code: 404, data: null, errors: [`the book with that id: ${bookId} does not exist.`] }) } return book } ``` As usual, here is our controller. Thanks to us doing the business logic validation in our domain layer, our controller is thin and free of clutter. *File: src/controllers/book.controller.js* ```javascript const getBookById = catchException(async (req, res, next) => { const book = await bookService.getBookById(req.params.id) return res.json( globalResponseDTO({ status: 'success', code: 200, message: `Book with the specified id.`, data: book, errors: null }) ) }) ``` # Summary In this chapter, we used the 8 step recipe to plan out our endpoints for retrieving all books in the database and retrieving a single book by ID. We then proceeded to creating and implementing those endpoints in our RESTful API. Next, we will focus on the authentication endpoints. # Chapter 8: Adding Authentication to our API Now it is time for something more complicated, you'll find that this chapter is going to be a lot more engaging than the last one. Because we are dealing with input fields now, our 8 step recipe is going to look a lot more involved than before. Get ready, because this is going to be a long and exciting chapter! # Registering the User - Planning Refer back to [*Chapter 6, The Bookstore API Endpoint Specifications, API Endpoint #1: User Registration*](/part-3/chapter-6/1-the-bookstore-api-endpoint-specifications#api-endpoint-1-user-registration) section for the API endpoint specifications. ## 1 - Route Name Following the guidelines in chapter 6, [*Method Verbs*](/part-2/chapter-3/3-method-verbs) and [*URI Design*](/part-2/chapter-3/2-uri-design), we will create the following URI. Since it is a resource creation, using `POST` over any other method verbs would make sense here. As for the namespace, we will follow [*Guideline #4: Avoid actions and verbs in the URI in chapter 3, URI Design, URI Path Design*.](/part-2/chapter-3/2-uri-design#guideline-4-avoid-actions-and-verbs-in-the-uri) This means we should not use any verbs, but rather let the `POST` method verb speak for itself on what resource it will create. A lot of developers will make the mistake of making the URI something like this `/api/v1/auth/regster` or `/api/v1/users/create`. Don't do this as it is the exact opposite of the guideline we are trying to follow. Also, don't forget to always pluralize the resources your endpoints. In our case, the resource being `users`. The following URI is what we will be going with. `POST /api/v1/users` ## 2 - Input Request As stated from the endpoint specifications in the previous chapter, we will accept the following input fields from the client. * `first_name` * `last_name` * `email` * `password` * `password_confirmation` * `phone_number` ## 3 - Middleware Since this is a going to be a public user registering for an account, no middleware is necessary. However, since this chapter has a bunch of input validation, we'll be adding in a global middleware that will catch and handle all exceptions in our application. You will see more of this in the next section. ## 4 - Validation From the looks of the specification, it would seem like we'll need to perform the following validation rules for each of the fields. * `first_name` - required. * `last_name` - required. * `email` - required, must be a valid email format. * `password` - required, at least 6 characters. * `password_confirmation` - required, password and password\_confirmation must match. * `phone_number` - required, must be a valid phone number format. ## 5 - Domain From the specifications, it looks like we will need the following user entity in our domain layer. **userModel** * `first_name` * `last_name` * `email` * `password` * `phone_number` We will also need some sort of way to make a query to the database to insert the user in, so we'll create a `userRepository` layer with the method `createUser()` to do so. **userRepository** * `createUser()` On top of that, our controller will call a service to `registerUser()`, in this case, it will be the `authService` which will use the `userRepository` layer. We'll also need to keep in mind that we will have to do some sort of database validation here since there can only be one unique email in the users table. **authService** * `registerUser()` This is a rough outline of how we are going to be implementing these functions, we'll take a deeper dive at the implementations in the next section of this chapter. ***\*Note: for the purposes of demonstration in this book, we will not be hashing our passwords. If you want to deploy any sort of authentication system into production, always remember to hash your passwords.*** ## 6 - Events Send an email to the registered user's email letting them know that they have successfully registered. ## 7 - Response There are 4 possible responses we could possibly return from this endpoint. The first is a successful response in which we have successfully inserted the newly registered user into the database and fired off an email to that user. ```json { "status": "success", "code": 201, "message": "User has successfully been registered." "data": { "id": "61f889bbc6bbf81a97ba69d6", "first_name": "John", "last_name": "Doe", "email": "johndoe@email.com", "phone_number": "1234567890" }, "errors": null } ``` The second would be an error on an invalid input request payload. ```json { "status": "error", "code": 422, "message": "Input fields are of not the correct form.", "data": null, "errors": [ "This DTO's property is required: email.", "This DTO's property is required: password.", "This DTO's property is required: password_confirmation.", "This DTO's property is required: phone_number." ] } ``` *\*Note: I won't be mentioning any more DTO validations in the ensuing chapters as the response is quite routine for most API calls with an input request body. So for login user, create a book, and update a book API endpoints, it's implied that a DTO will yield a particular response depending on the input from the client.* The third would be an error on the form validation. ```json { "status": "error", "code": 400, "message": "There were errors with the validation.", "data": null, "errors": [ "The email format is invalid.", "The password confirmation and password fields must match." ] } ``` Lastly, the fourth would be a backend database validation checking if an email address is already taken. ```json { "status": "error", "code": 409, "message": "This email is already taken.", "data": null, "errors": ["This email is already taken."] } ``` # Registering the User - Implementation ## 1 - Route Name `GET /api/v1/users` Let's add an additional `users` route to the `routes/index.js file`. *File: src/routes/index.js* ```javascript const express = require('express') const router = express.Router() const userRoutes = require('./user.route') const bookRoutes = require('./book.route') function getRouter() { router.use('/books', bookRoutes) router.use('/users', userRoutes) // our new route return router } module.exports = getRouter ``` Here, we will then add a `registerUser` controller function. *File: src/routes/user.route.js* ```javascript const express = require('express') const router = express.Router() const isAuthenticated = require('../middleware/auth.middleware') const { registerUser } = require('../controllers/auth') router.post('/', registerUser) module.exports = router ``` We will then proceed to fill up the rest of the controller below. *File: src/controllers/auth.controller.js* ```javascript /** * Inserts the user into the database and fires off an email notification * to that user's email if successful. */ const registerUser = catchExceptions(async (req, res) => { // Our code goes here... }) ``` ## 2 - Input Request Now we will add in a custom DTO that will help us specify the contract between us and the client. This layer can be seem as a validation layer, but what we are ultimately trying to accomplish here is to get the frontend and backend aligned on the same page. Notice there is no real validation of having the `password` being a certain length or the `phone_number` being a certain format. The DTO simply helps the client who's calling our API to provide the correct structure in the payload. *File: src/requests/registerUserDTO.js* ```javascript const ApiException = require('../utils/ApiException') const fields = [ 'first_name', 'last_name', 'email', 'password', 'password_confirmation', 'phone_number' ] /** * @param Object data */ const registerUserRequestDto = (data) => { const errors = [] fields.forEach((field) => { if (!(field in data)) { errors.push(`This DTO's property is required: ${field}.`) } }) if (errors.length > 0) { throw new ApiException({ status: 'error', code: 422, message: 'Input fields are not in the correct form.', data: null, errors }) } return data } module.exports = registerUserRequestDto ``` ## 3 - Middleware As stated in the previous section, we will be adding in a global middleware to help us catch errors. You may have noticed a utility wrapper being used in the controllers up until now called `catchExceptions`. *File: src/controllers/auth.controller.js* ```javascript const registerUser = catchExceptions(async (req, res) => { // Our code goes here... }) ``` *File: src/utils/catchExceptions.js* ```javascript const catchExceptions = (func) => { return (req, res, next) => { Promise.resolve(func(req, res, next)).catch(next) } } module.exports = catchExceptions ``` The reason for this is due to our `globalErrorHandler` that we'll be putting as follows. *File: src/server.js* ```javascript const globalErrorHandler = require('./utils/globalErrorHandler') app.use(globalErrorHandler) ``` *File: src/utils/globalErrorHandler.js* ```javascript const globalResponseDTO = require('../responses/globalResponseDTO') const globalErrorHandler = async (err, req, res, next) => { console.log('===============================') console.log('Global Error Catcher:', err.name) console.log('===============================') if (err.name === 'ApiException') { console.error('ApiException', err) res.status(err.code).json( globalResponseDTO({ status: err.status, code: err.code, message: err.message, data: err.data, errors: err.errors }) ) } else { console.error('Other Error', err) } } module.exports = globalErrorHandler ``` This is really going to make our lives easier, because every time we throw an `ApiException` anywhere in our application, it is going to automatically catch the error we have thrown and output it out as JSON output. ## 4 - Validation Next is the form validation, we'll be using the [*validatorjs library*](https://www.npmjs.com/package/validatorjs) to help us achieve the validation rules we specified in our planning section of this endpoint. *File: src/validators/registerUserValidator.js* ```javascript const Validator = require('validatorjs') const ApiException = require('../utils/ApiException') /** * @param {*} data { * - first_name * - last_name * - email * - password * - password_confirm * - phone_number * } * * @returns Validator */ const registerUserValidator = (data) => { const rules = { first_name: 'required', last_name: 'required', email: 'required|email', password: 'required|min:6', password_confirmation: 'required|min:6|same:password', phone_number: 'required|telephone' } const validator = new Validator(data, rules) if (validator.fails()) { let errors = [] for (const field in validator.errors.errors) { errors = errors.concat(validator.errors.errors[field]) } throw new ApiException({ status: 'error', code: 422, message: 'There were errors with the validation.', data: null, errors }) } return validator } // This is our custom 'telephone' validation rule Validator.register( 'telephone', function (value) { return value.match(/^\(?([0-9]{3})\)?[-.●]?([0-9]{3})[-.●]?([0-9]{4})$/) }, 'The :attribute field is not in a correct format.' ) module.exports = registerUserValidator ``` ## 5 - Domain For our domain layer, we'll first create our user model as follows. *File: src/domain/models/user.model.js* ```javascript const mongoose = require('mongoose') const Schema = mongoose.Schema const userModel = new Schema({ first_name: { type: String, required: true }, last_name: { type: String, required: false }, email: { type: String, required: true, unique: true }, password: { type: String, required: true }, phone_number: { type: String, required: true }, createdAt: { type: Date, default: Date.now } }) module.exports = mongoose.model('user', userModel) ``` Then, we will create the `createUser` method, which will simply save the `userData` into our database. *File: src/domain/services/userRepository.js* ```javascript const UserModel = require('../models/user.model') /** * * @param {*} user { * - name * - email * - password * } * * @returns user */ const createUser = async (userData) => { const user = new UserModel(userData) const userReturn = await user.save(userData) return userReturn } module.exports = { createUser } ``` Lastly, we will create the `authService` layer and use our `userRepository.createUser()` utility to create our service for registering our user. *File: src/domain/services/authService.js* ```javascript /** * @returns User */ const registerUser = async (user) => { try { const createdUser = await userRepository.createUser(user) return createdUser } catch (err) { if (err.code === 11000 && err?.keyPattern?.email === 1) { throw new ApiException({ message: `This email is already taken.`, status: 'error', code: 409, data: null, errors: [`This email is already taken.`] }) } } } ``` Notice how we do a try-catch here where we throw a custom `ApiException` in order to catch a specific type of error, in this case it's that the email must be unique. ## 6 - Events As noted in the planning section of this endpoint, when the user has successfully registered, an event will fire off. We can take advantage of Node.js' built-in `eventEmitter` to create a pub-sub structure for us. *File: src/events/index.js* ```javascript const events = require('events') const eventEmitter = new events.EventEmitter() const userHasRegistered = require('./userHasRegisteredEvent') eventEmitter.on('userHasRegistered', userHasRegistered) ``` *File: src/events/userHasRegisteredEvent.js* ```javascript const mailer = require('../utils/mailer') /** * * @param {*} user * * @returns boolean */ const userHasRegisteredEvent = (user) => { let bodyText = `Hello ${user.firstName}. Thanks for registering!` return mailer.sendEmailToUser(user.email, 'Welcome aboard!', bodyText) } module.exports = userHasRegisteredEvent ``` Notice here that we are not really going to be sending any real emails, we will just be stubbing it out. Of course, if you really wanted to, I suggest using [SendGrid](https://sendgrid.com) or [Amazon SES](https://aws.amazon.com/ses). *File: src/utils/mailer.js* ```javascript /** * Sends an email to a user, if it was successfully sent, then return true, else return false * * @param {*} toEmail * @param {*} subject * @param {*} bodyText * * @returns boolean */ const sendEmailToUser = () => { return false } module.exports = { sendEmailToUser } ``` ## 7 - Response Recall in the previous section [*Registering the User - Planning*](/part-3/chapter-8/1-registering-the-user-planning#7-response). We indicated that there were going to be 4 different responses. Remember, our `globalErrorHandler` is able to catch all these `ApiException`s. The first, ***DTO validation*** is triggered when we throw the following `ApiException`. *File: src/requests/registerUserDTO.js* ```javascript throw new ApiException({ status: 'error', code: 422, message: 'Input fields are of not the correct form.', data: errors }) ``` ```json { "status": "error", "code": 422, "message": "Input fields are of not the correct form.", "data": null, "errors": [ "This DTO's property is required: email.", "This DTO's property is required: password.", "This DTO's property is required: password_confirmation.", "This DTO's property is required: phone_number." ] } ``` The second, ***form validation*** is triggered when we throw the following `ApiException`. *File: src/validators/registerUserValidator.js* ```javascript throw new ApiException({ status: 'error', code: 422, message: 'There were errors with the validation.', data: null, errors }) ``` ```json { "status": "error", "code": 422, "message": "There were errors with the validation.", "data": null, "errors": [ "The email format is invalid.", "The password confirmation and password fields must match." ] } ``` The third, **service layer validation** in which we check if email already taken, is triggered when we throw the following `ApiException`. *File: src/domain/services/authService.js* ```javascript /** * @returns User */ const registerUser = async (user) => { try { const createdUser = await userRepository.createUser(user) return createdUser } catch (err) { if (err.code === 11000 && err?.keyPattern?.email === 1) { throw new ApiException({ message: `This email is already taken.`, status: 'error', code: 409, data: null, errors: [`This email is already taken.`] }) } } } ``` ```json { "status": "error", "code": 409, "message": "This email is already taken.", "data": null, "errors": ["This email is already taken."] } ``` Finally, the last response, which is the **success** response. If everything goes through and there are no errors, then we will return a `User` object. For that, we will use a DTO as follows. *File: src/responses/userResponseDTO.js* ```javascript const userResponseDTO = (user) => { return { id: user['id'], first_name: user['first_name'], last_name: user['last_name'], email: user['email'], phone_number: user['phone_number'] } } module.exports = userResponseDTO ``` ```json { "status": "success", "code": 201, "message": "The email: yichenzhu13371@email.com has successfully registered.", "data": { "id": "61f889bbc6bbf81a97ba69d6", "first_name": "Yichen", "last_name": "Zhu", "email": "yichenzhu13371@email.com", "phone_number": "1234567890" }, "errors": null } ``` ## Putting It All Together If you've followed a long, this is how our controller should look like by the end. Nice and clean :sunglasses:. ```javascript /** * Inserts the user into the database and fires off an email notification * to that user's email if successful. */ const registerUser = catchExceptions(async (req, res) => { // request const registerUserRequest = registerUserRequestDTO(req.body) // validation registerUserValidator(registerUserRequest) // domain logic const user = await authService.registerUser(registerUserRequest) // events eventEmitter.emit('userHasRegistered', user) // response - success return res.json( globalResponseDTO({ status: 'success', code: 201, message: `The email: ${registerUserRequest.email} has successfully registered.`, data: userResponseDTO(user), errors: null }) ) }) ``` # Logging the User In - Planning Refer back to [*Chapter 6, The Bookstore API Endpoint Specifications, API Endpoint #2: User Login*](/part-3/chapter-6/1-the-bookstore-api-endpoint-specifications#api-endpoint-2-user-login) section for the API endpoint specifications. ## 1 - Route Name For logging the user into our application, I propose the following URI. `POST /api/v1/auth/login` Now if you've been paying close attention, you'll notice that we've been following the guidelines from chapter 6, [*Method Verbs*](/part-2/chapter-3/3-method-verbs) and [*URI Design*](/part-2/chapter-3/2-uri-design) very closely. In this particular case, two questions come to mind. 1. We are not creating a resource, so which method verb should we use? 2. For the URI, [*Guideline #4: Avoid actions and verbs in the URI in chapter 3, URI Design, URI Path Design*](/part-2/chapter-3/2-uri-design#guideline-4-avoid-actions-and-verbs-in-the-uri) tells us that we should not use verbs in our URI, in this case, we are using "login". To answer the first question, it's simple, we can use either POST or PUT. As long as the method verb we are using has some sort of way for us to pass a payload to the server. In terms of semantics, I would argue either is fine, both can be used. As for the second question, the guideline is to avoid action verbs. In this case, we will be pragmatic, but also follow the rule of thumb, which is to use an action verb only when a URI **does not adhere** to a CRUD operation. In our case a login action does not creating any new resource, therefore it is fine to put that in our URI. ## 2 - Input Request As stated from the endpoint specifications in the previous chapter, we will accept the following input fields from the client. * `email` * `password` ## 3 - Middleware Since the user doesn't need to be logged in, we can leave the middleware part out. ## 4 - Validation From the looks of the specification, it would seem like we will need to perform the following validation rules for each of the fields. * `email` - required, must be a valid email format. * `password` - required, at least 6 characters. ## 5 - Domain Since we already created the `userModel`, we'll just be building on top of that. First, we'll need some sort of way to do a check in our database on whether or not a row exists with the given email and password. **userRepository** * `findUserByEmailAndPassword()` Then, we'll have some sort of `authService` to call the `findUserByEmailAndPassword()` in `userRepository`. We'll fit this in a `loginUser` function. Don't forget, we'll need to check weather the credentials are valid or not based on `findUserByEmailAndPassword()`. **authService** * `loginUser()` This is a rough outline of how we are going to be implementing these functions, we'll take a deeper dive at the implementations in the next section of this chapter. ## 6 - Events There are no events based on the specifications in the previous chapter. ## 7 - Response There are 3 possible responses that we can output from this API. The first is from the form validation. ```json { "status": "error", "code": 400, "message": "There were errors with the validation", "data": null, "errors": ["The email format is invalid."] } ``` The second is when we do a check from our `authService` seeing if we could find a possible row in the database with a given pair of email and password. ```json { "status": "error", "code": 400, "message": "Invalid credentials, please try a different email and password combination.", "data": null, "errors": [ "Invalid credentials, please try a different email and password combination." ] } ``` The third is if all validation passes, we'll log the user in and output the user in a success response. ```json { "status": "success", "code": 200, "message": "The user has successfully logged in.", "data": { "id": "61f888043b95e61830a98da0", "first_name": "Yichen", "last_name": "Zhu", "email": "yichen@yichen.com", "phone_number": "1234567890" }, "errors": null } ``` # Logging the User In - Implementation ## 1 - Route Name Let's add the route and the controller in first. `POST /api/v1/auth/login` *File: src/routes/index.js* ```javascript const express = require('express') const router = express.Router() const userRoutes = require('./user.route') const authRoutes = require('./auth.route') function getRouter() { router.use('/users', userRoutes) router.use('/auth', authRoutes) // our new route return router } module.exports = getRouter ``` *File: src/routes/auth.route.js* ```javascript const express = require('express') const router = express.Router() const { logUserIn } = require('../controllers/auth') router.post('/login', logUserIn) module.exports = router ``` *File: src/controllers/auth/logUserIn.js* ```javascript const catchExceptions = require('../utils/catchExceptions') const logUserIn = catchExceptions(async (req, res, next) => { // Code goes here... }) ``` ## 2 - Input Request Next we'll add in the DTO, let's call it `loginUserRequestDto`. ```javascript const ApiException = require('../utils/ApiException') const fields = ['email', 'password'] const loginUserRequestDto = (data) => { const errors = [] fields.forEach((field) => { if (!(field in data)) { errors.push(`This DTO's property is required: ${field}.`) } }) if (errors.length > 0) { throw new ApiException({ status: 'error', code: 422, message: 'loginUserRequestDto failed.', data: null, errors }) } return data } module.exports = loginUserRequestDto ``` ## 3 - Middleware None. ## 4 - Validation Now time for the validator. ```javascript const Validator = require('validatorjs') const ApiException = require('../utils/ApiException') /** * @param {*} data { * - email * - password * } * * @returns Validator */ const loginUserValidator = (data) => { const rules = { email: 'required|email', password: 'required' } const validator = new Validator(data, rules) if (validator.fails()) { let errors = [] for (const field in validator.errors.errors) { errors = errors.concat(validator.errors.errors[field]) } throw new ApiException({ message: 'There were errors with the validation', status: 'error', code: 400, data: null, errors: validator.errors.errors }) } return validator } module.exports = loginUserValidator ``` ## 5 - Domain For checking if there is a match in our database, we'll use our `userModel` that was created in the last section. We'll add a new method called `findUserByEmailAndPassword` which will simply use our `userModel` to do a find in our database. *File: src/domain/services/userRepository.js* ```javascript const UserModel = require('../models/user.model') /** * * @param {*} user { * - name * - email * - password * } * * @returns user */ const findUserByEmailAndPassword = async (userData) => { const foundUser = await UserModel.findOne(userData) return foundUser } ``` Once we have database query done, we'll add a new method called `loginUser` and use it in our `authService`*.* Notice here that we throw and exception if we do not find any users in the database. This would mean that the client's request has failed. *File: src/domain/services/authService.js* ```javascript /** * @returns user */ const loginUser = async (user) => { const loginUser = await userRepository.findUserByEmailAndPassword(user) if (!loginUser) { throw new ApiException({ status: 'error', code: 400, message: `Invalid credentials, please try a different email and password combination.`, data: null, errors: [ `Invalid credentials, please try a different email and password combination.` ] }) } return loginUser } ``` ## 6 - Events None. ## 7 - Response Now to put everything all together. We'll reuse our `userResponseDto` from last section and log the user into our session with a simple `req.session.user`. Once again, our code looks nice and clean :sunglasses:. ```javascript /** * Logs the user in and set a session for it. */ const logUserIn = catchExceptions(async (req, res) => { const loginUserRequest = loginUserRequestDto(req.body) loginUserValidator(loginUserRequest) // if the user's email and password match in our database // then set the current session to that user const loggedInUser = await authService.loginUser(loginUserRequest) // If there we find a user with authService.loginUser, then // we'll set the current session to that user req.session.user = loggedInUser const userDto = userResponseDto(loggedInUser) res.status(200).json( globalResponseDto({ status: 'success', code: 200, message: `The user has successfully logged in.`, data: userDto, errors: null }) ) }) module.exports = logUserIn ``` # Getting Authenticated User - Planning Refer back to [*Chapter 6, The Bookstore API Endpoint Specifications, API Endpoint #4: Get Authenticated User*](/part-3/chapter-6/1-the-bookstore-api-endpoint-specifications#api-endpoint-4-get-authenticated-user) section for the API endpoint specifications. ## 1 - Route Name Now things are starting to get interesting, because recall that we just broke [*Guideline #4: Avoid actions and verbs in the URI in chapter 6, URI Design, URI Path Design*](/part-2/chapter-3/2-uri-design#guideline-4-avoid-actions-and-verbs-in-the-uri) for our login route. Is it time to do the same thing here as well? Well if you were thinking of doing something like this... `GET /api/v1/auth/user`, you, would not be the only one. Remember what talked about? Always think of the URIs as resources and entities that we can fetch. In this case, I think it's best to create the URI as follows. **`GET /api/v1/users/auth`** This would suffice many different guidelines we proposed in chapter 3. This would let us think of `users` as the resource, which means it would then be pluralized. This would also let us have the option of creating a dynamic URI such as this, `GET /api/v1/users/:user_id` in the future if we were to extend this endpoint. ## 2 - Input Request None. ## 3 - Middleware We are finally ready to write our first middleware! For this endpoint, I'm thinking we should write a basic auth middleware that will check if there exists an authenticated user currently. If the current user is not logged in, then return a response with a 401 status code. Otherwise, business as usual, the request will go through. ## 4 - Validation None. ## 5 - Domain You'll see in the next section that since we are using Express and the `express-session` library, fetching the currently authenticated user is quite simple, there's no real reason to make any database calls. ## 6 - Events None. ## 7 - Response There are two possible responses from this endpoint. The first is if the user is not logged in. This will be caught somewhere in our middleware. For future use cases, we will be using the same middleware for creating, updating, and deleting a book listing. ```json { "status": "error", "code": 401, "message": "Access denied: you must be logged in to access this API endpoint.", "data": null, "errors": ["You must be logged in."] } ``` And if the user is logged in, we will get the user's information in the response. ```json { "status": "success", "code": 200, "message": "The currently authenticated user's information.", "data": { "first_name": "Yichen", "last_name": "Zhu", "email": "yichen@yichen.com", "phone_number": "1234567890" }, "errors": null } ``` # Getting Authenticated User - Implementation ## 1 - Route Name We'll firs start off by adding the route and the controller in. `GET /api/v1/users/auth` *File: src/routes/user.route.js* ```javascript const express = require('express') const router = express.Router() const isAuthenticated = require('../middleware/auth.middleware') const { registerUser, getAuthUser } = require('../controllers/auth') router.post('/', registerUser) router.post('/auth', getAuthUser) // This is our new route module.exports = router ``` *File: src/controllers/user/getAuthUser.js* ```javascript const catchExceptions = require('../../utils/catchExceptions') /** * Gets the currently authenticated user in the current session. */ const getAuthUser = catchExceptions((req, res) => { // our code goes here... }) module.exports = getAuthUser ``` ## 2 - Input Request None. ## 3 - Middleware Now it's time to add our first middleware. This will essentially protect any route against unauthenticated users. *File: src/middleware/auth/auth.middleware.js* ```javascript const globalResponseDto = require('../responses/globalResponseDto') const isAuthenticated = (req, res, next) => { if (!req.session.user) { return res.status(401).json( globalResponseDto({ status: 'error', code: 401, message: 'Access denied: you must be logged in to access this API endpoint.', data: null, errors: ['You must be logged in.'] }) ) } next() } module.exports = isAuthenticated ``` We can then add this `isAuthenticated` middleware to our `GET /users/auth` route as follows. Remember that this is a reusable middleware, we'll be using this all throughout the next chapter. *File: src/controllers/auth/getAuthUser.controller.js* ```javascript const { getAuthUser } = require('../controllers/auth') router.get('/auth', isAuthenticated, getAuthUser) ``` ## 4 - Validation None. ## 5 - Domain None, but because we are using the `express-session` library, we can simply just retrieve the user from the `req` object given to us in our controller, see below. ## 6 - Events None. ## 7 - Response Putting it all together, we get the follow. Do note also that we are reusing our `userResponseDto` to help us out with outputting the correct user information to our client. *File: src/controllers/user/getAuthUser.js* ```javascript const globalResponseDto = require('../../responses/globalResponseDto') const userResponseDto = require('../../responses/userResponseDto') const catchExceptions = require('../../utils/catchExceptions') /** * Gets the currently authenticated user in the current session. */ const getAuthUser = catchExceptions((req, res) => { const user = req.session.user // This is essentially our logged in user res.status(200).json( globalResponseDto({ status: 'success', code: 200, message: `The currently authenticated user's information.`, data: userResponseDto(user), errors: null }) ) }) module.exports = getAuthUser ``` # Summary Phew\... now that was a long and difficult chapter. In this chapter, we both planned and implemented all of the user authentication endpoints of our RESTful API. We added user registration, login, and current authenticated user to our API. In the next chapter, we'll be going over the rest of the CRUD operations for our book listings endpoints, specifically creating, updating, and deleting book listings. # Chapter 9: Adding the Create, Update, and Delete Operations to our API In the previous 2 chapters, we implemented most of the functionalities for our Bookstore API. In this chapter, we'll be finishing off the last 3 endpoints related to the book listings. Since we've already had plenty of practice, there will be less explanations on guidelines and best practices, we'll jump straight to the point when planning and implementing our endpoints. # Creating A Book Listing - Planning Refer back to [*Chapter 6, The Bookstore API Endpoint Specifications, API Endpoint #7: Create a Book Listing* ](/part-3/chapter-6/1-the-bookstore-api-endpoint-specifications#api-endpoint-8-update-a-book-listing)section for the API endpoint specifications. ## 1 - Route Name Referring back to the sections under chapter 3, [*Method Verbs*](/part-2/chapter-3/3-method-verbs) and [*URI Design*](/part-2/chapter-3/2-uri-design), we can continue to build upon our `books` namespace. In fact we'll be using that for the rest of the chapter. We'll use `POST` as our method verb of choice since we are creating a resource, and this is what we'll be sticking with as our route name. `POST /api/v1/books` ## 2 - Input Request From the specifications, these are the fields that will be input to us by the user. * `title` * `description` * `price` * `author` * `datePublished` ## 3 - Middleware This middleware part of this feature will be quite simple. Recall in the previous chapter, when we went over the section [*Getting Authenticated User - Implementation*](/part-3/chapter-8/6-getting-authenticated-user-implementation#3-middleware). We built ourselves a nice little helper that will let us guard against requests from users who are not authenticated in our application. We'll be reusing that middleware for this feature and for the rest of the remaining sections of this chapter. ## 4 - Validation From the looks of the specification, it would seem like we will need to perform the following validation rules for each of the fields. * `title` - required. * `description` - required. * `price` - required, numeric, minimum number is 1. * `author` - required. * `datePublished`- required, valid date format. ## 5 - Domain If you've been paying attention in the previous 2 chapters of the book, you'll probably guess what 3 files we are going to create. From the specifications, it looks like we will need the following book entity in our domain layer. **bookModel** * `title` * `description` * `price` * `author` * `datePublished` We'll also need some sort of way to make a query to the database to insert the book in, so we'll create a `bookRepository` layer with the method `create()` to do so. **bookRepository** * `create()` On top of that, our controller will call a service to `createBook()`, in this case, it will be the `bookService` which will use the `userRepository` layer. **bookService** * `createBook()` This is a rough outline of how we are going to be implementing these functions, we'll take a deeper dive at the implementations in the next section of this chapter. ## 6 - Events None. ## 7 - Response There are 3 possible responses that can be output with this endpoint. The first is the authentication middleware that we'll be adding in. ```json { "status": "error", "code": 401, "message": "Access denied: you must be logged in to access this API endpoint.", "data": null, "errors": ["You must be logged in."] } ``` The second has to deal with whether or not the client has put in the correct fields to pass the form validation. ```json { "status": "error", "code": 400, "message": "There were errors with the validation.", "data": null, "errors": ["The price must be at least 1.", "The author field is required."] } ``` The third, of course, is the successful scenario. ```json { "status": "success", "code": 200, "message": "Book has successfully been added to the database.", "data": { "id": "620066e1f84bfd64e7125830", "title": "'Harry Potter and the Awesome Book of Nothing", "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.", "price": 100, "author": "J.K. Rowling", "datePublished": "December 25, 2010" }, "errors": null } ``` # Creating A Book Listing - Implementation ## 1 - Route Name `POST /api/v1/books` Just like we planned, let's add in the route. *File: src/routes/book.route.js* ```javascript const express = require('express') const router = express.Router() const { getAllBooks, getBookById, createABook } = require('../controllers/book') router.get('/', getAllBooks) router.get('/:id', getBookById) // This is the new route we are adding in router.post('/', createABook) ``` Let's also create our controller so we can fill in the details later on. *File: src/controllers/book/createABook.js* ```javascript const catchException = require('../../utils/catchExceptions') /** * Creates a new book listing. */ const createABook = catchException(async (req, res) => { // we'll fill in the details after we get each of the other layers ready }) module.exports = createABook ``` ## 2 - Input Request Very similar to how we implemented our `registerUserRequestDto`, we'll create a `createBookRequestDto` that has the fields *title, description, price, author,* and *datePublished* in it. *File: src/requests/createBookRequestDto.js* ```javascript const ApiException = require('../utils/ApiException') const fields = ['title', 'description', 'price', 'author', 'datePublished'] const createBookRequestDto = (data) => { const errors = [] fields.forEach((field) => { if (!(field in data)) { errors.push(`This DTO's property is required: ${field}.`) } }) if (errors.length > 0) { throw new ApiException({ status: 'error', code: 422, message: 'Input fields are of not the correct form.', data: null, errors }) } return data } module.exports = createBookRequestDto ``` ## 3 - Middleware We are going to be reusing the `isAuthenticated` middleware from before and add it in right before our controller. *File: src/routes/book.route.js* ```javascript const express = require('express') const router = express.Router() const { createABook } = require('../controllers/book') const isAuthenticated = require('../middleware/auth.middleware') // The isAuthenticated middleware is going to protect this route // from unauthenticated users from accessing it router.post('/', isAuthenticated, createABook) ``` ## 4 - Validation Let's not forget about the validation layer. Again, this is very similar to the `registerUserValiator` function we created in the previous chapter except it's just going to be for different fields. *File: src/validators/createBookValidator.js* ```javascript const Validator = require('validatorjs') const ApiException = require('../utils/ApiException') /** * @param {*} data { * - title * - description * - price * - author * - datePublished * } * * @returns Validator */ const createBookValidator = (data) => { const rules = { title: 'required', description: 'required', price: 'required|numeric|min:1', author: 'required', datePublished: 'required' } const validator = new Validator(data, rules) if (validator.fails()) { let errors = [] for (const field in validator.errors.errors) { errors = errors.concat(validator.errors.errors[field]) } throw new ApiException({ message: 'There were errors with the validation', status: 'error', code: 400, data: null, errors }) } return validator } module.exports = createBookValidator ``` ## 5 - Domain As mentioned in the previous planning section, this will be the model we create. *File: src/domain/models/book.model.js* ```javascript const mongoose = require('mongoose') const Schema = mongoose.Schema const bookModel = new mongoose.Schema({ userId: { type: Schema.Types.ObjectId, ref: 'user' }, title: { type: String, required: true }, description: { type: String }, price: { type: Number, required: true }, author: { type: String }, datePublished: { type: String }, createdAt: { type: Date, default: Date.now } }) module.exports = mongoose.model('book', bookModel) ``` This will be the repository layer on top of that model. *File: src/domain/repositories/book.repository.js* ```javascript const Model = require('../models/book.model') // Saves a book in the database const create = async (newBook) => { const book = new Model(newBook) return await book.save() } module.exports = { create } ``` Finally, our service layer that will use the `bookRepository` layer. *File: src/domain/services/book.service.js* ```javascript const bookRepository = require('../repositories/book.repository') const ApiException = require('../../utils/ApiException') const mongoose = require('mongoose') // Create a book const createBook = async (book) => { return bookRepository.create(book) } module.exports = { createBook } ``` ## 6 - Events None. ## 7 - Response If we put this all together now in the controller, this is what we'll get. ```javascript const mongoose = require('mongoose') const globalResponseDto = require('../../responses/globalResponseDto') const createBookRequestDto = require('../../requests/createBookRequestDto') const catchException = require('../../utils/catchExceptions') const bookService = require('../../domain/services/book.service') const bookResponseDto = require('../../responses/bookResponseDto') const createBookValidator = require('../../validators/createBookValidator') /** * Creates a new book listing. */ const createABook = catchException(async (req, res) => { const createBookRequest = createBookRequestDto({ id: req.session.user.id, ...req.body }) createBookValidator(createBookRequest) const book = await bookService.createBook({ userId: mongoose.Types.ObjectId(req.session.user._id), ...req.body }) res.status(200).json( globalResponseDto({ status: 'success', code: 200, message: `Book has successfully been added to the database.`, data: bookResponseDto(book), errors: null }) ) }) module.exports = createABook ``` The `createBookRequestDto` and the `createBookValidator` will naturally throw an `ApiException` which will yield the correct output response as we stated previously in the planning section of this endpoint. In case of no errors and we get a successful pass, we will reuse our `bookResponseDto` from which we created previously on the output of the `bookService.create` method we just implemented and return the newly created book in our response. # Updating A Book Listing By ID - Planning Refer back to [*Chapter 6, The Bookstore API Endpoint Specifications, API Endpoint #8: Update a Book Listing*](/part-3/chapter-6/1-the-bookstore-api-endpoint-specifications#api-endpoint-8-update-a-book-listing) section for the API endpoint specifications. Note that updating a book is quite similar to creating a book, so most of the planning here in this section will be quite similar to the last one. ## 1 - Route Name Referring back to the sections under chapter 3, [*Method Verbs*](/part-2/chapter-3/3-method-verbs) and [*URI Design*](/part-2/chapter-3/2-uri-design), we can continue to build upon our `books` namespace. We'll use `PUT` as our method verb of choice since we are updating a resource, and we'll also put the `:id` as one of the URI parameters. `PUT /api/v1/books/:id` ## 2 - Input Request Since updating a book will have the same fields as creating one, these are the fields that will be input to us by the user. * `title` * `description` * `price` * `author` * `datePublished` ## 3 - Middleware There are two layers of middleware that we will add here. The first reusing our `isAuthenticated` helper that will protect us from unauthenticated user from accessing this endpoint. The second will be a backend service validation check where we'll be checking if the currently authenticated user who is requesting to update said book is fact the owner of that book. There is also one more thing I would like to add into this middleware, which is a check on whether or not the client's request book ID exists. You'll see how this will be done in implementation section of this endpoint. You might be wondering why we didn't put this logic somewhere in our domain layer. Well, we could if we wanted to, there wouldn't have been anything wrong with it. Remember that middleware is flexible and can be used in many different ways. ## 4 - Validation Same as creating a book, we will need to perform the following validation rules for each of these fields. * `title` - required. * `description` - required. * `price` - required, numeric, minimum number is 1. * `author` - required. * `datePublished`- required, valid date format. ## 5 - Domain We'll built on top of the previous section and use the **bookModel**. We'll also need some sort of way to make a query to the database in order to find the existing book and update it, so we'll use the `bookRepository` layer and add the method `updateById()` to do so. **bookRepository** * `updateById()` On top of that, our controller will call a service to `updateBookById()`, in this case, it will be the `bookService` which will use the `bookRepository` layer. **bookService** * `updateBookById()` ## 6 - Events None. ## 7 - Response Again, very similar to the previous section, except there will be 4 possible responses that can be output with this endpoint. The first is the authentication middleware that we'll be reusing. ```json { "status": "error", "code": 401, "message": "Access denied: you must be logged in to access this API endpoint.", "data": null, "errors": ["You must be logged in."] } ``` The second has to deal with whether or not the client has put in the correct fields to pass the form validation. ```json { "status": "error", "code": 400, "message": "There were errors with the validation", "data": null, "errors": ["The price must be at least 1.", "The author field is required."] } ``` The third is whether or not the client has inputted an existing book ID. ```json { "status": "error", "code": 404, "message": "That book with the specified ID does not exist." "data": null, "errors": [ "Book listing not found." ] } ``` The fourth is the successful scenario. Do keep in mind that since we are using a `PUT` request, we'll be returning the entire representation back, meaning that the update response should have the exact same fields as the create response. ```json { "status": "success", "code": 200, "message": "The book has successfully been updated.", "data": { "id": "620066e1f84bfd64e7125830", "title": "'Harry Potter and the Awesome Book of Nothing", "description": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.", "price": 100, "author": "J.K. Rowling", "datePublished": "December 25, 2010" }, "errors": null } ``` # Updating A Book Listing By ID - Implementation ## 1 - Route Name `PUT /api/v1/books/:id` Just like we planned, let's add in the route. *File: src/routes/book.route.js* ```javascript const express = require('express') const router = express.Router() const { getAllBooks, getBookById, createABook, updateABook } = require('../controllers/book') router.get('/', getAllBooks) router.get('/:id', getBookById) router.post('/', createABook) // This is the new route we are adding in router.put('/:id', updateABook) ``` Let's also create our controller so we can fill in the details later on. *File: src/controllers/book/updateABook.js* ```javascript const catchException = require('../../utils/catchExceptions') /** * Updates an existing book listing by id. */ const updateABook = catchException(async (req, res) => { // we'll fill in the details after we get each of the other layers ready }) module.exports = updateABook ``` ## 2 - Input Request For the input request, this will seem very familiar. It's because this is the exact code we used for our `createBookRequestDto.js` with the exception of an additional `id` field. *File: src/controllers/requests/updateBookRequestDto.js* ```javascript const ApiException = require('../utils/ApiException') const fields = [ 'id', 'title', 'description', 'price', 'author', 'datePublished' ] const updateBookRequestDto = (data) => { const errors = [] fields.forEach((field) => { if (!(field in data)) { errors.push(`This DTO's property is required: ${field}.`) } }) if (errors.length > 0) { throw new ApiException({ status: 'error', code: 422, message: 'Input fields are of not the correct form.', data: null, errors }) } return data } module.exports = updateBookRequestDto ``` ## 3 - Middleware *File: src/routes/book.route.js* ```javascript // We'll be adding in the isAuthenticated middleware and also our // new middleware, the bookPermission middleware router.put('/:id', isAuthenticated, bookPermission, updateABook) ``` As stated previously, we need a way to check if the request book to be updated does indeed belong to the currently authenticated user. This will help us do so. Notice also that we are going to be using the `bookService.getBookId` method we created previously. This is a great functionality to reuse because the `getBookById` inherently has a check to see if the `bookId` is correct, otherwise it will throw an exception. *File: src/middleware/bookPermission.middleware.js* ```javascript const globalResponseDto = require('../responses/globalResponseDto') const catchExceptions = require('../utils/catchExceptions') const bookService = require('../domain/services/book.service') const bookPermission = catchExceptions(async (req, res, next) => { // When updating or deleting a book, the book must belong to the user that created it const bookId = req.params.id const book = await bookService.getBookById(bookId) // check to see if the current authenticated user's the owner of the // requested bookId if (req.session.user._id !== book.userId.toString()) { res.status(401).json( globalResponseDto({ status: 'error', code: 401, message: 'Access denied: you must be the owner of this book when updating or deleting it.', data: null, errors: [ 'Access denied: you must be the owner of this book when updating or deleting it.' ] }) ) } next() }) module.exports = bookPermission ``` ## 4 - Validation Again, this is very similar to the `createBookValidator` with the exception of an additional `id` field. You are probably wondering why we require all fields instead of a subset of them. Do recall that this is a `PUT` request, which means we must accept the entire resource into our API as we are re-updating the entire entity in database. *File: src/validators/updateBookValidator.js* ```javascript const Validator = require('validatorjs') const ApiException = require('../utils/ApiException') /** * @param {*} data { * - id * - title * - description * - price * - author * - datePublished * } * * @returns Validator */ const updateBookValidator = (data) => { const rules = { title: 'required', description: 'required', price: 'required|numeric|min:1', author: 'required', datePublished: 'required' } const validator = new Validator(data, rules) if (validator.fails()) { let errors = [] for (const field in validator.errors.errors) { errors = errors.concat(validator.errors.errors[field]) } throw new ApiException({ message: 'There were errors with the validation', status: 'error', code: 400, data: null, errors }) } return validator } module.exports = updateBookValidator ``` ## 5 - Domain Yet again, this is similar to what we had before when we wrote the `create` method in our `bookRepository` and the `createBook` method in our `bookService`. We now do this for updating a book. *File: src/domain/repositories/book.repository.js* ```javascript const Model = require('../models/book.model') // Update const updateById = async (id, book) => { const updatedBook = await Model.findByIdAndUpdate(id, book) return updatedBook } ``` *File: src/domain/services/book.service.js* ```javascript const bookRepository = require('../repositories/book.repository') // Update a book const updateBookById = async (book) => { const newlyUpdatedBook = await bookRepository.updateById(book.id, book) return newlyUpdatedBook } ``` ## 6 - Events None. ## 7 - Response Now for the response. Do note that our `isAuthenticated` and `bookPermission` middleware will handle most of the error messages for us. For our success message we will put have the following in our controller. *File: src/controllers/book/updateABook.js* ```javascript const catchException = require('../../utils/catchExceptions') const globalResponseDto = require('../../responses/globalResponseDto') const updateBookRequestDto = require('../../requests/updateBookRequestDto') const updateBookValidator = require('../../validators/updateBookValidator') const bookService = require('../../domain/services/book.service') const bookResponseDto = require('../../responses/bookResponseDto') /** * Updates an existing book listing by id. */ const updateABook = catchException(async (req, res) => { const updateBookFields = updateBookRequestDto({ id: req.params.id, ...req.body }) updateBookValidator(updateBookFields) const updatedBook = await bookService.updateBookById(updateBookFields) res.json( globalResponseDto({ status: 'success', code: 200, message: `The book has successfully been updated.`, data: bookResponseDto(updatedBook), errors: null }) ) }) module.exports = updateABook ``` Once again, this is nice and thin, and easy on the eyes. # Deleting A Book Listing By ID - Planning Refer back to [*Chapter 6, The Bookstore API Endpoint Specifications, API Endpoint #9: Delete a Book Listing*](/part-3/chapter-6/1-the-bookstore-api-endpoint-specifications#api-endpoint-9-delete-a-book-listing) section for the API endpoint specifications. ## 1 - Route Name Finally, our final API endpoint. For the delete a book by ID, it will be very similar to updating. We'll just use `DELETE` instead of `PUT` as our method verb of choice. `DELETE /api/v1/books/:id` ## 2 - Input Request None. ## 3 - Middleware This will be the exact same as the last section. For the first layer, we'll be reusing our `isAuthenticated` helper that will protect us from unauthenticated user from accessing this endpoint. For the second layer, we'll be reusing our `bookPermission` helper to verify that the requested book ID exists and that the currently authenticated user is in fact the owner of it. ## 4 - Validation None. ## 5 - Domain Just like the last section, we'll built on top of the previous section and use the **bookModel**. We'll also need some sort of way to make a query to the database in order to find the existing book and delete it, so we'll use the `bookRepository` layer and add the method `deleteById()` to do so. **bookRepository** * `deleteById()` On top of that, our controller will call a service to `deleteBookById()`, in this case, it will be the `bookService` which will use the `bookRepository` layer. **bookService** * `deleteBookById()` ## 6 - Events None. ## 7 - Response Again, very similar to the previous section, except there will be 3 possible responses that can be output with this endpoint. The first is the authentication middleware that we'll be reusing. ```json { "status": "error", "code": 401, "message": "Access denied: you must be logged in to access this API endpoint.", "data": null, "errors": ["You must be logged in."] } ``` The second is whether or not the client has inputted an existing book ID. ```json { "status": "error", "code": 404, "message": "That book with the specified ID does not exist." "data": null, "errors": [ "Book listing not found." ] } ``` The third is the successful scenario. Do keep in mind that since we are using a `DELETE` request, so won't be returning any representation back. ```json { "status": "success", "code": 200, "message": "The book has successfully been deleted.", "data": {}, "errors": null } ``` # Deleting A Book Listing By ID - Implementation ## 1 - Route Name This will be our final API endpoint! `DELETE /api/v1/books/:id` Just like we planned, let's add in the route and our controller. *File: src/routes/book.route.js* ```javascript const express = require('express') const router = express.Router() const { getAllBooks, getBookById, createABook, updateABook } = require('../controllers/book') router.get('/', getAllBooks) router.get('/:id', getBookById) router.post('/', createABook) router.put('/:id', updateABook) // This is the new route we are adding in router.delete('/:id', deleteABook) ``` *File: src/controllers/book/deleteABook.js* ```javascript const catchException = require('../../utils/catchExceptions') /** * Deletes an existing new book listing. */ const deleteABook = catchException(async (req, res) => { // Our code goes here }) ``` ## 2 - Input Request None. ## 3 - Middleware As mentioned before, we will be reusing our `isAuthenticated` and `bookPermission` middleware. If we plug those in, our route should look something like this. *File: src/routes/book.route.js* ```javascript const express = require('express') const router = express.Router() const { deleteABook } = require('../controllers/book') const isAuthenticated = require('../middleware/auth.middleware') const bookPermission = require('../middleware/bookPermission.middleware') router.delete('/:id', isAuthenticated, bookPermission, deleteABook) module.exports = router ``` ## 4 - Validation None. ## 5 - Domain If you've followed along till this point, then the following will be of no surprise to you. Do note that because we are deleting a book, the `deleteOne` method from mongoose will either return a 1 for successful deletion or a 0 for a failed deletion. For a failed deletion to happen, it will have been an invalid `bookId` was inputted. Recall that we have already taken care of that in our `bookPermission` middleware, so no error handling is necessary here. *File: src/domain/repositories/book.repository.js* ```javascript const Model = require('../models/book.model') // Delete const deleteById = async (id) => { return await Model.deleteOne({ _id: id }) } module.exports = { deleteById } ``` *File: src/domain/services/book.service.js* ```javascript const mongoose = require('mongoose') const ApiException = require('../../utils/ApiException') const bookRepository = require('../repositories/book.repository') const checkBookId = require('../../utils/checkBookId') // Delete a book const deleteBookById = async (bookId) => { return await bookRepository.deleteById(bookId) } module.exports = { deleteBookById } ``` ## 6 - Events None. ## 7 - Response Once all our error handling will be taken care of by the `isAuthenticated` and `bookPermission` middleware. If there are no errors, we should show a success message like the following. ```javascript const globalResponseDto = require('../../responses/globalResponseDto') const catchException = require('../../utils/catchExceptions') const bookService = require('../../domain/services/book.service') /** * Deletes an existing new book listing. */ const deleteABook = catchException(async (req, res) => { const book = await bookService.deleteBookById(req.params.id) res.json( globalResponseDto({ status: 'success', code: 200, message: `The book with the id: ${book.id} was successfully deleted.`, data: null, errors: null }) ) }) module.exports = deleteABook ``` # Summary We've finally done it, we have finished implementing the Bookstore API, but wait, we're not done yet. To put forth our due diligence, we have to think long term and backup our codebase with tests. In the next chapter, that's exactly what we'll be doing. We'll be diving deep into writing unit and integration tests for our API. # Chapter 10: Testing our API > Every time you've ever filled a form and you expect to see some kind of notification panel at the top, you are testing. Everyone agrees that testing is good, the only question is, is automated testing good? Why would the answer ever be no? Or think of it this way, why would it ever be a bad thing to have this huge group of tests that backup everything you write. > > *-Jeffrey Way, Founder of Laracasts.com* At this point, we've written the code for our RESTful API. The only problem is that we don't have any automated tests to back them up. In this chapter, ***we'll demonstrate how to test each layer of our API*** and we'll see just how easy and efficient a layered architecture application is able to incorporate automated tests. Since there are way too many files and features to test, we'll keep it simple and consistent, we'll be mainly focused on writing tests for the **user registration endpoint**. ### Before We Begin We will be using [JEST](https://jestjs.io) as our testing framework of choice. All tests will be written in a `__tests__` folder depending on which layer we'll be testing except for API integration tests which will be located in the `tests` directory of the root folder. You can find all the configuration settings in the `config/jest.config.js` file. Tests can be ran by following the instructions on the [README page of the GitHub repository.](https://github.com/restful-node/restful-nodejs-app#running-tests) Make sure to install the application and have it running locally before running the tests. There is a Git Actions setup for continuous integration for this project. You may find it here at: . Do note that this is not a complete course on testing, if you want to learn more about testing and testing in JavaScript, I recommend taking the course by Kent C. Dodds. # Testing the Request Recall how we implemented the `registerUserRequestDto`. *File: src/requests/registerUserRequestDto.js* ```javascript const path = require('path') const ApiException = require('../utils/ApiException') const fields = [ 'first_name', 'last_name', 'email', 'password', 'password_confirmation', 'phone_number' ] /** * @param Object data */ const registerUserRequestDto = (data) => { const errors = [] fields.forEach((field) => { if (!(field in data)) { errors.push(`This DTO's property is required: ${field}.`) } }) if (errors.length > 0) { throw new ApiException({ status: 'error', code: 422, message: 'Input fields are of not the correct form.', data: null, errors }) } return data } module.exports = registerUserRequestDto ``` We'll first setup our test suite. We'll be working mainly in this file for the rest of this section. *File: src/requests/\_\_tests\_\_/registerUserRequestDto.test.js* ```javascript const registerUserRequestDto = require('../registerUserRequestDto') describe('Test Suite: registerUserRequestDto', () => { // tests go here... }) ``` I personally always like to write at least 1 test for both a passing and failing scenario. Of course you can always write more tests to cover more additional scenarios and edge cases . For our purposes, we'll just be sticking to writing 1 passing test and 1 failing test. ### The Passing Test The first one is simple, let's add in all the required fields in `registerUserRequestDto` and expect what we want out of it. ```javascript test('Request 1 - all required fields', () => { // 1. Arrange const inputRequest = { first_name: 'Yichen', last_name: 'Zhu', email: 'yichen@yichen.com', password: 'yichen-and-his-awesome-password', password_confirmation: 'yichen-and-his-awesome-password', phone_number: '1234567890' } // 2. Act const registerUserRequest = registerUserRequestDto(inputRequest) // 3. Assert expect(registerUserRequest).toEqual({ first_name: 'Yichen', last_name: 'Zhu', email: 'yichen@yichen.com', password: 'yichen-and-his-awesome-password', password_confirmation: 'yichen-and-his-awesome-password', phone_number: '1234567890' }) }) ``` ### The Failing Test The next test case is what happens if we don't pass the correct inputs into the `registerUserRequestDto` function. We would expect an error, and not just any error, but an `ApiException` error being thrown. ```javascript test('Request 2 - missing input fields', () => { // 1. Arrange const inputRequest = { message: 'this is awesome!' } // 2. Act + 3. Assert expect(() => { registerUserRequestDto(inputRequest) }).toThrow(ApiException) }) ``` # Testing the Middleware Recall how the user registration endpoint did not have any middleware. For the purposes of demonstrating how to test a middleware, we'll just be grabbing the `isAuthenticated` middleware that we created and test that instead. *File: src/middleware/auth.middleware.js* ```javascript const globalResponseDto = require('../responses/globalResponseDto') const isAuthenticated = (req, res, next) => { if (!req.session.user) { return res.status(401).json( globalResponseDto({ status: 'error', code: 401, message: 'Access denied: you must be logged in to access this API endpoint.', data: null, errors: ['You must be logged in.'] }) ) } next() } module.exports = isAuthenticated ``` Testing the middleware is a lot trickier than just a regular function. This reason why is because we are expected to stick to a particular function signature `(req, res, next)`. The answer to this is to mock out those particular parameters. The way to do it is to use `jest.fn`, which we will use to help us write out helpers like `mockRequest`, `mockResponse`, and `mockNext` functions. *File: src/middleware/\_\_tests\_\_/auth.middleware.test.js* ```javascript const isAuthenticated = require('../auth.middleware') const globalResponseDto = require('../../responses/globalResponseDto') const mockRequest = (userData) => ({ session: { user: userData } }) const mockResponse = () => { const res = {} res.status = jest.fn().mockReturnValue(res) res.json = jest.fn().mockReturnValue(res) return res } const mockNext = () => { return jest.fn() } describe('Test Suite: isAuthenticated middleware', () => { // our tests go here... }) ``` ### The Passing Test Now watch as we pass the `mockRequest`, `mockResponse`, and `mockNext` functions into our `isAuthenticated` function. If everything goes well, then we should expect the `next` to be invoked. ```javascript test('Access granted, next() should be invoked in express', async () => { // 1. Arrange const req = mockRequest({ first_name: 'john' }) const res = mockResponse() const next = mockNext() // 2. Act await isAuthenticated(req, res, next) // 3. Assert expect(next).toHaveBeenCalled() }) ``` ### The Passing Test Now for the failing test, we should expect the correct status code and output. ```javascript test('Access denied, respond with a status 401', async () => { // 1. Arrange const req = mockRequest() const res = mockResponse() // 2. Act await isAuthenticated(req, res) // 3. Assert expect(res.status).toHaveBeenCalledWith(401) expect(res.json).toHaveBeenCalledWith( globalResponseDto({ status: 'error', code: 401, message: 'Access denied: you must be logged in to access this API endpoint.', data: null, errors: ['You must be logged in.'] }) ) }) ``` # Testing the Validation *File: src/requests/registerUserValidator.js* ```javascript const Validator = require('validatorjs') const ApiException = require('../utils/ApiException') /** * @param {*} data { * - first_name * - last_name * - email * - password * - password_confirm * - phone_number * } * * @returns Validator */ const registerUserValidator = (data) => { const rules = { first_name: 'required', last_name: 'required', email: 'required|email', password: 'required|min:6', password_confirmation: 'required|min:6|same:password', phone_number: 'required|telephone' } const validator = new Validator(data, rules) if (validator.fails()) { let errors = [] for (const field in validator.errors.errors) { errors = errors.concat(validator.errors.errors[field]) } throw new ApiException({ status: 'error', code: 400, message: 'There were errors with the validation.', data: null, errors }) } return validator } Validator.register( 'telephone', function (value) { return value.match(/^\(?([0-9]{3})\)?[-.●]?([0-9]{3})[-.●]?([0-9]{4})$/) }, 'The :attribute field is not in a correct format.' ) module.exports = registerUserValidator ``` *File: src/validators/\_\_tests\_\_/registerUserValidator.test.js* ```javascript const registerUserValidator = require('../registerUserValidator') describe('Test Suite: registerUserValidator.test.js', () => { test('The happy path - everything works', () => { // 1. Arrange const data = { first_name: 'John', last_name: 'Doe', email: 'johndoe@gmail.com', password: 'superduperpassword', password_confirmation: 'superduperpassword', phone_number: '1234567890' } // 2. Act const validator = registerUserValidator(data) // 3. Assert expect(Object.keys(validator.errors.errors).length).toEqual(0) }) test('Not matching passwords', () => { // 1. Arrange const data = { email: 'johndoe@gmail.com', password: 'superduperpassword', password_confirmation: 'superduperpassword1' } // 2. + 3. Act and Assert expect(() => { registerUserValidator(data) }).toThrow(Error) }) }) ``` # Testing the Domain ### Repository Recall how we implemented the `userRepository` in one of the earlier chapters. *File: src/domain/repositories/user.repository.js* ```javascript const UserModel = require('../models/user.model') /** * * @param {*} user { * - name * - email * - password * } * * @returns user */ const createUser = async (userData) => { const user = new UserModel(userData) const userReturn = await user.save(userData) return userReturn } module.exports = { createUser } ``` Now it's time to write out test cases. For testing repositories, what we are mainly interested in is if the database operations are working. For that to work, we'll have to arrange to connect to the database on every test. Here we have a couple of helpers that will help us test for database-based test cases. The `db` function let's us create a connection to the database, it's simply a wrapper around the mongoose database driver, that is why you see a `disconnect` function in the `afterAll` function. We then have a `dbTestUtils` object that has a `clearDatabase` function which allows us to reset the database and let's us start off from a clean slate. For a closer look at what those helper functions do, you can dive deeper by looking at the source code of the repository. *File: src/domain/repositories/\_\_tests\_\_/user.repository.test.js* ```javascript const db = require('../../../utils/db') let dbConnection const dbTestUtils = require('../../../../tests/testUtils/dbTestUtil') const userRepository = require('../user.repository') beforeAll(async () => { dbConnection = await db() }) afterEach(async () => { await dbTestUtils.clearDatabase() }) afterAll(async () => { await dbConnection.disconnect() }) describe('Test Suite: User Repository', () => { // Our tests go here... }) ``` Now Moving on to the first successful test case, we have the following. We are simply calling the `createUser` method and passing in all of the necessary fields. This is a nice, clean, and simple test. ```javascript test('User Repository - createUser - success', async () => { const testUser = { first_name: 'Yichen', last_name: 'Zhu', email: 'yichen@yichen.com', password: 'password123', phone_number: '1234567890' } const user = await userRepository.createUser(testUser) const expectedName = 'Yichen' const actual = user.first_name expect(actual).toEqual(expectedName) }) ``` For the failing test case, we can decide to not pass in certain required fields. Recall that our UserModel had certain required fields when we created it. Here is what a possible failing test case would look like. ```javascript test('User Repository - createUser - error', async () => { try { const testUser = { email: 'yichen@yichen.com' } await userRepository.createUser(testUser) } catch (error) { /* eslint-disable-next-line */ expect(error.message).toBe( 'user validation failed: phone_number: Path `phone_number` is required., password: Path `password` is required., first_name: Path `first_name` is required.' ) } }) ``` Although this seems a little redundant due to the fact that we already have validation in another layer, this is simply to illustration how one might go about writing tests for a repository layer. When you start creating more complicated repositories that use multiple different models, you'll find more complicated tests are needed. ### Service Recall how we implemented the `authService` in one of the earlier chapters. *File: src/domain/services/auth.service.js* ```javascript const ApiException = require('../../utils/ApiException') const userRepository = require('../repositories/user.repository') /** * @returns User */ const registerUser = async (user) => { try { const createdUser = await userRepository.createUser(user) return createdUser } catch (err) { if (err.code === 11000 && err?.keyPattern?.email === 1) { throw new ApiException({ message: `This email is already taken.`, status: 'error', code: 409, data: null, errors: [`This email is already taken.`] }) } } } ``` The service layer is going to be all about us mocking out the repository layer. We do so in order to keep the tests as separate and isolated as possible. For a successful test case, take a look at the following. We do a simple mock of both the `userRepository` and its method `createUser` functions and see if they've been called properly. *File: src/domain/services/\_\_tests\_\_/auth.service.test.js* ```javascript const userRepository = require('../../repositories/user.repository') const authService = require('../auth.service') beforeAll(async () => {}) beforeEach(() => { userRepository.createUser = jest.fn(() => { return {} }) }) afterEach(async () => {}) afterAll(async () => {}) describe('Test Suite: Auth Service', () => { test('Auth Service - registerUser', async () => { const testUser = { first_name: 'john', last_name: 'doe', email: 'john@john.com', password: 'password', phone_number: '4168561988' } await authService.registerUser(testUser) expect(userRepository.createUser).toHaveBeenCalledWith(testUser) expect(userRepository.createUser).toHaveBeenCalledTimes(1) expect(userRepository.createUser).toHaveReturnedWith({}) }) }) ``` For the failing test, we'll simply want to throw an `ApiException` artificially. ```javascript test('Auth Service - registerUser - error', async () => { userRepository.createUser = jest.fn(() => { class CustomError extends Error { constructor() { super() this.code = 11000 this.keyPattern = { email: 1 } } } throw new CustomError() }) try { await authService.registerUser({}) } catch (err) { /* eslint-disable-next-line */ expect(err.message).toBe('This email is already taken.') } }) ``` # Testing the Event Recall how we implemented the `userHasRegisteredEvent`. *File: src/events/userHasRegisteredEvent.js* ```javascript const mailer = require('../utils/mailer') /** * * @param {*} user * * @returns boolean */ const userHasRegisteredEvent = (user) => { const bodyText = `Hello ${user.firstName}. Thanks for registering!` return mailer.sendEmailToUser(user.email, 'Welcome aboard!', bodyText) } module.exports = userHasRegisteredEvent ``` Now for this test, we actually don't care that much about the `mailer` utility and whether or not we actually sent an email. What we are really interested in is whether or not the `mailer.sendEmailToUser` was called or not. For this particular test, it would be sufficient to just mock up the `mailer` and the `sendEmailToUser` method to check if they've been called. ### The Test The mocking part for this test is a lot tricky, but notice how we mock both the `mailer` and the `sendEmailToToUser` below. *File: src/events/\_\_tests\_\_/userHasRegisteredEvent.test.js* ```javascript const event = require('../userHasRegisteredEvent') const mailer = require('../../utils/mailer') const sentEmailSuccessful = !!Math.round(Math.random(0, 1)) jest.mock('../../utils/mailer') mailer.sendEmailToUser.mockImplementation(() => sentEmailSuccessful) describe('Test Suite: userHasRegisteredEvent', () => { test("Email has (un)successfully been sent to the user's email address", () => { const eventResult = event({ firstName: 'John', email: 'john@john.com' }) mailer.sendEmailToUser.mockImplementation(() => true) expect(mailer.sendEmailToUser).toHaveBeenCalledTimes(1) expect(eventResult).toBe(sentEmailSuccessful) }) }) ``` This might look a little strange at first, but one aspect to consider is that you should not be sending an actual email when running the test. Imagine a scenario in which we did not mock the `mailer` utility and its `sendEmailToUser` method, and we actually stubbed in real inputs into the function, then we would be making real life calls that would send real emails. # Testing the Response Recall how we implemented the `userResponseDto`. *File: src/responses/userResponseDto.js* ```javascript const userResponseDto = (user) => { return { id: user['id'], first_name: user['first_name'], last_name: user['last_name'], email: user['email'], phone_number: user['phone_number'] } } module.exports = userResponseDto ``` You may look at this and feel a sense of déjà vu, and you would be right, because writing a test for the `userResponseDto` is quite similar to writing a test for the `registerUserRequestDto`. As always we'll setup our test suite as follows. *File: src/responses/\_\_tests/userResponseDto.test.js* ```javascript const userResponseDto = require('../userResponseDto') describe('Test Suite: userResponseDto', () => { // Tests go here }) ``` ### The Tests Since this is quite an easy function to test, let's add in some more test cases this time. ```javascript const userResponseDto = require('../userResponseDto') describe('Test Suite: Response', () => { test('Correct request 1 - empty parameters', () => { // 1. Arrange const user = {} // 2. Act const response = userResponseDto(user) // 3. Assert expect(response).toEqual({}) }) test('Correct request 2 - extra useless parameters', () => { // 1. Arrange const user = { id: '123', first_name: 'john', last_name: 'doe', email: 'johndoe@email.com', phone_number: '1234567890', password: 'undefined', extraParam1: null, extraParam2: 'whatzup', body: 'John Doe is cool' } // 2. Act const response = userResponseDto(user) // 3. Assert expect(response).toEqual({ id: '123', first_name: 'john', last_name: 'doe', email: 'johndoe@email.com', phone_number: '1234567890' }) }) test('Correct request 3 - add password fields, hopefully they do not show up', () => { // 1. Arrange const user = { id: '123', first_name: 'Yichen', last_name: 'Zhu', email: 'yichen@yichen.com', password: 'yichen-and-his-awesome-password', password_confirmation: 'yichen-and-his-awesome-password', phone_number: '1234567890' } // 2. Act const response = userResponseDto(user) // 3. Assert expect(response).toEqual({ id: '123', first_name: 'Yichen', last_name: 'Zhu', email: 'yichen@yichen.com', phone_number: '1234567890' }) }) }) ``` # Testing the Controller Recall how we implemented the `registerUser` controller. *File: src/controllers/auth/registerUser.js* ```javascript const catchExceptions = require('../../utils/catchExceptions') const globalResponseDto = require('../../responses/globalResponseDto') const userResponseDto = require('../../responses/userResponseDto') const registerUserRequestDto = require('../../requests/registerUserRequestDto') const registerUserValidator = require('../../validators/registerUserValidator') const authService = require('../../domain/services/auth.service') const EventEmitter = require('events') const eventEmitter = new EventEmitter() /** * Inserts the user into the database and fires off an email notification to that user's email if successful. */ const registerUser = catchExceptions(async (req, res) => { const registerUserRequest = registerUserRequestDto(req.body) registerUserValidator(registerUserRequest) const user = await authService.registerUser(registerUserRequest) eventEmitter.emit('userHasRegistered', user) res.json( globalResponseDto({ status: 'success', code: 200, message: `The email: ${registerUserRequest.email} has successfully registered.`, data: userResponseDto(user), errors: null }) ) }) module.exports = registerUser ``` ### The Test Now because our controller has been kept nice and thin up until this point and have barely any logical statements other than the functions and services we've created, there's actually very little to test. If you've done your job correctly, then you should not have to test the controller what so ever because all of your tests would have been done in those other layers. However, since this is an educational book, we will do it for demonstration purposes. In order to test the controller, the key is to mock every service that its using and see if they've been called or not. Check out the following test case as we mock the `registerUserRequestDto`, the `registerUserValidator`, the `authService.register` function, and the express `req` and `res`. *File: src/controllers/auth/\_\_tests\_\_/registerUser.test.js* ```javascript const registerUserController = require('../registerUser') const registerUserRequestDto = require('../../../requests/registerUserRequestDto') const registerUserValidator = require('../../../validators/registerUserValidator') const authService = require('../../../domain/services/auth.service') jest.mock('../../../requests/registerUserRequestDto', () => jest.fn((data) => data) ) jest.mock('../../../validators/registerUserValidator', () => jest.fn((data) => data) ) jest.mock('../../../domain/services/auth.service', () => { return { registerUser: jest.fn((data) => data) } }) const mockRequest = () => ({ body: { first_name: 'john', last_name: 'doe' } }) const mockResponse = () => { const res = {} res.status = jest.fn().mockReturnValue(res) res.json = jest.fn().mockReturnValue(res) return res } describe('Controler - Register User', () => { test('User should be registered successfully', async () => { const req = mockRequest() const res = mockResponse() await registerUserController(req, res) const randomUserFromBodyRequest = { first_name: 'john', last_name: 'doe' } expect(registerUserRequestDto).toHaveBeenCalledWith( randomUserFromBodyRequest ) expect(registerUserValidator).toHaveBeenCalledWith( randomUserFromBodyRequest ) expect(authService.registerUser).toHaveBeenCalledWith( randomUserFromBodyRequest ) expect(res.status).toHaveBeenCalledWith(201) expect(res.json).toHaveBeenCalled() }) }) ``` # Integration Test Now comes my favorite part of the test, the high level API integration test. This is where we call the API endpoint directly as if we are posing as a real client. The idea here is to set the application up with an open port and build a new database along with some fake seed data. Then, make the API call programmatically with an HTTP library like `fetch`. Finally, we make an assertion and check if the response is what we would expect from the client. Here is how we'll be setting this test up. *File: tests/api/auth/register.test.js* ```javascript const fetch = require('node-fetch') const api = require('../../../src/server') const apiPort = Math.round(Math.random() * 65535) const baseURL = `http://localhost:${apiPort}/api/v1` const db = require('../../../src/utils/db') let dbConnection const dbTestUtils = require('../../../tests/testUtils/dbTestUtil') /** * 1. Arrange * - setup the world */ beforeAll(async () => { await api.listen(apiPort) // start the application dbConnection = await db() // start the database }) beforeEach(async () => { await dbTestUtils.setUpDatabase() }) afterEach(async () => { await dbTestUtils.clearDatabase() }) afterAll(async () => { await api.close() await dbConnection.disconnect() }) /** * 2. Act * - make the http call * 3. Assert * - response check */ describe('API Test - Register User', () => { // Tests go here }) ``` ### The Passing Test The first test is our happy path, so no errors. This is simple, make sure all the input requests are correct and that when we call the API, the user is successfully registered. ```javascript test('POST /api/v1/users - user successfully registered', async () => { const user = { first_name: 'Yichen', last_name: 'Zhu', email: 'yichen@yichen.com', password: 'password123', password_confirmation: 'password123', phone_number: '1234567890' } const response = await ( await fetch(`${baseURL}/users`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(user) }) ).json() delete user.password delete user.password_confirmation expect(response).toMatchObject({ status: 'success', code: 201, message: `The email: ${user.email} has successfully registered.`, data: user }) }) ``` ### The Failing Test For the second test, let's have the API throw some sort of error. One possible scenario is if the user enters passwords do not match. ```javascript test('POST /api/v1/users - fail due to invalid form inputs', async () => { const user = { first_name: 'Yichen', last_name: 'Zhu', email: 'yichen@yichen.com', password: 'password', password_confirmation: 'password_not_same', phone_number: '1234567890' } const response = await ( await fetch(`${baseURL}/users`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(user) }) ).json() expect(response).toMatchObject({ code: 400, data: null, message: 'There were errors with the validation.', status: 'error', errors: ['The password confirmation and password fields must match.'] }) }) ``` # Summary That's a wrap everyone, we're now finished with all the chapters of the book. Although we weren't able to get 100% coverage of our Bookstore API, we have learned how to write automated tests in each layer of our application. You'll be able to use this new-found knowledge going forward and have much more confidence in your code when building new RESTful API endpoints. To end off at a high note, why don't we all run our tests one more time and appreciate this automated work of beauty. ![I love it when all my tests are passing and all I see is green!](/files/TztnclC0KOihCmm2swD1) # Final Words Hopefully you actually read this book and didn't just skip to the final chapter XD. If you've read this book thoroughly, then you would by now have a pretty good understanding of REST and what it means for an API to be truly RESTful. You now know some of the "best practices" and guidelines that can be followed and hopefully you will use and apply those guidelines in your next project. You should also know how to structure your Node.js application with a layered architecture approach and have a nifty little 8 step recipe by your side to aid you in planning any future RESTful API endpoints. To all the readers, take what you learned from this book and try ***NOT** to give your boss, your CTO, and your client all a near death heart attack*. -Yichen 😊 # Refactoring to HATEOAS This bonus chapter is still a work in progress. Check out for updates on the codebase. # Sources & References ## Chapter 1 1. [History of APIs](https://history.apievangelist.com) 2. [JSConf EU - 2009](https://www.jsconf.eu/2009/about.html) 3. [Computing Conversations with Brendan Eich](https://www.youtube.com/watch?v=IPxQ9kEaF8c) 4. [Microsoft REST API Guidelines](https://github.com/microsoft/api-guidelines/blob/vNext/Guidelines.md) 5. [Google API Design Guide](http://apistylebook.com/design/guidelines/google-api-design-guide) 6. [GitHub REST API Docs](https://docs.github.com) 7. [Slack API Docs](https://api.slack.com/methods/chat.getPermalink) 8. [Twitter API Docs](https://developer.twitter.com/en/docs/twitter-api/enterprise/historical-powertrack-api/api-reference/historical-powertrack)