Serverless Queues

One of the most useful components in scaling a serverless architecture is the queue. Queues are a key asynchronous processing concept. In a queue, data points are added to the back of the queue and taken from the front. An example of this is a first-in-first-out data structure. In a cloud architecture, a queue will generally store events or messages from an upstream process until subscribers downstream have the time to process them.

An important thing to understand before deciding to use a queue is the compromise you are making in using one. In a synchronous operation (that is, inserting a record into a database all in one operation), if something goes wrong, you have instant feedback to your user. In an asynchronous operation (that is, dropping an event onto a queue that a worker later picks up to insert into the same database), if the later operations go wrong, then the user has usually left the application and cannot fix their mistake or call support.

A good way to try and square that particular circle is to do your data validation synchronously and your data retention asynchronously. So, if your product is only allowed to have a size between XS and XL, check that synchronously before loading the product onto the queue. This way, the number of errors can be reduced heavily, and usually you are left with just connection errors.

Another thing to consider is whether it is vastly important that the user fixes their input straight away. If they are inputting a multimillion-pound time-sensitive trade, then yes. If they are submitting a profile picture to a social networking site, then an email or other notification after that fact will probably be fine.

There are four main fully managed queue providers in Azure. They differ in focus, generally in making the distinction between business logic events and data points:

• Azure Event Hubs: Event Hubs is designed for handling vast quantities of data point messages, for example, a temperature readout from a fleet on Internet of Things devices.
• Azure Event Grid: Event Grid focuses on business logic events, such as purchases of products being made.
• Azure Service Bus: Service Bus is generally suitable for either of the aforementioned use cases, but isn't as well-geared up for each as it doesn't make the assumptions that allow the other services such efficiency in their relative areas.
• Azure Queue Storage: Azure Queue Storage is the lowest maintenance service by far, as it uses an Azure Storage account that's already been created. It can store as much data as a storage account, which is 500 Terabytes by default. Because it is very simple, it won't suit the more specialized use cases that the others will. It is, for example, limited to a 64 KB message size and cannot guarantee correct queue ordering. However, it is the closest fit to serverless with no maintenance.

In a serverless architecture, you should always use a managed service over one you manage yourself, unless there is a compelling business requirement to do otherwise. You can use a managed queue service such as Azure Storage Queues for this.

Exercise 5: Creating an Azure Storage Queue and Submitting Messages from an Azure Function

In this exercise, you will be learning how to create an Azure Storage Queue instance and submit data to it from an Azure Function using Postman. This Azure Function will convert the data into a C# object before adding it to a queue:

Note

Ensure that you've installed Postman by following the instructions in the preface before beginning with this exercise.

1. First, create the Azure Storage Queue. But, before that, create a storage account called advancedserverless (refer to Exercise 1, Creating an Azure Function, in case you need help). Now, open the Azure Portal and navigate to the Azure Storage account that you created earlier. You should be able to see a box on the user interface that says Queues. Click on Queues:
   

   Figure 2.1: A screenshot of the Azure Storage home screen

2. You will now be on the Queues screen. Click the + Queue button to create a new queue, as shown in the following screenshot:
   

   Figure 2.2: Queues home screen

3. Name the queue product-queue and click on OK:
   

   Figure 2.3: Creating a new queue

4. You can manually add messages to the queue in this view. Open the queue you created and click on + Add message:
   

   Figure 2.4: Queue home screen

5. Now, you can add an example message. In this example, it is a JSON formatted string with the property "name" that has a value of "bob," as shown in the following screenshot. The message has an expiry time on it (we've set it to 7 days here), which is the amount of time it will stay on the queue without being processed before being removed automatically:
   

   Figure 2.5: Creating an example message

6. Now you have a message in your queue. As you will be using this queue for proper processing, having this test message in here is likely to break your later work, so dequeue it by selecting the message and then clicking the Dequeue message button. This is to demonstrate how the queue works. Using this button imitates the working of a piece of software that's listening for messages on the queue:
   

   Figure 2.6: Test message on a queue

7. Next, we'll create an Azure Function that can submit messages to the queue we just created. Create a new folder called QueueFunctions and open a new Visual Studio Code instance in there:
   

   Figure 2.7: Visual Studio Code showing the QueueFunctions folder

8. Referring back to Exercise 1, Creating an Azure Function, if necessary, create a C# Azure Functions project in the QueueFunctions folder and restore any dependencies, as requested:
   

   Figure 2.8: Scaffolded Azure Functions project

9. Create an Azure Function using the HTTP trigger, referring to Exercise 1, Creating an Azure Function, if necessary. Call the function AddProducts and put it in the QueueFunctions.Products namespace. Your scaffolded Azure Function should look as follows:
   

   Figure 2.9: Scaffolded Azure Function

10. An important difference between the first and second version of Azure Functions is that, now, all connectors to external services arrive as separate packages in version 2 (rather than being bundled as they are in version 1). Therefore, you will need to install the Azure Storage package using the following code in the terminal that's in the QueueFunctions folder:

    dotnet add package Microsoft.Azure.WebJobs.Extensions.Storage --version 3.0.1

    Your terminal will appear as follows:

    

    Figure 2.10: Installing Microsoft.Azure.WebJobs.Extensions.Storage 3.0.1

11. Now, we need to prepare our data model so that the function can return and display. Create a folder inside QueueFunctions called Models, and a file inside this called Product.cs. Create a new Product class inside the Product.cs file in the QueueFunctions.Models namespace. Include a using statement for Newtonsoft.Json. Create the five properties with camel-cased JSON property names, as follows:

    using Newtonsoft.Json;

    namespace QueueFunctions.Models {

      public class Product {

        [JsonProperty(PropertyName = "typeId")]

        public string TypeId { get; set; }

        [JsonProperty(PropertyName = "name")]

        public string Name { get; set; }

        [JsonProperty(PropertyName = "size")]

        public string Size { get; set; }

        [JsonProperty(PropertyName = "colour")]

        public string Colour { get; set; }

        [JsonProperty(PropertyName = "id")]

        public string Id { get; set; }

      }

    }

    The following screenshot shows the contents of the Product.cs file:

    

    Figure 2.11: Product class with camel-cased JSON properties

12. Now, open the AddProducts.cs file. Annotate the class with [StorageAccount("AzureQueueStorageAccount")]. This tells the Azure Function to look for the connection string to the storage account in an app setting called AzureQueueStorageAccount (the connection string encodes everything that's required to connect to the storage account, including authentication).
13. Add an extra annotation above the Run method, that is, [return: Queue("product-queue")]. This tells the Azure Function that the result of the execution should be added to a queue called product-queue.
14. Remove the "get" string and the body of code. Change the return type to Product and add the following three lines instead:

    String requestBody = await new StreamReader(req.Body).ReadToEndAsync();

    Product product = JsonConvert.DeserializeObject<Product>(requestBody);

    return product;

    This code deserializes a JSON object from the payload into a C# object and returns it. This adds a simple schema to the object, which could be enforced more thoroughly by adding in tests for string length and so on if so desired. This is then added to the queue by the annotation above the method:

    

    Figure 2.12: Azure Function adding messages onto a queue

15. Go to the Azure Portal and retrieve the primary connection string for your Azure Storage account from the Keys section. Add it to your local.settings.json file in a property called AzureQueueStorageAccount:
    

    Figure 2.13: Local.settings.json with the Azure Storage connection string property

16. Run the Azure Function using the play button, as shown earlier in this book. The terminal should show the address that the HTTP trigger is listening on. Copy and paste it into Postman. Change the Postman request type to POST, click on Body, and change the type from form-data to raw (see the radio buttons in the following screenshot). Add the following code into the body and send the request by clicking on the Send option in the top-right corner of the screen:

    {

      "typeId": "tshirt",

      "id": "tshirt_metallica_black_xl",

      "colour":"black",

      "size": "XL",

      "name": "Metallica"

    }

    Your screen will look as follows:

    

    Figure 2.14: Postman screen showing successful request in 79 ms

17. Send the request a few times and feel free to change the data and send different messages. Now, open the Azure Portal and go to your queue. There should be lots of messages, as shown in the following screenshot:

Figure 2.15: Queue with multiple messages

Congratulations! You have added several messages to your queue. This queue required no maintenance and very little effort to set up, making it a good companion for serverless applications. It will happily scale to incredible levels of data storage without any issue. The messages are also confined to a schema, albeit in a fairly loose way, by being converted to C# objects on the way. This will help prevent errors in downstream processing, but it would be advisable to add far more thorough checks (for instance, you could add null checks on properties that your application can't survive being null or enums for size).

Exercise 6: Triggering an Azure Function on a Message Arriving on an Azure Storage Queue and Inserting it into Cosmos DB

In this exercise, you will create your first event-driven function using a trigger other than the HTTP trigger. This method can be used to reduce the real-time strain on a non-serverless database such as Oracle or PostgreSQL. It allows the entire application to continue to scale, despite the lack of scalability of one component. While we will be using a serverless database in this instance for convenience, this is a pattern you should look at using when you can't use one. You will create a function using a Queue trigger, which will input a document into a Cosmos DB database:

1. Open Visual Studio Code and click the Azure logo button to create a function. Choose QueueTrigger as the trigger for the function:
   

   Figure 2.16: Creating a QueueTrigger Azure Function

2. Next, you will be prompted to enter a name for the function. Name it DequeueProducts and set the namespace to QueueFunctions.Products:
   

   Figure 2.17: Creating an Azure Function called DequeueProducts

3. Set the name of the app setting containing your storage connection to AzureQueueStorageAccount, when prompted:
   

   Figure 2.18: Setting the name of the Azure Storage connection string

4. A function will be templated for you, as shown in the following screenshot. Also, a small box will come up on the bottom right saying that you need an Azure Storage account for any non-HTTP-triggered functions. Click on it and select the advancedserverless storage account:
   

   Figure 2.19: Setting the storage account used by the Function itself

5. Change the type of the incoming message from string to Product and import the QueueFunctions.Models namespace with a using statement. This tells the Azure Function to serialize the message on arrival itself, which means less code for you to write. If you need custom or clever serialization, then you will need to keep this as a string and do it manually in the function:
   

   Figure 2.20: Changing the input type to Product

6. Install the package for Cosmos DB using the following code in the terminal that's in the QueueFunctions folder:

   dotnet add package Microsoft.Azure.WebJobs.Extensions.CosmosDB --version 3.0.1

   

   Figure 2.21: Installing the Cosmos DB extension

7. Add the following code as a second argument to the Run function:

   [CosmosDB(

   databaseName: "serverless",

   collectionName: "products",

   ConnectionStringSetting = "CosmosDBConnectionString",

   Id = "{ProductId}",

   PartitionKey = "black")]Product

   product,

   This sets the output of this function to be a Product object. This is inserted into the products collection in the serverless database in the Cosmos DB database that's defined in the CosmosDBConnectionString app setting.

8. Add the following line of code below the logging statement to assign a value to the product being submitted to the database:

   outProduct = product;

   You function will look as follows:

   

   Figure 2.22: Azure Function submitting messages from the queue to Cosmos DB

9. Copy the Cosmos DB connection string from the earlier function app and add it to local.settings.json before running the function and testing using Postman again. Open the Cosmos DB instance in the portal and inspect the collection. You should now have one entry there. Even if you continually send requests, this will not change because the ID has been set to tshirt_metallica_black_xl and each request has the same ID. It will update the record instead:

Figure 2.23: Entry in Cosmos DB showing successful message processing

Congratulations! You have successfully used a queue to add asynchronous processing to your application. While Cosmos DB could happily scale to any input in a loose consistency mode, this method is very useful in general for non-serverless databases or Cosmos DB in the most restrictive consistency mode. If the database has maxed out and has stopped accepting new inputs, the function would retry five times by default before putting messages onto a new queue called {current queue name}-poison. You can then have a serverless function process those, possibly putting them into an archive and sending out an alert. You can also increase the time between retries on failed executions to further reduce the load on the database by adding the following to the host.json file:

{

  "queues":

{

  "visibilityTimeout" : "00:01:00",

  }

}

This sets the retry time to one minute, by which time the database would usually have recovered. Other possibilities include hosting the function on a non-consumption plan to prevent it horizontally scaling at all, but that's not ideal as the backlog of messages may simply keep growing without anything apparently being wrong. It's probably better to allow the messages to reach the poison queue in that situation and be handled specifically.