Published on

Creating MQTT client code generator with AsyncAPI

In case you want to generate something more than just models out of your AsyncAPI document, sooner or later you will reach out for AsyncAPI Generator.

if ('I only want to generate models from my schemas) return "https://modelina.org/"

Intro to AsyncAPI Generator

AsyncAPI Generator is a library that you need to use to generate apps, SDKs or docs. Generator itself is just a set of helpers and configurations that do not really have any logic aobut SDKs or other stuff that you'd like to generate. The part that defines what should be generated out of your AsyncAPI document is what we call Template.

What the heck is this? Generator and Template :thinking:

You know Johannes Gutenberg and his printing press? not personally of course, he is dead...like almost 600y ago. Gutenberg is not important in this story, but the printing press.

picture showing old printing press

So Generator is like printing press. It helps you to print stuff at scale, but alone, it is useless. Printing press needs input, a Template :tadam: and your AsyncAPI document.

picture showing printing press template

In your AsyncAPI document you describe your application, and in the Template you say what part of the document should be used for a specific print output. So basically when you feed Generator with your AsyncAPI document and the Template, in Template you specify where asyncapi.info.title() should be rendered.

Community Templates

You do not have to create a Template to generate something out of your AsyncAPI document. There are many community maintained templates, but for sure many are missing. Sometimes you need to create your custom template, tailored for your use case, which is completely fine. Just please remember, maintaining a template together with the community makes maintainance easier and also helps other community members to save on development time.

Prerequisite before template development

As I explained earlier, all the fun makes sense when the generator gets the template and AsyncAPI document. Therefore before you start writing the template, you definitely need some sample AsyncAPI document that you will test your template agains :smiley:

The template that I develop as part of this article is based on the following example:

asyncapi: 2.6.0

info:
  title: Comments Service
  version: 1.0.0
  description: This service is in charge of processing all the events related to comments.

servers:
  dev:
    url: test.mosquitto.org
    protocol: mqtt
    bindings:
      mqtt:
        clientId: comment-service

channels:
  comment/liked:
    description: Updates the likes count in the database.
    publish:
      operationId: commentLiked
      message:
        description: Message that is being sent when a comment has been liked by someone.
        payload:
          type: object
          additionalProperties: false
          properties:
            commentId:
              type: string

I created asyncapi.yml document in new test/fixtures directory in my template project

Creating first template

For the sake of this article I will create a Template that will support MQTT protocol and generate a Python client from your AsyncAPI document.

I'm not a Python developer. This will actually be my first time I write some Python code, but I'm not worried, I'll try to get some inspiration from articles like this one and see if AI will be able to help a bit.

Why Python then and not my beloved JavaScript?

The noisy sound of people leaving the room because they've heard the author loves JavaScript!

picture of people leaving the room

I wanted to use Python as my lovely wife is learning it now, and I want to be able to mentor her on a long run.

Template tech stack

Template is a standalone project that is not part of the Generator library logic and its codebase. You can store Template code wherever you want.

AsyncAPI Generator is written in JavaScript.

Java people already left the room anyway, but now there rest, TypeScript lovers, left

Ok, now we have only people with strong character left :muscle:

AsyncAPI Generator is written in JavaScript and uses npm libraries to pass Template logic through the generator. In other words, when you tell AsyncAPI Generator to use Template A, the generator basically does npm install of the template. Thanks to this approach templates can be stored anywhere, like package registries, remote repository or just a tarball file - basically whatever npm install supports now.

Basic template structure

I store all template code in folder called python-mqtt-client-template.

  1. First I create a package.json like you see below:

    {
  "generator": {
    "renderer": "react",
    "apiVersion": "v1",
    "generator": ">=1.10.0 <2.0.0",
    "supportedProtocols": ["mqtt"]
  },
  "dependencies": {
    "@asyncapi/generator-react-sdk": "^0.2.25"
  }
}

    AsyncAPI Generator supports two way of writing templates:

    Don't use the first one. Every AsyncAPI Generator maintainer lost a lot of hair debugging Nunjucks templates. One day it will be deprecated.

    Lets see what is there in the package.json:

    • generator is where I specify generator specific configuration.
      • renderer is where I specify that the generator should push my template through react rendering enginge,
      • apiVersion is where I specify that I want to use in my template the latest and gratest Parser API for extracting information from AsyncAPI documents,
      • generator is where I specify what versions of generator is my template compatible with. I don't know if AsyncAPI Generator 2.0 will bring changes that will break my template. I want my template users to have good experience and get a nice human-readable message from the generator that my template is not compatible with certain version of the generator. Also the apiVersion was introduced in the generator 1.10.0 so I know my template won't work with older version,
      • supportedProtocols is where I specify what protocols my template supports. Again purely developer experience related setting. I could leave that out, but no. I do not want user of my template to get undefined errors if they use kafka but a nice human-readable info that my template is at the moment compatible only with the mqtt protocol?
    • dependencies > @asyncapi/generator-react-sdk is where I specify what version of react sdk should be used. This package contains a set of useful components that you MUST use in your template.

  2. Since there are dependencies added, I need them locally, so I run npm install

  3. Now I create template folder where all my template code will be located

Create highway to heaven

Or to hell, if you are a developer that loves strongly typed languages but agile drawing machine drum picked you to work on the template.

picture of crying emoji looking at drawing machine

Everything starts in index.js

import { File } from '@asyncapi/generator-react-sdk'

export default function ({ asyncapi }) {
  return <File name="client.py">{asyncapi.info().title()}</File>
}

First template file and you already see @asyncapi/generator-react-sdk dependency is a must have as among other components it contains a File component that you need to use to specify that given template file must return a file.

Focus your eyes on { asyncapi.info().title() } as this is most important. Template files get access to many different goodies thanks to the generator, one of those is asyncapi that is not representing you AsyncAPI document 1:1 but it is an instance of the AsyncAPI Parser that is responsible for validating AsyncAPI documents and giving you an API that makes it much easier to extract information from AsyncAPI document.

In other words { asyncapi.info().title() } means that asyncapi contains a set of helper functions like info() that returns the Info object from an AsyncAPI document:

# ...(redacted for brevity)
info:
  title: Comments Service
  version: 1.0.0
  description: This service is in charge of processing all the events related to comments.
# ...(redacted for brevity)

And the title() helper let's you extract the title from the Info object.

Lets see if it will work and the template is valid.

Test the highway

AsyncAPI CLI is a terminal tool that integrates different AsyncAPI tools that allow you to do different things with your AsyncAPI documents, like validation, optimization, bunding and others. You probably guessed already that AsyncAPI Generator is there too.

Install AsyncAPI CLI

Well, yes this one is in JavaScript as well. Like the source code is in TypeScript, but whatever, installation sill requires you do run:

npm install -g @asyncapi/cli

But calm down, we know people that do not like JS, also can't imagine installing node and npm. AsyncAPI CLI is also published in a much friendlier way, with binaries dedicated to different operating systems.

I'm Mac user so I like this one the most: brew install asyncapi

For more options just check out the docs.

Use AsyncAPI CLI to test Template

tl;dr just run asyncapi generate fromTemplate test/fixtures/asyncapi.yml ./ --output test/project

  • asyncapi generate fromTemplate is the way to use AsyncAPI Generator
  • asyncapi.yml is how you provide the AsyncAPI document to the generator. Can be URL if you want.
  • ./ since this article describes how to write template, I assume you develop it locally, and it is not available on any server. I assume you call asyncapi generate fromTemplate from the directory where your template project is located
  • --output test/project is how you specify the folder where code should be generated

In case all is good, success looks like below:

Generation in progress. Keep calm and wait a bit... done
Check out your shiny new generated files at test/project.

Looks the same on your side, right?

meme saying that it works on my computer

I can see client.js file in test/project directory and the only content is Comments Service.

Let's finally create that client

Nobody does it this way, like write template from scratch directly. At least I can't imagine doing it. My brain can get on that level of abstraction.

Anyway, this is my article and we do it my way:

  1. Create some real working client code
  2. Create some test code that uses the client
  3. Update template with working client code
  4. Setup some script that will help you run this code quickly to test
  5. Start templating your code
mandalorian image with this is the way inscription

Create some real working client code

I'm not a Pyton dev and to save my time to get some real working code I used ChatGPT for the first time. I could get some code with additional refactoring in just few minutes.

This is the module for the client I have in client.py file in the test/project:

import paho.mqtt.client as mqtt

mqttBroker = "test.mosquitto.org"
commentLikedTopic = "comment/liked"

class CommentLikedClient:
    def __init__(self):
        self.client = mqtt.Client()
        self.client.connect(mqttBroker)


    def sendCommentLiked(self, id):
        self.client.publish(commentLikedTopic, id)

I'm using here an MQTT client from Eclipse Paho project to connect to the MQTT broker. In this article I will not explain what MQTT is and its use case, instead, I send you to article about MQTT in IoT.

Instead of spinning my own broker instance, I'm using a public test instance of Eclipse Mosquitto broker.

You will have to install that Paho package with pip install paho-mqtt.

The idea is that if somebody wants to interact with my Comments Service they do it using my client module. Just create instance of the client like client = CommentLikedClient() and then use sendCommentLiked function to publish messages that Comments Service is subscribed to.

Pretty neat and this is what the purpose of the client is right? User do not need the info about the server and topic name, but then only need a helper function with clear info on what data should be sent.

Create some test code that uses the client

Let me try it in action. I created test.py in test/project directory that looks like below:

from client import CommentLikedClient
from random import randrange
import time

client = CommentLikedClient()

id_length = 8
min_value = 10**(id_length-1)  # Minimum value with 8 digits (e.g., 10000000)
max_value = 10**id_length - 1  # Maximum value with 8 digits (e.g., 99999999)

while True:
    randomId = randrange(min_value, max_value + 1)
    client.sendCommentLiked(randomId)
    print("New like for comment " + str(randomId) + " sent to comment/liked")
    time.sleep(1)

The code focus only on sending the message, no need for any configurations, all is provided by the client. Rest of the code is only focused on generating some random comment ID and then endlessly sending it to the broker.

After calling python test.py I see the following logs:

New like for comment 39097159 sent to comment/liked
New like for comment 13144095 sent to comment/liked
New like for comment 19962377 sent to comment/liked
New like for comment 10554318 sent to comment/liked

But how do I know it works? There are no errors comming from the broker, but how can I be 100% sure that the message I'm sending is actually reaching the broker and potential consumers?

diagram showing event driven flow with broker in the middle and unknown consumer

I need to emulate some consumer that will subscribe to comment/liked and will log information about received message. I could write another python script that subscribes to the topic, but that would complicate the article too much and better is just use some MQTT CLI, and best to do it through docker.

docker run hivemq/mqtt-cli sub -t comment/liked -h test.mosquitto.org

Nothing more is needed. CLI connects to broker and subscribes to comment/liked and I should see incomming comment IDs:

$ docker run hivemq/mqtt-cli sub -t comment/liked -h test.mosquitto.org
60982511
93401458
47315214
30191547

All the dots are connected:

diagram showing event driven flow with broker in the middle and known consumer

Update template with working client code

This part is the most simple, I copy contents of my client.py and replace what I had in my index.js, to be exact, I replace asyncapi.info().title() and result is:

import { File } from '@asyncapi/generator-react-sdk'

export default function ({ asyncapi }) {
  return (
    <File name="client.py">
      {`import paho.mqtt.client as mqtt

mqttBroker = "test.mosquitto.org"
commentLikedTopic = "comment/liked"

class CommentLikedClient:
    def __init__(self):
        self.client = mqtt.Client()
        self.client.connect(mqttBroker)


    def sendCommentLiked(self, id):
        self.client.publish(commentLikedTopic, id)`}
    </File>
  )
}

This is not all yet. There are hardcodes, right?

Setup script that runs test code

Last step before actuall template work. During template development I will go back and forth quite a lot, so it would be nice to have a script that I can run to use the client, if it is still working.

Let's laverage the fact that templates are Node.js projects and that we use package.json.

In package.json you can have scripts property that later you can invoke by calling npm run <yout_script>.

Before I show you my scripts, lets have a look at the current structure of the project:

picture showing project structure

This are the scripts:

    "scripts": {
        "test:clean": "rimraf test/project/client.js",
        "test:generate": "asyncapi generate fromTemplate test/fixtures/asyncapi.yml ./ --output test/project --force-write",
        "test:start": "python test/project/test.py",
        "test": "npm run test:clean && npm run test:generate && npm run test:start"
    }

For clarity I created four scripts, to make it clear what each step of test script is doing.

You probably noticed something confusing, that is called rimraf. This is pretty standard package Node.js folks use in npm scripts for cleanup. The rimraf package is used to perform deletion. You will need to add it as development dependency to the template like this: npm install rimraf --save-dev

  • test:clean is needed because every time you run your test, you need to remove the old version of generated client.py
  • test:generate is responsible for calling AsyncAPI CLI to generate fresh version of client.py
  • test:start is for callin test python code that uses the client.py

And the only thing I have to do in terminal to check if my generated code works is: npm test

This time we can really start templating

The first part of the code that requires templating is:

mqttBroker = 'test.mosquitto.org'

The URL of the broker is not a static information, especially in professional environment where you have different instance for broker for production and testing.

In AsyncAPI document you can provide a list of different servers, but when you generate client code, it is for one of servers only.

AsyncAPI Generator has cool feature that allows you to make template configurable in runtime. It is called parameters.

Adding first parameters config

In package.json under generator property (the one where I already specified that the template is created with react render engine), I can create parameters object where I define a set of different parameters that user can pass to template in runtime and dynamically modify the output.

There is one special parameter that the generator supports and it is called server. The in the runtime I can say that "please generate code for me, but for server A" so in template code it is later much easier to extract server details from the AsyncAPI document.

Ok, bla bla bla. Give examples as for not it sounds misterious.

I added below to my package.json:

    "generator": {
        # ...(redacted for brevity)
        "parameters": {
            "server": {
              "description": "The server you want to use in the code.",
              "required": true
            }
        }
    }

"required": true makes the parameter mandatory and once user forgets about it, proper error message is yeld

This means that later when I trigger generation with AsyncAPI CLI, I can pass to the generator information about server to be used in generated client:

--param=server=dev

It simplifies template code a lot because I can then extract the URL of the server like this:

// "params" is a parametes object that generator injects into the template in runtime
// "asyncapi" is the instance of the parsed AsyncAPI document that I mentioned earlier in the article
asyncapi.servers().get(params.server).url()

So the template for now looks like:

import { File } from '@asyncapi/generator-react-sdk'

// notice that now the template now only gets the instance of parsed AsyncAPI document but also the parameters
export default function ({ asyncapi, params }) {
  return (
    <File name="client.py">
      {`import paho.mqtt.client as mqtt

mqttBroker = "${asyncapi.servers().get(params.server).url()}"
commentLikedTopic = "comment/liked"

class CommentLikedClient:
    def __init__(self):
        self.client = mqtt.Client()
        self.client.connect(mqttBroker)


    def sendCommentLiked(self, id):
        self.client.publish(commentLikedTopic, id)`}
    </File>
  )
}

If you follow the article the right way, you did npm test now and the test code created few chapters below works.

Templating of channel information

Next thing to template is commentLikedTopic = "comment/liked". Looks innocent, just a variable, but underneets it is a beast as it forces me to do some components composition using React.

Side small talk about React

This is the moment where I feel like I need to share more about React before I throw you at code like this:

<Text newLines={2}>
    import paho.mqtt.client as mqtt
</Text>

<Text newLines={1}>
    mqttBroker = "{ asyncapi.servers().get(params.server).url() }"
</Text>

<Text newLines={1}>
    <TopicVariable
        channels={asyncapi.channels().filterByReceive()}
    />
</Text>

This is the moment Java developer says: "what the hell is it, HTML in a template for code generator?"

I hope this code do not look scary, but if it does, lemme sell it to you.

First of all do not be scared of word REACT. You do not have learn a lot about it to create templates. You will not have to write code like this:

class ClassComponent extends React.Component {
  constructor(props) {
    super(props)
  }

  render() {
    // In `childrenContent` prop is stored `text wrapped by custom component\n\n`.
    // The content of the `children` prop is transformed to string and saved to the `childrenContent` prop.
    return this.props.childrenContent
  }
}

Above code scares the hell out of me too. You only need to know the generic concept of components in modern frontend frameworks. They exist to enable composition and reusability.

You need to remember that:

  • Your component should return string
  • You can reuse components and they can behave differently depending on the props you pass to them

Maybe I'm biased, but please look at below samples of template, written with classical templating enginge, Nunjucks, and React:

//Nunjucks

{% macro list(data, type = "-") %}
{% for item in data %}
{{type}} {{item}}
{% endfor %}
{% endmacro %}

{% from "partials/list.njk" import list %}
{{ list(["one", "two", "three"]) }}

//React

function List({ list = [], type = '-' }) {
  return list.map((item) => `${type} ${item}\n`)
}

// use `List` component in another component
export function SimpleList() {
  return <List list={['one', 'two', 'three']} />
}

Look into a mirror and answer that super important question every developer has to answer: "which code would you prefer to maintain in the future".

React toolset, community support, IDEs plugings. Yeah, AsyncAPI Generator still supports Nunjucks, but yeah, don't go this way.

Whey you encounter error like I did during development of this article, it will be super clear like below:

Generator Error: /Users/wookiee/sources/example-template/template/index.js: Unexpected token (41:67)

  39 |
  40 |       <Text indent={0} newLines={1}>
> 41 |         mqttBroker = "{ asyncapi.servers().get(params.server).url( }"
     |                                                                    ^
  42 |       </Text>
  43 |
  44 |       <Text indent={0} newLines={1}>

:heart:

Good luck having the same experience with Nunjucks.

I have spoken mandalorian meme
Small refactoring

In React the only disadvantage is that whatever component returns in a one line string. This means you need to care for new lines and indents.

Luckily the dependency I added at the beginning to my template have what we need. I import a Text component that I will use to wrap strings that need to be indent properly:

import { File, Text } from '@asyncapi/generator-react-sdk'

Maybe when you generate code you do not care that much on structure and proper indents, but yeah, I picked Python for that excercise,

     this is why
         indents
             are pretty
     important here

First I will refactor just part of this code:

// ...(redacted for brevity)
<File name="client.py">
  {`import paho.mqtt.client as mqtt

mqttBroker = "${asyncapi.servers().get(params.server).url()}"
commentLikedTopic = "comment/liked"
// ...(redacted for brevity)
      `}
</File>

It doesn't look nice because of indents and I need to refactor the topic name variable.

Paho module import import paho.mqtt.client as mqtt is static. I turn it into:

<Text newLines={2}>import paho.mqtt.client as mqtt</Text>

This way I assure that once this line of code is rendered in client.py file, it will add two extra new lines below that line.

Next is broker URL:

mqttBroker = '${asyncapi.servers().get(params.server).url()}'

I turn it into:

<Text newLines={1}>mqttBroker = "{asyncapi.servers().get(params.server).url()}"</Text>

You can see a slight difference between how dynamic information is templating in a string (first version) with $ and inside components it is slightly different.

Creating first component

And finally something more advanced, I need to template commentLikedTopic = "comment/liked".

My goal is to get working template code like:

<Text newLines={1}>
  <TopicVariable channels={asyncapi.channels().filterByReceive()} />
</Text>

You can see that I want to have TopicVariable component that I could reuse in other scenarios to return a list of topic variables.

In my scenario, I'm specifying that I want topic variables only for channels that Comment Service application is receiving messages from.

Recommended path is to put reusable components outside template directory in a new directory called components, surprise surprise.

I end up with code in components/TopicVariable.js that looks like this:

/*
 * This component returns a list of variables that hold information about topic name.
 * As input it requires a list of Channel models from the parsed AsyncAPI document
 */
export function TopicVariable({ channels }) {
  const topicsDetails = getTopics(channels)
  let variables = ''

  topicsDetails.forEach((t) => {
    variables += `${t.name}Topic = "${t.topic}"\n`
  })

  return variables
}

Not that scarry right? component is just another lovely JavaScript function.

I don't think there is much to explain here other than below:

{
  channels
}

This component argument is a standard way of enabling component to accept custom props. You can see here the channels prop I showed in action in a component few paragraphs above:

<Text newLines={1}>
  <TopicVariable channels={asyncapi.channels().filterByReceive()} />
</Text>

The component must return a string and this is what it is doing.

You can now ask what is getTopics(channels) function responsible for.

/*
 * This component returns a list of objects, one for each channel with two properties, name and topic
 * name - holds information about the operationId provided in the AsyncAPI document
 * topic - holds information about the address of the topic
 *
 * As input it requires a list of Channel models from the parsed AsyncAPI document
 */
function getTopics(channels) {
  const channelsCanSendTo = channels
  let topicsDetails = []

  channelsCanSendTo.forEach((ch) => {
    const topic = {}
    topic.name = ch.operations().filterByReceive()[0].id()
    topic.topic = ch.address()

    topicsDetails.push(topic)
  })

  return topicsDetails
}

Channel in AsyncAPI document holds a lot of different information. It is good for clarity to have separate testable code where you extract specific information from the AsyncAPI docs.

I won't explain entire JavaScript code but focus on the part where data is extracted from single channel:

topic.topic = ch.address()
topic.name = ch.operations().filterByReceive()[0].id()

Using Parser API I extract the channel address and record it as topic. In case of name I decided to use operationId of the operation associated with the channel.

You can see that I assume that there is just one operation. Which is completely fine prior 3.0 AsyncAPI spec and most pobably even in 3.0 people will still not have multiple receiving operations on the same channel, not in MQTT. To not complicate article and code more than needed, I just pick first operation from the list.

Final version of components/TopicVariable.js is:

/*
 * This component returns a list of variables that hold information about topic name.
 * As input it requires a list of Channel models from the parsed AsyncAPI document
 */
export function TopicVariable({ channels }) {
  const topicsDetails = getTopics(channels)
  let variables = ''

  topicsDetails.forEach((t) => {
    variables += `${t.name}Topic = "${t.topic}"\n`
  })

  return variables
}

/*
 * This component returns a list of objects, one for each channel with two properties, name and topic
 * name - holds information about the operationId provided in the AsyncAPI document
 * topic - holds information about the address of the topic
 *
 * As input it requires a list of Channel models from the parsed AsyncAPI document
 */
function getTopics(channels) {
  const channelsCanSendTo = channels
  let topicsDetails = []

  channelsCanSendTo.forEach((ch) => {
    const topic = {}
    topic.name = ch.operations().filterByReceive()[0].id()
    topic.topic = ch.address()

    topicsDetails.push(topic)
  })

  return topicsDetails
}

And now, in my template, in index.js I can import the component and this is how half-refactored code looks so far:

import { File, Text } from '@asyncapi/generator-react-sdk';
import { TopicVariable } from '../components/TopicVariable';

export default function({ asyncapi, params }) {



  return (
    <File name="client.py">

      <Text newLines={2}>
        import paho.mqtt.client as mqtt
      </Text>

      <Text newLines={1}>
        mqttBroker = "{ asyncapi.servers().get(params.server).url() }"
      </Text>

      <Text newLines={1}>
        <TopicVariable
          channels={asyncapi.channels().filterByReceive()}
        />
      </Text>


{`class CommentLikedClient:
    def __init__(self):
        self.client = mqtt.Client()
        self.client.connect(mqttBroker)


    def sendCommentLiked(self, id):
        self.client.publish(commentLikedTopic, id)`}
    </File>
  );
}

If you follow the article the right way, you did npm test now and the test code created few chapters below works.

Refactoring the rest

TODO: to be continued