You try to find reasons to believe you are exempt from suffering.

You want to believe that you are in control, that the world is fundamentally fair, and that you get what you deserve.

Terrible things can happen to you, for reasons that you cannot possibly control.

There is no fate to deserve. There is no lesson to learn. There is no reason to anything.

You will eventually be disassembled and spread across the vast cosmos to be anything else that could be except you.

The Universe will pour you back into the soup of everything.

Through a powerful abstraction like this, we could bring different stages of the generative pipeline of a Transformer-like architecture all the way up to the application layer.

Now that we are no longer limited by the static nature of prompts and hyperparameter configuration, we are able to apply advanced techniques for controlling the outputs of LLMs as a consumer.

# Constrained Decoding

My focus for this post is constrained decoding, specifically for text-based LLMs, but this could be applicable for any sequence-to-sequence generative model and may be extended beyond that.

In simple terms, constrained decoding consists of reducing the number of possible next-token predictions to guide the model to produce a more desireable outcome. This is usually done by providing logit biases for specific tokens at each decoding step.

An example of this is OpenAI's "JSON Mode", which, among other techniques, applies a grammar-based constraint in the decoding phase of GPT to reduce occurrences of invalid JSON syntax outputs. This methodology is likely also applied for OpenAI's function calling feature, to ensure function calls are correct.

To illustrate the ideas of this post, I decided to use JavaScript, so I could express my ideas in a langage that is more accessible to a wider audience, but this could be implemented in any language.

In fact, the AICI project has a JavaScript interface JsCtrl which you can use to write these controllers.

This is how an AICI controller looks like:

async function main() {
    await $`The answer to the ultimate question of life, the universe, and everything is `
    await gen({ regex: /\d\d/ })
}

start(main)

A likely output for this inference would be:

The answer to the ultimate question of life, the universe, and everything is 42

For general-purpose code generation, however, I would lean towards a more precise approach for constraint decoding that explicitly presents the abstract syntax tree (AST). This will improve readability and maintainability of the code, and will also allow for more advanced features like context-awareness and type-safety.

In general, the idea is to use functions with tagged template literals to express the AST of the code that we want to generate and the constraints that we want to apply to the LLM.

For example, a function called membersOf would be used to express a MemberExpression tree in JavaScript, and a function called property would be used to express a Property tree that is a child of a MemberExpression tree.

/**
 * 
 * @example Syntax Tree
 * MemberExpression(
 *   object: Identifier('user')
 *   property: Identifier('name')
 * )
 * @example Outputs
 * user
 * user.name
 * user.email
 */
membersOf`user`(
    property`name`,
    property`email`
)

Here's a more practical example of how this could be used to express a constraint to match one of five possible JavaScript member expressions:

import { oneOf, membersOf, property } from "?"

async function main() {
    await oneOf(
        /**
         * @example Output
         * meaningOfLife.answer
         * meaningOfLife.question
         */
        membersOf`meaningOfLife`(
            property`answer`,
            property`question`
        ),
        /**
         * @example Output
         * meaningOfLife.author.name
         * meaningOfLife.author.born
         * meaningOfLife.author.died
         */
        membersOf`meaningOfLife.author`(
            property`name`,
            property`born`,
            property`died`
        )
    )
}

start(main)

The content in the tagged template literals would work the same way as AICI's $ function, but with the added benefit of being able to mix the AST with the grammar constraints.

We could also express other syntactic features like function calls, computed properties, or even variable references.

import { oneOf, membersOf, callOf, property, arg, computedProperty } from "?"

async function main() {
    await oneOf(
        membersOf`meaningOfLife`(
            property`answer`,
            property`question`
        ),
        membersOf`meaningOfLife.author`(
            property`name`,
            property`born`,
            property`died`
        ),
        membersOf`inventory.fruit`(
            computedProperty`0`,
            computedProperty`1`,
            computedProperty`2`
        ),
        membersOf`userMap`(
            computedProperty`userId`,
        ),
        /**
         * @example
         * window.localStorage.getItem(userMap[userId])
         */
        callOf`window.localStorage.getItem`(
            memberOf`userMap`(
                computedProperty`userId`
            )
        )
    )
}

start(main)

At a surface level, these could be expressed as a grammar, in fact here's how it would look like in EBNF:

RULE_0_A := "name" | "born" | "died"
RULE_0   := "meaningOfLife" "." [ "answer" "question" "author" [ "." RULE_0_1 ] ]
RULE_1_A := "0" | "1" | "2"
RULE_1   := "inventory" [ "." "fruit" "[" RULE_1_1 "]" ]
RULE_2   := "userMap" [ "[userId]" ]
RULE_3   := "window.localStorage.getItem" [ "(" "userMap" [ "[userId]" ] ")" ]
OUTPUT   := RULE_0 | RULE_1 | RULE_2 | RULE_3

However, expressing the AST in our decoding process has the added benefit of being able to express constraints that are also semantically significant.

# Semantically-Aware Constrained Decoding

So far, we've only been talking about static semantic constraints. But what if we could go further? Could we handle dynamic semantic constraints?

Consider this example, which constrains the LLM to pick between two different API calls fetchUsersByEmail and postUser. This illustrates how str() could be used to express the AST of the code that we want to generate while also providing the LLM opportunities to fill in the blanks.

import { oneOf, callOf, obj, property, str } from "?"

const name = str()
const email = str()

/**
 * @example Output
 * fetch("https://api.example.com/users?email=john.doe@example.com")
 */
const fetchUsersByEmail = callOf`fetch`(
    str`https://api.example.com/users?email=${email}`
)

/**
 * @example Output
 * fetch("https://api.example.com/users", {
 *   method: "POST",
 *   body: {
 *     name: "John Doe",
 *     email: "john.doe@example.com"
 *   }
 * })
 */
const postUser = callOf`fetch`(
    str`https://api.example.com/users`,
    obj(
        property`method`("POST"),
        property`body`(
            obj(
                property`name`(name),
                property`email`(email)
            )
        )
    )
)

async function main() {
    await oneOf(
        fetchUsersByEmail,
        postUser
    )
}

start(main)

But generating literals is... well, too literal (and let's be honest, table stakes).

Of course the model would generate values for name and email, but in the real world we may want to use values that are present in the context of the application like variables and other expressions.

What if we used a special contextOf function to express the context of the application in a way that can be referenced by the LLM in a semantically-accurate way?

import { contextOf, callOf, obj, property, str, num, bool } from "?"

/**
 * @example Output
 * session.user.name
 * session.user.email
 * api.url
 */
const context = contextOf(
    membersOf`session.user`(
        property`name`(str),
        property`email`(str),
        property`age`(num)
    ),
    membersOf`api`(
        property`url`(str),
        property`public`(bool)
    )
)

const postUser = callOf`fetch`(
    str(),
    obj(
        property`method`("POST"),
        property`body`(
            obj(
                property`name`(str()),
                property`email`(str())
            )
        )
    )
)

async function main() {
    await context;
    await postUser;
}

start(main)

This contextOf function is not a function that generates a context, but a function that references a context and makes it available to future generative steps.

We would implement our str handler to prompt the model to generate a string literal or reference a string from the context.

# Speculative Implementation Details

Behind the scenes, a type mapping could be generated from the contextOf call, and the LLM would be guided to generate values that are semantically accurate and type safe.

string:
    - session.user.name
    - session.user.email
number:
    - session.user.age
boolean:
    - api.public
// ...
SomeType:
    - some.path.to.value

Practically-speaking, the real context would be provided by the application at runtime, but it wouldn't even be necessary to provide it at inference time.

# Type-Safe Generative Variables

If we want to allow the LLM to output variable declarations, we could define a variable function.

Then, when generating arguments for function calls, we would constrain the LLM to use a variable that is already in the context or a literal value that satisfies the type of the argument.

Here's a simple arithmetic calculator used by contraining the model to only output variable declarations of explicitly number type and function invocations of explicitly the add, sub, mul, and div functions.

import { manyOf, callOf, returnOf, variable, id, num } from "?"

async function main() {
    /**
     * @example Prompt
     * I bought 3 apples (3.99 ea) and 4 oranges (4.39 ea).
     * 
     * How much did I spend?
     * @example Output
     * let apples = 3
     * let oranges = 4
     * let applePrice = 3.99
     * let orangePrice = 4.39
     * let appleTotal = mul(apples, applePrice)
     * let orangeTotal = mul(oranges, orangePrice)
     * let total = add(appleTotal, orangeTotal)
     * return total
     */
    await manyOf(
        variable`${id()}`(num()),
        variable`${id()}`(callOf`add`(num(), num()))
        variable`${id()}`(callOf`sub`(num(), num())),
        variable`${id()}`(callOf`mul`(num(), num())),
        variable`${id()}`(callOf`div`(num(), num())),
    )
    await returnOf(num())
}

The LLM would still need to generate the variable declarations and function invocations, but it would be guided to generate semantically-accurate and type-safe code.

# Bonus: Refactoring

We could express the same code in a more readable, less-verbose way by abstracting the variable function.

import { manyOf, callOf, returnOf, variable, id, num } from "?"

const varFor = variable`${id()}`

async function main() {
    await manyOf(
        varFor(num()), // e.g. let apples = 3
        varFor(callOf`add`(num(), num())) // e.g. let totalFruit = add(apples, oranges)
        varFor(callOf`sub`(num(), num())), // e.g. let oneMinusTwo = sub(1, 2)
        varFor(callOf`mul`(num(), num())), // e.g. let applesTotal = mul(apples, applePrice)
        varFor(callOf`div`(num(), num())), // e.g. let orangeToAppleRatio = div(oranges, apples)
    )
    await returnOf(num()) // e.g. return orangeTotal
}

# Conclusion

I know that this is purely a theoretical idea at this point, but I think it's a very interesting one that would allow us to get semantically-accurate and type-safe code generation from LLMs.

This works in theory, but there are many practical challenges to overcome. For example, how would we handle the context at runtime? How would we handle the type mapping? How would we know when to pause and resume the inference process to provide the context?

I deeply believe these are all solvable problems, and I'm excited to see where this goes.

The Artificial Intelligence Controller Interface (AICI) lets you build Controllers that constrain and direct output of a Large Language Model (LLM) in real time. Controllers are flexible programs capable of implementing constrained decoding, dynamic editing of prompts and generated text, and coordinating execution across multiple, parallel generations. Controllers incorporate custom logic during the token-by-token decoding and maintain state during an LLM request. This allows diverse Controller strategies, from programmatic or query-based decoding to multi-agent conversations to execute efficiently in tight integration with the LLM itself.

This is such a clever idea. A couple months ago I talked about being able to control the output of an LLM in real time. This does exactly that, but better because you can write your own custom logic to control the output.

What is most interesting to me is not only the what but also the how:

AICI is designed for both local and cloud execution, including (eventually) multi-tenant LLM deployments. Controllers are implemented as light-weight WebAssembly (Wasm) modules which run on the same machine as the LLM inference engine, utilizing the CPU while the GPU is busy with token generation. AICI is one layer in the inference stack, and is designed to allow control libraries such as Guidance, LMQL, and others to run on top of it and gain both efficiency and performance improvements, as well as portability across LLM inference and serving engines.

WebAssembly is a perfect fit for this. It's standard, fast, and embeddable. More importantly, it is relatively easy to run arbitrary code in a sandboxed, memory-safe execution environment, which is ideal for this type of use-case.

Once again, the Web Platform is the future of software. I'm glad to see Microsoft embracing it. I hope OpenAI and other LLM developers follow suit and implement something like this as well.

Apparently they are not announcing new foundation models (i.e. GPT-5), but are instead going to focus on new features and tools for their APIs.

This is my wishlist, categorized in three buckets: Likely Suspects, Big Moves, Moonshots.

# Likely Suspects

# General Performance Improvements

This one feels obvious to me, faster gpt-4, it's a no-brainer. Faster time-to-first-token, faster inference times, faster everything. GPT-4 is very slow. It's still usable with streaming, but it's too slow.

I'm not sure if this will be in the form of a "distilled" version of existing models, a completely new model, or just general infrastructure improvements across the board. Will gpt-4 just be faster or will we see a gpt-4-turbo?

Maybe they will also release an even faster gpt-3.5-turbo?

# GPT-4 Fine-Tuning API

This one is also a no-brainer. We already have the fine-tuning API for gpt-3.5-turbo, so it's only natural that we get one for gpt-4. Right? Right!?

I'm interested in how the economics are going to look like, but the general idea is that you can fine-tune gpt-4 to your specific domain or use case and get better results using fewer tokens, which generally means it's going to be cheaper for specific cases.

# Public DALL·E 3 API

We already have an API for DALL·E 2, of course they're working on making DALL·E 3 public! In fact, they already said it was going to be released late fall when they publicly announced it.

What I'm really wondering is the pricing, though. Will it be the same as DALL·E 2? Will it be more expensive? Will it be cheaper because they achieved some sort of economies-of-scale win? I'm not sure, but I'm excited to find out.

# Multimodal API

GPT-V, as OpenAI has been marketing it, is already making its way into ChatGPT and Bing, so it is possible that they bring it to the API.

I'm not sure if it will be a completely new model or just a modification to the API that allows sending images and text together.

I'm curious about how the API would look like and how the pricing would work. Are images encoded as base64 strings? Are they sent as URLs? Are they sent as binary blobs? Are they sent as a combination of all of the above?

# Playground Improvements

I'm hoping for some improvements to the playground, especially if multimodality comes through. I'd like to see a token counter, support for function calling, and maybe even templating?

# Big Moves

# Ability to Pause, Insert Into, and Resume Inferences

I don't know of a clever way to call this. But I would like to have longer-lived inferences that can span multiple requests.

Edit You might be thinking, "oh so you want a stateful API?". Short answer is yes, but that is an implementation detail. I just want the ability to have longer-lived inferences. I don't care how this is achieved, I just want it to be possible.

A general idea is that you could provide some sort of token to pause the inference, freeze the state of the infrerence in place, provide more context to bias the rest of the inference, and then resume it.

If possible, maybe even save in token costs along the way?

llama.cpp has a similar feature, where you can pause the inference at any time, inject some text, and then resume the inference as if the model produced the text you injected.

This would open the door for one-shot reACT loops. Fewer tokens, but longer inference lifespan. I think it'd be worth it.

We'd be able to do things like:

Sure, I'll unsubscribe you from the
/unsubscribe(User)⏸️Successfully unsubscribed User⏩️Done!

The ability to pause, insert into, and resume the inference.

The ability to pause... are you getting it? These aren't three separate inferences, it's one single inference that can be paused, inserted into, and resumed.

# Moonshots

# Output Templates

It would be great to be able to provide a template to the API and have the LLMs exclusively sample tokens in a way that the full output conforms to it. This would be amazing for things like biasing the model to output code, JSON, HTML, etc.

A good medium to allow this feature would be through the usage of a grammar file. llama.cpp, for example, supports this feature.

# Regular Updates to Training Data

What if OpenAI had a regular cadence of releasing new models with updated information about the outside world (e.g. current events, new people, new places, new things, etc.)?

# More Deterministic Models

I know, I know... LLMs are inherently stochastic, but hear me out. I have a more practical solution to this problem.

I am under the impression that OpenAI's models are not frozen in time, that they are constantly being updated and retrained. This means that the same input prompt can produce different outputs over time.

OpenAI kinda, sorta versions their models, but it is widely understood that they are making changes under the hood all the time.

Here's a comprehensive list of things that could be altering the output of the model:

• Adding new training data
• Changing the model architecture
• Modifying the hyperparameters
• Altering the training process
• Applying learnings from RLHF
• Updating the tokenization process
• Scaling the API depending on load
• Changing how the API is deployed

It must be happening, no?

I'd like to see them release frozen models that are guaranteed to produce similar output for the same input prompt (as much as its stochastic nature allows).

Or, if they already guarantee this inherently, I would like to see them make it more clear.

# Built-In Code Interpreter API

What if the API had a built-in runtime to execute code for popular programming languages (Python, JavaScript, TypeScript)? It would be amazing to have the model run the code for you in a sandboxed environment and use the output.

Here's an imaginary transcript of the prompt and response:

[
  {
    "role": "user",
    "content": "Give me a random number from 1 to 10"
  },
  {
    "role": "assistant",
    "to": "javascript",
    "content": "Math.floor(Math.random() * 10) + 1"
  },
  {
    "role": "interpreter",
    "name": "javascript",
    "content": "3"
  },
  {
    "role": "assistant",
    "content": "How about 3?"
  }
]

How cool would that be?

QQ is my proposal for a new editing modality that allows the user make quick consults with a language model that has been fine-tuned for chat within the editor. It offers a frictionless way to ask for explanations, examples, and documentation without interrupting the user's workflow.

Here is a brief demo of QQ in action:

# How to Trigger QQ

When the user types qq at the beginning of a line or comment, the editor shows a floating tooltip below the cursor line. The text editor then functions like a text input where the user can type a question like "what does this code do?".

When the user presses enter (or return on macOS), the question is sent to a language model – in the case of the demo above, OpenAI's gpt-3.5-turbo, – which streams back a response to answer the user's question. The response is directly displayed in the tooltip, so the user doesn't have to leave the editor to see the answer.

The user can then choose to either close the conversation by pressing esc or continue sending replies to the model to iterate on the generated responses. When the user closes the tooltip, the edited line is cleared and the editor returns to its regular state.

# Why Modal Editing?

Modal editing is a way to preserve flow by reducing the number of context switches. It's a way to "stay in the zone" and keep your hands on the keyboard while you're working. With modal editing there's no need to reach for the mouse, or to switch to a different pane to look something up. You will be able to type stuff out, and then ask for help, all without leaving the editor or in many cases switching focus away from the text editor itself.

# Presenting the Conversation

In screen readers, the tooltip is announced as a live region, and the user is informed that they are in a conversation with a language model. I still need to do some research on how to best present the conversation to screen readers, but live regions seem to work better than a modal dialog, for example.

The content of the conversation is presented as a list of message bubbles, analogous to a two-way chat conversation between the user and another individual. Like in an instant-messaging application, the user's questions are shown as accent-colored message bubbles on the right, and the responses from the language model are shown as message grayscale bubbles on the left.

When the conversation is empty, the user is presented with an empty state that shows "No messages".

As the user types their question, the dialog shows the user's question as a message bubble, but is presented in an indeterminate state, communicating that the question still has not been sent to the language model.

When the user presses enter, the question is sent and the message bubble's visual state is updated to indicate it. A live announcement is dispatched to the screen reader through ARIA live regions as well, communicating that the message has been sent.

As the response from the language model is received, the response is shown as a message bubble in a similar indeterminate state. When the response is received, the message bubble is updated to show the response.

Responses from the language model are expected to be richer in content and format than the user's questions, so the message bubbles are styled differently to communicate that. Responses from the LLM also support markdown formatting and syntax highlighting of code-blocks to make the content more digestable and reable for the user.

# Managing Conversations

The user can close the conversation by pressing esc or enter on an empty line. When the conversation is closed, the tooltip is hidden and the editor returns to its regular state, but the conversation is still available in the editor's state. The user can trigger qq again to reopen the conversation and continue where they left off, or to review the conversation again.

Conversations are grouped by line number, so that the user can have multiple conversations open at the same time. This is useful for when the user is working on multiple lines of code at the same time, and wants to ask questions at different parts of the code.

# Prompting

The contents of the active document are sent alongside the user's question to the language model. This is done to provide context to the language model, and to help it generate more relevant responses for the user.

Here is the prompt that is sent to the language model:

As "Loop", you're a programming assistant driven by a language model.
Respect user's instructions strictly, but avoid discussing opinions, existence, sentience, or engage in arguments.
Provide accurate, logical, brief, and Markdown-formatted responses with named language in code blocks.
Respond to technical inquiries with code suggestions only within code blocks.
Account for the user's language in code blocks when using the text editor, Printloop.
Limit responses to one per interaction turn.
---
location: {window.location.origin}{window.location.pathname}
editing document: printloop.md
contents:
# A Simple SQLite Database

Here is a simple SQLite database:

```sqlite!
CREATE TABLE messages (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  text TEXT
);

<- user's cursor is here

INSERT INTO messages (text) VALUES ('Hi!'), ('How are you?');

UPDATE messages SET text = 'Hi there!' WHERE id = 1;

SELECT * FROM messages;
```

I've been using this prompt for a while now, and it's been working well. I'm sure there are ways to improve it. I'm also thinking of adding a way for the user to customize the prompt, or to add their own prompts.

# Next Steps

I'm still working on improving the experience of using QQ, and I'm looking forward to hearing feedback from users. My main focus right now is on improving accessibility, and making sure that the conversation is presented in a way that is easy to follow and understand for screen reader users and users with other cognitive and physical disabilities.

I also want to improve the triggering mechanism. Right now, the user has to type qq at the beginning of a line or comment, but I want to make it easier to trigger the conversation. I'm thinking of adding a keyboard shortcut, or a way to trigger the conversation by listening to qq as a series of keystrokes instead of parsing it from the text buffer per se.

# Try It!

Type qq in printloop.dev to open the QQ modal. Then, type a question, like "what is a monad?".

h(x, y): |hate| in {x} for {y}
t(x, y): |trait|{x} in {y}
a(x): |amount| of {x}

T: any
Y: |you|
O: |other people|

A: a(h(Y, t(T, Y)))
B: a(h(Y, t(T, O)))

A ∝ B

I've been building a lot of stuff with them, mainly Printloop, but also BORI.

This week is expected to be big for language models as GPT-4 is rumored to be announced, but also high-performance language models are very close to land in consumer hardware!

# Language Computer

I've been looking for ways to build a "Language Computer" for my home; I want to run a service on my own hardware that I can use to build personal language tools, services, and home automation workflows.

I also want to host other AI services at home (like speech-recognition), but my focus is on language models first because I believe it's the most useful.

Turns out, I am very close to making this a reality! We'll soon be able to run a language model with performance comparable to OpenAI's Davinci GPT models on our own hardware.

I highly recommend reading Simon Wilson's "Stanford Alpaca" post discussed the acceleration of large language model development on-device, particularly talking about Stanford Alpaca and llama.cpp.

I already so many ideas for things I will be able to do with my own language service at home:

• Digitize, summarize, categorize, and sort my snail mail
• Digest, summarize, and present the news to me
• An army of mechanical historians that indefinitely browse the web, recording findings about topics that I am interested in, keeps track of trends, and writes me a report every day

Well, not so secret, as I've been regularly posting my progress on Twitter. I've been working on it since March 20 and I'm pretty happy with my progress so far.

# What I'm Building

All I have is a faint idea of the type of product I want to build: a beautiful, low-code, statechart editor. Doesn't have to be full-featured, but enough to be dangerous.

Of course, a statechart editor is not a new idea at all, many software products like these exist already. My favorite one so far is Stately, founded by David K. Piano, creator of XState and a popular advocate for state machines and statecharts.

But I'm not trying to build a power tool like Stately! I want this to be more of a delightful toy tool. Or at least I want that to be the driving idea.

# What Do You Mean by Beautiful?

I want UN1 to be nice to look at, even if it's at the expense of some performance. I want it to be easy to use, visually-appealing, and interactive. I do not care if that complicates my project or I have to sacrifice some feature for it.

The good thing about having no commercial incentive for working on a project is that I can make decisions that don't make economical sense. I'm just in it for the fun of building it.

I want to work on rendering the arrows between elements, to build the infinite-canvas layout with pinch-to-zoom and panning.

I want this to have eye-catching visuals, a great UX, and a playful tone.

# Why Low-Code?

I am currently working on a very ambitious low-code product at Airkit, so I think about problems like these all the time. Having side-projects like UN1 help me explore ideas beyond the scope of my work while still exercising my creativity in a similar context to the problem I try to solve for a living. It does wonders for my easily-distracted brain.

I want the graphical user interface to allow me to build statecharts without having to even think about code – except for maybe side-effects and guards. And even then, try to reduce those to a GUI whenever possible.

# Let's See How it Goes

I am explicitly avoiding naming this project until I figure out the finer details of the design. The last time I was so excited about a project was back in 2018 when I was working on M1N, which turned into Minimo.

When writing a stylesheet, :scope is the same thing as :root, because there is no way to write scoped stylesheets. Scoped stylesheets was a feature that was proposed some time ago, and some browsers experimented with, but have since been removed from browsers and removed from the spec.

Even though we cannot do anything useful with :scope in our stylesheets, we can use the :scope pseudo-class with the DOM API!

When using DOM API methods that expect CSS selectors, you can use :scope to refer to the element from which you are calling it!

element.matches(":scope") === true; // element is itself, so true!

# Things You Can Do with :scope

# Selecting Direct Descendants

:scope is particularly useful when selecting direct descendants of the current element that match some criteria!

element.querySelector(":scope > input#username");

An alternative, would be to get all the children of element and match them against a selector.

Array.from(element.children).find((child) => child.matches("input#username"));

Another approach would be to refer to the current element using an ID or class:

element.setAttribute("id", "myid");
element.querySelector("#myid > input#username");

But this would be a bit harder to maintain, as the selector and whatever attribute we are using to match the current element would need to be in sync.

# Assertions That Involve Hierarchy

Another use case for :scope is to elegantly assert whether the current element matches a particular selector without needing to use an ID or class to refer to the current element.

// element is a direct child of `div#wepa`
element.matches("div#wepa > :scope"); // => `boolean`

// element is descendant of #content
element.matches("#content :scope"); // => `boolean`

// element is the next sibling of a `label`
element.matches("label + :scope"); // => `boolean`

// the next sibling of the element is a `label`
element.matches(":scope + label"); // => `boolean`

// element is a descendant of a `form` element,
// and has a `textarea` as a child
element.matches("form :scope > textarea"); // => `boolean`

Again, there are other ways to do this, but this way is probably more expressive, concise, and easier to maintain.

# Limitations

A noteworthy limitation that I've noticed, however, is that using Element.querySelector() or Element.querySelectorAll() to match siblings of the current element does not work.

This likely due to the fact that querySelector and querySelectorAll only consider the descendants of the current element, so it makes sense.

// try to get all siblings of the current element
element.querySelectorAll(":scope ~ *"); // => Empty NodeList

// try to get the next sibling of the current element
element.querySelectorAll(":scope + *"); // => Empty Nodelist

In that case we still have to use the traditional methods, like getting all the children of the parent element, matching them against a selector, and filtering out our "target element".

Likely for the same reason, :scope cannot be used to match the current element itself. Which would not be useful, but interesting to know about.

element.querySelector(":scope"); // => null

# Browser Support

I expected :scope to have limited browser support. Surprisingly, at the time of this writing :scope has very good browser support.

The TL;DR is if you don't have to support Internet Explorer, you can use :scope!

# What's Next?

I hope you enjoyed this quick writeup. I'm going to write a couple more about other interesting Web features that I've found myself using recently.

My blog posts are simple MDX files, so all I really need is a decent text editor with MDX syntax highlighting. Unfortunately, I haven't found many MDX editors out there, so I've had to use VSCode all this time.

But now I can just use VSCode in the browser!!! There's a very nice MDX extension for VSCode, which I can also install on github.dev. So now I just open my site's GitHub repo page, press . and boom, I'm in VSCode in seconds.

From there, I can just create a new .mdx file, edit it, and commit the changes straight from my browser. Committing my changes immediately triggers a new release. Netlify builds my site, generates a new open graph image for my post, and we're done!

From now on, I'll be publishing blog posts from my browser using github.dev. Pretty neat.

Needless to say, I no longer look like myself 5 years ago. I've put on a few pounds and my beard is a mess these days.

I don't take many pictures of myself, at least not many that I would use as my avatar. But I joined a new job last fall with a bunch of people who I haven't met in person so I figured that the honest thing to do would be to change it. You know, to put a face to my name.

I took it with my webcam, because why would I take it with my very good iPhone camera when I can take it with a terrible potato camera instead. It's good enough, though. And it's more honest, I look friendlier (I think), and I'm wearing my favorite hoodie.

I've been using it exclusively as my internal headshot at Airkit since I joined, but I think I'm finally ready to let go of my old one and update every social media profile I have.

Here is to the next 5 years.

I'm ashamed to admit that more often than not, I'm trying to find an elegant, yet grandiose solution that will change everything. An explanation so concise that it can be expressed with a simple sentence. An approach so fantastic that I have to share with everyone.

The problem with this kind of idea is that they are usually vague, immature, and almost always rooted in fantasy. You couldn't pierce such an idea with an invisible needle, because it would evaporate right before your eyes.

But these ideas keep me alive, they provide me with enough incentive to keep me engaged. They take fascinating shapes and make interesting noises. They are invisible but always feel like I am three diagrams and two essays away from making them concrete.

At the beginning, I find myself lost in thought in the middle of my day fantasizing about the dream solution. But as time progresses, the problem becomes more concrete and the novelty of the idea slowly fizzles out. At the end of it I'm left with the final product. Not groundbreaking as I imagined it, but pretty solid.

For me, clever ideas are not the vehicle, they are the fuel.

I used Puppeteer to take a screenshot of a very simple HTML template, hooked it into my website's deployment pipeline, and voilà: dynamic OG images!

There's still some tweaks I wanna make to the template to make it nicer, but it's looking clean!

I wrote generator code that gets executed as part of my next export process, this is the full script:

import chromium from "chrome-aws-lambda";
import format from "date-fns/format";
import fs from "fs";
import path from "path";
import posts from "./posts";

export async function generate({ outDir }) {
  const browser = await chromium.puppeteer.launch({
    args: chromium.args,
    executablePath: await chromium.executablePath,
    headless: true,
  });

  const page = await browser.newPage();

  // Open the HTML template as a data URI
  await page.goto(
    `data:text/html;charset=UTF-8,${encodeURIComponent(TEMPLATE)}`,
    { waitUntil: "networkidle0" }
  );

  // Wait until the document is fully rendered
  await page.evaluateHandle("document.fonts.ready");

  // Set the viewport size to match
  await page.setViewport({
    width: 1200,
    height: 632,
    // Netlify build machines do not have a
    // retina display, apparently xD
    deviceScaleFactor: process.env.NETLIFY ? 1 : 2,
  });

  const ogImagesDir = path.resolve(outDir, OG_IMAGES_PATH);

  // iterate over each post
  for (const post of posts) {
    const info = {
      title: post.title,
      date: format(post.date, "dddd, MMMM Do, YYYY"),
    };

    await page.evaluate((info) => {
      const titleElement = document.querySelector("#title");
      const dateElement = document.querySelector("#date");
      titleElement.innerHTML = info.title;
      dateElement.innerHTML = info.date;
    }, info);

    await page.screenshot({
      path: `${ogImagesDir}/${post.id}.png`,
      type: "png",
      clip: { x: 0, y: 0, width: 1200, height: 632 },
    });
  }

  // Wrap it up
  await browser.close();
}

This code heavily inspired by vercel/og-image's code.

Microsoft is building a universal Outlook client for Windows and Mac that will also replace the default Mail & Calendar apps on Windows 10 when ready. [...] The project will deliver Outlook as a single product, with the same user experience and codebase whether that be on Windows or Mac.

This most definitely means they're going to be using something like Electron.

And who could blame them? For a software product that is expected to run everywhere, having only one desktop client codebase just makes sense.

Shipping features frequently and consistently across platforms is a really complex problem that takes a remarkable level of coordination and engineering discipline. Not everybody can pull it off, and even those who do, still have to deal with big tradeoffs.

For a market as competitive as email, having cross-platform desktop applications that work consistently for every platform can be a huge differentiator. For enterprise customers, this might even be expected of your product (see how many platforms Slack supports).

People often criticize when a big company switches their native clients to Electron, and with good reason. Since the framework bundles apps with its own instance of Chromium, Electron apps are usually bloated, glitchy, and resource intensive.

But even so, Electron's value proposition is too attractive.

This wouldn't be the first time Microsoft moves a fully native app to Electron, and it certainly won't be the last. They did it with Skype, and were quite successful at it.

Last year, lots of things changed for me (as it did for many of us). This post is for those changes I do want to share.

# I Moved

Small move. I moved from Oakland to San Jose. The weather is even nicer, rent is cheaper, etc. No particular reason why I moved from Oakland other than it didn't make sense for me to live up there.

# Claimbot Was Acquired

In a remarkable turn of events, last fall, Claimbot's virtual chat engine was acquired by Mobile Tech Rx! This also means that I'm no longer working at Claimbot.

Closing this chapter of my life was a bitter-sweet moment. I feel very lucky for having the opportunity to build a great product alongside very talented teammates. I'll miss it dearly.

I am happy that all the work that we've put on the platform for the past four years will live on and be useful in different ways for the very talented folks at Mobile Tech Rx. Our work is in good hands, and I can't wait to see what they build on top of it.

Having worked on Claimbot is an experience I'll cherish for the rest of my career. I have grown exponentially as a professional and as a person, and I owe much of it to my co-founder, partner, and mentor, Miguel Fernandez, who started and gracefully lead this whole endeavor.

Gracias, broder.

# I Got a New Job

Finally, as a direct – yet somehow also parallel – consequence of Claimbot's news, I started working at a new company, Airkit. In fact, most of my teammates from Claimbot are now working at Airkit too!

Airkit is a platform that empowers teams to build customer experiences without having to write a lot of code.

I'll be working as a Full Stack Engineer on Airkit's core platform. Particularly with the Developer Experience team, which mainly focuses on the Studio portion of Airkit, the interactive tool that companies use to build these low-code customer experiences!

There are lots of places where Airkit and Claimbot intersect, from go-to-market strategies and target industries, to tech stack and architecture design. Customer experience automation, configurable software, and full stack JavaScript is proving to be a career theme for me, which is awesome too!

I have worked at Airkit for the past ~3 months. So far, my experience has been delightful. Airkit is a great company, with a great team and product, and lots of potential. I've learned a lot so far and I'm excited to see what I'll learn next!

Also, we're hiring!

# Onwards

Anyways!

As I said earlier, 2020 was objectively bad for the world and my expectations for 2021 are not that high either. However, on a personal level, I am thankful for all the good things that did happen last year and I'm looking forward for the good parts of this one.

Take care, everyone.

Eric Yuan (Zoom's CEO) published what looks like an apology reacting to the criticism they've been receiving lately and acknowledging most of Zoom's mistakes. In the post, he announced that Zoom will be stopping the development of any new features to focus on addressing current security and privacy concerns.

I found this post reassuring and sincere. Large companies rarely — if ever — admit that many mistakes in a single post. It gave me hope that they are trying to make a secure and private product — even if their incentive to do so comes from the market, and after the fact, rather than their proactive willingness to build a quality product. Today, market pressure seems to be the only way to hold corporations accountable.

The thing is, Zoom knows better than any of us what are the internal forces that are causing all these issues. They seem to be cutting corners that a lot of startups usually cut, but they are doing it at an unforgivable scale. Zoom has become too popular, too important to ignore these mistakes, and serious customers are starting to lose trust in the platform.

In Eric's post, he also mentioned that they set themselves a 90-day deadline to figure it out. With the global demand for video conferencing software skyrocketing to unprecedented levels due to the global COVID-19 pandemic, Zoom needs more than ever to focus on two main things: anticipating scale and sacrificing user experience, all in favor of security and privacy.

# Anticipating Scale

We all know that the scale they are dealing with is unprecedented. In the month of March 2020, Zoom was handling around 200 million meeting participants a day.

During a global pandemic that made most businesses and schools — at least those who can — switch to video conferencing as their main method of communication, their video conferencing product still works really well.

It would simply not be fair to say that Zoom has not handled scale correctly. Even before the global COVID-19 pandemic, their video conferencing software was among the most reliable consumer-grade video communication platforms out there.

That said, there are also catastrophically incorrect ways to handle scale that are not necessarily linked to audio/video performance, like generating guessable meeting IDs, or not enabling meeting passwords by default.

You've probably read of the consequences of these design decisions. Hundreds of stories of "Zoom-bombings", a type of prank that consists of an unknown party joining a meeting they do not belong to and saying or presenting something obscene or disturbing to the participants of the call in an act of crude vandalism.

There have been multiple attacks of this nature in virtual classrooms and remote thesis defenses, which led the FBI to issue a warning about this new fenomenon.

Thankfully, this issue is easily fixed by manually password-protecting your Zoom meetings, but they could've made Zoom-bombing harder from the beginning. They could have enabled password protection by default on all meetings or at least used longer, harder-to-guess, alphanumeric keys as meeting IDs. At the very least they could have hired a security firm to audit their security and privacy practices; any internal team that was proactively looking for a vulnerability would have found this one.

Whether they anticipated this problem at scale or not, this kind of behavior was inevitable and Zoom should've made these decisions months or maybe years ago. When it comes to security and privacy, you should be able to anticipate scale, especially when designing something as crucial, yet easily-avoidable, as unique IDs.

# Stop Obsessing with Reducing Friction

Not so long ago, a security researcher found a security flaw that allowed any website to start a Zoom meeting without any interaction — or request for consent — from the user. It took advantage of a Zoom "feature" that exposed a local, ever-running web server that enabled the native Zoom client to open a Zoom window on demand. All of this to save Zoom users a single click.

I get it. The state of video conferencing these days is confusing and complicated; every click matters. There are many operating systems and platforms to support, hundreds of services to integrate with, millions of businesses to connect, and many more millions of people with different setups trying to figure out how to enter these damn meetings.

Services like Microsoft Teams or Slack require you to have a common communication platform with your co-participants (i.e. they must all be on Slack or Microsoft Teams) — which usually isn't the case. Services like Skype require you to have a Microsoft account. And services that allow you to dial into a video conferencing meeting without signing-in or having a common platform sign-in could be rare, expensive, and/or unreliable — in some cases all three.

There is a need for affordable, easy-to-use, and reliable video conferencing software and that's what Zoom's competitive advantage is all about.

But practicality and ease of use usually comes at a great cost: security and privacy. Skipping passwords, settling for easy-to-spell meeting IDs, and even saving a single click could mean sacrificing our natural expectations of privacy and security.

Zoom needs to start adding more security controls in place, by default, even if it meant sacrificing the user experience by adding a little friction. This aggresiveness against user friction is no longer helping build Zoom's case, it's deteriorating their reputation.

At this point Zoom clearly has all the demand it needs to be successful for a while, so it's absurd to believe that a couple clicks will sacrifice any significant part of your sales. And even if it did, they wouldn't compare to the business they are missing right now because of their... insecure or at the very least suspicious practices.

# Wrapping Up

I know that everything is clear in hindsight. It's super easy for me to sit here and point all the flaws that Zoom has, but I believe that they could've done better. Zoom is an excellent product, with high demand right now, that also performs really well. If they fix all this mess, they have the opportunity to position themselves among the best consumer video-conferencing products out there.

Today we have some important and exciting news to share: Dark Sky has joined Apple.

Our goal has always been to provide the world with the best weather information possible, to help as many people as we can stay dry and safe, and to do so in a way that respects your privacy.

There is no better place to accomplish these goals than at Apple. We’re thrilled to have the opportunity to reach far more people, with far more impact, than we ever could alone.

There's no need to speculate on why this is a sound acquisition for Apple. They will always need to improve their weather capabilities on their operating systems and Dark Sky is a premium product that is precise and offers reliable forecasts. It checks out.

The [Android] app will no longer be available for download. Service to existing users and subscribers will continue until July 1, 2020, at which point the app will be shut down. Subscribers who are still active at that time will receive a refund.

Yikes. It makes sense that they are doing this, but it's a huge stab in the back for their long-time Android customers.

The optics are not great for small tech businesses / startups in general. As customers, what guarantees do we have that the products we rely on won't be acquired and immediately shut down by Google, Amazon, Microsoft, or Apple?

Even small, profitable, award-winning products like Dark Sky are not immune to the ever-expanding reach of The Tech Giants.

Our API service for existing customers is not changing today, but we will no longer accept new signups. The API will continue to function through the end of 2021.

Speaking of reliability, what about all those businesses that rely on the Dark Sky API? What are they gonna do? They now have a couple months to figure it out.

There is a glimmer of hope, though. Although it's clear that Apple is not particularly interested in Dark Sky's API business, seeing recent Apple Maps API developments leads me to believe that they could build their own branded version of Dark Sky's weather API – an Apple Weather API. Same as with their Maps API, they could offer it at competitive prices with first-class support for their operating systems and the web.

I hope this is what they're gonna do with it.

The thing I'm curious about is whether this means that Siri will finally warn me if it's gonna rain in 7 minutes before I walk my dog.

Sometimes I spend several weeknights or entire weekends contributing to open source. And sometimes months go by and I don't contribute to any project. It all depends on my availability, interest, and energy. And it's all okay.

You don't have to give all your free time away to be a useful part of an open source community. You don't even need a regular contribution schedule. In fact, it's common to see developers who have only made a couple contributions in a project's lifetime, and their contributions are still valuable!

If you've been thinking about contributing to open source, but you are not sure if you want to do it regularly, this post is for you. This is a simple guide on how to contribute to open source when you don't have much free time on your hands or simply don't want a long term commitment.

# Finding Projects & Problems to Solve

Finding a project to contribute to can be time-consuming and exhausting. If you don't have a specific system, you'll end up asking yourself questions ranging from, "Am I in a position to contribute to this project?" to "Is this a meaningful project?", and it can become complicated!

What project should you contribute to? And even after you find the perfect project: which problem should you solve?

I've found three approaches to finding the right problem in the right project.

# The Active Approach

The simplest way to start is to take a look at open source software you are already using and actively look for things you can help with. It can be an app you use daily, a dependency for one of your hobby projects or a build tool you use regularly.

Think about your favorite open source project, open the project's page (GitHub, GitLab, BitBucket, etc.), and try to find out what's the best contribution you can make! Keep in mind that it doesn't have to be a code-related contribution, there are many ways to contribute!

# The Passive Approach

Wait for the problem to come to you. I believe that most casual open source developers prefer this method. Report problems, fix issues or propose features that affect you directly.

• Is your build step failing because of one bug in an open source project you depend on?
• Is there something missing in a project's documentation?
• Is there a feature you would like to see in your favorite open source app?

As long as the problem is not too big, these are good reasons to contribute.

The best part of this approach is that you have a personal incentive towards solving a particular problem, making it a highly-rewarding activity.

The only downside is that this type of contributions are spontaneous by nature and don't happen very often, which is troubling if your responsibilities don't allow you to find the perfect time to solve the problem.

# The Vigilante Approach

The final one is a variant of the Active approach. It consists of using apps, dashboards or feeds to stalk watch your favorite organizations, developers, and projects. See what problems they are facing, and when the opportunity arises, contribute!

In my case, I use DevHub (basically TweetDeck for GitHub). DevHub provides a horizontal dashboard with feeds from my favorite OSS organizations, developers, and/or projects. I usually watch out for issues that I can reproduce, bugs I can fix or docs I can write.

This is my favorite way of finding open source projects to contribute to and problems to solve, because it's on-demand, rewarding, and easy to squeeze into my schedule from time to time.

Whenever I feel like dedicating some time to open source (i.e. an uneventful Sunday evening), I check my dashboard and see if there's any small problem I can help with. It's not guaranteed to work, but it frequently does.

# How to Contribute

There are multiple ways to contribute to an open source project, but generally speaking you should focus on discrete problems rather than recurrent tasks such as maintenance or content moderation. Those tasks are much better suited for regular contributors and maintainers.

# Code Contributions

• Bug Reports – open new issues, file bug reports, reproduce issues
• Feature Requests – request new features, propose solutions, draft proposals or RFCs
• Bug Fixes – fix known issues, patch minor bugs, solve performance issues
• Security – search for vulnerabilities, patch them, update dependencies
• Features – add requested features, enhance existing features
• Automated Testing – write tests, integrate with CI/CD, refactor tests
• Accessibility – fix accessibility issues, provide accessibility feedback, point to useful resources

# Non-Code Contributions

• Manual Testing / QA – test usability, solve non-trivial testing issues, develop automation strategies
• Documentation – document features, write how-to guides/tutorials, write examples, write translations
• Mentoring & Support – help in community forums or chat channels occasionally, provide learning resources, participate in community discussions
• Design – make logos/graphical assets, propose branding strategies, provide constructive design feedback
• Outreach – promote a project, write instuctional articles, showcase a projects' usage

# Financial Contributions

• Project Donations – donate thorugh open source funding programs such as Open Collective
• Developer Funding – sponsor individual developers through subscription-based platforms such as Patreon

Learn more on how to contribute to open source here.

# Choosing Tasks & Problems to Solve

Contributions for any open source project can be split into two main categories: general contributions and specific contributions. As a casual open source contributor, the type of problems you tackle and the tasks you complete for a project should be determined by your knowledge, skills, availability, and interests.

# General Contributions

This type of contribution focuses on tasks that are general in nature (i.e. they apply to many projects of the same kind) and require little to no domain-specific knowledge.

General contributions are suitable for you if you are familiarized with the project, but only at the surface level. This type of contribution is usually non-critical which means that your contribution will probably only enhance the project.

This is my favorite way of contributing to a project because contributions of this type are generally easy to solve, don't require a big time investment, and aren't domain-specific.

I also like that this type of contribution is usually non-controversial. It's likely that your contribution will be welcome without much push-back or debate, which saves you some precious time.

Examples for general contributions include but are not limited to:

• Fixing simple code errors
• Writing reference or usage documentation
• Translating content/docs to other languages
• Authoring usage guides ("How to do [...]")
• Updating dependencies that bring no breaking changes
• Adding surface-level features
• Comprehensive bug reports
• Reproducing and documenting simple issues
• Fixing typos and errors in documentation
• Adding supporting resources
• Translations, formatting tools, etc.

# Project-Specific Contributions

Project-specific contributions are focused on the core functionality of the project. This type of contribution is suitable for you if you have domain-specific knowledge that could be leveraged for solving project-specific problems.

This one is the trickiest, but in my opinion it's also the most rewarding type of task you could find. The biggest downside is that it requires more skill and time. If you are familiar with a project's inner-workings and structure, you might be able to do this from time to time.

It's also important to note that this type of contribution also means that complex debates and brainstorming sessions might be necessary. It could take weeks or sometimes even months for your contribution to be accepted. If you are limited on time or are not willing to commit temporarily to solving a specific problem, this might not be the best type of contribution you could make.

Examples for specific contributions include but are not limited to:

• Fixing a critical bug that affects a specific group of users
• Writing feature requests
• Drafting proposals or RFCs
• Refactoring a major part of the project
• Updating core dependencies that bring breaking changes
• Adding support for a particular platform/integration
• Reproducing and documenting complex issues

# Conclusion

I've been a casual open source contributor for ~6 years and as time passes, I dedicate more time to it. I do it with confidence because I know that if at any point in my life I need to take a break, I can do it without letting anyone down.

You don't have to be a 10x developer or a full-time contributor to be a useful part of an open source community. There are ways to contribute to open source without dedicating your life to it or being exceptionally good at it.

If you know anyone who is thinking about contributing to open source share this article with them! If you have any other tips or hints on how to make casual open source contributions let me know (@KristianMuniz on Twitter) and I'll add them here!

# Supporting Resources

Although this post is mainly subjective, the following resources served me as a source of inspiration and references for its development:

• Open Source Guides
• How to manage casual contributors to open source projects

# Why Have a Blog?

I want to have a place to share longform writing without the restrictions of a publishing platform (like Medium) or the hassle of maintaining a self-hosted blogging platform like WordPress or Ghost.

I want to have full control over the reading experience.

# Features

Here is a general list of feature requirements that all iterations of my personal site must adhere to:

• Security - Follow strict security practices, stay current, and adapt to our threat landscape to protect visitors.
• Privacy - Never track visitors and do not collect visitor data unless explicitly asking for permission to do so (e.g. newsletter signup forms, contact forms, preferences, etc.).
• Accessibility - Be accessible to as many visitors as possible and follow the latest Web Content Accessibility Guidelines (WCAG) as well as other generally-accepted accessibility practices, normative or otherwise.
• Syndication - Provide alternative means of consuming site contents via Web syndication.
• Durability - To the extent that it is possible, the contents of this site should be built to be archiveable and last until the end of the Internet.
• Compatibility - Web Platform features used by this site must be supported by at least 90% of the global browser market share, even if its by means of a Polyfill. The site should work well without JavaScript enabled.

# Anyways!

Welcome to my blog! Make yourself at home, read some of my posts, learn more about me, and subscribe to my feed. You can also follow me on Twitter or see my open source projects on GitHub.

Minimo is a humble project with one objective: making the new tab page unobstrusive yet beautiful. It's also a fully open source project that anyone can contribute to!

As its name suggests, Minimo was designed to be simple, intuitive, and straightforward. There are no widgets, no complicated layouts, no account integrations, and no complicated settings pages. It's just a new tab, with a couple shortcuts, and basic information at a glance. No distractions.

# Features

Minimo provides a simple interface to access your favorite websites and have useful information at a glance. It displays the current date & time, connection status, estimated download speed (in Mbps), battery level, and even synced tabs from other devices.

Thanks to the Bookmarks APIs provided by modern browsers, shortcuts are synced across devices and automatically backed-up in a Shortcuts folder on your bookmarks bar. If you export/import bookmarks from other platforms, it will work as expected. Your shortcuts will survive even if you switch browser vendors!

Minimo also has a simple, yet powerful, rich-text scratchpad called "writing mode" that allows you to jot down notes for later (stored locally). You can open "writing mode" from a configurable keyboard shortcut to have quick access to your notes.

# Results

The development of Minimo is far from finished. There are still major things left in the roadmap. I'm definitely biased, but I consider Minimo the simplest new tab alternative that does not compromise user experience or aesthetics.

It feels good to be able to ship something so tangible. I hadn't shipped a product like this in years, so it's a rather refreshing experience. There's lots to learn from this space, and I believe Minimo is in the right spot.

The extension is available on the Chrome Webstore and supports Brave and Chrome browsers. Support for more platforms (like Firefox) is coming soon!

Minimo was featured on Product Hunt as Product of the Day on November 4, 2018. It has positive reviews both in Product Hunt and in the Chrome Webstore.

Note: This is an archived post I recovered from my old blog.

This design is a theoretical solution for modeling complex conversational flows as directional graphs. Conversational graphs will be expressed in JSON using JSON Graph Format spec as a reference.

As a rule of thumb, conversational graph declarations should be serializable to JSON, portable, and contain absolutely no code. This reduces platform and language dependency and facilitates extendability.

# High Configurability

In theory, directed graphs can be arranged to satisfy virtually any conversational flow. Even parallel processes such as performing asynchronous requests while other flows are happening may be possible.

Even if in practice not all configurations were possible, graphs will be able to describe most conversational flows. These flows will not be limited to trees and can be arranged in a natural, non-linear fashion.

# Graphs as Modular Components

Each graph can be connected to other graphs via abstract nodes (entry_node and exit_node) that can lead to other flows. This facilitates portability, modularity, and composability.

Graphs will be arranged as atomic, specialized flows that will be part of a greater graph that connects them together through their entry and exit nodes.

The following graph could be a possible declaration for a simplified, modular flow that specializes in getting a user's email:

Note that not all graphs will have only one edge to exit. Any edge can flow to/from the exit/entry node; the flow could be adapted to exit on an I don't have an email response, for example. Additional metadata could be passed through the exit edge to serve the entry node and edge of the following flow.

# Definitions

# Graphs

A graph describes the relations between nodes through edges. In the context of this model, all graphs are directional and describe a conversational flow.

Example of a simplified graph JSON object:

{
  "graph": {
    "id": "example_graph",
    "label": "An example graph.",
    "nodes": [
      {
        "id": "node_a",
        "type": "example_node",
        "label": "Example of a simple node A"
      },
      {
        "id": "node_b",
        "type": "example_node",
        "label": "Example of a simple node B"
      }
    ],
    "edges": [
      {
        "type": "entry_edge",
        "source": "entry_node",
        "target": "node_a",
        "relation": "flow"
      },
      {
        "type": "example_edge",
        "source": "node_a",
        "target": "node_b",
        "relation": "flow"
      },
      {
        "type": "exit_edge",
        "source": "node_b",
        "target": "exit_node",
        "relation": "flow"
      }
    ]
  }
}

# Nodes

A node (or vertex) represents a discrete, atomic, single-purpose piece of a flow, fundamentally an action.

Example of a simplified node JSON object:

{
  "id": "ask_email",
  "type": "ask",
  "label": "What is your email address?"
}

The JSON object above describes an ask action. Note that the node object itself does not describe the next or previous steps. That's what edges are for.

# Nodes as Actions

For example, nodes might be treated as discrete actions; conversational actions that can include but may not be limited to:

• ask – ask a question expecting a response that is sent to the next node.

• say – send a message, expecting no response.

• store – store a message, status response may be sent to next node.

Naturally, custom actions might be added to the model. Such as actions specialized on performing asynchronous tasks like http requests, for example.

# Abstract Nodes

In order to achieve modularity in conversational graphs, specialized flows should make no assumptions on the rest of the flow. However, while this might be true, graphs aren't designed to solve the modularity problem.

The proposed solution: abstract nodes.

Abstract nodes are nodes that serve as placeholders for the preceding or following nodes in a higher-order graph (a graph of graphs, so to speak).

There are two types of abstract nodes:

• entry_node – the first node of any flow. An abstract reference (a placeholder) that represents the last node of the previous graph.

• exit_node – an abstract reference to a node that follows the last node in a graph.

# Edges

An edge (or link) represents a directional control and/or data flow. Each edge must have a source node and a destination node and only one of them may be an abstract node. Edges represent a data or control flow from a source node to a target node; every edge is directed, the whole graph is directed.

Example of a simplified edge JSON object:

{
  "type": "action",
  "source": "valid_email",
  "target": "store_email",
  "relation": "flow"
}

# Edges for Flow Control

Edges might serve as a way to control which nodes to execute. An edge of type control might use assertion metadata to determine whether to move to a node or not.

Example of an edge as a flow control mechanism:

{
  "type": "control",
  "source": "ask_email",
  "target": "valid_email",
  "relation": "flow",
  "metadata": {
    "assertion": {
      "==": [{ "var": "response.is_email" }, true]
    }
  }
}

The edge above directs the flow from ask_email to valid_email only if response.is_email is true.

In this example JsonLogic is used as an assertion mechanism, but other mechanisms (such as an NLP text-classification layer) could be employed to control the flow of a conversation.

# Entry and Exit Edges

Entry and exit edges are in charge of connecting an abstract node to a regular node and transferring data to and from a graph.

# Example: Example JSON Conversational Graph

If we wanted the following flow to happen:

bot: What's your email address?
user: My email is wrong-email.com
bot: What is your email address?
user: My email is bob@gmail.com
bot: Great! I will store your email now.

We would declare the flow in a JSON as follows:

{
  "graph": {
    "directed": true,
    "label": "Ask for email flow.",
    "nodes": [
      {
        "id": "ask_email",
        "type": "ask",
        "label": "What is your email address?"
      },
      {
        "id": "invalid_email",
        "type": "say",
        "label": "The email you provided is invalid."
      },
      {
        "id": "valid_email",
        "type": "say",
        "label": "Great! I will store your email now."
      },
      {
        "id": "store_email",
        "type": "store",
        "label": "* saves email in db *",
        "metadata": {
          "take": "response.text",
          "save_as": "user.email"
        }
      }
    ],
    "edges": [
      {
        "type": "entry_edge",
        "source": "entry_node",
        "target": "ask_email",
        "relation": "flow"
      },
      {
        "type": "control",
        "source": "ask_email",
        "target": "valid_email",
        "relation": "flow",
        "metadata": {
          "assertion": {
            "==": [{ "var": "response.is_email" }, true]
          }
        }
      },
      {
        "type": "control",
        "source": "ask_email",
        "target": "invalid_email",
        "relation": "flow",
        "metadata": {
          "assertion": {
            "==": [{ "var": "response.is_email" }, false]
          }
        }
      },
      {
        "type": "action",
        "source": "valid_email",
        "target": "store_email",
        "relation": "flow"
      },
      {
        "type": "action",
        "source": "invalid_email",
        "target": "ask_email",
        "relation": "flow"
      },
      {
        "type": "exit_edge",
        "source": "store_email",
        "target": "exit_node",
        "relation": "flow"
      }
    ]
  }
}

This flow could be conceptually visualized as follows: