On Notational Programming for Notebook Environments

An in-line handwritten diagram recognizer for quantum circuits, returning a Qiskit circuit. Examples use the Notate Jupyter extension interface, available here.

When we speak of “writing” code, we — almost always — imagine typing it. But will that always be the case?

Notational programming is a vision of programming that involves handwritten input. I say ‘involves,’ because it’s not that handwriting is universally ‘better,’ but rather that in the future, there should be a role for it in programming. That we should consider handwriting again — as Konrad Zuse, and John von Neumann “coded” in the 1940s.

In this article, I provide some thoughts on notational programming and demonstrate a key principle — “variables travel,” where typewritten text can be referenced in handwritten code. My goal is to provoke discussion around “traveling” variables as powerful tools for bridging handwritten and typewritten contexts, for a future version of programming practice less constrained by typewritten input.

You got handwriting all over my IDE

To begin with, we must solve a pretty basic issue — that no current IDE allows you handwrite in the middle of your code! Of course, why would it? That would be crazy!

But what if I told you that there is an extension to Jupyter notebooks that let you draw in your code? You simply tear open a canvas in the middle of a line and start drawing. The canvas acts as a numpy image array, which you can do whatever you want with (in this instance, to feed to a recognizer).

Obligatory example.

The Notate extension doesn’t come with fancy recognizers — that’s up to us — but it provides the “barest of bones” interface for future experimenting. I use this plugin in the examples that follow (the recognizers are private, though I don’t claim that they are especially robust.)

Case study: Quantum circuit diagrams

Quantum programming is a growing domain that involves coding quantum computers to solve important problems. The practice differs substantially from coding classical computers. One key difference is representational: quantum programs are often first expressed as quantum circuit diagrams, and the process of coding is the process of iteratively building such circuits. The centrality of a 2d notation suggests a useful domain for a case study of notational programming.

For instance, consider one of the most basic circuits, the Bell state |ϕ+>:

This can be expressed in the Qaw* system as:

Let’s break this down: I’ve torn open a canvas in my code, and drawn the circuit with a Microsoft Surface pen. The corresponding image is fed into the QCR function (short for quantum-circuit-recognition), which outputs an IBM Qiskit QuantumCircuit object. I then use Qiskit’s built-in draw() function to check that the circuit was accurately recognized.

Seeing this working is pretty cool, but setting a Bell state is perhaps the simplest thing one can do in quantum programming, requiring only a .h() and .x() call. This might lead one to believe that drawing circuits is cute, but if you want to do anything more complex, you’re better off typing. You might reason that, unlike diagrams, typewritten code can interface with the Python code around it to, say, embed a subcircuit in a larger circuit.

But what if the hand-drawn circuit could access variables in the local scope?

Variables travel

In a notational programming system, typewritten variables can “travel” to the handwritten context and vice-versa, blending the distinctions between them. Consider the following example:

A handwritten symbol references the typewritten variable in the local Python context.

Here I’ve drawn a two circuits A and B. The handwritten A in circuit B’s definition references the typewritten A as a local variable. (Yes, this actually runs.) If I decompose the B gate one level, we see that it’s interpreted A correctly (ignore other decompositions):

Were the recognizer extremely robust (*cough* like it is now *cough*), you wouldn’t worry about how well you’ve drawn the circuit: the infrastructure that connects the typewritten and handwritten scopes would fade into the background of your consciousness, allowing you to focus on the problem at hand.

Practical example

Of course, it’s still not clear how useful notational programming is on real problems. Take, for instance, the diffusion subcircuit in Grover’s algorithm (shown here for n=3, taken from the Qiskit textbook):

The general diffusion circuit is represented in IBM Qiskit (Python) code as:

Now consider the Qaw representation of the same circuit:

Return the interpretation of the hand-drawn circuit, with variable “n” in local scope. The slash-n notation is common practice in quantum circuit diagrams, standing for “n qubit lines.”

Typewritten text may be more editable, but less readable: you have to read each line closely, and remember the type of certain variables. By contrast, the Qaw system may have other issues— such as a cumbersome input method, depending on who you ask— but the notation is surely more readable, provided the reader understands certain conventions (e.g. that the bottom controlled dot is short for “control on all n bundled wires”). Using IBM Qiskit or Microsoft Q#, we might spend a lot of time figuring out what methods we can call and how the syntax fits together. In other words, we perform a lot of translation work: how to convert our 2d diagram to linear sequences of API calls. Syntax constraints still exist when we draw the circuit, but we perform much less translation work.

“But but… *something* GUIs *something*…”

Now, the rebuttal some readers might have is —

“Well, that’s nice, but there are quantum circuit editors with fancy GUIs, so why not just embed one of those in a computational notebook? It’s much easier to edit and less error-prone.”

And they would be right, to an extent. In quantum computing, IBM Composer is a great drag-n-drop GUI for circuits. And recent work by researchers at Apple, CMU, UCBerkeley, MIT and Microsoft have begun to integrate GUIs within notebook environments (such as the amazing “mapping” widgets of livelit).

The point of notational programming is not that domain-specific GUIs can’t improve interaction. The point is the immediacy of handwriting. It doesn’t require that you learn a new UI or click through sub-windows. And we can imagine that, once a notation is recognized, a domain-specific GUI could replace the drawing to facilitate later editing. (Or handwriting would be used as shortcuts: draw a gate, and an AI tool guesses what you mean and replaces it with typewritten code. There’s tons of possibilities at the intersection of pen-based computing and programming.)

Beyond just “recognition”

But we can go slightly further than the current capabilities of those GUIs, too. Handwriting code pushes us not just to “recognize” what came before, but to extend notations with new features. For instance, in a future version of the Qaw system, we might define a circuit F recursively as:

This example relies upon abstract extensions** to regular-old quantum circuit notation. Now I could use this F gate in another recursive circuit Q that staggers F-gates and applies a Hadamard gate to the highest qubit:

This is actually the meat of the Quantum Fourier Transform (QFT). We’re just missing the swap function, which we can also define recursively:

A recursively-defined Swap gate, which swaps highest and lowest qubits. This works in the general case because of the “default” base cases for n=0 and n=1.**

Putting it all together, we define the QFT circuit as:

Specifying the # of input qubits=4 to QFT would generate the following:

An imagined output for our QFT circuit, from Qiskit documentation.

This way of defining quantum circuits is vastly different than typing code. It’s not always the most readable — recursive functions rarely are — but the one thing it has eliminated is the need to specify qubit indices. The index information is already present in the 2d notation.

2D notations offer more freedom?

I’d like to provide one last provocative example, a simple one.

Consider one of the most basic problems in coding — defining a rectangle. Across libraries, the expected parameters to a rect() maker function can differ widely. Here are just a few examples:

• rect(x, y, width, height)
• rect(x0, y0, x1, y1)
• rect(x0, x1, y0, y1)
• rect((x0, y0), (x1, y1))
• rect(center_x, center_y, width, height)

But suppose I could define a box by sketching it. I have more freedom here: the box might be defined with two points; or I can put the (x, y) point in the bottom-right corner — there’s many ways one can go about it:

Three different ways to define a rectangle.

All this is to say, notational programming isn’t simply “we’re going to handwrite what we could’ve typed.” In typing code, your input better fit the expected ordering + type of arguments. In handwriting code, there are still constraints, but notations have more freedom.

Towards “Writing” Code

Notational programming isn’t just about recognizing existing notations — it’s about extending them or developing new notations, with the full capabilities of AI technology. It’s about imagining a future where programming doesn’t just look like this:

Image source

But (sometimes) like this:

Image source

And with that, I’m out of examples. I hope you enjoyed this brief, if nascent discussion of notational programming. More soon.

— Ian

Ian Arawjo is a PhD candidate at Cornell University in Information Science. He studies at the intersection of programming and culture.

*Yes, “Qaw” is meant to evoke the call of a pestering crow. It’s short for quantum-draw.

**In the Qaw notation, slash-wires mean bundles of n≥0 wires and do not always need to be explicitly labeled. If n=0, the wire disappears. If n=1 and the gate is undefined for that # of input wires, the gate stands for identity. For gate F, the “n” labels the # of input wires as size n. The circular gate stands for a controlled phase shift (.cp in Qiskit) with rotation pi/2^(n-1), and the circular shape is inspired by a book on quantum programming.