Recording clicks into replayable browser workflows: selectors, event mapping and MV3 gotchas

The lowest-effort way to automate a web task is to do it once while something watches. AWFlow, my browser-automation extension, has a recorder: hit record, fill the form, stop, and your actions become steps in a “UI Automation” node you can replay, parameterize, or feed a list of rows into.

This post covers the parts that matter: which events you listen to, how you turn an element into a selector that still works tomorrow, and how you keep the content script from losing actions. If the DOM and selectors are new to you, DOM automation without code is the friendlier introduction; this one is about building the recorder.

Listen early, map to intent

Listen in the capture phase, to a short list of events

The recorder is a content script injected into the page. It listens on document, in the capture phase, so page handlers that call stopPropagation() can’t hide events from it:

for (const type of ['click', 'dblclick', 'contextmenu', 'input', 'change', 'keydown']) {
  document.addEventListener(type, this.handle, true); // true = capture
}

Two guards keep the recording clean:

• Ignore your own UI. The recorder shows a floating badge with an action count, and the extension may have other UI injected in the page. Any event whose target is inside those elements is dropped (el.closest('[data-awflow-recorder], …')).
• Inject once. Content scripts can be injected twice (navigation, re-record). A window.__actionRecorder guard makes the second injection a no-op.

Map raw events to intent

A raw event stream is noisy. Typing “hello” produces five input events and a change. A select fires both input and change. Most keydowns are just characters you already captured via input. So each event is mapped to an action, or to nothing:

switch (event.type) {
  case 'click':       return { action: 'click', domSelector };
  case 'dblclick':    return { action: 'dblclick', domSelector };
  case 'contextmenu': return { action: 'rightclick', domSelector };
  case 'input':
  case 'change':
    if (el instanceof HTMLSelectElement)       // only on change, or you record it twice
      return event.type === 'change' ? { action: 'select', domSelector, value: el.value } : null;
    if (isCheckboxOrRadio(el))
      return event.type === 'change' ? { action: el.checked ? 'check' : 'uncheck', domSelector } : null;
    if (isTextField(el))                       // only on input; ignore the trailing change
      return event.type === 'input' ? { action: 'type', domSelector, text: el.value } : null;
    return null;
  case 'keydown':
    // only "meaningful" keys (Enter, Escape, Tab, arrows, Backspace…) or combos with Ctrl/Meta/Alt
    return isMeaningful(ke) ? { action: 'keypress', domSelector, key: ke.key, ctrl, shift, alt, meta } : null;
}

Then a tiny coalescer merges the stream: consecutive type actions on the same selector collapse into one with the final value, and repeated scrolls to the same element are dropped.

Recording type with the field’s value rather than individual keystrokes is the key decision. It replays correctly even when the page reformats input as you type.

Selectors that survive tomorrow’s deploy

This is where most recorders fail. The obvious selector, the nth-child path from body, breaks the first time someone adds a banner to the page.

The recorder tries candidates from most to least stable:

It takes the first one that matches exactly one element, and that element is the one clicked:

const uniq = (s: string) => {
  try {
    const found = document.querySelectorAll(s);
    return found.length === 1 && found[0] === el;
  } catch { return false; } // invalid selector → not a candidate
};

Here is that check running on a single click, on a button with no id and no test hook.

Two details that matter in practice:

• CSS.escape everything. Ids like email:1 or class names from CSS-in-JS (css-1x2y3z) are valid HTML but break naive selectors.
• Validate uniqueness at record time. A selector that matched one element when you recorded is your best guess at what you meant. If it matches zero or two at replay, that’s a real signal, not a silent misclick.

Test hooks being second on the ladder is a nice side effect: apps with good test coverage are also the easiest to automate.

Don’t lose actions, then make replay robust

Reconnect when the background goes to sleep

The content script sends each action to the extension’s background through a long-lived runtime.connect port. In MV3 the background is a service worker that Chrome suspends when idle (Firefox’s event page does the same). When it’s suspended, the port disconnects, and an action posted to a dead port just disappears.

The fix is boring but essential: track disconnects, and reconnect before posting if the port is gone.

private post(action) {
  if (!this.port) this.connect();   // the worker may have been suspended mid-recording
  this.port?.postMessage({ action: 'actionRecorded', recordedAction: action, uid: this.uid });
}

Before this, long recordings occasionally “skipped” a step with no visible error. That’s the worst kind of bug in a recorder.

Replay is where you add robustness

A recording is a starting point, not a finished automation. On replay, the UI Automation node gives you Wait for element steps (and a per-action wait timeout) so you’re not racing a slow page, expressions to swap a recorded value for one read from the incoming item, and normal workflow logic around it: conditions, AI steps, schedules, error branches.

With an expression in the typed value, the same recording fills the form for every row of a sheet. For a complete form-filling workflow that also asks before it touches the page, the docs example below is the closest: it uses a form-filling node rather than a recording, but the shape (values in, approve, fill, submit) is the same one a recording slots into. Build an AI form filler goes further on that use case.

Honest limits

• Sites that redesign will break recorded selectors sometimes. The stable-first strategy reduces that a lot but doesn’t eliminate it. Re-recording one step is usually the fix.
• Canvas-heavy apps and closed shadow DOM aren’t well covered by DOM events.
• Anything behind a login works only because it runs in your session, which is also why the extension asks for per-site access.

If you’re building a recorder, this is most of the reliability:

AWFlow is free to install. Questions and “here’s how my recorder broke” stories are welcome.