Skip to content
Change language: EN
My learningAdd to Chrome
Track progress 0%

Lesson 2 6 min

Scrape a list into clean data

Fetch a page by its address and pull each item's title and link into data you can reuse, without opening the page.

Try it in AWFlow

Opens this lesson's workflow in AWFlow. You review it before anything is saved.

Import the recipe (opens in a new tab)

How the recipe works

  1. When Started runs it when you click Test Workflow.
  2. Get HTML from Link downloads https://news.ycombinator.com and puts its HTML in html. The page doesn’t need to be open.
  3. HTML Extract in Repeating list mode finds every .titleline and, inside each, reads the link text as title and its href as link. It builds one item per match.
  4. Display Object shows the items.
The settings of the Pull out the items step: HTML, Mode Repeating list, Selector, Return, Attribute, Container Selector .titleline, and Fields with a title field read from the a element.The settings of the Pull out the items step: HTML, Mode Repeating list, Selector, Return, Attribute, Container Selector .titleline, and Fields with a title field read from the a element.
The Container Selector field

Try it

Import the recipe, then click Make it mine at the bottom of the canvas, tick the box and confirm, so you can edit it. Click Test Workflow and open Show the data in the run results: you see about 30 items, each with a title and a link.

Make it yours

Change URL to the page you want. Set Container Selector to one repeated item on that page (one product, one result) and add a field for each value: price, date, author. Each field’s selector is searched inside its container, so keep it short, such as a.title.

Key takeaways

  1. Get HTML from Link fetches a page; HTML Extract in Repeating list mode reads every repeated item.
  2. The Container Selector points at one repeated item; add one field per value.
  3. Display Object shows the items as a tree you can expand.