Notes on Writing Tutorial Code

// Add import of CheerioCrawler
import { RequestQueue, CheerioCrawler } from "crawlee";

const requestQueue = await RequestQueue.open();
await requestQueue.addRequest({ url: "https://crawlee.dev" });

// Create the crawler and add the queue with our URL
// and a request handler to process the page.
const crawler = new CheerioCrawler({
  requestQueue,
  // The `$` argument is the Cheerio object
  // which contains parsed HTML of the website.
  async requestHandler({ $, request }) {
    // Extract <title> text with Cheerio.
    // See Cheerio documentation for API docs.
    const title = $("title").text();
    console.log(`The title of "${request.url}" is: ${title}.`);
  },
});

// Start the crawler and wait for it to finish
await crawler.run();

Good pedantic code is written with comments. Those comments should be written in a way to set the user up to intuitively write the correct code based on the tutorial/docs information they have just consumed. So for example, in the above snippet, I had just read about how to instantiate the CheerioCrawler class.

It is extremely effective, at least for me, to be able to then skim the code snippet for that section, read the comments, and make an attempt at writing the correct code as described in the comments. I should be able to do this because the preceding paragraph would have contained specifically the correct information I need to deduce such code.

As a pedagogy nerd I was oddly tickled by how well the docs are written for Crawlee.