This tutorial shows a clean way to do that in bulk, with one output row per website, and pay only when something useful is found.

Before you start — stay on the right side of the line

• Use public business contact details for legitimate outreach. Do not scrape personal inboxes to spam people.
• Honour each site’s terms and applicable privacy law (UK GDPR / PECR if you email people in the UK).
• Prefer role addresses (hello@, press@, support@) when a tool can filter personal mailboxes.
• Cap pages per site so you do not crawl an entire blog looking for a mailto link.

This is not a Google Maps scraper. You paste company websites; you do not search “plumbers near me”.

Step 1 — Prepare a domain list

A CSV or plain list is enough:

https://apify.com
https://www.python.org
https://www.gov.uk

Normalise them first:

• Add https:// if missing.
• Drop tracking paths (?utm_…).
• Skip social profile URLs (LinkedIn company pages are not “the company website” for this method).

Step 2 — Run the contact extractor

Console

1. Open Company Website Contact Scraper – Emails, Phones, Socials.
2. Paste the websites.
3. Set Max pages per website to 3 or 4 (homepage + contact + about is usually enough).
4. Optionally enable Only emails on the website’s own domain so gmail.com personal addresses drop out.
5. Optionally enable Leave out websites with no contacts found to keep the dataset clean.
6. Start and export CSV or JSON.

On this Actor you pay $1.50 per 1,000 websites that return contacts (plus a small start fee). Sites with nothing found can be skipped from charging when that option is on — check the pricing tab for the exact event rules before a big run.

Python

pip install apify-client

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("ryumoon-data/company-contact-details-scraper").call(
    run_input={
        "websites": [
            "https://apify.com",
            "https://www.python.org",
        ],
        "maxPagesPerWebsite": 4,
        "onlyEmailsOnSiteDomain": True,
        "skipWebsitesWithoutContacts": True,
        "includePersonalContacts": False,
    }
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(
        row.get("website") or row.get("url"),
        "| emails:", row.get("emails") or row.get("email"),
        "| phones:", row.get("phones") or row.get("phone"),
        "| socials:", list((row.get("socials") or row.get("socialProfiles") or {}).keys())[:5],
    )

Step 3 — Sample output

Trimmed example shape (field names can vary slightly by version — use the dataset column picker in Console):

{
  "website": "https://apify.com",
  "emails": ["hello@apify.com"],
  "phones": [],
  "socials": {
    "twitter": "https://twitter.com/apify",
    "linkedin": "https://www.linkedin.com/company/apifytech",
    "github": "https://github.com/apify"
  },
  "pagesScanned": 2,
  "scannedUrls": ["https://apify.com", "https://apify.com/contact"]
}

One row per site keeps enrichment simple: join back to your CRM on domain.

Step 4 — Turn results into an outreach sheet

1. Export CSV from the Apify dataset.
2. Add columns: company, domain, email, source_page, date_found, permission_basis.
3. Deduplicate emails case-insensitively.
4. For UK email marketing, record a lawful basis before you send anything. Cold B2B email has rules; when unsure, ask a solicitor or stick to LinkedIn / contact forms.

Quality tips that actually move the needle

What this method is bad at

• Finding the personal email of a specific employee (use a people database, carefully).
• Sites that hide contacts behind forms or logins.
• Marketplaces and social networks (wrong tool).

Cost sketch

Twenty company sites, most with a public contact@, cost well under $0.10 in contact events on this pricing. That is enough to validate your sheet format before you paste 2,000 domains.

Wrap-up

1. Clean a list of company websites.
2. Crawl a few public pages per site for emails, phones and socials.
3. Keep one row per domain and record how you plan to use the data.

Worked example Actor: apify.com/ryumoon-data/company-contact-details-scraper.

Disclosure: I built the Apify Actor in the worked example (Company Website Contact Scraper). The workflow is tool-agnostic; swap the Actor if you prefer another contact extractor.

This post shows a practical path: fetch pages as Markdown, keep only the main content, then load a few pages into a tiny retrieval demo. You do not need a YouTube API key, a browser farm on your laptop, or a paid Firecrawl plan to follow along.

What “good” Markdown for RAG looks like

A useful page export usually has:

1. The article or docs body, not the whole chrome of the site.
2. Real # / ## headings in reading order.
3. Links and code fences preserved.
4. Metadata you can store alongside the text (URL, title, description).
5. A hard cap on pages so a runaway crawl does not empty your wallet.

HTTP-first fetchers are enough for most docs and blogs. Reach for a headless browser only when the first response is empty (heavy client-side apps).

Step 1 — Pick a small, public start URL

Use a docs page you are allowed to read. Two safe examples:

• https://docs.apify.com/academy/web-scraping-for-beginners
• https://en.wikipedia.org/wiki/Retrieval-augmented_generation

Keep the first run to 5 pages or fewer and crawl depth 0 (only the URLs you paste). Expand later once the shape of the output looks right.

Step 2 — Run a Markdown crawler

Option A: Apify Console (no code)

1. Open Website to Markdown for LLM & RAG.
2. Paste the start URLs.
3. Set Max pages in total to 5.
4. Leave rendering on auto / HTTP-first if the Actor offers it.
5. Click Start, then open the default dataset.

You are charged per successfully saved page on this Actor ($1 per 1,000 HTTP pages; browser-rendered pages are a separate, slightly cheaper add-on event). Failed pages are not charged as page events.

Option B: Python with the Apify client

pip install apify-client

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("ryumoon-data/website-to-markdown-rag").call(
    run_input={
        "startUrls": [
            {"url": "https://docs.apify.com/academy/web-scraping-for-beginners"},
            {"url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"},
        ],
        "maxPages": 5,
        "maxCrawlDepth": 0,
        "mainContentOnly": True,
        "includeLinks": True,
    }
)

rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
print(f"Got {len(rows)} pages")
for row in rows:
    print(row.get("url") or row.get("request", {}).get("url"), "→", len((row.get("markdown") or "")) , "chars")

Field names above match the Actor’s public input schema (startUrls, maxPages, maxCrawlDepth, mainContentOnly, includeLinks). If a field is missing on a future version, check the input tab on the Store page.

Step 3 — Inspect a sample row

A typical successful item looks like this (trimmed):

{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation",
  "markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (RAG) is a technique…\n\n## History\n\n…",
  "description": "…",
  "links": ["https://en.wikipedia.org/wiki/Large_language_model"]
}

Check three things before you scale up:

• Headings survived (#, ##).
• Boilerplate is gone (no “Skip to content”, cookie text, or footer link farms).
• The Markdown is short enough to chunk (tens of KB per page is fine; multi-MB dumps usually mean you forgot main-content mode).

Step 4 — Chunk and load into a tiny RAG demo

Header-aware splitting keeps section context:

from langchain_text_splitters import MarkdownHeaderTextSplitter

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]
)

docs = []
for row in rows:
    md = row.get("markdown") or ""
    if not md.strip():
        continue
    parts = splitter.split_text(md)
    for p in parts:
        p.metadata["source"] = row.get("url")
        docs.append(p)

print(f"{len(docs)} chunks from {len(rows)} pages")
print(docs[0].metadata, docs[0].page_content[:240])

From here, push docs into whatever vector store you already use (Chroma, Pinecone, pgvector). The important part was getting clean Markdown before embedding.

Common failure modes

When not to use a generic Markdown crawler

• Pages behind login or paywalls.
• Sites that forbid scraping in their terms — respect robots and the law where you operate.
• Binary-heavy sites (raw PDFs only); use a document parser instead or in addition.

Cost sanity check

For this Actor, five ordinary HTTP pages cost on the order of one US cent of page events plus the small Actor-start fee. That is enough to validate a pipeline. Scale the page cap only after you like the Markdown quality.

Wrap-up

1. Start with a handful of public URLs.
2. Export Markdown with main content only.
3. Split on headings, attach the source URL, then embed.

The same pattern works with other crawlers; the Store listing used above is one HTTP-first option aimed at LLM and RAG use: apify.com/ryumoon-data/website-to-markdown-rag.

Disclosure: I built the Apify Actor used in the worked example (Website to Markdown for LLM & RAG). The steps below still work as a pattern if you swap in another Markdown crawler.

This post walks through a reliable bulk path: collect video URLs, pull captions with language fallback, keep timestamps when you need them, and skip videos that have no captions so you do not pay for empty rows.

What you will get

For each public video that has captions:

• Full plain-text transcript.
• Optional timestamped segments (start / end / text).
• Language actually used (handy when you ask for English but only Spanish exists).
• A clear miss when the video is private, deleted, or caption-free.

You will not get a speech-to-text engine for silent uploads. No captions means no transcript from this method.

Step 1 — Collect video URLs or IDs

Start small (3–10 videos). Mix:

• Single watch URLs: https://www.youtube.com/watch?v=…
• Short links: https://youtu.be/…
• Later: a channel or playlist URL if your tool supports expanding them.

Example seed (3Blue1Brown neural-network intro — public, has captions):

https://www.youtube.com/watch?v=aircAruvnKk

Step 2 — Run a bulk transcript job

Console

1. Open YouTube Transcript Scraper – Captions, Timestamps, Bulk.
2. Paste video URLs into Video URLs or IDs.
3. Set preferred languages to en (or your list).
4. Turn Include timestamped segments on if you need SRT-like timing.
5. Start the run and open the dataset.

Pricing on this Actor is pay-per-event: about $3 per 1,000 successful transcripts. Videos without captions are free (you still pay the small Actor-start fee).

Python

pip install apify-client

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("ryumoon-data/youtube-transcript-scraper").call(
    run_input={
        "videoUrls": [
            "https://www.youtube.com/watch?v=aircAruvnKk",
            "https://www.youtube.com/watch?v=arj7oStGLkU",
        ],
        "languages": ["en"],
        "preferManualCaptions": True,
        "allowAutoGenerated": True,
        "fallbackToAnyLanguage": True,
        "includeSegments": True,
        "maxVideosTotal": 10,
    }
)

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    text = row.get("transcript") or row.get("text") or ""
    print(
        row.get("url") or row.get("videoUrl"),
        "| lang:", row.get("language"),
        "| chars:", len(text),
        "| segments:", len(row.get("segments") or []),
    )

Input field names follow the live schema (videoUrls, languages, includeSegments, channelOrPlaylistUrls, maxVideosPerChannel). Always re-check the Store input tab if a run rejects a key.

Step 3 — Read a sample output

Trimmed success shape:

{
  "url": "https://www.youtube.com/watch?v=aircAruvnKk",
  "title": "But what is a neural network? | Deep Learning Chapter 1",
  "language": "en",
  "transcript": "This is a 3. It's sloppily written and falling apart…",
  "segments": [
    { "start": 4.2, "end": 5.4, "text": "This is a 3." },
    { "start": 5.4, "end": 8.1, "text": "It's sloppily written and falling apart…" }
  ]
}

If a row has empty text, treat it as a miss: private video, region block, or no captions. Do not retry those in a tight loop — you will just burn start fees.

Step 4 — Useful follow-ons

Search inside a channel’s transcripts

1. Expand a channel URL with channelOrPlaylistUrls and a low maxVideosPerChannel (for example 20).
2. Concatenate transcripts.
3. Grep for product names or objections.

Rough chapter markers

Group consecutive segments into 30–60 second windows and print the first line of each window. Good enough for internal notes; not a substitute for creator chapters.

Content-gap research

Pull transcripts from several competitor videos, count frequent phrases, and compare to your own outline. Pair with Google Trends later if you care about rising queries.

Why bulk jobs fail on a laptop

The popular youtube-transcript-api library works from home, then dies on cloud IPs with IpBlocked / RequestBlocked. YouTube rate-limits caption endpoints. Hosted runners that already rotate residential proxies exist so you do not have to maintain a proxy pool yourself. If you self-host, budget for residential IPs and polite concurrency (a handful of parallel videos, not hundreds).

Limits to accept up front

• Members-only, private and deleted videos cannot be read.
• Auto-captions can mis-hear names and numbers — spot-check anything customer-facing.
• Some live or brand-new uploads have no captions yet; wait and retry once.

Cost example

Ten successful English transcripts ≈ $0.03 in transcript events plus one Actor-start ($0.005 at 1 GB). That is cheaper than most Store peers that charge around $5–$10 per 1,000 items — verify competitor prices on their pages before you rely on the comparison.

Wrap-up

1. Gather public video URLs.
2. Request preferred languages with fallback.
3. Keep timestamps only when you need them.
4. Ignore caption-free rows instead of hammering retries.

Actor used in the examples: apify.com/ryumoon-data/youtube-transcript-scraper.

Disclosure: I built the Apify Actor used in the worked example (YouTube Transcript Scraper). Treat this as a tutorial that happens to use that tool; the checklist applies to any caption exporter.