An AI agent needs a web tool when its answer depends on information newer than its model context or specific to a live page. MCP provides a standard tool boundary; it does not make retrieved content trustworthy by itself.
A minimal crawl MCP server can expose one read-only read_web_page(url) tool backed by Nstdata Crawl. The tool fetches an authorized public URL and returns clean Markdown with its source URL.
The working Python example below uses the current MCP 2.x MCPServer API. It was installed and tested with a real in-process MCP client, which discovered read_web_page and its required url input.
Cursor can register the local server in .cursor/mcp.json; Claude Desktop’s current path is Settings → Extensions. Treat copied legacy FastMCP and claude_desktop_config.json tutorials as version-sensitive.
Keep the agent bounded. Validate URLs, restrict allowed hosts where possible, retain source citations, cache repeated reads, and require approval before a tool reaches private or sensitive data.
An AI model cannot know whether a product page changed this morning, a status page recovered five minutes ago, or a policy was revised after its training cutoff. A live web tool closes that evidence gap by retrieving the page at answer time.
The useful architecture is deliberately small: the host decides when a page is needed, MCP describes and invokes the tool, and a crawl API handles retrieval and extraction. This guide builds that connection with current MCP 2.x interfaces and Nstdata Crawl, then attaches it to Claude Desktop or Cursor.
Nstdata Crawl gives an AI agent a managed page-retrieval layer that can return cleaned Markdown, page metadata, links, and other configured artifacts. The agent receives content it can read without operating browser workers, extraction rules, queues, or proxy routing itself.
MCP supplies the integration contract. An MCP server publishes named tools with descriptions and typed input schemas; a compatible host lists those tools and decides when to call them. The current MCP specification defines the protocol boundary, while the official MCP Python SDK documentation supplies the server and client APIs used here.
This separation matters operationally. The model should never receive the Nstdata token. The local server reads the secret from its environment, submits the page request, and returns only the requested content plus provenance.
Nstdata’s earlier web search MCP server tutorial demonstrates a two-tool search-then-read pattern. This article focuses on the safer onboarding primitive: one read-only URL tool that you can attach to multiple hosts and test before adding discovery.
Why AI Agents Need Live Web Data?
AI agents need live web data for freshness, source specificity, and verification. A static model can explain a concept, but it cannot reliably report a current changelog, inventory state, documentation parameter, or public incident without retrieving evidence.
Typical bounded use cases include:
reading a current documentation page before generating integration code;
comparing an approved set of public product pages;
checking a public status or policy page;
collecting fresh evidence for a research memo;
refreshing one document in a RAG knowledge base;
reading the page behind a user-supplied URL before summarizing it.
Live access does not remove the need for judgment. Pages can be stale, manipulated, region-specific, or hostile to agents. Returned text can contain prompt injection that asks the model to ignore instructions or disclose data. Treat web content as evidence, never as authority over the host’s rules.
For broader tool design principles, see Nstdata’s guide to agent tools, security, and evaluation. A well-designed web tool has a narrow name, explicit input contract, predictable output, timeout, audit log, and clear failure behavior.
Prerequisites and Version Check
You need Python 3.10 or newer, an Nstdata account with a Crawl API token, and an MCP-compatible host. Create a virtual environment and install the exact versions verified for this article on September 4, 2026:
The version pins make the example reproducible. Check the official SDK release notes before upgrading: MCP 2.x replaced the older FastMCP class and import path with MCPServer. An older tutorial can fail immediately even if the underlying design is still sound.
Export your Crawl token only in the server’s environment:
exportNSTDATA_API_TOKEN="replace-with-your-token"
Do not commit the token to the server file, project configuration, or a public repository. The example’s authenticated page fetch is a prerequisite gap in this article because no account token was available during verification; the MCP server import and tool-discovery round trip were executed successfully.
Give Your Agent Fresh Web Context
Use Nstdata Crawl to turn approved public URLs into clean Markdown for MCP tools.
The server below exposes one tool that accepts a public HTTP(S) URL and returns source-labelled Markdown. Save it as mcp_server.py.
import os
from mcp.server import MCPServer
from nstdata_ai_crawl import Format, NstDataClient, ScrapeRequestDto
server = MCPServer("nstdata-live-web", instructions="Fetch authorized public web pages as clean Markdown.",)@server.tool()defread_web_page(url:str)->str:"""Fetch one authorized public HTTP(S) page and return Markdown."""ifnot url.startswith(("https://","http://")):raise ValueError("url must use http:// or https://") token = os.environ["NSTDATA_API_TOKEN"]with NstDataClient(token)as client: result = client.submit_scrape_task_sync( ScrapeRequestDto( url=url, formats=[Format.MARKDOWN], onlyMainContent=True, timeout=60000,))ifnot result.success ornot result.data: message = result.errorMessage or result.errorCode or"unknown error"raise RuntimeError(f"page fetch failed: {message}") markdown = result.data.get_markdown()ifnot markdown:raise RuntimeError("page fetched but returned no Markdown")returnf"Source: {url}\n\n{markdown}"if __name__ =="__main__": server.run(transport="stdio")
The function is intentionally synchronous because the current Nstdata SDK method waits for the single-page result. For longer site crawls, expose a submit tool and a separate status tool instead of holding one MCP call open indefinitely.
The scheme check is only a starting point. A production server that accepts arbitrary URLs must also block localhost, private IP ranges, cloud metadata endpoints, nonstandard ports, DNS rebinding, and redirects to disallowed destinations. An allowlist of approved domains is safer than an open URL fetcher.
Verify the MCP Tool Before Opening Claude or Cursor
Verify the server contract with an in-process client before adding host configuration. Save this as test_server.py beside the server:
import asyncio
from mcp import Client
from mcp_server import server
asyncdefmain()->None:asyncwith Client(server)as client: tools =await client.list_tools()for tool in tools.tools:print(tool.name, tool.input_schema.get("required"))asyncio.run(main())
Run python test_server.py. The exact code produced:
read_web_page ['url']
This result proves that the installed MCP client can connect to the server object and discover the generated tool schema. It does not prove that the Nstdata token is valid, because listing tools does not call the Crawl API.
You can also inspect the server using the official MCP development tooling described in the MCP Python SDK getting-started guide. Test invalid schemes, missing environment variables, empty Markdown, timeouts, and provider errors before giving a host access.
Add the MCP Server to Cursor
Cursor registers project-specific MCP servers in .cursor/mcp.json and global servers in ~/.cursor/mcp.json. The official Cursor MCP documentation describes the supported transports and configuration locations.
Use an absolute interpreter and script path so Cursor does not depend on its working directory:
Restart or reload Cursor, open the available-tools panel, and confirm that read_web_page appears. Ask: “Use read_web_page to summarize the current public page at https://example.com, and cite the source URL.” Review the tool arguments before approval.
Claude Desktop’s current onboarding path is Settings → Extensions. Anthropic now recommends desktop extensions for local servers and Settings → Connectors for remote MCP services, rather than relying on old manual JSON instructions for every installation. Follow the current Claude Desktop local MCP guide.
For local development, open Settings → Extensions → Advanced settings and use the developer controls appropriate to your Claude Desktop build. Package the tested server as a desktop extension before distributing it to other users. Mark the token field as sensitive so Claude Desktop stores it through the operating system’s protected credential storage.
After installation, start a new conversation, verify that the tool is listed, and request one known public page. A successful tool listing confirms MCP registration; a successful page response confirms the Nstdata credential and crawl path. Keep those as separate onboarding checks.
Use Live Web Data Without Losing Control
A production agent should choose a URL from an approved source or a user request, call the tool, inspect the returned source label, and answer only from content relevant to the question. Require citations in the agent instruction and preserve the final URL if redirects occur.
Add these controls before expanding usage:
Domain policy: allow only approved public hosts or require confirmation for a new host.
Network safety: reject local and private addresses before and after DNS resolution and redirects.
Data minimization: request only the content format the task needs.
Caching: avoid fetching an unchanged page repeatedly in one session.
Size limits: truncate or store oversized results rather than filling the model context.
Prompt-injection resistance: never let page text override system policy or authorize another tool.
Observability: log tool name, normalized URL, duration, result state, and non-secret request ID.
The MCP specification’s security guidance emphasizes user control and clear authorization around tools. Web retrieval is read-only from the agent’s perspective, but it still transmits the target URL to a third-party service and can expose private URLs if input is not constrained.
Respect target-site terms, robots instructions, copyright, privacy, and applicable law. Keep request volume tied to actual user tasks rather than turning an interactive agent tool into an unattended bulk crawler.
Onboarding Checklist
A reliable onboarding sequence separates protocol, credential, retrieval, and answer quality:
pin and record the installed package versions;
run the in-process MCP tool-discovery test;
add the local server to one host;
confirm the tool name and url input appear;
invoke a permitted public test page;
verify source URL and meaningful Markdown are returned;
test an invalid scheme and missing token;
enable domain, size, timeout, and logging controls;
evaluate whether the final answer cites and follows the retrieved evidence.
Start with one read tool. Add a search tool only when discovery is a real requirement, and add site crawling as a separate asynchronous workflow. This keeps cost, permission, and failure behavior legible to both the user and the agent.
Give Your Agent a Verifiable Web Tool
A useful crawl MCP integration is a narrow evidence channel, not unrestricted browsing. The current MCP 2.x server above provides a tested tool contract, Nstdata Crawl handles page retrieval, and Claude Desktop or Cursor supplies the host experience.
Use Nstdata Crawl pricing to choose the appropriate usage model, then validate one representative page end to end with your own token. Only expand the tool’s domain scope or autonomy after audit logs and approval behavior are working.
A crawl MCP server exposes web retrieval as typed tools that an MCP-compatible AI host can discover and call, while keeping API credentials inside the server process.
Q: Why does an AI agent need live web data?
An AI agent needs live web data when an answer depends on current, source-specific information that is not guaranteed to exist in the model’s training data or conversation context.
Q: Does the MCP server expose the Nstdata token to the model?
No. The token is read by the local server from its environment and used for the Crawl API request; the tool should return only page content, provenance, and controlled error details.
Q: Can the same MCP server work with Claude Desktop and Cursor?
Yes. Both support MCP tools, although their installation flows differ: Cursor uses mcp.json, while current Claude Desktop versions manage local servers through Extensions.
Ivy Lin
Sep. 4th 2026
110M+ real IPs with 99.9% access success
Blazing-fast average response ~0.5s for high-concurrency tasks
From only $0.1/GB
Get immediate access to premium residential, datacenter, IPv6 and ISP proxy pools.
