FRP Intranet Penetration for Web Crawling: Expose Internal Services Safely

Start from our web crawling and data collection basics guide if you’re new

TL;DR

• Intranet penetration (NAT traversal) lets external networks reach services that are inside a private LAN and have no public IP.
• frp (Fast Reverse Proxy) is a popular, open-source choice with TCP/UDP/HTTP/HTTPS forwarding and P2P mode. 
• You typically run frps on a public VPS and frpc on the internal device, then map ports/domains as needed.

What Is Intranet Penetration and Why Do You Need It?

Proxy technology is often used in web crawling, where a server acts as an intermediary to access other host servers—commonly when a crawler IP is blocked.

Intranet penetration works in the opposite direction: it uses a relay (jump server/跳板) so that devices in different locations can access internal services (NAS, home PC, dev machine) that do not have a public IP.

Typical needs:

• Remote access to a home computer without a public IP
• Company blocks ports 22/3389 (SSH/RDP not reachable)
• WeChat debugging needs 80/443, but local dev is only localhost
• Access internal logs/services over a dedicated line

Intranet penetration (NAT penetration) is a method to expose intranet services to external networks via controlled forwarding.

Common Implementation Methods (Quick Comparison)

1) Port Mapping

Maps a router/firewall public port to an internal device port.

2) VPN (Virtual Private Network)

Creates an encrypted tunnel so remote users can access intranet resources like they are on the same LAN.

3) Reverse Proxy

A public-facing proxy forwards requests to internal servers, hiding internal IPs and adding caching/filtering/load balancing.

4) Intranet Penetration Tools (Client–Server Relay)

A public server acts as relay; intranet devices connect outward to it; external requests are forwarded through the relay to internal services.

Tool #1: frp (Fast Reverse Proxy)

frp is a fast reverse proxy that helps you expose a local server behind NAT/firewall to the Internet, supporting TCP/UDP/HTTP/HTTPS and also offering a P2P mode. 

Official repo:

https://github.com/fatedier/frp

Version note: at the time of writing, the latest GitHub Releases tag shows v0.64.0. 

(If you prefer stability, you can pin a slightly older version in production.)

Preparation Materials

Step 1: Download frp (Linux server example)

Use GitHub Releases to choose the version you want. Example (keep your own version pinning policy):

wget https://github.com/fatedier/frp/releases/download/v0.60.0/frp_0.60.0_linux_amd64.tar.gz
tar -xzf frp_0.60.0_linux_amd64.tar.gz
cd frp_0.60.0_linux_amd64

Step 2: Server Configuration (frps.toml)

Starting from v0.52.0, frp supports TOML/YAML/JSON configs; INI is deprecated. 

Create:

/etc/frp/frps.toml

Example:

bindPort = 7000         # frpc connects to this port
vhostHTTPPort = 80      # HTTP virtual host (e.g., web debug)
vhostHTTPSPort = 443    # HTTPS virtual host
auth.token = "abc123"   # do not use weak tokens

Step 3: Enable systemd Auto-Start (Server)

Copy binary:

sudo cp frps /usr/local/bin/

Create unit file:

sudo vim /etc/systemd/system/frps.service

[Unit]
Description=FRP Server
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/frps -c /etc/frp/frps.toml
Restart=always

[Install]
WantedBy=multi-user.target

Enable + start:

sudo systemctl enable --now frps
sudo systemctl status frps

Step 4: Client Configuration (frpc.toml)

Example: expose Windows RDP (3389) through the public server port 13389:

serverAddr = "x.x.x.x"   # your VPS public IP
serverPort = 7000
auth.token = "abc123"

[[proxies]]
name = "rdp"
type = "tcp"
localIP = "127.0.0.1"
localPort = 3389
remotePort = 13389

Step 5: Windows Startup on Boot (Client)

Place frpc.exe and frpc.toml into:

C:\frp\

Win + R → shell:startup → create a frpc.vbs that starts frpc.exe silently (keep your existing method).

After restart:

PublicIP:13389 → RDP to home PC

5 High-Frequency Scenario Templates (frp)

Optional: Peer-to-Peer Intranet Penetration (XTCP)

frp provides XTCP, a P2P intranet penetration proxy. If P2P succeeds, subsequent traffic does not need to be relayed through the server (server coordinates the connection). 

Note: P2P success depends on NAT type. If it fails, you may need to fall back to relay/proxy modes.

Minimal Server (Coordinator) Config

bindPort = 7000
auth.token = "demo123321"

Start:

./frps -c ./frps.toml

Client A (Active Side) Example

serverAddr = "Public server IP"
serverPort = 7000

# XTCP visitor connects to peer
[[visitors]]
name = "p2p_remote"
type = "xtcp"
serverName = "p2p_remote"
secretKey = "your_shared_secret"
bindAddr = "127.0.0.1"
bindPort = 6000

Client B (Passive Side) Example

serverAddr = "Public server IP"
serverPort = 7000

[[proxies]]
name = "p2p_remote"
type = "xtcp"
secretKey = "your_shared_secret"
localIP = "127.0.0.1"
localPort = 3389

Then on Client A:

• connect to 127.0.0.1:6000 to reach Client B’s 3389.

(If your current configs are INI-based, migrate to TOML/YAML/JSON to align with frp’s current direction.  )

Key Notes (Troubleshooting + Safety)

1. Config parsing

• If the config fails to load, remove inline comments, ensure UTF-8 encoding, and use a proper editor.

2. NAT restrictions

• P2P modes depend on NAT type; strict NAT may fail.
• If P2P fails, consider relay mode or adjust router settings (UPnP/port mapping) where appropriate.

3. Security

• Always enable auth.token (and avoid weak secrets).
• Expose only necessary ports/services; keep OS/services patched.