Using Squid on Intermittent Connections

Listen Print Discuss

Using Squid on Intermittent Connections

by Jennifer Vesperman
08/02/2001

One of the more frequent requests on the Squid mailing lists is for help configuring Squid to operate well on dial-up or demand-dial networks. Offline mode will function for some of these networks, but is far from ideal. Unfortunately many of the features of Squid's offline mode appear to have largely vanished during the development of the Squid 2.x series. In the 2.3 STABLE 4 version, the offline mode has nearly no effect at all.

Squid can be patched to work well with dial-up and other intermittent connections. Having a cache on the intermittent side of the link can take some of the curse off these connections, providing access to cached information and reducing bandwidth use on the link. Unpatched, Squid can work reasonably well on dial-on-demand connections, but dialing in each time it needs to resolve a query can become expensive.

Standard mode

Squid is designed for permanent connections. Making Squid work on intermittent connections requires changing how Squid handles stale web pages and stale DNS lookups.

Pages are considered stale when their TTL (time to live) has expired. Many web pages have expiry data in the headers -- information on how often the page can be expected to be updated, and when to consider it "old." Squid has default expiry as well -- if there is no data in the headers, Squid sets its own. Stale pages are kept in the cache -- if the cache still has room for them.

If the cache runs out of space, stale pages are thrown out on an oldest-first basis. This algorithm is called LRU (least recently used). Other algorithms are available in later versions of Squid. The compile time option --enable-heap-replacement allows you to choose other options.

If this doesn't clear enough space, pages that are still fresh get thrown out as well. This is bad for cache efficiency -- if this is happening and there is room on the disk, or you can squeeze more disk space and RAM from your boss, adjust your configuration.

Stale pages that remain in the cache are validated when a client requests the page. Squid sends a request, asking "Hey, has this page changed since [time]?", and the origin server responds with "Yeah, here's the new page," or "No, it's still good." The page is then considered fresh again, whether it's the new page or the old one. This is called an IMS request, shorthand for the term "if modified since." Since when? Since Squid last verified that it was fresh.

DNS lookups are cached as well, and have a configurable TTL. When a cached Fully Qualified Domain Name (FQDN) or IP DNS entry expires, it is removed from the cache.

Offline mode

So why not just use offline mode?

Offline mode is designed for complete disconnection from the network -- for reading only from what is already in the cache. It works well for demonstrations and presentations. If you were a guest on "60 Minutes" or were working offline from a trade show floor, you could display parts of your web site that are stored in the cache and they will appear with marketable speed. Just be sure to avoid anything that isn't in the cache!

How do you deal with dial-up connections? Where does Squid fit in?
Post your comments

Related articles:

Installing and Configuring Squid

Deploying Squid, Part 2 of 2

Deploying Squid, Part 1 of 2

Squid does this by switching off expiry of stale pages and stale lookups, and blocking off its attempts to retrieve web pages. If the web page is a "cache miss," it returns an error to the client rather than trying to retrieve the page.

Offline mode also never expires cached DNS information, and never refreshes stale web pages. This is usually not what dial-up users and others with intermittent connections want.

In the unmodified Squid configuration, offline_mode on signals Squid to never attempt to validate, and offline_mode off signals normal operation.

Note: In Squid 2.3 STABLE 4, the offline mode no longer has most of these functions. If you add the "intermittent connections" patch, this functionality will be restored and extended.

Intermittent mode

To work with intermittent connections, Squid must be patched. The patch must be able to tell whether or not the connection is up.

If it is connected, it must run as if in standard mode: expiring web pages; replacing stale web pages, if requested; and expiring and removing stale DNS entries.

If disconnected, it must run almost as if in offline mode -- except it must mark old pages and DNS entries as stale, so they can be removed or replaced when Squid is next connected.

Intermittent connections patch

A patch to enable intermittent connections is available from here.

The file name is squid-tristate-offline-patch-1.0.txt. If any bugs (or even more miraculously, patches or bug fixes) turn up, please report them to dancer@users.sourceforge.net.

The patch is currently against Squid 2.3 STABLE 4, a widely deployed Squid release. Some manual changes are required to patch against later versions. To install the patch:

Download the patch and the source code for Squid.
Unpack your Squid source-tree, and go into its top directory.
Run patch -p1 --dry-run < ../squid-tristate-offline-patch-1.0.txt
If no errors appear, remove the --dry-run parameter and run it again.
Compile, reinstall (make install), and modify your squid.conf file. Then restart Squid.

Patch details

This patch has the following effects:

In offline mode, no object is ever considered stale. Thus, no validation occurs.
In offline mode, no ipcache object is ever expired.
In offline mode, no fqdncache object is ever expired.
If a DNS lookup takes place in offline mode, Squid inserts a dummy cache entry with an immediate expiry (that cannot be expired until Squid is no longer in offline mode).
In offline mode, any attempt to go forward to query another server results in a "503 - Cannot Forward" error. This covers all MISS cases.
Offline mode is now tri-state. It can be on, off, or dependent on the status of a network interface.

In a modified Squid, the offline_mode directive works differently. Formerly it was a simple boolean type with the values on or off. Now we use the values 0, 1, or 2.

# Normal online operation.
offline_mode 0

# Offline operation. Use cached information only.
# Do not talk to the world.
offline_mode 2

# Conditionally offline.
# Online if a monitored network interface 
# exists AND is up. If either condition is 
# false, Squid will act in offline mode.
offline_mode 1

The new directive network_interface takes a string parameter, which is the name of the network interface to check. If we are in mode 1, Squid will act in offline(2) or online(0) mode depending on the status of this interface. Squid will test the status of this interface no more frequently than once per second.

# watch ppp0
network_interface ppp0

You can use a dummy interface to manually toggle Squid between online and offline modes.

In squid.conf, set

network_interface dummy0

At a prompt, run

% ifconfig dummy0 [up|down]

Because PPP connections take several seconds to properly establish after the interface is created, you may prefer to switch modes using a dummy interface and control that dummy interface from your ip-up and ip-down scripts. (See man 8 pppd for more information.)

Caveats and gotchas

Offline mode may not work properly in unmodified versions of Squid.

Offline store and forward web proxy
2002-09-17 03:40:59 anonymous2 [Reply | View]

I use wwwoffle when trying to use a web proxy/cache behind an offline/ dialup link.

http://www.gedanken.demon.co.uk/wwwoffle/

Works great if your remote site only dials up once a day to download requested Web pages.

mark@purcell.homeip.net

Link has been updated
2002-08-02 23:04:30 Jennifer Vesperman | [Reply | View]

The link to the intermittent connection patch has been updated. Thanks!

dial-up
2002-03-25 03:38:54 accelicim_ho [Reply | View]

good

dail-up
2002-03-24 20:39:39 accelicim_ho [Reply | View]

Dial-up connectios Squid

Link is out of date
2002-03-12 18:48:59 Jennifer Vesperman | [Reply | View]

The simegen.com link in the article is out of date. The correct URL is now http://www.anthill.echidna.id.au/~dancer/patches

Jenn Vesperman

squid
2001-10-22 08:21:49 rooh [Reply | View]

How do you deal with dial-up connections? Where does Squid fit in?

squid on dial-up
2003-01-28 15:18:08 Jennifer Vesperman | [Reply | View]

If you have a dial-up connection and an in-house lan, you can put squid (with the patch) between the dial-up and the workstations.

You can then browse as normal, and if it's connected squid will fetch stale web pages from the net, and if not, it will serve them from the cache.

Does that explain it better?

Jenn V.

squid on dial-up
2003-10-20 11:41:31 anonymous2 [Reply | View]

I can see, in general, how the scheme is supposed to work; I'm less sure of the nuts 'n bolts of how you put it together.

Presumably, you give your browser the hostname and port on which squid listens as its proxy server details.

How does squid 'know' where to go for its information stream from the 'net? Again, presumably, the default is to use something like ppp to manage the modem link, but how does squid know that its ppp (or whatever) that will be doing the talking to the modem? I didn't see a squid.conf entry that might define this, or am I overlooking something obvious?

Equally, presumably (but wandering to the very edge of the topic...) if this is happening on a remote machine (I'm assuming that ppp & squid are running on a server) you need to do something like a perl script with sockets to remotely turn ppp on and off as appropriate. Or have I overcomplicated this?

m a r k underscore w at techie dot com
(delete spaces, underscore = _, dot = .)

squid on dial-up
2003-10-20 17:54:53 Jennifer Vesperman | [Reply | View]

To answer 'how does Squid know where to go', check the paragraph starting:
"The new directive network_interface takes a string parameter, which is the name of the network interface to check."

Also, you've missed how the patch works. All a patched Squid cares about is 'can I connect to a remote site or not?'. If so, it runs in connected mode. If not, in disconnected mode.
A patched Squid does NOT turn the modem on itself, if you want to do that you need to set up dial-on-demand at the OS level, and you DON'T want this patch.

To quote from the article:

"The patch must be able to tell whether or not the connection is up.
If it is connected, it must run as if in standard mode: expiring web pages; replacing stale web pages, if requested; and expiring and removing stale DNS entries.
If disconnected, it must run almost as if in offline mode -- except it must mark old pages and DNS entries as stale, so they can be removed or replaced when Squid is next connected."

If a patched Squid can't reach past your gateway, it (probably) won't trigger a dial-on-demand.

(If this seems nonsensical to you - well, we live in a place where local calls are charged per call. Without having tested it, I assume unpatched Squid will already trigger dial-on-demand if dial-on-demand is set up in the OS; which is useful for people charged per unit time. This patch provides a method of proxying over dialup which is useful for people who are charged per call. Make more sense now?)

squid
2001-10-21 07:53:20 rooh [Reply | View]