Alter.Org.UA  
 << Back Home RU ru   Donate Donate www/www1/www2

Local HTTP caching (Proxy)

Intro

We shall cache HTTP traffic with all possible optimizations. This would speed up reload of pages, images and even video. It is possible to save about 35% of bandwith in average. Some sites may be cached woth 90% efficiency. It is significant for slow internet channel (e.g. GPRS, DSL). Also, you can filter out unnecessary content (e.g. banners).

Download

Squid27-win-20130910.rar - Squid Proxy package
Squid27-win-conf-20131214.rar - fresh updates

Local proxy (Squid)

Setup HOWTO for Windows:

  • Download this Squid27-win-20130910.rar and Squid27-win-conf-20131214.rar
  • unpack both to C:\squid. Second one contains fresh config updates Do not try to run from Program Files, Squid doesn't correctly handle path with spaces.
  • Look in C:\squid\etc\squid.conf
    There are some important options:
    • dns_nameservers - DNS servers to use. It you comment it out, system default should be used.
    • acl banners - Banner filters. My own. Paranoic :). You may manage the on your own. Uncomment (remove #) the following line in squid.conf to enable:
      #http_access deny banners !allow_banners
            
    • acl allow_banners - exceptions for filters. Also adjustable.
  • Run
    C:\squid\sbin\squid -z
    C:\squid\sbin\squid -i
    net start squid
    
  • Configure your browser to use proxy for all protocols. Proxy IP 127.0.0.1, port 3128
  • Enjoy.

If you want to see what resources are actually accessed by your browser, see C:\squid\var\logs\access.log

  • TCP_DENIED - blocked
  • TCP_MISS - was downloaded
  • TCP_REFRESH_HIT, TCP_IMS_HIT, TCP_MEM_HIT - cache hit, local copy was used instead or remote server access

Error messages you can found in C:\squid\var\logs\cache.log


after changing C:\squid\etc\squid.conf you should restart Squid:

net stop squid
net start squid
or
sbin/squid.exe -k reconfigure -n squid

PS. the original distribution was taken from http://squid.acmeconsulting.it/download/squid-2.7.STABLE8-bin.zip
Original setup manual is here: http://squid.acmeconsulting.it/SquidNT25.html. It was written for 2.5, there were no significant changes since that time.

Under Windows url_rewrite.exe with config special file is used instead of squid_rewrite.pl. You may use either rewrite.pl (whih is derived from squid_rewrite.pl) or squid_rewrite.pl itself as config file. Both .pl files are located in etc directory, url_rewrite.exe - in libexec.

Local proxy (Squid) for Unix

Under Linix/FreeBSD we have different setup steps, but config format is same. Same cache management options, same refresh_pattern, storeurl_access and store_rewrite_list, same acl's, see related squid.conf options below. You must build Squid without --disable-http-violations, or event better - with --enable-http-violations. Fresh config and squid_rewrite.pl are in Squid27-win-conf-20131214.rar in etc directrory.

squid.conf

Final version for Squid 2.6 or 2.7:

squid.conf

acl store_rewrite_list url_regex -i \.youtube\.com\/get_video\?
acl store_rewrite_list url_regex -i \.youtube\.com\/videoplay.*
acl store_rewrite_list url_regex -i \.youtube\.[a-z][a-z]\/videoplayback
acl store_rewrite_list url_regex -i \.youtube\.[a-z][a-z]\/get_video\?
acl store_rewrite_list url_regex -i \.googlevideo\.com\/videoplayback\?
acl store_rewrite_list url_regex -i \.googlevideo\.com\/get_video\?
acl store_rewrite_list url_regex -i \.google\.com\/videoplayback
acl store_rewrite_list url_regex -i \.google\.com\/get_video\?
acl store_rewrite_list url_regex -i \.google\.[a-z][a-z]\/videoplayback
acl store_rewrite_list url_regex -i \.google\.[a-z][a-z]\/get_video\?
acl store_rewrite_list url_regex -i \.ytimg\.com\/.*\.(jpg|jpeg|gif|png|ico|mp3|flv|mp4)
acl store_rewrite_list url_regex -i \.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/(videoplayback|videoplay|get_video)\?
acl store_rewrite_list url_regex -i (kh|mt)(.?)\.google\.com
acl store_rewrite_list url_regex -i maps\.googleapis\.com
acl store_rewrite_list url_regex -i mt\d+.googleapis\.com
acl store_rewrite_list url_regex -i vec([0-9]+)\.maps\.yandex\.net\/tiles
acl store_rewrite_list url_regex -i pvec([0-9]+)\.maps\.yandex\.net
acl store_rewrite_list url_regex -i static\.video\.yandex\.net\/(.*)\.(jpg|jpeg|gif|png|ico|mp3|flv|mp4|wmw|avi|mpg|mpeg)
acl store_rewrite_list url_regex -i tub-ua\.yandex\.net
acl store_rewrite_list url_regex -i s\d+\.dotua\.org\/fsua_items.*\.(jpg|jpeg|gif|png|ico|mp3|flv|mp4)
acl store_rewrite_list url_regex -i fbcdn\.net.*\.(jpg|jpeg|gif|png|ico)
acl store_rewrite_list url_regex -i cdn(.?)/[0-9a-zA-Z_-]*.?\.flv
acl store_rewrite_list url_regex -i st(.*)\.userapi\.com
acl store_rewrite_list url_regex -i ecn\.dynamic\..*\.tiles\.virtualearth\.net\/comp
acl store_rewrite_list url_regex -i static\.video\.yandex\.ru\/swf
acl store_rewrite_list url_regex -i video\.meta\.ua\/players
acl store_rewrite_list url_regex -i \.vkadre\.ru\/.*\.(jpg|jpeg|gif|png|ico|mp3|flv|mp4)
acl store_rewrite_list url_regex -i (st|cs)(.?)\.vk\.me\/.*\.(jpg|jpeg|gif|png|ico|mp3|flv|mp4)
acl store_rewrite_list url_regex -i img\d+.slando\.ua\/.*\.(jpg|jpeg|gif|png|ico|mp3|flv|mp4)
acl store_rewrite_list url_regex -i .*s\d*\.staticclassifieds\.com\/static

storeurl_access allow store_rewrite_list
storeurl_access deny all
storeurl_rewrite_program /usr/local/bin/squid_rewrite.pl
#storeurl_rewrite_program C:/Squid/libexec/url_rewrite.exe -l G:\\Squid\\var\\logs\\rewrite.log C:\\Squid\\etc\\rewrite.pl
storeurl_rewrite_children 5
storeurl_rewrite_concurrency 10 # use only with newer rewrite helpers, v1.3 or above.
#cache allow store_rewrite_list
#cache deny all 

#	usage: refresh_pattern [-i] regex min percent max [options]
#	The refresh_pattern lines are checked in the order listed here.
refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern youtube.*videoplay  14400   90%     24400   ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern youtube.*get_video  14400   90%     24400   ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern google.*videoplay   14400   90%     24400   ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern googlevideo.*get_video  14400   90%     24400   ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern ytimg\.com\/.*\.(jpg|jpeg|gif|png|ico|mp3|flv|mp4)  14400   90%     24400   ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern (mt|kh|pap).*\.google\.com  14400   90%     24400   ignore-no-cache override-expire override-lastmod ignore-reload ignore-private ignore-auth
refresh_pattern (mt|kh|pap).*\.googleapis\.com  14400   90%     24400   ignore-no-cache override-expire override-lastmod ignore-reload ignore-private ignore-auth
refresh_pattern s\d+\.dotua\.org\/fsua_items.*\.(jpg|jpeg|gif|png|ico|mp3|flv|mp4)  14400   90%     24400   ignore-no-cache override-expire override-lastmod ignore-reload ignore-private ignore-auth
refresh_pattern .*static\.video\.yandex\.ru\/swf\/.*&r=.*  14400   90%     24400   ignore-no-cache override-expire override-lastmod ignore-reload ignore-private ignore-auth
refresh_pattern vec.*\.maps\.yandex\.net\/tiles\?	14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private ignore-auth
refresh_pattern static.*\.maps\.yandex\.	14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private ignore-auth
refresh_pattern pvec.*\.maps\.yandex\.net	14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private ignore-auth
refresh_pattern lrs\.maps\.yandex\.net\/tiles\?		14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private ignore-auth
refresh_pattern yandex\.st\/.*(jpg|jpeg|gif|png|ico|mp3|flv|mp4)		14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private ignore-auth
refresh_pattern static\.video\.yandex\.net\/.*(jpg|jpeg|gif|png|ico|mp3|flv|mp4).*		14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private ignore-auth
refresh_pattern .*ecn\.dynamic.*\.tiles\.virtualearth\.net\/comp   14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern fbcdn\.net.*\.(jpg|jpeg|gif|png|ico|mp3|flv)	14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern static\.ak\.fbcdn\.net.*\.(jpg|jpeg|gif|png|ico|mp3|flv)	14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern (st|cs)\d+\.vk\.me\/.*\.(jpg|jpeg|gif|png|ico|mp3|flv|mp4)	14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern img\d+.slando\.ua\/.*\.(jpg|jpeg|gif|png|ico|mp3|flv)	14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern .*s\d*\.staticclassifieds\.com\/static	14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern \.vkadre\.ru\/assets\/.*\.(jpg|jpeg|gif|png|ico|mp3|flv|mp4)	14400	90%	20080 ignore-no-cache override-expire override-lastmod ignore-reload ignore-private
refresh_pattern .*\.(css)$	1440	90%	1440 ignore-no-cache override-expire override-lastmod ignore-private
refresh_pattern .*\.(js)$	1440	90%	1440 ignore-private
refresh_pattern -i (/cgi-bin/|\?) 10	20%	120
refresh_pattern .		10	20%	4320
#	see also refresh_pattern for a more selective approach.

cache_mem 400 MB
maximum_object_size_in_memory 4 MB
maximum_object_size 32 MB
read_ahead_gap 64 KB
range_offset_limit -1

Note: Pay attention to proper placement of settings.
acl's must be defined prior to options using them. It is better to find original location of each option in squid.conf and place it imemdiatly after or instead of default value.

I2P

Additional settings for I2P resources

acl i2p_urls url_regex -i .*://.*\.i2p\/.*

cache_peer 127.0.0.1            parent    4444  4444  no-digest allow-miss no-query
cache_peer_access 127.0.0.1 allow i2p_urls

never_direct allow i2p_urls
never_direct deny all
always_direct deny i2p_urls
always_direct allow all

or even like this, to forward all requests via anonymizer:

cache_peer 127.0.0.1            parent    4444  4444  no-digest allow-miss no-query
cache_peer_access 127.0.0.1 allow all

never_direct allow all
always_direct deny all

Anti-banner

Here you can found part of squid.conf, with banner filters. Note, that it is very aggressive filter, it may deny some thinsg you would like to see. Be attentive.

History

youtube fixes. Requests with redirect options must be kept intact.

Squid27-win-conf-20131214.rar

2013.12.14

Fixes for facebook images, some tricky youtube video and storeurl_rewrite performance.

Squid27-win-conf-20130924v2.rar

2013.09.24

The 1st verison.

Squid27-win-20130910.rar
Squid27-win-conf-20130916.rar

2013.09.16

See also:




Mail to alterX@alter.org.ua (remove X)  
<< Back designed by Alter aka Alexander A. Telyatnikov powered by Apache+PHP under FBSD © 2002-2017