Frontier squid release notes ============================ frontier-squid-2.7.STABLE9-27 - 30 August 2016 (dwd@fnal.gov) - When using compressed logs and SQUID_CLEAN_CACHE_ON_START is true (both of which are default), then truncate the swap.state file in ufs cache directories each time logs are rotated. Otherwise the file grows without bounds. - When using the 'restart' function, clean ufs cache directories the same way as when doing 'start'. - Change the default cache_dir size in squid.conf.proto to 10000 MB in case someone deletes the default 10000 MB line in customize.sh. frontier-squid-2.7.STABLE9-26 - 16 June 2016 (dwd@fnal.gov) - Change default minimum_expiry_time to 0 seconds. Without this change, squid will not cache any objects that are loaded for the first time during the last minute of their lifetime; instead, all requests for that object during that minute are sent upstream. It only affects squids that are fed by other squids. frontier-squid-2.7.STABLE9-25 - 11 May 2016 (dwd@fnal.gov) - Apply fix for reported security vulnerability CVE-2016-4554. It is only for transparent proxies which is not the way frontier-squid is normally used, but the patch is applied just in case. Also discussed in http://bugs.squid-cache.org/show_bug.cgi?id=4501 and http://bugs.squid-cache.org/show_bug.cgi?id=4515. - Add hepvm.cern.ch to the MAJOR_CVMFS acl. frontier-squid-2.7.STABLE9-24 - 3 June 2015 (dwd@fnal.gov) - Fix the disabling of log compression with the SQUID_COMPRESS_LOGS=false setting. The variable was not being honored in the nightly log rotate cron, so log compression still happened at night. It was being honored in the every 15 minute check, so if the maximum log size was reached during the day it removed all the compressed files that were rotated overnight, losing log history. frontier-squid-2.7.STABLE9-23 - 22 April 2015 (dwd@fnal.gov) - Back out the dividing up of file descriptors between multiple squids that was added in frontier-squid-2.7.STABLE9-20. It was based on a confusion over how the limit worked; the file descriptor limit works per process and not per user. - Support new configuration option SQUID_MULTI_PEERING=false to not insert cache_peer parent settings when there are multiple squids. By default when there are multiple squids, any squid other than the first one reads from the first one like it always used to. - Support using multiple squids for a reverse proxy. Formerly it clobbered the http_port and cache_peer parent settings when using multiple squids. Now it preserves any extra parameters on http_port and sets SQUID_MULTI_PEERING=false if a cache_peer parent setting already exists. - Support awstats with multiple squids: invoke run_awstats.sh if it exists (installed by frontier-awstats rpm) for the logs of all of the squids and not just the first one. Requires frontier-awstats rpm version 6.9-3.2 or newer to work properly. - Don't invoke awstats if SQUID_SUFFIX is set (that is, in the frontier-squid2 rpm) so it won't get invoked twice when it is installed simultaneously with frontier-squid. - Support a "daemon:" prefix on access_log and cache_log, a poorly documented squid feature that uses a separate process to handle writing to log files so the main squid process doesn't have to wait for disk I/O. This was added because log compression was observed on one machine to interfere with squid I/O accesses. Make this the default for access_log. - Run log rotation with ionice -n7. frontier-squid-2.7.STABLE9-22 - 1 January 2015 (dwd@fnal.gov) - Apply patch from Debian that was backported from a denial of service vulnerability reported for squid3 in CVE-2014-3609, having to do with an invalid "Range" header request. - Make slight correction to MAJOR_CVMFS acl regular expression. - Expand the server names allowed at RAL in the ATLAS_FRONTIER acl. frontier-squid-2.7.STABLE9-21 - 3 December 2014 (dwd@fnal.gov) - Fix redirection of stderr in the hourly and daily crons so error messages properly go to squidcron.log. - Only require the minimum 4096 file descriptors when doing one of the commands that contain "start" (that is, "start", "restart", or "condrestart") with multiple squids. - Add the script that generates squid.conf to the list of files that trigger regenerating squid.conf if they're newer than squid.conf. - Only generate the per-squid configuration files used with multiple squids when squid.conf is newer than them. frontier-squid-2.7.STABLE9-20 - 6 November 2014 (dwd@fnal.gov) - Increase the maximum number of squids that may be started from 4 to 16 - When running N > 1 squids, limit each squid to the hard limit on file descriptors divided by N. This limiting greatly reduces or eliminates the number of failed accesses to cache files indicated by TCP_SWAPFAIL_MISS entries in access.log. Require a minimum of 4096 file descriptors for each squid, unless customize.sh sets a value below the calculated limit. - Support use of SQUID_SUFFIX to add a suffix to all the files. This is not supported for use in the standalone tarball, just for use within an rpm. frontier-squid-2.7.STABLE9-19 - 15 September 2014 (dwd@fnal.gov) - Make the default SQUID_MAX_ACCESS_LOG be 5G instead of 1G unless log compression is disabled. This should take about the same maximum space (~11Gbytes) as uncompressed log files did with a max size of 1G. - Add the TRIUMF CVMFS stratum 1 to the list in MAJOR_CVMFS. frontier-squid-2.7.STABLE9-18 - 3 September 2014 (dwd@fnal.gov) - Protect rotate operations with a lock, because now that they compress files they can take a long time. This is especially important for the one minute between the daily cron and the first hourly cron. - Fix bug introduced in last release where if access_log is set to "none", the cache log is rotated every 15 minutes. frontier-squid-2.7.STABLE9-17 - 22 August 2014 (dwd@fnal.gov) - Update the CERN Hungary Data Center's LHCOPN IP address range in the HOST_MONITOR access control list from the incorrect 188.185.0.0/17 to the correct 188.184.128.0/17 and 188.185.128.0/17. - Add commented-out acls CMS_FRONTIER, ATLAS_FRONTIER, and MAJOR_CVMFS that can be uncommented and used in place of RESTRICT_DEST to restrict outgoing connections to the corresponding servers. This allows updating the lists via frontier-squid package upgrades rather than requiring individual administrators to know how to keep the lists up to date. - Include the real time zone in the access.log timestamp instead of always +0000, and include milliseconds after the seconds. - Add the "cvmfs-info" header to the same double-quoted log entry that now has "X-Frontier-Id". Since no client sends both headers, only one will show at a time; frontier entries will end with " -" and cvmfs entries will start with "- ". Cvmfs clients currently only send cvmfs-info if configured with CVMFS_SEND_INFO_HEADER=yes so if that's not the case their log entries will show "- -". - Accept SQUID_MAX_ACCESS_LOG as the variable setting the maximum access log file size in place of LARGE_ACCESS_LOG (which is still accepted for backward compatibility). Also if the value ends in 'M' it indicates megabytes and if it ends in 'G' it indicates gigabytes; the default is bytes. - Run the "hourly" cron 4 times an hour, to catch faster when a log file has gone over the max size limit. - Compress log files by default, using logrotate. If environment variable SQUID_COMPRESS_LOGS is exported and set to 'false', fall back to the previous method of telling squid to rotate the log files without compression. In either case the logfile_rotate configuration parameter is used to define the maximum number of rotated files. If frontier-awstats is also installed, the first file is left uncompressed. When switching either way between compressed and uncompressed, removes all log files of the old type. - Rotate cache.log even if the access_log configuration parameter is "none". - Add a new "removecache" option to the init script to simply remove all of the cache, for use when removing the package. frontier-squid-2.7.STABLE9-16 - 10 May 2013 (dwd@fnal.gov) - Rearrange the new code in the /etc/init.d startup script to be easier to modify at post-install time by the rpm - Change default chkconfig levels to "-" to match the Redhat standard - Change the delay after starting squid from 3 to 10 seconds before checking to see if it is running. That was the previous delay, and a system with very slow disk access running two squids didn't get both of them started within 3 seconds while a cache cleaning was happening in the background. frontier-squid-2.7.STABLE9-15 - 8 May 2013 (dwd@fnal.gov) - Put squidcron.log in the same directory as cache.log rather than access.log, in case the access_log option is set to "none". In the previous version it would put squidcron.log in the squid user's home directory if access_log was "none". - Change the init.d startup script to abort with an error message if the squid user's home directory does not exist, because on RHEL6-derived systems if a user's home directory doesn't exist then cron won't run the user's jobs. - Run squid without the -S option so it will never run an audit of the cache files. During a normal start the cache is deleted so it doesn't matter, and the audit operation can take a very long time on a large cache during a restart. Also an analysis showed that the typical types of errors the audit catches (missing files) are survivable. - Allow multiple background cache cleans to be happening at the same time, in case the cache is very large and someone does multiple stop/start operations. - Add environment variable SQUID_CLEAN_CACHE_ON_START which defaults to true and when set to false prevents clearing the cache on start. It can be set and exported in the package's /etc/sysconfig file. frontier-squid-2.7.STABLE9-14 - 29 Mar 2013 (dwd@fnal.gov) - Put the tarball version string into the string reported as the version in the SNMP-based monitoring. Requires keeping RELEASE variable up to date squid/Makefile. - Supply defaults for cache memory size and cache dir size in the configure scripts, if people just hit enter. frontier-squid-2.7.STABLE9-13 - 28 Jan 2013 (dwd@fnal.gov) - Move the output from the cron jobs from daily.log in the cron directory to squidcron.log in the log directory frontier-squid-2.7.STABLE9-12 - 25 Jan 2013 (dwd@fnal.gov) - Change the comment in customize.sh so the rpm can refer to the standard init script instead of the nonstandard fn-local-squid.sh, and also mention doing reload for a running squid. - Create the default cache directory when doing a make install. frontier-squid-2.7.STABLE9-11 - 11 Jan 2013 (dwd@fnal.gov) - Add the Referer and User-Agent headers to the end of the default log format. Each are in double quotes, and if the headers are missing a hyphen is inserted. - Include the upstream squid source in the tarball. Source rpms are expected to be self-contained, and this tarball is included in the frontier-squid rpm so it has to be self-contained too. frontier-squid-2.7.STABLE9-10 - 20 Dec 2012 (dwd@fnal.gov) - Change the default allowed monitoring hosts to be the "WLCG" address ranges at the main CERN data center and the new backup data center in Hungary - Disable the icp port by default frontier-squid-2.7.STABLE9-9 - 19 Sep 2012 (dwd@fnal.gov) - Fix sed command in frontier-squid-utils/Makefile that caused tarball installation to fail. The bug was introduced in version STABLE-6. It does not affect the rpm. frontier-squid-2.7.STABLE9-8 - 4 Sep 2012 (dwd@fnal.gov) - Change the init.d 'rotate' command to first remove the oldest log files, rather than asking squid to do it, because it can take a long time to delete large access logs and squid stops servicing requests during the rotate process. frontier-squid-2.7.STABLE9-7 - 2 Aug 2012 (dwd@fnal.gov) - Fix bug that prevented the hourly cron from reading an /etc/sysconfig file for the setting of LARGE_ACCESS_LOG, making it always rotate at the default 1GB. Now it will use the setting of that variable from /etc/sysconfig/frontier-squid or /etc/sysconfig/frontier-squid.sh. frontier-squid-2.7.STABLE9-6 - 25 Jun 2012 (dwd@fnal.gov) - Support running up to 4 squids listening on the same port. In order to start multiple squids, subdirectories have to be created for the cache directory of the form squidN where N goes from 0 to the number of squids minus 1. Corresponding directories for the logs and pid file are also used and created if necessary. See more details in the comments at the beginning of fn-local-squid.sh. - Whenever SETSQUIDAFFINITY=true, the affinity of squid processes are tied to individual CPU cores with the use of the taskset command. This has been observed to increase throughput on squids that are CPU-limited by about 15%. When $SETSQUIDAFFINITY is not set, the default is to enable it if and only if more than one squid is being started and the number of cores is greater than or equal to the number of squids. Cores are assigned beginning at number 0 and going up 1 at a time. - Fix the detection of whether or not squid is already running on RHEL6-based systems. - If the maximum number of open file descriptors (ulimit -n) is less than 4096 per process when starting squid, the limit will be increasd to that number. A higher number can be set in the /etc/sysconfig file matching the name of the init.d script (that is, frontier-squid for the rpm and frontier-squid.sh for the tarball) or by setting the 'nofile' parameter in /etc/security/limits.conf. - Eliminate error from 'find' in the init.d script that occurred when generating squid.conf from within a working directory that's not accessible to the user id that squid runs under. - Require access_log to be either "none" or to be in the same directory as cache_log's directory. This simplifies especially the handling of running multiple squids and is usual practice anyway. - Do better checking for error conditions in fn-local-squid.sh: make sure it is not run as root; make sure cache_dir is not set to '/' (so the clean-up of the cache can't try to delete everything); make sure that cache_dir, cache_log, and pid_filename are set and not empty; and make sure that cache_dir, the directory of cache_log, and the directory of pid_filename all exist and are writable. - Integrate better with the rpm distribution so it doesn't need to patch any of the .proto files. - Allow SNMP queries from the two individual IP addresses of the central frontier monitoring machines in addition to the shared virtual IP address. frontier-squid-2.7.STABLE9-5 - 29 Nov 2010 (dwd@fnal.gov) - In order to prevent accidental clobbering of a squid.conf from pre-customize.sh squid installations, print an error and refuse to start if squid.conf has write permission. Also when squid.conf is overwritten, print the path to the saved squid.conf.old. - Make detection of an existing running squid process more robust by matching the process ID in the saved squid.pid file with ones found by the "pidof" Linux command. This prevents an old squid.pid from matching some other process using the same pid after a reboot. - When cleaning out an old cache, move the old cache directories out of the way, remove them in the background, and then proceed so squid will start faster. - Remove the hidden default of always allowing incoming access to private network addresses 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16. Instead, make that default explicit in the first customize.sh that is generated so if the administrator doesn't want that it can be easily changed. - Add to the init.d script to source in /etc/sysconfig/frontier-squid.sh if it exists (for the tarball packaging, that is; for the rpm packaging it is /etc/sysconfig/frontier-squid). - Explicitly choose a shell for runuser from inside the init.d script in case the login shell of the user is not bash - Add new 'proto_install' Makefile targets in frontier-squid-utils and squid source directories that make it easier to create an rpm with installation paths that fit well with standard Linux. Also change the .proto files to have independently specifiable installation paths so the same files can be used for both tarball and rpm packages. frontier-squid-2.7.STABLE9-4 - 18 May 2010 (bjb@jhu.edu) - Replacement SNMP monitor IP address for Fermilab (131.225.240.232) frontier-squid-2.7.STABLE9-3 - 5 April 2010 (dwd@fnal.gov) - The cache.log error message Failed SNMP agent query from : 128.142.202.212. was happening every 3 hours, every time the DNS Time-To-Live ran out on cmsdbsfrontier.cern.ch. To work around that, added frontier.cern.ch to the HOST_MONITOR acl, which will work until that IP address changes, and then the above error message will come back until squid restarts. Details are in http://bugs.squid-cache.org/show_bug.cgi?id=2894. - Add "rotate" and "rotateiflarge" commands to fn-local-squid.sh. Change daily.sh to use "rotate", and add an hourly.sh to use "rotateiflarge". Also add a cron entry to the default crontab to call hourly.sh every hour. This will keep access access.log from growing excessively. The size of access.log that will trigger the hourly rotate is a gigabyte by default, but can be overridden by setting the environment variable LARGE_ACCESS_LOG to the number of bytes desired before invoking hourly.sh from cron. Also change daily.sh to clean up the log file written by both daily.sh and hourly.sh once a week. - Change the "status" command in fn-local-squid.sh to be more robust. frontier-squid-2.7.STABLE9-2 - 1 April 2010 (dwd@fnal.gov) - There's now often a new fairly harmless message Failed SNMP agent query from : 128.142.202.212. that appears in cache.log the first time squid is polled by the MRTG server after startup. That was interfering with the logic in fn-local-squid.sh which was checking for whether or not squid started successfully. This can cause the start script to wait for 5 minutes if the message appears in the first 10 seconds after startup. Change the script to allow for the success message to be in any of the last 5 lines. - Note that having update_headers on sometimes results in a small number of other harmless messages to cache.log of the form storeUpdateCopy: Aborted frontier-squid-2.7.STABLE9-1 - 31 March 2010 (dwd@fnal.gov) - Change the name of the package to make it obvious what the underlying squid release is. - Upgrade to squid-2.7.STABLE9. - Apply patches for the following squid bugs. These are needed only for squids that feed other squids: - http://bugs.squid-cache.org/show_bug.cgi?id=2831 - needed for changing expiration time of cached objects. - http://bugs.squid-cache.org/show_bug.cgi?id=2833 - needed for correct collapsed forwarding of simultaneous If-Modified-Since requests on previously cached objects. - Add a new way to customize squid.conf that gets preserved across upgrades. A script called squid/etc/customize.sh is now used to edit squid.conf, and it is guaranteed to not be overwritten after upgrades. The 3 customization questions asked at configure time are now used to generate customize.sh the first time only, and on upgrades the questions no longer need to be asked. The user may add any additional edits as desired, and functions that understand the squid.conf are provided to make it fairly easy to do. See the comments there and in customhelps.awk for details. Questions from configure can be avoided completely if the user passes a new '--prefix' parameter defining where to install, and also the new '--oldprefix' parameter defining where the old installation is if it is different than --prefix. Whenever customize.sh is changed, squid.conf is automatically recreated when fn-local-squid.sh is run. - Make several changes to the function of init.d/frontier-squid.sh: - Change its implementation to use runuser instead of su, and to always pass all parameters to $INSTALL_DIR/utils/bin/fn-local-squid.sh after switching to the non-privileged user id. - Attempting to start squid while squid is running was supposed to abort before deleting the cache, but if there was a parse error in squid.conf there was a bug that made it proceed to delete the cache even while squid was running. That bug is now fixed: any start attempt while there's a squid.conf parse error will abort before deleting the cache. - Change restart command to not delete the cache. - Implement the condrestart command to only attempt to restart if squid is already running. - Change the previously undocumented reload command to tell squid to reconfigure rather than restarting it. - Add a cleancache command. - Make error codes from all the commands be returned to the shell. - Make several changes to the default squid.conf: - Instead of having a CMSFRONTIER acl that was unused by default, add a commented-out RESTRICT_DEST acl that gives an example on how to restrict the destinations to only cmsfrontier.cern.ch servers. This may be customized via customize.sh. Uses dstdom_regex instead of dst because the latter only works on IP addresses that are cached at the time squid starts. - Remove the cmsdbsfrontier.cern.ch from the HOST_MONITOR acl and instead put it in a HOST_MONITOR_NAME acl that is of type srcdomain rather than type src, because src (like dst) doesn't recognize an IP address change without restarting squid. Would have preferred to use the alias frontier.cern.ch, but unfortunately squid has no source acl type that will both recognize changing IP addresses and work with DNS aliases. Details are in http://bugs.squid-cache.org/show_bug.cgi?id=2837. - Added explicit path settings for all options that locate files, to make it easier for the rpm version of this package to be relocatable. - Stop turning the update_headers option off by default. This is needed by squids that feed other squids in order to support If-Modified-Since, and even though currently non-launchpad squids don't feed other squids that will likely change in the future. - Change the default umask to 022 so log files will be readable by other. - Add a 'make distclean' target to clean out all generated files and get the source directory ready for distribution. frontier_squid4.0rc9 - 17 September 2009 (bjb@jhu.edu) - Upgrade to squid-2.7.STABLE7. The only significant change is that it it is a stable release that includes patch2451. frontier_squid4.0rc8 - 13 September 2009 (bjb@jhu.edu) - Include patch2451 to fix a squid performance bug that causes already-cached items at a site to triple the number of transactions. This especially harms sites that are far away from the central server. frontier_squid4.0rc7 - 29 July 2009 (bjb@jhu.edu) - Add an extra safety check in the start script frontier_squid4.0rc6 - 8 February 2009 (bjb@jhu.edu) - Upgrade to squid-2.7.STABLE6 to close a potential Denial Of Service vulnerability. frontier_squid4.0rc5 - 31 January 2009 (bjb@jhu.edu) - Set "update_headers off" by default because it was suspected to be the cause of some problems seen and is only needed to be on for squids that feed other squids. frontier_squid4.0rc4 - 31 October 2008 (bjb@jhu.edu) - Upgrade to squid-2.7.STABLE5 - Update monitor IP address - New default access.log format - New restart command