Version 4.2.1

Version 4.2.1

Postby sudipta » Wed Oct 24, 2007 4:46 pm

SafeSquid produces logs in three distinct formats.
We traditionally name them as safesquid.log (Native Log Format), access.log (Access Log Format) & extended.log (NCSA / Extended log format).
Versions prior to SafeSquid 4.1.x only Advanced Edition produced all these three formats. The other editions produced only Native & Squid log formats.
However since 4.1.0 all editions of SafeSquid are capable of producing all the three formats.
Of course, the application owners have a choice of deciding the name and path of each of these logs. The users can also choose if they do not want to produce any one or all of these three logs.
These logs contain a wealth of information, that can be used by application managers, to understand the overall operation and functioning of the application.
The access.log has been traditional favorite, because it can be used by a variety of log analyzers like Calamaris, SARG, Squint, SquidTailD, etc. The reports produced by these log analyzers reveal useful details of the overall usage and the pattern of access of the application. The Native Log records the conditions encountered by the application and processes undertaken by SafeSquid. The Extended log records maximum details of each request handled by the proxy application.

Improvisation -
For SafeSquid 4.2.1 we focused on improvisations in logs, and handling of forwarded requests.

Until SafeSquid 4.2.0 SafeSquid needed a restart when the logfiles reached 2GB size.
From 4.2.1 this limitation has been removed. The limitation was enforced by using the older 32-bit referencing while opening files. Most SafeSquid users have now migrated to newer kernel, and modern distros, that support 64-bit referencing of files, so this shouldn't effect users using Linux Kernel 2.6.x

The Native log now records the application of "profiles". Profiles are undoubtedly one of the most important aspects of SafeSquid. From 4.2.1 onwards addition or removal of profiles is logged for every request, in the Native Logs. The Extended Logs will now, also record the profiles that were applied to each request that was handled by the application. Unfortunately there isn't yet a log analyser completely compatible to generate reports from the extended logs. Though Awstats may be used for some benefits. A uniue record identifier is now printed with every line of extended log, to easily prevent duplication of records when imported into SQL databases.

The new format for extended log:

"UNIQUE_RECORDID" ELAPSED_TIME_IN_MSEC CLIENT_IP "USER_NAME" "CLIENT_CONNECTION_ID" [DATE_TIME_OF_REQUEST] "METHOD URL" "HTTP_STATUS_CODE" BYTES_TRANSFERRED "REFERRER_URL" "USER_AGENT" MIME_TYPE "FILTER_NAME FILTERING_REASON" "COMMA_SEPARATED_LIST_OF_PROFILES_APPLIED" "INTERFACE_IP:INTERFACE_PORT"


Example:

"1191586598.504-7-192.168.0.221-8888" 929 192.168.0.150 "anonymous" "7" [05/Oct/2007:17:46:39] "GET http://updates.f-prot.com:80/cgi-bin/ch ... protocol=1" 200 750 "-" "FPAV_Update_Monitor/3.16f (Windows; WINNT; 2000 Professional; SP4)" text/plain "- -" "-" "192.168.0.221:8888"


Details:

"UNIQUE_RECORDID" - A unique record identifier, to prevent duplication of records when imported into SQL databases. E.g. "1191586598.504-7-192.168.0.221-8888"

ELAPSED_TIME_IN_MSEC - Elapsed time of the request, in milliseconds. E.g. 929

CLIENT_IP - The IP address of the requesting client. E.g. 192.168.0.150

"USER_NAME" - The username, (or user ID) used by the client for authentication. If no value is present, "anonymous" is substituted. E.g. "anonymous"

"CLIENT_CONNECTION_ID" - The internal SafeSquid ID associated with this connection. E.g. "7".

[DATE_TIME_OF_REQUEST] - The date and time stamp of the HTTP request.
The fields in the date/time field are [dd/MMM/yyyy:hh:mm:ss +-hhmm], where the fields are defined as follow:
dd is the day of the month, MMM is the month, yyy is the year, hh is the hour, mm is the minute, ss is the seconds.
E.g. [05/Oct/2007:17:46:39]

"METHOD URL" - The HTTP request. The request field contains three pieces of information. The main piece is the requested resource. The request field also contains the HTTP method. E.g. "GET http://updates.f-prot.com:80/cgi-bin/ch ... protocol=1"

"HTTP_STATUS_CODE" - The status code is the numeric code indicating the success or failure of the HTTP request. E.g. 200.

BYTES_TRANSFERRED - This field is a numeric field containing the number of bytes of data transferred as part of the HTTP request, not including the HTTP header. E.g. 750.

"REFERRER_URL" - The referrer is the URL of the HTTP resource that referred the user to the resource requested. "-" is substituted when there are no referrers.

"USER_AGENT" - An HTTP client that makes HTTP requests. It is customary for an HTTP client, such as a Web browser, to identify itself by name when making an HTTP request. It is not required, but most HTTP clients do identify themselves by name. E.g. "FPAV_Update_Monitor/3.16f (Windows; WINNT; 2000 Professional; SP4)"

MIME_TYPE - The MIME-type of the requested object. E.g. text/plain.

"FILTER_NAME FILTERING_REASON" - If the request get blocked, then this field contains the name of the filter, or the reason for which the request was blocked. "- -" is substituted when there are no blocks.

"COMMA_SEPARATED_LIST_OF_PROFILES_APPLIED" - The comma separated list of profiles that were applied to the request. "-" is substituted when no profiles are applied.

"INTERFACE_IP:INTERFACE_PORT" - The IP:PORT that received the request. This can be important when SafeSquid is listening on multiple IPs or Ports. E.g. "192.168.0.221:8888".


The Access Logs are rather legacy in nature, and any change in their content or structure could break the compatibility with the popular and legacy analysers. The structure of Access logs therefore remain unchanged. HOWEVER, prior to 4.2.1, the time spent by the user on an HTTPS / SSL session was not recorded. From 4.2.1 the CONNECT requests will be appropriately logged. Unfortunately, at this time, the quantity of data transferred (bytes) is shown as "0", and will be substituted with the correct value in a future release.

4.2.1 also introduces new variables for use in Custom Templates and External Parsers.
The following is the list of all variables, that may be used with SafeSquid 4.2.1 and subsequent versions.

VERSION - The Version of SafeSquid being used.
INTERFACE - The I.P. address of the SafeSquid Service, that recieved the request
PORT - The I.P. address of the SafeSquid Service, that recieved the request
IP - The I.P. address of the source of the request.
CLIENTID - The unique Client ID allocated to the connection handled.
USERNAME - The username of the client.
TIME - The time of the request.
URL - The full URL requested
HTTP_HOST - The target Host that served the response.
HTTP_FILE - The File that was served as a response by the target web-server
HTTP_PORT - The port of the web-server to which the request was made.
HTTP_METHOD - The HTTP method ( GET / POST / CONNECT ) used for the request
HTTP_PROTO - The protocol, over HTTP used to make the request, ( HTTP / FTP / CONNECT )
MIME - The mime-type of the downloaded content
SIZE - The size in bytes of the downloaded data
DOWNLOADLIMIT - The effective maximum downloadlimit in bytes
TRANSFERRED - The amount of data transferred in bytes
UPLOADLIMIT - The effective maximum uploadlimit in bytes
MTIME - The last-modified time of the cached file
FILTER -The name of the Filter that blocked the content
THRESHOLD - The Threshold limit defined for keyword filtering
SCORE - The total score resulting from all the keyword filtering rules
AVSCANNER - The name of the virus scanner that detected virus
VIRUSNAME - The name of the virus that was detected
CATEGORY - The name of the category that was determined by the UrlBlacklist Filter
IMAGESCORE - The score applied to the image by the Image Filter
IMAGETHRESHOLD - The Threshold limit set for Image Filter

Obviously some of the above listed variables, would be available, if content was blocked. Additionally, for every header received from the remote website and set by a client, an environment variable is set. All the environment variables for the server's headers start with SERVER_, and the client's start with CLIENT_; All '-' (dashes) in the header type are converted to '_' (underscores), and all characters are in uppercase. If an executable returns with a non-zero status code, the original content is returned.

Please view the sample shell scripts and templates available along with the installation package, for a better understanding.


Enhancement -
SafeSquid 4.2.1 and later will now be -

# compatible to systems running 64bit operating systems

# easily installable on all distros, without the inconvenience of locating and fixing the ssl libraries.


BugFix -
A bug was identified in SafeSquid's ICAP Client. Some legacy ICAP services exhibit "Response Modified" characteristics, without providing a modified response content. SafeSquid versions prior to 4.2.1, suffered crashes in such situations. A fix has been implemented against this in SafeSquid 4.2.1. If you are using SafeSquid with an ICAP service and application frequently crashes, you surely must upgrade to SafeSquid 4.2.1

If you are already using SafeSquid 4.2.0. and your application does not use ICAP, you may safely skip migration to SafeSquid 4.2.1., in case the enhancements in 4.2.1 do not impress you. Maybe 4.2.2 will attract you more.
sudipta
Sales Admin
 
Posts: 639
Joined: Wed Apr 14, 2004 8:50 pm
Location: India

Return to Changelog

Who is online

Users browsing this forum: No registered users and 0 guests

cron