LiveZilla Live Help

You are here

SafeSquid log analyzers

SafeSquid Log Format:

SafeSquid produces logs in three distinct formats.
We traditionally name them as access.log (Access Log Format), extended.log (NCSA / Extended log format) and safesquid.log (Native Log Format).
The path to the log files, and soft link that is created during installation, are as follows:

 

Log File Path Soft Link
access.log /var/log/safesquid/safesquid/access/ /opt/safesquid/safesquid/logs/access/
safesquid.log /var/log/safesquid/safesquid/native/ /opt/safesquid/safesquid/logs/native/
extended.log /var/log/safesquid/safesquid/extended/ /opt/safesquid/safesquid/logs/extended/

Access Log

The access.log has been traditional favorite, because it can be used by a variety of log analyzers like Calamaris, SARG, Squint, SquidTailD, etc.
The reports produced by these log analyzers reveal useful details of the overall usage and the pattern of access of the application.
See Sample Log Reports

Access Log fields:
start_time_in_seconds.milliseconds elapsed_time client cachecode/status size method url username peercode/peer mime

Example:
1189403858.675 654 192.168.0.21 TCP_MISS/200 246 GET http://ds.ds3ps.co.uk:80/refer/surebrowse/operator/chat-server.xml?time=1189404101675 sudipta DIRECT/ds.ds3ps.co.uk text/xml

The details of the fields in access.log are as follows:

Field Explanation
Time UNIX time stamp as Coordinated Universal Time (UTC) seconds with a millisecond resolution.
Elapsed Time Length of time in milliseconds that the cache was busy with the transaction. The information is  logged after the reply has been sent, not during the lifetime of the  transaction.
Client IP address of the requesting host.
Cachecode/Status Two entries separated by a slash. Code specifies the result of the transaction: the kind of request, how it was satisfied, or in what way it  failed. The second entry contains the HTTP result codes.
Bytes Amount of data delivered to the client. This does not constitute the net object size, because headers are also counted. Also, failed requests may deliver an  error page, the size of which is also logged here.
Method Request method to obtain an object, e.g. GET, POST, CONNECT.
URL URL requested.
Username Authenticated username
Peerstatus/Peerhost Two entries separated by a slash. The first entry represents a code that explains how the request was handled, for example, by forwarding it to a peer, or returning the request to the source. The second entry contains the name of the host from which the object was requested. This host may be the origin site, a parent, or any other peer. Also note that the host name may be numerical.
Mime Mime type of the object.

Extended Log

The extended.log (NCSA / Extended log format) records maximum details of each request handled by the proxy application.
Log Analyzers like Sawmill can generate analysis reports using the extended log, and give lots more information, than the ones using access.log.
See Sawmill sample report.

FORMAT:
"UNIQUE_RECORDID" ELAPSED_TIME_IN_MSEC CLIENT_IP "USER_NAME" "CLIENT_CONNECTION_ID" [DATE_TIME_OF_REQUEST] "METHOD URL" "HTTP_STATUS_CODE" BYTES_TRANSFERRED "REFERRER_URL" "USER_AGENT" MIME_TYPE "FILTER_NAME FILTERING_REASON" "COMMA_SEPARATED_LIST_OF_PROFILES_APPLIED" "INTERFACE_IP:INTERFACE_PORT"

Example:
"1191586598.504-7-192.168.0.221-8080" 929 192.168.0.150 "anonymous" "7" [05/Oct/2007:17:46:39] "GET http://updates.f-prot.com:80/cgi-bin/check-updates?run_as=check_updates&protocol=1" 200 750 "-" "FPAV_Update_Monitor/3.16f (Windows; WINNT; 2000 Professional; SP4)" text/plain "- -" "-" "192.168.0.221:8080"

The details of the fields in extended.log are as follows:

Field Explanation
Unique Record ID A unique record identifier, to prevent duplication of records when imported into SQL databases.Here in e.g. 1215419711.460 
Elapsed time in milliseconds Elapsed time of the request, in milliseconds. 
Client IP The IP address of the requesting client.
User name The username, (or user ID) used by the client for authentication. If no value is present, "anonymous" is substituted.
Client connection ID The internal SafeSquid ID associated with this connection. 
Date & time of request The date and time stamp of the HTTP request.The fields in the date/time field are [dd/MMM/yyyy:hh:mm:ss +-hhmm], where the fields are defined as  follows:
dd is the day of the month, MMM is the month, yyyy is the year, hh is the hour, mm is the minute, ss is the seconds.
Method URL The HTTP request. The request field contains three pieces of information. The main piece is the requested resource. The request field also contains  the HTTP method. 
HTTP Status Code  The status code is the numeric code indicating the success or failure of the HTTP request. 
Bytes Transferred  This field is a numeric field containing the number of bytes of data transferred as part of the HTTP request, not including the HTTP header. E.g. 750.
Referrer URL  The referrer is the URL of the HTTP resource that referred the user to the resource requested. "-" is substituted when there are no referrers.
User agent An HTTP client that makes HTTP requests. It is customary for an HTTP client, such as a Web browser, to identify itself by name when making an HTTP request. It is not required, but most HTTP clients do identify themselves by name. 
Mime type MIME-type of the requested object. E.g. text/plain.
Filter name & Filtering reason If the request get blocked, then this field contains the name of the filter, or the reason for which the request was blocked. "- -" is  substituted when there are no blocks.
Comma separated list of profiles applied Comma separated list of profiles that were applied to the request. "-" is substituted when no profiles are applied.
Interface IP:Interface port IP:PORT that received the request. This can be important when SafeSquid is listening on multiple IPs or Ports.

Native Log

This is SafeSquid's native log format.
It  records various functional aspects like REQUESTS, SECURITY, REDIRECT etc. that are effected by the various features and their configuration.
You can control the verbosity of the Native log by specifying LOGLEVEL, as shown in the table below.
The LOGLEVEL parameter affects only the SafeSquid's Native log.

Value Process logged Value Process logged
1 Requests 16384 Forwarding
2 Network 32768 Config synchronization
4 URL filtering 65536 Antivirus
8 Header filtering 131072 External parsers
16 Mime filtering 262144 ICAP 
32 Cookie filtering   524288 DNS blacklist
64 Redirections 1048576 URL blacklist
128 Templates 2097152 URL commands
256 Keyword filtering 4194304 Modules
512 Rewriting 8388608 Security
1024 Limits 16777216 Warnings
2048 Caching 33554432 Errors
4096 Prefetching 67108864 Profiles
8192 ICP 134217728 Debug

So, if you wish to record only the requests set LOGLEVEL to 1, if you wish to record only caching related activities set LOGLEVEL to 2048.
If you wish to record all the three activities of rewriting, limits and forwarding, you would simply set LOGLEVEL to 512 + 1024 + 16384 i.e. 17920.
Similarly, if you wished to view absolutely everything (and run the risk of generating a very huge log file in a very short time!), you could set LOGLEVEL to a total of all the values in the table, i.e. 134217727 which  is also the default LOGLEVEL if you simply comment the LOGLEVEL specification!.
If you wished to produce just debug logs you should set the LOGLEVEL to 134217728.
If you wished to record all activities and debug information, you should set the LOGLEVEL to 268435455.

NOTE: Adjusting this value requires a restart of SafeSquid service.


Log Rotation

There obviously needs to be a control on log file size. SafeSquid  executable cannot start if the size of any of the log files exceeds 2147483648 bytes (2GB).The parameter <LOG_SIZE_LIMIT> sets the maximum size in bytes for a log file, exceeding which, the <INIT_SCRIPT> logrotate (/etc/init.d/safesquid logrotate) will automatically truncate and compress all the three types of log files. The same command can be also run manually to rotate your logs in case any situation demands.