From 059b52759e0df118039afdec1a9ad421482767de Mon Sep 17 00:00:00 2001 From: zeripath Date: Thu, 4 Apr 2019 09:44:01 +0100 Subject: [PATCH] Some logging documentation (#6498) --- .../advanced/logging-documentation.en-us.md | 396 ++++++++++++++++++ 1 file changed, 396 insertions(+) create mode 100644 docs/content/doc/advanced/logging-documentation.en-us.md diff --git a/docs/content/doc/advanced/logging-documentation.en-us.md b/docs/content/doc/advanced/logging-documentation.en-us.md new file mode 100644 index 0000000000..db8dcbba36 --- /dev/null +++ b/docs/content/doc/advanced/logging-documentation.en-us.md @@ -0,0 +1,396 @@ +--- +date: "2019-04-02T17:06:00+01:00" +title: "Advanced: Logging Configuration" +slug: "logging-configuration" +weight: 55 +toc: true +draft: false +menu: + sidebar: + parent: "advanced" + name: "Logging Configuration" + weight: 55 + identifier: "logging-configuration" +--- + +# Logging Configuration + +The logging framework has been revamped in Gitea 1.9.0. + +## Log Groups + +The fundamental thing to be aware of in Gitea is that there are several +log groups: + +* The "Default" logger +* The Macaron logger +* The Router logger +* The Access logger +* The XORM logger +* A logger called the `GitLogger` which is used during hooks. + +There is also the go log logger. + +### The go log logger + +Go provides its own extremely basic logger in the `log` package, +however, this is not sufficient for our purposes as it does not provide +a way of logging at multiple levels, nor does it provide a good way of +controlling where these logs are logged except through setting of a +writer. + +We have therefore redirected this logger to our Default logger, and we +will log anything that is logged using the go logger at the INFO level. + +### The "Default" logger + +Calls to `log.Info`, `log.Debug`, `log.Error` etc. from the `code.gitea.io/gitea/modules/log` package will log to this logger. + +You can configure the outputs of this logger by setting the `MODE` +value in the `[log]` section of the configuration. + +Each output sublogger is configured in a separate `[log.sublogger]` +section, but there are certain default values. These will not be inherited from the `[log]` section: + +* `FLAGS` is `stdflags` (Equal to +`date,time,medfile,shortfuncname,levelinitial`) +* `FILE_NAME` will default to `%(ROOT_PATH)/gitea.log` +* `EXPRESSION` will default to `""` +* `PREFIX` will default to `""` + +The provider type of the sublogger can be set using the `MODE` value in +its subsection, but will default to the name. This allows you to have +multiple subloggers that will log to files. + +### The "Macaron" logger + +By default Macaron will log to its own go `log` instance. This writes +to `os.Stdout`. You can redirect this log to a Gitea configurable logger +through setting the `ENABLE_MACARON_REDIRECT` setting in the `[log]` +section which you can configure the outputs of by setting the `MACARON` +value in the `[log]` section of the configuration. `MACARON` defaults +to `file` if unset. + +Each output sublogger for this logger is configured in +`[log.sublogger.macaron]` sections. There are certain default values +which will not be inherited from the `[log]` or relevant +`[log.sublogger]` sections: + +* `FLAGS` is `stdflags` (Equal to +`date,time,medfile,shortfuncname,levelinitial`) +* `FILE_NAME` will default to `%(ROOT_PATH)/macaron.log` +* `EXPRESSION` will default to `""` +* `PREFIX` will default to `""` + +NB: You can redirect the macaron logger to send its events to the gitea +log using the value: `MACARON = ,` + +### The "Router" logger + +There are two types of Router log. By default Macaron send its own +router log which will be directed to Macaron's go `log`, however if you +`ENABLE_MACARON_REDIRECT` you will enable Gitea's router log. You can +disable both types of Router log by setting `DISABLE_ROUTER_LOG`. + +If you enable the redirect, you can configure the outputs of this +router log by setting the `ROUTER` value in the `[log]` section of the +configuration. `ROUTER` will default to `console` if unset. The Gitea +Router logs the same data as the Macaron log but has slightly different +coloring. It logs at the `Info` level by default, but this can be +changed if desired by setting the `ROUTER_LOG_LEVEL` value. + +Each output sublogger for this logger is configured in +`[log.sublogger.router]` sections. There are certain default values +which will not be inherited from the `[log]` or relevant +`[log.sublogger]` sections: + +* `FILE_NAME` will default to `%(ROOT_PATH)/router.log` +* `FLAGS` defaults to `date,time` +* `EXPRESSION` will default to `""` +* `PREFIX` will default to `""` + +NB: You can redirect the router logger to send its events to the Gitea +log using the value: `ROUTER = ,` + +### The "Access" logger + +The Access logger is a new logger for version 1.9. It provides a NCSA +Common Log compliant log format. It's highly configurable but caution +should be taken when changing its template. The main benefit of this +logger is that Gitea can now log accesses in a standard log format so +standard tools may be used. + +You can enable this logger using `ENABLE_ACCESS_LOG`. Its outputs are +configured by setting the `ACCESS` value in the `[log]` section of the +configuration. `ACCESS` defaults to `file` if unset. + +Each output sublogger for this logger is configured in +`[log.sublogger.access]` sections. There are certain default values +which will not be inherited from the `[log]` or relevant +`[log.sublogger]` sections: + +* `FILE_NAME` will default to `%(ROOT_PATH)/access.log` +* `FLAGS` defaults to `` or None +* `EXPRESSION` will default to `""` +* `PREFIX` will default to `""` + +If desired the format of the Access logger can be changed by changing +the value of the `ACCESS_LOG_TEMPLATE`. + +NB: You can redirect the access logger to send its events to the Gitea +log using the value: `ACCESS = ,` + +#### The ACCESS_LOG_TEMPLATE + +This value represent a go template. It's default value is: + +`{{.Ctx.RemoteAddr}} - {{.Identity}} {{.Start.Format "[02/Jan/2006:15:04:05 -0700]" }} "{{.Ctx.Req.Method}} {{.Ctx.Req.RequestURI}} {{.Ctx.Req.Proto}}" {{.ResponseWriter.Status}} {{.ResponseWriter.Size}} "{{.Ctx.Req.Referer}}\" \"{{.Ctx.Req.UserAgent}}"` + +The template is passed following options: + +* `Ctx` is the `macaron.Context` +* `Identity` is the `SignedUserName` or `"-"` if the user is not logged +in +* `Start` is the start time of the request +* `ResponseWriter` is the `macaron.ResponseWriter` + +Caution must be taken when changing this template as it runs outside of +the standard panic recovery trap. The template should also be as simple +as it runs for every request. + +### The "XORM" logger + +The XORM logger is a long-standing logger that exists to collect XORM +log events. It is enabled by default but can be switched off by setting +`ENABLE_XORM_LOG` to `false` in the `[log]` section. Its outputs are +configured by setting the `XORM` value in the `[log]` section of the +configuration. `XORM` defaults to `,` if unset, meaning it is redirected +to the main Gitea log. + +XORM will log SQL events by default. This can be changed by setting +the `LOG_SQL` value to `false` in the `[database]` section. + +Each output sublogger for this logger is configured in +`[log.sublogger.xorm]` sections. There are certain default values +which will not be inherited from the `[log]` or relevant +`[log.sublogger]` sections: + +* `FILE_NAME` will default to `%(ROOT_PATH)/xorm.log` +* `FLAGS` defaults to `date,time` +* `EXPRESSION` will default to `""` +* `PREFIX` will default to `""` + +### The Hook and Serv "GitLoggers" + +These are less well defined loggers. Essentially these should only be +used within Gitea's subsystems and cannot be configured at present. + +They will write log files in: + +* `%(ROOT_PATH)/hooks/pre-receive.log` +* `%(ROOT_PATH)/hooks/update.log` +* `%(ROOT_PATH)/hooks/post-receive.log` +* `%(ROOT_PATH)/serv.log` +* `%(ROOT_PATH)/http.log` + +In the future these logs may be rationalised. + +## Log outputs + +Gitea provides 4 possible log outputs: + +* `console` - Log to `os.Stdout` or `os.Stderr` +* `file` - Log to a file +* `conn` - Log to a keep-alive TCP connection +* `smtp` - Log via email + +Certain configuration is common to all modes of log output: + +* `LEVEL` is the lowest level that this output will log. This value +is inherited from `[log]` and in the case of the non-default loggers +from `[log.sublogger]`. +* `STACKTRACE_LEVEL` is the lowest level that this output will print +a stacktrace. This value is inherited. +* `MODE` is the mode of the log output. It will default to the sublogger +name. Thus `[log.console.macaron]` will default to `MODE = console`. +* `COLORIZE` will default to `true` for `file` and `console` as +described, otherwise it will default to `false`. + +### Non-inherited default values + +There are several values which are not inherited as described above but +rather default to those specific to type of logger, these are: +`EXPRESSION`, `FLAGS`, `PREFIX` and `FILE_NAME`. + +#### `EXPRESSION` + +`EXPRESSION` represents a regular expression that log events must match to be logged by the sublogger. Either the log message, (with colors removed), must match or the `longfilename:linenumber:functionname` must match. NB: the whole message or string doesn't need to completely match. + +Please note this expression will be run in the sublogger's goroutine +not the logging event subroutine. Therefore it can be complicated. + +#### `FLAGS` + +`FLAGS` represents the preceding logging context information that is +printed before each message. It is a comma-separated string set. The order of values does not matter. + +Possible values are: + +* `none` or `,` - No flags. +* `date` - the date in the local time zone: `2009/01/23`. +* `time` - the time in the local time zone: `01:23:23`. +* `microseconds` - microsecond resolution: `01:23:23.123123`. Assumes +time. +* `longfile` - full file name and line number: `/a/b/c/d.go:23`. +* `shortfile` - final file name element and line number: `d.go:23`. +* `funcname` - function name of the caller: `runtime.Caller()`. +* `shortfuncname` - last part of the function name. Overrides +`funcname`. +* `utc` - if date or time is set, use UTC rather than the local time +zone. +* `levelinitial` - Initial character of the provided level in brackets eg. `[I]` for info. +* `level` - Provided level in brackets `[INFO]` +* `medfile` - Last 20 characters of the filename - equivalent to +`shortfile,longfile`. +* `stdflags` - Equivalent to `date,time,medfile,shortfuncname,levelinitial` + +### Console mode + +For loggers in console mode, `COLORIZE` will default to `true` if not +on windows, or the windows terminal can be set into ANSI mode or is a +cygwin or Msys pipe. + +If `STDERR` is set to `true` the logger will use `os.Stderr` instead of +`os.Stdout`. + +### File mode + +The `FILE_NAME` defaults as described above. If set it will be relative +to the provided `ROOT_PATH` in the master `[log]` section. + +Other values: + +* `LOG_ROTATE`: **true**: Rotate the log files. +* `MAX_SIZE_SHIFT`: **28**: Maximum size shift of a single file, 28 represents 256Mb. +* `DAILY_ROTATE`: **true**: Rotate logs daily. +* `MAX_DAYS`: **7**: Delete the log file after n days +* NB: `COLORIZE`: will default to `true` if not on windows. +* `COMPRESS`: **true**: Compress old log files by default with gzip +* `COMPRESSION_LEVEL`: **-1**: Compression level + +### Conn mode + +* `RECONNECT_ON_MSG`: **false**: Reconnect host for every single message. +* `RECONNECT`: **false**: Try to reconnect when connection is lost. +* `PROTOCOL`: **tcp**: Set the protocol, either "tcp", "unix" or "udp". +* `ADDR`: **:7020**: Sets the address to connect to. + +### SMTP mode + +It is not recommended to use this logger to send general logging +messages. However, you could perhaps set this logger to work on `FATAL`. + +* `USER`: User email address to send from. +* `PASSWD`: Password for the smtp server. +* `HOST`: **127.0.0.1:25**: The SMTP host to connect to. +* `RECEIVERS`: Email addresses to send to. +* `SUBJECT`: **Diagnostic message from Gitea** + +## Default Configuration + +The default empty configuration is equivalent to: + +```ini +[log] +ROOT_PATH = %(GITEA_WORK_DIR)/log +MODE = console +LEVEL = Info +STACKTRACE_LEVEL = None +REDIRECT_MACARON_LOG = false +ENABLE_ACCESS_LOG = false +ENABLE_XORM_LOG = true +XORM = , + +[log.console] +MODE = console +LEVEL = %(LEVEL) +STACKTRACE_LEVEL = %(STACKTRACE_LEVEL) +FLAGS = stdflags +PREFIX = +COLORIZE = true # Or false if your windows terminal cannot color +``` + +This is equivalent to sending all logs to the console, with default go log being sent to the console log too. + +## Log colorization + +Logs to the console will be colorized by default when not running on +Windows. Terminal sniffing will occur on Windows and if it is +determined that we are running on a terminal capable of color we will +colorize. + +Further, on *nix it is becoming common to have file logs that are +colored by default. Therefore file logs will be colorised by default +when not running on Windows. + +You can switch on or off colorization by using the `COLORIZE` value. + +From a development point of view. If you write +`log.Info("A %s string", "formatted")` the `formatted` part of the log +message will be Bolded on colorized logs. + +You can change this by either rendering the formatted string yourself. +Or you can wrap the value in a `log.ColoredValue` struct. + +The `log.ColoredValue` struct contains a pointer to value, a pointer to +string of bytes which should represent a color and second set of reset +bytes. Pointers were chosen to prevent copying of large numbers of +values. There are several helper methods: + +* `log.NewColoredValue` takes a value and 0 or more color attributes +that represent the color. If 0 are provided it will default to a cached +bold. Note, it is recommended that color bytes constructed from +attributes should be cached if this is a commonly used log message. +* `log.NewColoredValuePointer` takes a pointer to a value, and +0 or more color attributes that represent the color. +* `log.NewColoredValueBytes` takes a value and a pointer to an array +of bytes representing the color. + +These functions will not double wrap a `log.ColoredValue`. They will +also set the ResetBytes to the cached resetBytes. + +Be careful not to change the contents of resetBytes or boldBytes as this +will break rendering of logging elsewhere. You have been warned. + +## Log Spoofing protection + +In order to protect the logs from being spoofed with cleverly +constructed messages. Newlines are now prefixed with a tab and control +characters except those used in an ANSI CSI are escaped with a +preceding `\` and their octal value. + +## Creating a new named logger group + +Should a developer wish to create a new named logger, `NEWONE`. It is +recommended to add an `ENABLE_NEWONE_LOG` value to the `[log]` +section, and to add a new `NEWONE` value for the modes. + +A function like `func newNewOneLogService()` is recommended to manage +construction of the named logger. e.g. + +```go +func newNewoneLogService() { + EnableNewoneLog = Cfg.Section("log").Key("ENABLE_NEWONE_LOG").MustBool(false) + Cfg.Section("log").Key("NEWONE").MustString("file") // or console? or "," if you want to send this to default logger by default + if EnableNewoneLog { + options := newDefaultLogOptions() + options.filename = filepath.Join(LogRootPath, "newone.log") + options.flags = "stdflags" + options.bufferLength = Cfg.Section("log").Key("BUFFER_LEN").MustInt64(10000) + generateNamedLogger("newone", options) + } +} +``` + +You should then add `newOneLogService` to `NewServices()` in +`modules/setting/setting.go`