-
Notifications
You must be signed in to change notification settings - Fork 48
Overview
This repository contains a collection of support libraries for Java applications. They serve two main purposes:
- Provide means to emit structured application log messages in JSON format.
- Instrument parts of your application stack to collect request metrics in a JSON request log.
The libraries take an opinionated approach on the fields contained in the generated JSON. It produces a standard format, that can be easier parsed and analysed in log analysis services. Details on the standard fields can be found in the fields.yml files for application logs and request logs. With proper configuration those fields should be populated automatically. Users can change the field values add additional fields known as custom fields to the generated messages. See the article on Custom Fields for details.
The library contains special support to add application metadata from a Cloud Foundry runtime environment, such as application id and name. This support is now optional, since the runtime usually sends these metadata in a side-channel when exporting the logs. See the article on Writing Application Logs for configuration details.
cf-java-loggging-support will generate two kind of log messages. While the full specification of the log formats can be found in the beats folder, we start off with a short overview of what messages would look like when using these support libraries.
You can run the sample application locally and create a simple log message with the following curl command:
curl -X POST -u user:secret 'localhost:8080/log/test/info?m=Hello+cf-java-logging-support!'
The response will be `Generated info log with message: "Hello cf-java-logging-support!" with the following associated application log:
{"msg":"Hello cf-java-logging-support!","level":"INFO","written_ts":"1648619448171070000","logger":"test","thread":"http-nio-8080-exec-1","type":"log","message_count":19,"correlation_id":"6e72144d-ff3f-4d67-93e9-a5b57726017d","written_at":"2022-03-30T05:50:48.171070Z"}
This line actually stems from the following log statement in the code
logger.info(message);The fields that should be familiar to you are logger, thread, level, and msg and directly correspond to pattern fields known from log4j2 or logback. There is, of course, also a "date" included in that message, but we've decided to call that field written_at. Next to it is a field written_ts which represents the "timestamp" when the message was written with nanosecond precision.
The field categories may contain additional category keys that can be later used to filter log messages by specific categories. How this field can be filled is covered in more detail in the section Writing Application Logs.
The remaining fields provide context information that is either derived from information provided by the Cloud Foundry runtime environment or via request instrumentation and is kept in mapped diagnostic contexts.
In the console output of the sample application, you will also find a log line like this:
{"categories":["request"],"request":"/log/test/info?m=Hello+cf-java-logging-support!","request_size_b":-1,"logger":"com.sap.hcp.cf.logging.servlet.filter.RequestLogger","type":"request","layer":"[SERVLET]","protocol":"HTTP/1.1","remote_ip":"redacted","response_content_type":"text/plain;charset=ISO-8859-1","request_received_at":"2022-03-30T05:50:47.986380Z","written_at":"2022-03-30T05:50:48.196559Z","response_time_ms":208.262,"direction":"IN","response_sent_at":"2022-03-30T05:50:48.194642Z","response_status":200,"method":"POST","level":"INFO","response_size_b":66,"written_ts":"1648619448196559000","remote_port":"redacted","thread":"http-nio-8080-exec-1","message_count":20,"remote_host":"redacted","correlation_id":"6e72144d-ff3f-4d67-93e9-a5b57726017d"}
As you can see, it shares all the context fields with application log messages. The log pattern fields though have been replaced by a set of request metric fields, like, e.g., response_status, response_time_ms, and response_size_b.
With version 2.1.3, the fields remote_ip, remote_host, remote_port and x_forwarded_for as well as the fields referer and remote_user are no longer logged by default in the request logs created by this library. Logging of these fields can be activated by adding the environment variables:
| Environment Variable | Optional Fields |
|---|---|
LOG_SENSITIVE_CONNECTION_DATA: true |
activates the fields remote_ip, remote_host, remote_port, x_forwarded_for |
LOG_REMOTE_USER: true |
activates the field remote_user |
LOG_REFERER: true |
activates the field referer |
An example for this can be found in the manifest.yml of the sample app: https://github.com/SAP/cf-java-logging-support/blob/master/sample/manifest.yml
Caution: in Version 2.1.2 logging of the remote_ip field could be activated via the environment variable LOG_REMOTE_IP: true. This variable has been renamed to LOG_SENSITIVE_CONNECTION_DATA: true since it switches logging of more fields than just the remote_ip field.
You may also notice that both log messages contain a field correlation_id which allows you to correlate an incoming request to application log messages that have been emitted during the cause of processing that request.
Note that an application log will only report a meaningful correlation ID if that has been injected into the context information via proper instrumentation.
The available instrumentation libraries will be covered in the section Request Instrumentation.