Troubleshooting

Increase the log verbosity of a Canary using `trace` annotation

To add detailed debug information related to a specific Canary to the log output, add a debug: true or trace: true annotation to the Canary resource:

apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: http-check
  annotations:
    trace: "true"
spec:
  http:
    - url: https://httpbin.demo.aws.flanksource.com/headers

Sensitive Data & Excessive Logging

Trace level logging will return the HTTP response body which may contain sensitive data (The authorization headers will be sanitized)

Trace Levels

Level	Logs
`debug`	- HTTP Request and Response Header
`trace`	- HTTP Request and Response Header - HTTP Response Body - Custom Metrics

Run a Canary immediately using `next-runtime` annotation

To run a canary outside of its normal schedule add the annotation:

 kubectl annotate canary <canary> next-runtime=$(date -Iseconds)

Temporarily pause a Canary using `suspend` annotation

 kubectl annotate canary <canary> supend=true

Run a Canary from the CLI

The easiest method of troubleshooting is to run the canary-checker run command with a copy of the canary resource locally. This increases velocity of troubleshooting iteration, and also prevents the possibility of sensitive information in verbose output being leaked to logging systems by the canary pod.

Local permissions

The local canary-checker execution may need to be provided with the permissions required to run the check.
Depending on the check this might include:

The ability to read secrets or other objects in the cluster
Access to cloud APIs, such as AWS, GCP or Azure API interfaces
Locally installed binaries invoked by exec scripts The following local settings will affect the permissions available for use by the canary-checker executable:
The active kubectl context as specified in KUBECONFIG
The AWS profile specified in AWS_DEFAULT_PROFILE or AWS_PROFILE, or the AWS credentials specified by AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
The GCP credentials configured through gcloud auth application-default login
The $PATH configuration

Local Canary execution

Install the CLI as per the Installation instructions Generate a local resource file with the trace annotation enabled:

    kubectl get canary test-canary -n test-namespace -o yaml | yq '.metadata.annotations.trace = "true"' > test-canary.yaml

Or generate a resource file by hand, then run the canary-checker CLI:

    canary-checker run -v test-canary.yaml

Troubleshooting

Increase the log verbosity of a Canary using trace annotation​

Trace Levels​

Run a Canary immediately using next-runtime annotation​

Temporarily pause a Canary using suspend annotation​

Run a Canary from the CLI​

Local permissions​

Local Canary execution​