Security Benchmarking Authorization Policy Engines: Rego, Cedar, OpenFGA & Teleport ACD

Intro

Back in 2024, Amazon Web Services (AWS) engaged Trail of Bits (ToB) to perform a comparative assessment between several authorization and access management policy languages. If you're unfamiliar with the concept of a policy engine, it's essentially a fully-featured engine that offloads authorization decisions in an application. Instead of implementing an authorization framework from scratch, it is possible to define policies written in specific languages that dictate which people or machines can run which actions on which resources.

The goals of the ToB research work were to identify broadly applicable threats to policy languages, to identify language features that partially or fully mitigate those threats, and to provide security recommendations to improve the general design of policy languages. In practice, this was an in-depth threat modeling exercise involving Cedar, Rego, and OpenFGA policy languages. The final deliverable of their research can be downloaded from ToB's Github repository.

Though many security engineering aspects have been covered in this analysis and it has served as the foundation for our work, it merely resulted in a static threat model. Given that we're evaluating custom programming languages and their execution environments, we thought that it would be interesting to explore the possibility of developing a dynamic evaluation framework.

Thanks to the support of Teleport, we developed an automated framework for evaluating the security of such authorization engines. Our Security Policy Evaluation Framework (SPEF) is a testing and benchmarking system designed to evaluate the robustness and correctness of various authorization policy engines. It provides a consistent, automated environment for executing policy test cases across multiple languages. This framework is primarily intended for researchers, security engineers, and policy developers who want to benchmark how different policy engines behave under predefined test conditions.

Before introducing SPEF, let's briefly review the policy frameworks currently integrated into our evaluation framework:

Rego is a high-level, declarative query language, created by Styra. It extends Datalog and is used to write policies for decision-making in applications. It is primarily used with the Open Policy Agent (OPA) and was designed to define access control, resource validation, and data filtering policies.

How Rego Works:

• Write policies in .rego files.
• Load them into OPA.
• Query OPA via an HTTP API or embedded SDK with an input (JSON).
• OPA evaluates the policy and returns a decision (e.g., allow = true).

Cedar is a declarative open-source authorization language and engine developed by AWS. It was designed to allow defining fine-grained, role-based access control (RBAC) and attribute-based access control (ABAC) policies in a safe, fast, and auditable way and evaluating those policies.

How Cedar Works:

• Define schema (entities, actions, resources, attributes).
• Write policies using the Cedar language.
• Evaluate policies using the Cedar engine with a JSON input request.
• Get a decision (e.g., allow or deny).

OpenFGA is an open-source fine-grained authorization system built for scalable relationship-based access control (ReBAC), inspired by Google's Zanzibar system. It includes a policy engine based on relationships (who is related to what and how) and was designed for high-scale, low-latency access control. It is maintained by OpenFGA (part of the CNCF sandbox as of 2023) and initially created by Auth0/Okta.

How OpenFGA Works:

• Define your model (types, relations, permissions).
• Store relationships (e.g., user X is a writer on document Y).
• Query the API: "Can user X perform action Y on object Z?".
• OpenFGA returns true or false based on the relationship graph.

Teleport ACD (Access Control Decision) is a component of the Teleport Platform, which is available as an open-source project, or an Enterprise option, created and maintained by Teleport. It moves the access-control decisions in Teleport from the agent to a centralized service representable as a gRPC API.

How Teleport ACD Works:

• Teleport agents establish trust with an internal Auth service.
• Agents then verify user-provided certificates, which encode a user's roles.
• The agents then query the Teleport Access Control Decision API for access-control decisions (e.g., allow/deny) using Teleport's Predicate language. Access-control decisions are made by querying role specifications stored as YAML files, which define resources permissions and implement a default-deny strategy.

Key Features

Security Policy Evaluation Framework (SPEF) Architecture

The framework is built in a modular design that keeps orchestration, execution, evaluation, and result processing as separated steps. At a high level, it starts by loading available test cases and filtering them based on the arguments provided - such as specific test case IDs, ID ranges, or an explicit list.

Each selected test is executed inside a dedicated Docker container that runs the corresponding policy engine. For every test case and policy language, the evaluation takes place in an isolated environment specific to that engine. This approach not only avoids conflicts with the host environment but also makes it easier to support additional engines down the line, since they operate independently from one another.

Under the hood, the internal modules handle different stages of the process: some manage the orchestration of tests, others take care of interacting with policy engines, running evaluations, and collecting outputs. Each engine interaction is abstracted, so whether it communicates via HTTP, system shell calls, or custom logic, the test execution remains consistent. Each test case in the framework is designed to be evaluated independently for every supported policy language. When a test is defined, the expected outcome is defined as well - this is known as the expected_result. The framework uses this information to compare the actual result produced by the policy engine with what was expected. Based on this comparison, the test is marked as one of several possible outcomes: PASS, FAIL, TIMEOUT, ERROR, or NOT APPLICABLE. This approach allows defining a consistent validation, whether a query/assertion should succeed, trigger an error or timeout.

Defining Test Cases

Each test case in the framework is defined through a manifest file that describes both the scenario to evaluate and the specific inputs required by each policy engine. The manifest includes metadata like the test ID, a short scenario title, and a description of what the test aims to verify. For each supported engine, a corresponding block specifies the necessary elements for execution:

• OpenFGA test cases require three main components to run a test: an authorization model (which defines types and relations), a query (including user, relation, and object), and a set of tuples that represent the current state of the authorization data. These elements work together to simulate a real-world decision scenario and allow the engine to check if access should be granted or denied based on the model's logic.

• Cedar test cases need a set of entities to define the actors, resources, and relationships involved in a scenario. Each test includes a policy file written in Cedar's policy language, and a query that specifies the principal, the action, and the accessed resource.

• Rego test cases evaluates test cases based on a combination of a policy file (written in the Rego language) and a corresponding query input. The engine then computes whether the query is allowed or denied, or leads to an error.

• Teleport ACD (Access Control Decision) tests handles evaluations using a bootstrap configuration file which contains definitions of roles, users, and permissions in a format specific to Teleport. The framework uses this file to spin up a test instance of the engine and simulate an access request based on the defined setup. Each test is paired with an expected_result, which not only defines whether the outcome should be a success or an error, but may also include a logical assertion on the output itself.

To better understand how the framework works in practice, let's look at a concrete example that focuses on a specific security and performance scenario. In this case, we want to assess how each policy engine handles the use of regular expression built-ins, and whether they introduce any unexpected behavior or performance degradation during the test evaluation.

The test case is defined as follows:

id: testcase-16
scenario: User Induces a Denial of Service on the Policy Engine
description: Regular Expression Built-ins Do Not Introduce Security and Performance Risks
rego:
  - query: query.json
    policy: policy.rego
    reference: https://www.openpolicyagent.org/docs/latest/policy-performance/#in-memory-store-read-optimization
    expected_result:
      - status: success
      - condition: evaluation_time <= 0.05 # expecting a reasonable eval time

openfga:
  - authorization_model: authorization-model.json
    tuples: tuples.json
    query: query.json
    expected_result:
      - status: success
      - condition: evaluation_time <= 0.05 # expecting a reasonable eval time

cedar:
  - reference: https://cedarland.blog/design/why-no-regex/content.html <<Regular expressions and string formatting operators were intentionally omitted from the language because they work against these safety goals. This blog post describes why they can be dangerous and alternative approaches to writing policies without them.>>
    applicable: false

teleportacd:
  - type: evaluate-ssh-access
    login: mohamed
    username: mohamed
    server-id: f48b739e-7b6c-47e5-997e-f52e43273fae
    bootstrap: bootstrap_dangerous_regex.yaml
    expected_result:
      - status: success
      - condition: evaluation_time <= 0.05 # expecting a reasonable eval time

This test case can be executed on its own by using the --only flag and passing the ID:

$ python main.py --only 16

This will run the evaluation across all policy engines that support this scenario and output the final result to an HTML report: