Skip to content

Commit

Permalink
Merge branch 'main' into split-query-config-files
Browse files Browse the repository at this point in the history
  • Loading branch information
galvana authored Jan 9, 2025
2 parents 12ce3d6 + a583252 commit a009dbb
Show file tree
Hide file tree
Showing 53 changed files with 1,508 additions and 176 deletions.
2 changes: 2 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ _Write some things here about the changes and any potential caveats_
* [ ] Issue requirements met
* [ ] All CI pipelines succeeded
* [ ] `CHANGELOG.md` updated
* [ ] Add a https://github.com/ethyca/fides/labels/db-migration label to the entry if your change includes a DB migration
* [ ] Add a https://github.com/ethyca/fides/labels/high-risk label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
* Followup issues:
* [ ] Followup issues created (include link)
* [ ] No followup issues
Expand Down
37 changes: 0 additions & 37 deletions .github/workflows/backend_checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,43 +75,6 @@ jobs:
path: /tmp/python-${{ matrix.python_version }}.tar
retention-days: 1

###################
## Static Checks ##
###################
Static-Checks:
strategy:
matrix:
session_name:
[
'"isort(check)"',
'"black(check)"',
"mypy",
"pylint",
"xenon",
"check_install",
'"pytest(nox)"',
]
runs-on: ubuntu-latest
continue-on-error: true
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Set Up Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
cache: "pip"

- name: Install Nox
run: pip install nox>=2022

- name: Install Dev Requirements
run: pip install -r dev-requirements.txt

- name: Run Static Check
run: nox -s ${{ matrix.session_name }}

##################
## Performance ##
##################
Expand Down
53 changes: 53 additions & 0 deletions .github/workflows/static_checks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: Backend Static Code Checks

on:
pull_request:
push:
branches:
- "main"
- "release-**"

env:
IMAGE: ethyca/fides:local
DEFAULT_PYTHON_VERSION: "3.10.13"
# Docker auth with read-only permissions.
DOCKER_USER: ${{ secrets.DOCKER_USER }}
DOCKER_RO_TOKEN: ${{ secrets.DOCKER_RO_TOKEN }}

jobs:
###################
## Static Checks ##
###################
Static-Checks:
strategy:
matrix:
session_name:
[
'"isort(check)"',
'"black(check)"',
"mypy",
"pylint",
"xenon",
"check_install",
'"pytest(nox)"',
]
runs-on: ubuntu-latest
continue-on-error: true
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Set Up Python
uses: actions/setup-python@v5
with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
cache: "pip"

- name: Install Nox
run: pip install nox>=2022

- name: Install Dev Requirements
run: pip install -r dev-requirements.txt

- name: Run Static Check
run: nox -s ${{ matrix.session_name }}
15 changes: 13 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,20 @@ The types of changes are:
- `Fixed` for any bug fixes.
- `Security` in case of vulnerabilities.

Changes can also be flagged with a GitHub label for tracking purposes. The URL of the label should be put at the end of the entry. The possible labels are:
- https://github.com/ethyca/fides/labels/high-risk: to indicate that a change is a "high-risk" change that could potentially lead to unanticipated regressions or degradations
- https://github.com/ethyca/fides/labels/db-migration: to indicate that a given change includes a DB migration

## [Unreleased](https://github.com/ethyca/fides/compare/2.52.0...main)

### Added
- Added Action Center MVP behind new feature flag [#5622](https://github.com/ethyca/fides/pull/5622)
- Added cache-clearing methods to the `DBCache` model to allow deleting cache entries [#5629](https://github.com/ethyca/fides/pull/5629)

### Fixed
- Fixed issue where the custom report "reset" button was not working as expected [#5649](https://github.com/ethyca/fides/pull/5649)
- Fixed column ordering issue in the Data Map report [#5649](https://github.com/ethyca/fides/pull/5649)
- Fixed issue where the Data Map report filter dialog was missing an Accordion item label [#5649](https://github.com/ethyca/fides/pull/5649)

## [2.52.0](https://github.com/ethyca/fides/compare/2.51.2...2.52.0)

Expand All @@ -26,7 +37,7 @@ The types of changes are:
- Added event based communication example to the Cookie House sample app [#5597](https://github.com/ethyca/fides/pull/5597)
- Added new erasure tests for BigQuery Enterprise [#5554](https://github.com/ethyca/fides/pull/5554)
- Added new `has_next` parameter for the `link` pagination strategy [#5596](https://github.com/ethyca/fides/pull/5596)
- Added a `DBCache` model for database-backed caching [#5613](https://github.com/ethyca/fides/pull/5613)
- Added a `DBCache` model for database-backed caching [#5613](https://github.com/ethyca/fides/pull/5613) https://github.com/ethyca/fides/labels/db-migration
- Adds "reclassify" button to discovery result tables [#5574](https://github.com/ethyca/fides/pull/5574)
- Added support for exporting datamaps with column renaming, reordering and visibility options [#5543](https://github.com/ethyca/fides/pull/5543)

Expand Down Expand Up @@ -95,7 +106,7 @@ The types of changes are:
- Allow hiding systems via a `hidden` parameter and add two flags on the `/system` api endpoint; `show_hidden` and `dnd_relevant`, to display only systems with integrations [#5484](https://github.com/ethyca/fides/pull/5484)
- The CMP override `fides_privacy_policy_url` will now apply even if the `fides_override_language` doesn't match [#5515](https://github.com/ethyca/fides/pull/5515)
- Updated POST taxonomy endpoints to handle creating resources without specifying fides_key [#5468](https://github.com/ethyca/fides/pull/5468)
- Disabled connection pooling for task workers and added retries and keep-alive configurations for database connections [#5448](https://github.com/ethyca/fides/pull/5448)
- Disabled connection pooling for task workers and added retries and keep-alive configurations for database connections [#5448](https://github.com/ethyca/fides/pull/5448) https://github.com/ethyca/fides/labels/high-risk
- Added timeout handling in the UI for async discovery monitor-related queries [#5519](https://github.com/ethyca/fides/pull/5519)

### Developer Experience
Expand Down
175 changes: 175 additions & 0 deletions clients/admin-ui/cypress/e2e/action-center.cy.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
import { stubActionCenter, stubPlus } from "cypress/support/stubs";

import {
ACTION_CENTER_ROUTE,
INTEGRATION_MANAGEMENT_ROUTE,
} from "~/features/common/nav/v2/routes";

describe("Action center", () => {
beforeEach(() => {
cy.login();
stubPlus(true);
stubActionCenter();
});

describe("disabled web monitor", () => {
beforeEach(() => {
cy.intercept("GET", "/api/v1/config*", {
body: {
detection_discovery: {
website_monitor_enabled: false,
},
},
}).as("getTranslationConfig");
cy.visit(ACTION_CENTER_ROUTE);
});
it("should display a message that the web monitor is disabled", () => {
cy.wait("@getTranslationConfig");
cy.contains("currently disabled").should("exist");
});
});

describe("empty action center", () => {
beforeEach(() => {
cy.intercept("GET", "/api/v1/plus/discovery-monitor/aggregate-results*", {
fixture: "empty-pagination.json",
}).as("getMonitorResults");
cy.visit(ACTION_CENTER_ROUTE);
});
it("should display empty state", () => {
cy.wait("@getMonitorResults");
cy.get("[data-testid='search-bar']").should("exist");
cy.get(`[class*='ant-empty'] [class*='ant-empty-image']`).should("exist");
cy.get(
`[class*='ant-empty'] a[href="${INTEGRATION_MANAGEMENT_ROUTE}"]`,
).should("exist");
});
});

describe("Action center monitor aggregate results", () => {
const webMonitorKey = "my_web_monitor_2";
const integrationMonitorKey = "My_New_BQ_Monitor";
beforeEach(() => {
cy.visit(ACTION_CENTER_ROUTE);
});
it("should render the current monitor results", () => {
cy.get("[data-testid='Action center']").should("exist");
cy.wait("@getMonitorResults");
cy.get("[data-testid*='monitor-result-']").should("have.length", 3);
cy.get("[data-testid^='monitor-result-']").each((result) => {
const monitorKey = result
.attr("data-testid")
.replace("monitor-result-", "");
// linked title
cy.wrap(result)
.contains("assets detected")
.should("have.attr", "href", `${ACTION_CENTER_ROUTE}/${monitorKey}`);
// last monitored relative date with real date in tooltip
cy.wrap(result)
.find("[data-testid='monitor-date']")
.contains(" ago")
.realHover();
cy.get(".ant-tooltip-inner").should("contain", "December");
});
// description
cy.getByTestId(`monitor-result-${webMonitorKey}`).should(
"contain",
"92 Browser Requests, 5 Cookies detected.",
);
// monitor name
cy.getByTestId(`monitor-result-${webMonitorKey}`).should(
"contain",
"my web monitor 2",
);
});
it("should have appropriate actions for web monitors", () => {
cy.wait("@getMonitorResults");
// Add button
// TODO: [HJ-337] uncomment when Add button is implemented
// cy.getByTestId(`add-button-${webMonitorKey}`).should("exist");
// Review button
cy.getByTestId(`review-button-${webMonitorKey}`).should(
"have.attr",
"href",
`${ACTION_CENTER_ROUTE}/${webMonitorKey}`,
);
});
it.skip("Should have appropriate actions for Integrations monitors", () => {
cy.wait("@getMonitorResults");
// Classify button
cy.getByTestId(`review-button-${integrationMonitorKey}`).should(
"have.attr",
"href",
`${ACTION_CENTER_ROUTE}/${integrationMonitorKey}`,
);
// Ignore button
cy.getByTestId(`ignore-button-${integrationMonitorKey}`).should("exist");
});
it.skip("Should have appropriate actions for SSO monitors", () => {
cy.wait("@getMonitorResults");
// Add button
cy.getByTestId(`add-button-${webMonitorKey}`).should("exist");
// Ignore button
cy.getByTestId(`ignore-button-${webMonitorKey}`).should("exist");
});
it.skip("Should paginate results", () => {
// TODO: mock pagination and also test skeleton loading state
});
});

describe("Action center system aggregate results", () => {
const webMonitorKey = "my_web_monitor_1";
beforeEach(() => {
cy.visit(`${ACTION_CENTER_ROUTE}/${webMonitorKey}`);
});
it("should display a breadcrumb", () => {
cy.getByTestId("page-breadcrumb").within(() => {
cy.get("a.ant-breadcrumb-link")
.should("contain", "All activity")
.should("have.attr", "href", ACTION_CENTER_ROUTE);
cy.contains("my_web_monitor_1").should("exist");
});
});
it("should render the aggregated system results in a table", () => {
cy.wait("@getSystemAggregateResults");
cy.getByTestId("column-system_name").should("exist");
cy.getByTestId("column-total_updates").should("exist");
cy.getByTestId("column-data_use").should("exist");
cy.getByTestId("column-locations").should("exist");
cy.getByTestId("column-domains").should("exist");
cy.getByTestId("column-actions").should("exist");
cy.getByTestId("search-bar").should("exist");
cy.getByTestId("pagination-btn").should("exist");
cy.getByTestId("row-0-col-system_name").within(() => {
cy.getByTestId("change-icon").should("exist"); // new result
cy.contains("Uncategorized assets").should("exist");
});
// data use column should be empty for uncategorized assets
cy.getByTestId("row-0-col-data_use").children().should("have.length", 0);
cy.getByTestId("row-1-col-system_name").within(() => {
cy.getByTestId("change-icon").should("not.exist"); // existing result
cy.contains("Google Tag Manager").should("exist");
});
// TODO: data use column should not be empty for other assets
// cy.getByTestId("row-1-col-data_use").children().should("not.have.length", 0);

// multiple locations
cy.getByTestId("row-2-col-locations").should("contain", "2 locations");
// single location
cy.getByTestId("row-3-col-locations").should("contain", "USA");

// multiple domains
cy.getByTestId("row-0-col-domains").should("contain", "29 domains");
// single domain
cy.getByTestId("row-3-col-domains").should(
"contain",
"analytics.google.com",
);
});
// it("should navigate to table view on row click", () => {
// cy.getByTestId("row-1").click();
// cy.url().should("contain", "fds.1046");
// cy.getByTestId("page-breadcrumb").should("contain", "fds.1046");
// });
});
});
44 changes: 40 additions & 4 deletions clients/admin-ui/cypress/e2e/datamap-report.cy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,15 @@ describe("Data map report table", () => {
it("should filter the table by making a selection", () => {
cy.getByTestId("filter-multiple-systems-btn").click();
cy.getByTestId("datamap-report-filter-modal").should("be.visible");
cy.getByTestId("filter-modal-accordion-button")
.eq(0)
.should("have.text", "Data use");
cy.getByTestId("filter-modal-accordion-button")
.eq(1)
.should("have.text", "Data categories");
cy.getByTestId("filter-modal-accordion-button")
.eq(2)
.should("have.text", "Data subject");
cy.getByTestId("filter-modal-accordion-button").eq(1).click();
cy.getByTestId("filter-modal-checkbox-tree-categories").should(
"be.visible",
Expand Down Expand Up @@ -389,14 +398,15 @@ describe("Data map report table", () => {
cy.get("#toast-datamap-report-toast")
.should("be.visible")
.should("have.attr", "data-status", "success");
cy.getByTestId("custom-reports-trigger")
.should("contain.text", "My Custom Report")
.click();
cy.getByTestId("custom-reports-trigger").should(
"contain.text",
"My Custom Report",
);
cy.getByTestId("fidesTable").within(() => {
// reordering applied to report
cy.get("thead th").eq(2).should("contain.text", "Legal name");
// column visibility applied to report
cy.get("thead th").eq(4).should("not.contain.text", "Data subject");
cy.getByTestId("column-data_subjects").should("not.exist");
});
cy.getByTestId("group-by-menu").should(
"contain.text",
Expand Down Expand Up @@ -442,10 +452,36 @@ describe("Data map report table", () => {
cy.getByTestId("custom-reports-reset-button").click();
cy.getByTestId("apply-report-button").click();
cy.getByTestId("custom-reports-popover").should("not.be.visible");

cy.getByTestId("custom-reports-trigger").should(
"contain.text",
"Reports",
);
cy.getByTestId("fidesTable").within(() => {
// reordering reverted
cy.get("thead th").eq(2).should("contain.text", "Data categories");
// column visibility restored
cy.getByTestId("column-data_subjects").should("exist");
});
cy.getByTestId("group-by-menu").should("contain.text", "Group by system");
cy.getByTestId("more-menu").click();
cy.getByTestId("edit-columns-btn").click();
cy.get("button#data_subjects").should(
"have.attr",
"aria-checked",
"true",
);
cy.getByTestId("column-settings-close-button").click();
cy.getByTestId("filter-multiple-systems-btn").click();
cy.getByTestId("datamap-report-filter-modal")
.should("be.visible")
.within(() => {
cy.getByTestId("filter-modal-accordion-button").eq(0).click();
cy.getByTestId("checkbox-Analytics").within(() => {
cy.get("[data-checked]").should("not.exist");
});
cy.getByTestId("standard-dialog-close-btn").click();
});
});
it("should allow the user cancel a report selection", () => {
cy.wait("@getCustomReportsMinimal");
Expand Down
Loading

0 comments on commit a009dbb

Please sign in to comment.