Skip to content

Commit

Permalink
chore(NA): upgrade to webpack 5 (#191106)
Browse files Browse the repository at this point in the history 
### Summary

- Closes #89741

This PR contains the resulting work of a massive effort that ports our
on top bundler abstraction (called @kbn/optimizer) from Webpack v4 into
Webpack v5. It's essential in terms of long term maintenance since v4
was not receiving updates any longer but will also unblock some new
features that could be beneficial for our future DevEx endeavours.

Next you can find a small list of all the accomplished tasks on this
journey.

### Completed Tasks
- [x] Upgrade dependencies to match the ones on webpack v5
- [x] Fix null-loader usages
- [x] Fix raw-loader usages
- [x] Fix file-loader usages
- [x] Fix url-loader usages
- [x] Fix `@kbn/optimizer-webpack-helpers` to support webpack v5 
- [x] Adopt previous webpack v4 polyfill-all strategy with
node-polyfill-webpack-plugin
- [x] Fix theme-loader on @kbn/optimizer
- [x] Migrate configurations and ad-hoc loader options on all webpack
configs from v4 to v5
- [x] Fix @kbn/test jest resolver for file-loader cases
- [x] Migrate public-path loader on UiSharedDeps
- [x] Fix all usages of webpack-merge
- [x] Migrate BundleRemoteModule
- [x] Migrate BundleRemotesPlugin
- [x] Correctly migrate PopulateBundleCachePlugin
- [x] Correctly migrate BundleMetricsPlugin
- [x] Check if the profiling plugins still work (--profile flag)
- [x] Recover if possible the previous webpack v4 cacheGroup chunks
rename to something like `data.plugin.chunk.0.js`
- [x] Run `/ci` and make sure we get our first green CI, otherwise work
on the errors until we do
- [x] Profile and solve bottlenecks until we get a cold build
performance similar to the one we had on webpack v4 (`node
scripts/build_kibana_platform_plugins --no-cache`).
- [x] OpenSSL Legacy Warnings: try to remove `--openssl-legacy-provider
` flags
- [x] Add Webpack to Renovate config
- [x] Explore removing `NodePolyfillPlugin`
([here](https://www.npmjs.com/package/node-polyfill-webpack-plugin)) and
add each polyfill needed individually per each webpack config to check
if we get smaller bundles. If we do it's better to go with the case by
case need approach instead of deploying a bunch of polyfills with
NodePolyfillPlugin. As another alternative, create a custom smaller
plugin with only the union of all needed polyfills.
- [x] Evaluate if we want to touch the resolutions on mainFields and
conditionNames
- [x] Understand why `@import 'src/core/public/mixins'` does not work
anymore (not a problem, we should use relative paths anyway but we want
to track why it changed from v4 to v5)
- [x] BUG: Child compilers are having errors hidden and/or changed from
error to warning
- [x] Fix license check for
[Artistic-2.0](https://spdx.org/licenses/Artistic-2.0.html) is the
license for
[domain-browser](https://github.com/bevry/domain-browser?tab=License-1-ov-file).
This package is a dependency of
[NodePolyfillPlugin](https://www.npmjs.com/package/node-polyfill-webpack-plugin).
Artistic 2.0 license is [classified as
yellow](https://github.com/elastic/open-source/blob/main/elastic-product-policy.md#yellow-list)
and should only be used for dev dependencies.
- [x] Make sure `resourceQuery: { not: /raw/ }` is not necessary on
other webpack configs like storybook one
- [x] Find what is being wrongly removed by usedExports optimization;
hint: I believe it is identifying a lot of exports inside the sync entry
of plugins as unused exports and removing them. Then `__kbnBootstrap__`
can't be found
- [x] Rebalance @kbn/optimizer pickMaxWorkerCount
- [x] Re-open the issue to fix sass-warnings
[#190345](#190345) or downgrade
sass-loader to v10
- [x] Remove previous esm no parse rules
- [x] Confirm esm support is working
- [x] Confirm console override is needed
- [x] Confirm react prod builds on ui shared deps for distributable
- [x] Remove customization for
[xyflow](https://github.com/xyflow/xyflow) from webpack configs
- [x] Clean all the code
- [x] Make sure collected metrics from stats are still aligned with what
we were collecting before; also verify if the modules used for optimizer
caches etc are well generated (@kbn/node-libs-browser)
- [x] Fix watch performance

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Brad White <brad.white@elastic.co>
  • Loading branch information
@mistic @kibanamachine @Ikuni17
3 people authored Feb 14, 2025
1 parent 200922a commit 203bc28
Show file tree
Hide file tree
Showing 91 changed files with 1,896 additions and 940 deletions.
2 changes: 1 addition & 1 deletion .buildkite/pipelines/pull_request/webpack_bundle_analyzer.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ steps:
- command: .buildkite/scripts/steps/webpack_bundle_analyzer/build_and_upload.sh
label: 'Build Webpack Bundle Analyzer reports'
agents:
machineType: n2-standard-4
machineType: n2-standard-8
preemptible: true
key: webpack_bundle_analyzer
timeout_in_minutes: 60
6 changes: 5 additions & 1 deletion .buildkite/scripts/steps/storybooks/build_and_upload.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,11 @@ const build = () => {
console.log('--- Building Storybooks');

for (const storybook of Object.keys(storybookAliases)) {
exec(`STORYBOOK_BASE_URL=${STORYBOOK_BASE_URL}`, `yarn storybook --site ${storybook}`);
exec(
`STORYBOOK_BASE_URL=${STORYBOOK_BASE_URL}`,
`NODE_OPTIONS=--max-old-space-size=6144`,
`yarn storybook --site ${storybook}`
);
}
};

Expand Down
1 change: 1 addition & 0 deletions .github/CODEOWNERS
View file
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ packages/kbn-management/storybook/config @elastic/kibana-management
packages/kbn-manifest @elastic/kibana-core
packages/kbn-mock-idp-plugin @elastic/kibana-security
packages/kbn-mock-idp-utils @elastic/kibana-security
packages/kbn-node-libs-browser-webpack-plugin @elastic/kibana-operations
packages/kbn-openapi-bundler @elastic/security-detection-rule-management
packages/kbn-openapi-generator @elastic/security-detection-rule-management
packages/kbn-optimizer @elastic/kibana-operations
Expand Down
2 changes: 1 addition & 1 deletion examples/embeddable_examples/public/app/overview.tsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import React from 'react';
import { css } from '@emotion/react';
import { EuiMarkdownFormat } from '@elastic/eui';
// @ts-ignore
import overviewMarkdown from '!!raw-loader!@kbn/embeddable-plugin/README.md';
import overviewMarkdown from '@kbn/embeddable-plugin/README.md?raw';

export const Overview = () => {
return (
Expand Down
8 changes: 4 additions & 4 deletions examples/embeddable_examples/public/app/register_embeddable.tsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@
import React from 'react';
import { EuiCodeBlock, EuiSpacer, EuiText } from '@elastic/eui';
// @ts-ignore
import registerSearchEmbeddableSource from '!!raw-loader!../react_embeddables/search/register_search_embeddable';
import registerSearchEmbeddableSource from '../react_embeddables/search/register_search_embeddable?raw';
// @ts-ignore
import registerAttachActionSource from '!!raw-loader!../react_embeddables/search/register_add_search_panel_action';
import registerAttachActionSource from '../react_embeddables/search/register_add_search_panel_action?raw';
// @ts-ignore
import registerFieldListEmbeddableSource from '!!raw-loader!../react_embeddables/field_list/register_field_list_embeddable';
import registerFieldListEmbeddableSource from '../react_embeddables/field_list/register_field_list_embeddable?raw';
// @ts-ignore
import registerReactEmbeddableSavedObjectSource from '!!raw-loader!../react_embeddables/register_saved_object_example';
import registerReactEmbeddableSavedObjectSource from '../react_embeddables/register_saved_object_example?raw';

export const RegisterEmbeddable = () => {
return (
Expand Down
10 changes: 5 additions & 5 deletions examples/field_formats_example/public/app.tsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,15 @@ import { FieldFormatsStart } from '@kbn/field-formats-plugin/public';
import * as example1 from './examples/1_using_existing_format';
import * as example2 from './examples/2_creating_custom_formatter';
// @ts-ignore
import example1SampleCode from '!!raw-loader!./examples/1_using_existing_format';
import example1SampleCode from './examples/1_using_existing_format?raw';
// @ts-ignore
import example2SampleCodePart1 from '!!raw-loader!../common/example_currency_format';
import example2SampleCodePart1 from '../common/example_currency_format?raw';
// @ts-ignore
import example2SampleCodePart2 from '!!raw-loader!./examples/2_creating_custom_formatter';
import example2SampleCodePart2 from './examples/2_creating_custom_formatter?raw';
// @ts-ignore
import example2SampleCodePart3 from '!!raw-loader!../server/examples/2_creating_custom_formatter';
import example2SampleCodePart3 from '../server/examples/2_creating_custom_formatter?raw';
// @ts-ignore
import example3SampleCode from '!!raw-loader!./examples/3_creating_custom_format_editor';
import example3SampleCode from './examples/3_creating_custom_format_editor?raw';

export interface Deps {
fieldFormats: FieldFormatsStart;
Expand Down
58 changes: 28 additions & 30 deletions package.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
"serverless-security": "node scripts/kibana --dev --serverless=security",
"spec_to_console": "node scripts/spec_to_console",
"start": "node scripts/kibana --dev",
"storybook": "node --openssl-legacy-provider scripts/storybook",
"storybook": "node --no-deprecation scripts/storybook",
"test:ftr": "node scripts/functional_tests",
"test:ftr:runner": "node scripts/functional_test_runner",
"test:ftr:server": "node scripts/functional_tests_server",
Expand Down Expand Up @@ -691,6 +691,7 @@
"@kbn/newsfeed-plugin": "link:src/platform/plugins/shared/newsfeed",
"@kbn/newsfeed-test-plugin": "link:test/common/plugins/newsfeed",
"@kbn/no-data-page-plugin": "link:src/platform/plugins/private/no_data_page",
"@kbn/node-libs-browser-webpack-plugin": "link:packages/kbn-node-libs-browser-webpack-plugin",
"@kbn/notifications-plugin": "link:x-pack/platform/plugins/shared/notifications",
"@kbn/object-utils": "link:src/platform/packages/shared/kbn-object-utils",
"@kbn/object-versioning": "link:src/platform/packages/shared/kbn-object-versioning",
Expand Down Expand Up @@ -1322,7 +1323,6 @@
"@babel/plugin-proposal-optional-chaining": "^7.21.0",
"@babel/plugin-proposal-private-methods": "^7.18.6",
"@babel/plugin-transform-class-properties": "^7.24.7",
"@babel/plugin-transform-logical-assignment-operators": "^7.24.7",
"@babel/plugin-transform-numeric-separator": "^7.24.7",
"@babel/plugin-transform-optional-chaining": "^7.24.8",
"@babel/plugin-transform-runtime": "^7.24.7",
Expand All @@ -1336,7 +1336,7 @@
"@bazel/typescript": "4.6.2",
"@cypress/debugging-proxy": "2.0.1",
"@cypress/grep": "^4.0.1",
"@cypress/webpack-preprocessor": "^6.0.1",
"@cypress/webpack-preprocessor": "^6.0.2",
"@elastic/eslint-plugin-eui": "0.0.2",
"@elastic/makelogs": "^6.1.1",
"@elastic/synthetics": "^1.17.2",
Expand Down Expand Up @@ -1549,11 +1549,13 @@
"@storybook/addon-storyshots": "^6.5.16",
"@storybook/addons": "^6.5.16",
"@storybook/api": "^6.5.16",
"@storybook/builder-webpack5": "^6.5.16",
"@storybook/client-api": "^6.5.16",
"@storybook/components": "^6.5.16",
"@storybook/core": "^6.5.16",
"@storybook/core-common": "^6.5.16",
"@storybook/core-events": "^6.5.16",
"@storybook/manager-webpack5": "^6.5.16",
"@storybook/node-logger": "^6.5.16",
"@storybook/preview-web": "^6.5.16",
"@storybook/react": "^6.5.16",
Expand Down Expand Up @@ -1680,11 +1682,10 @@
"@types/vinyl": "^2.0.4",
"@types/vinyl-fs": "^3.0.2",
"@types/watchpack": "^1.1.5",
"@types/webpack": "^4.41.3",
"@types/webpack": "^5.28.5",
"@types/webpack-bundle-analyzer": "^4.7.0",
"@types/webpack-env": "^1.15.3",
"@types/webpack-merge": "^4.1.5",
"@types/webpack-sources": "^0.1.4",
"@types/webpack-env": "^1.18.5",
"@types/webpack-sources": "^3.2.3",
"@types/xml2js": "^0.4.11",
"@types/yargs": "^15.0.0",
"@types/yauzl": "^2.9.1",
Expand All @@ -1699,12 +1700,13 @@
"autoprefixer": "^10.4.7",
"axe-core": "^4.10.0",
"babel-jest": "^29.7.0",
"babel-loader": "^8.2.5",
"babel-loader": "^9.1.3",
"babel-plugin-add-module-exports": "^1.0.4",
"babel-plugin-istanbul": "^6.1.1",
"babel-plugin-require-context-hook": "^1.0.0",
"babel-plugin-styled-components": "^2.1.4",
"babel-plugin-transform-react-remove-prop-types": "^0.4.24",
"babel-plugin-transform-require-default": "^0.1.7",
"babel-plugin-transform-typescript-metadata": "^0.3.2",
"backport": "^9.6.4",
"blob-polyfill": "^7.0.20220408",
Expand All @@ -1713,12 +1715,12 @@
"chance": "1.0.18",
"chromedriver": "^132.0.0",
"clarify": "^2.2.0",
"clean-webpack-plugin": "^3.0.0",
"clean-webpack-plugin": "^4.0.0",
"cli-progress": "^3.12.0",
"cli-table3": "^0.6.1",
"content-security-policy-parser": "^0.6.0",
"cpy": "^8.1.1",
"css-loader": "^3.4.2",
"css-loader": "^7.1.2",
"cssnano": "^5.1.12",
"cssnano-preset-default": "^5.2.12",
"cssstyle": "^4.1.0",
Expand Down Expand Up @@ -1756,7 +1758,7 @@
"eslint-traverse": "^1.0.0",
"exit-hook": "^2.2.0",
"expect": "^29.7.0",
"expose-loader": "^0.7.5",
"expose-loader": "^5.0.0",
"express": "^4.21.2",
"fetch-mock": "^10.1.0",
"file-loader": "^4.2.0",
Expand All @@ -1767,7 +1769,7 @@
"gulp-postcss": "^9.0.1",
"gulp-terser": "^2.1.0",
"has-ansi": "^3.0.0",
"html-loader": "^1.3.2",
"html-loader": "^5.1.0",
"http-proxy": "^1.18.1",
"http2-proxy": "^5.0.53",
"http2-wrapper": "^2.2.1",
Expand All @@ -1791,10 +1793,9 @@
"license-checker": "^25.0.1",
"listr2": "^8.2.5",
"lmdb": "^2.9.2",
"loader-utils": "^2.0.4",
"marge": "^1.0.1",
"micromatch": "^4.0.8",
"mini-css-extract-plugin": "1.1.0",
"mini-css-extract-plugin": "2.9.1",
"minimist": "^1.2.8",
"mocha": "^10.3.0",
"mocha-junit-reporter": "^2.0.2",
Expand All @@ -1807,7 +1808,7 @@
"mutation-observer": "^1.0.3",
"native-hdr-histogram": "^1.0.0",
"nock": "12.0.3",
"null-loader": "^3.0.0",
"node-libs-browser": "^2.2.1",
"nyc": "^17.1.0",
"oboe": "^2.1.7",
"openapi-types": "^12.1.3",
Expand All @@ -1821,28 +1822,27 @@
"playwright-chromium": "1.49.0",
"pngjs": "^7.0.0",
"postcss": "^8.4.31",
"postcss-loader": "^4.2.0",
"postcss-loader": "^8.1.1",
"postcss-prefix-selector": "^1.16.0",
"postcss-scss": "^4.0.4",
"prettier": "^2.8.8",
"proxy": "^2.1.1",
"raw-loader": "^3.1.0",
"react-is": "~18.2.0",
"react-test-renderer": "^17.0.2",
"recast": "^0.23.9",
"regenerate": "^1.4.0",
"resolve": "^1.22.0",
"rxjs-marbles": "^7.0.1",
"sass-embedded": "^1.78.0",
"sass-loader": "^10.5.1",
"sass-loader": "^10.5.2",
"selenium-webdriver": "^4.28.0",
"sharp": "0.32.6",
"simple-git": "^3.16.0",
"sinon": "^7.4.2",
"sort-package-json": "^1.53.1",
"source-map": "^0.7.4",
"string-replace-loader": "^2.2.0",
"style-loader": "^1.1.3",
"string-replace-loader": "^3.1.0",
"style-loader": "^4.0.0",
"stylelint": "^14.16.1",
"stylelint-scss": "^4.3.0",
"superagent": "^10.1.1",
Expand All @@ -1853,25 +1853,23 @@
"table": "^6.8.2",
"tape": "^5.9.0",
"terser": "^5.36.0",
"terser-webpack-plugin": "^4.2.3",
"terser-webpack-plugin": "^5.3.10",
"tough-cookie": "^5.0.0",
"trace": "^3.2.0",
"tree-kill": "^1.2.2",
"ts-morph": "^15.1.0",
"tsd": "^0.31.1",
"typescript": "5.1.6",
"url-loader": "^2.2.0",
"val-loader": "^1.1.1",
"val-loader": "^6.0.0",
"vinyl-fs": "^4.0.0",
"watchpack": "^1.6.0",
"web-streams-polyfill": "^4.0.0",
"webpack": "^4.41.5",
"webpack-bundle-analyzer": "^4.5.0",
"webpack-cli": "^4.10.0",
"webpack-dev-server": "^4.9.3",
"webpack-merge": "^4.2.2",
"webpack-sources": "^1.4.1",
"webpack-visualizer-plugin2": "^1.1.0",
"webpack": "^5.95.0",
"webpack-bundle-analyzer": "^4.10.2",
"webpack-cli": "^5.1.4",
"webpack-dev-server": "^5.0.4",
"webpack-merge": "^6.0.1",
"webpack-sources": "^3.2.3",
"xml-crypto": "^6.0.0",
"xmlbuilder": "15.1.1",
"yargs": "^15.4.1",
Expand Down
12 changes: 12 additions & 0 deletions packages/kbn-ambient-ui-types/index.d.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,15 @@ declare module '*.mdx' {
// eslint-disable-next-line import/no-default-export
export default MDXComponent;
}

declare module '*?asUrl' {
const content: string;
// eslint-disable-next-line import/no-default-export
export default string;
}

declare module '*?raw' {
const content: string;
// eslint-disable-next-line import/no-default-export
export default string;
}
1 change: 1 addition & 0 deletions packages/kbn-babel-preset/BUILD.bazel
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ RUNTIME_DEPS = [
"@npm//babel-plugin-styled-components",
"@npm//babel-plugin-transform-react-remove-prop-types",
"@npm//babel-plugin-transform-typescript-metadata",
"@npm//babel-plugin-transform-require-default",
]

js_library(
Expand Down
19 changes: 18 additions & 1 deletion packages/kbn-babel-preset/webpack_preset.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,12 @@
const { USES_STYLED_COMPONENTS } = require('./styled_components_files');

/** @type {import('@babel/core').ConfigFunction} */
module.exports = (api, options = {}) => {
module.exports = (
api,
options = {
useTransformRequireDefault: false,
}
) => {
return {
presets: [
[
Expand All @@ -27,6 +32,18 @@ module.exports = (api, options = {}) => {
],
[require('./common_preset'), options],
],
plugins: [
// Conditionally include babel-plugin-transform-require-default
//
// We need to include this plugin in the main worker webpack config that handles our
// non node modules code base in order to support resolving esm
// as a priority over cjs (if that's defined in the mainFields). Without that we might run into
// cases where we have a repo wide cjs code that requires an esm module (coming from the ui-shared-deps that also prioritizes esm)
// which will not be applying the .default key in the require itself.
...(options.useTransformRequireDefault
? [require.resolve('babel-plugin-transform-require-default')]
: []),
],
env: {
production: {
plugins: [
Expand Down
2 changes: 2 additions & 0 deletions packages/kbn-cypress-config/index.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import { v4 as uuid } from 'uuid';
import { defineConfig } from 'cypress';
import wp from '@cypress/webpack-preprocessor';
import { NodeLibsBrowserPlugin } from '@kbn/node-libs-browser-webpack-plugin';

export function defineCypressConfig(options?: Cypress.ConfigOptions<any>) {
return defineConfig({
Expand Down Expand Up @@ -43,6 +44,7 @@ export function defineCypressConfig(options?: Cypress.ConfigOptions<any>) {
},
],
},
plugins: [new NodeLibsBrowserPlugin()],
},
})(file);
});
Expand Down
4 changes: 3 additions & 1 deletion packages/kbn-cypress-config/tsconfig.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,7 @@
"exclude": [
"target/**/*"
],
"kbn_references": []
"kbn_references": [
"@kbn/node-libs-browser-webpack-plugin",
]
}
4 changes: 2 additions & 2 deletions packages/kbn-eslint-plugin-imports/src/rules/no_boundary_crossing.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,8 @@ export const NoBoundaryCrossingRule: Rule.RuleModule = {
return visitAllImportStatements((req, { node, importer, type }) => {
if (
req === null ||
// we can ignore imports using the raw-loader, they will need to be resolved but can be managed on a case by case basis
req.startsWith('!!raw-loader') ||
// we can ignore imports using the ?raw (replacing legacy raw-loader), they will need to be resolved but can be managed on a case by case basis
req.endsWith('?raw') ||
// type only imports can stretch across all the boundaries
isTypeOnlyImport(importer)
) {
Expand Down
4 changes: 2 additions & 2 deletions packages/kbn-eslint-plugin-imports/src/rules/no_group_crossing_imports.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ export const NoGroupCrossingImportsRule: Rule.RuleModule = {
return visitAllImportStatements((req, { node }) => {
if (
req === null ||
// we can ignore imports using the raw-loader, they will need to be resolved but can be managed on a case by case basis
req.startsWith('!!raw-loader')
// we can ignore imports using the ?raw (replacing legacy raw-loader), they will need to be resolved but can be managed on a case by case basis
req.endsWith('?raw')
) {
return;
}
Expand Down
4 changes: 2 additions & 2 deletions packages/kbn-import-resolver/src/import_resolver.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -231,8 +231,8 @@ export class ImportResolver {
resolve(req: string, dirname: string): ResolveResult | null {
// transform webpack loader requests and focus on the actual file selected
const lastExI = req.lastIndexOf('!');
if (lastExI > -1) {
const quesI = req.lastIndexOf('?');
const quesI = req.lastIndexOf('?');
if (lastExI > -1 || quesI > -1) {
const prefix = req.slice(0, lastExI + 1);
const postfix = quesI > -1 ? req.slice(quesI) : '';
const result = this.resolve(req.slice(lastExI + 1, quesI > -1 ? quesI : undefined), dirname);
Expand Down
Loading

0 comments on commit 203bc28

Please sign in to comment.