summaryrefslogtreecommitdiff
path: root/cli/cache/mod.rs
AgeCommit message (Collapse)Author
2024-10-01BREAKING: rename "deps" remote cache folder to "remote" (#25969)David Sherret
Closes https://github.com/denoland/deno/issues/25967 Closes #25968
2024-09-28refactor: move NpmCacheDir to deno_cache_dir (#25916)David Sherret
Part of the ongoing work to move more of Deno's resolution out of the CLI crate (for use in Wasm and other things) Includes: * https://github.com/denoland/deno_cache_dir/pull/60
2024-09-26feat: add `--allow-import` flag (#25469)Bartek Iwańczuk
This replaces `--allow-net` for import permissions and makes the security sandbox stricter by also checking permissions for statically analyzable imports. By default, this has a value of `--allow-import=deno.land:443,jsr.io:443,esm.sh:443,raw.githubusercontent.com:443,gist.githubusercontent.com:443`, but that can be overridden by providing a different set of hosts. Additionally, when no value is provided, import permissions are inferred from the CLI arguments so the following works because `fresh.deno.dev:443` will be added to the list of allowed imports: ```ts deno run -A -r https://fresh.deno.dev ``` --------- Co-authored-by: David Sherret <dsherret@gmail.com>
2024-09-18feat: default to TS for file extension and support ext flag in more ↵Leo Kettmeir
scenarios (#25472) Closes #11220 Currently does lint, fmt, and repl
2024-09-16refactor(permissions): split up Descriptor into Allow, Deny, and Query (#25508)David Sherret
This makes the permission system more versatile.
2024-09-05BREAKING: remove "emit" and "map" from deno info output (#25468)David Sherret
The map field has been empty for years now and we don't want the emit file to be exposed so it allows us to iterate on making the cache faster. Additionally, it's racy/unreliable to rely on this information. Instead, people should emit the TS files themselves using tools like deno_emit, typescript, esbuild, etc. Closes https://github.com/denoland/deno/issues/17703
2024-08-28fix: delete old Deno 1.x headers file when loading cache (#25283)David Sherret
2024-08-26perf(cache): single cache file for remote modules (#24983)David Sherret
This changes the global cache to store the cache file for remote modules in one file instead of two.
2024-08-08perf: skip saving to emit cache after first failure (#24896)David Sherret
2024-06-27fix(ext/node): discover .npmrc in user's homedir (#24021)Bartek Iwańczuk
This commit adds discovery of `.npmrc` files in user's homedir. This is not a perfect fix as it doesn't merge multiple `.npmrc` files together as per https://github.com/denoland/deno/issues/23954 but allows to fallback if no `.npmrc` file is discovered in the project root.
2024-06-07fix: make writing to the deps cache more reliable (#24135)David Sherret
I was able to reproduce this locally. ``` [error] Failed to execute snippet: import { validate } from "@std/uuid"; import { assert, assertFalse } from "@std/assert"; assert(validate("6ec0bd7f-11c0-43da-975e-2a8ad9ebae0b")); assertFalse(validate("not a UUID")); Download https://jsr.io/@std/uuid/meta.json Download https://jsr.io/@std/uuid/1.0.0-rc.1_meta.json Download https://jsr.io/@std/uuid/1.0.0-rc.1/mod.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/common.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/constants.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v1.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v3.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v4.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v5.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/_common.ts error: Access is denied. (os error 5) (for 'V:\.cache\deno\deps\https\jsr.io\2ae5bb614c7526d0876be0b76da1372fd51304ae27d6202ee94df720b3523d08') at file:///V:/deno_std/uuid/common.ts:43 [error] Failed to execute snippet: import { v5, NAMESPACE_DNS, NIL_UUID } from "@std/uuid"; import { assert, assertFalse } from "@std/assert"; const data = new TextEncoder().encode("deno.land"); const uuid = await v5.generate(NAMESPACE_DNS, data); assert(v5.validate(uuid)); assertFalse(v5.validate(NIL_UUID)); Download https://jsr.io/@std/uuid/meta.json Download https://jsr.io/@std/uuid/1.0.0-rc.1_meta.json Download https://jsr.io/@std/uuid/1.0.0-rc.1/mod.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/common.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/constants.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v1.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v3.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v4.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v5.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/_common.ts error: Access is denied. (os error 5) (for 'V:\.cache\deno\deps\https\jsr.io\63dd818c5fc1ac39c04df9b42bd9dd4bbc07f7d1b174e405d003731125778da1') at https://jsr.io/@std/uuid/1.0.0-rc.1/mod.ts:30:15 at file:///V:/deno_std/uuid/mod.ts:4 [error] Failed to execute snippet: import { isNil } from "@std/uuid"; import { assert, assertFalse } from "@std/assert"; assert(isNil("00000000-0000-0000-0000-000000000000")); assertFalse(isNil(crypto.randomUUID())); Download https://jsr.io/@std/uuid/meta.json Download https://jsr.io/@std/uuid/1.0.0-rc.1_meta.json Download https://jsr.io/@std/uuid/1.0.0-rc.1/mod.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/common.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/constants.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v1.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v3.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v4.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v5.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/_common.ts error: Access is denied. (os error 5) (for 'V:\.cache\deno\deps\https\jsr.io\fd3a12fc091d16ee29f10fa7a05eeeb8bd6c3cc014642e72478c757f00e7261e') at https://jsr.io/@std/uuid/1.0.0-rc.1/mod.ts:34:40 at file:///V:/deno_std/uuid/common.ts:23 [error] Failed to execute snippet: import { version } from "@std/uuid"; import { assertEquals } from "@std/assert/assert-equals"; assertEquals(version("d9428888-122b-11e1-b85c-61cd3cbb3210"), 1); assertEquals(version("6ec0bd7f-11c0-43da-975e-2a8ad9ebae0b"), 4); Download https://jsr.io/@std/uuid/meta.json Download https://jsr.io/@std/uuid/1.0.0-rc.1_meta.json Download https://jsr.io/@std/uuid/1.0.0-rc.1/mod.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/common.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/constants.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v1.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v3.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v4.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/v5.ts Download https://jsr.io/@std/uuid/1.0.0-rc.1/_common.ts error: Access is denied. (os error 5) (for 'V:\.cache\deno\deps\https\jsr.io\2ae5bb614c7526d0876be0b76da1372fd51304ae27d6202ee94df720b3523d08') at file:///V:/deno_std/uuid/common.ts:66 4 errors found ``` It occurs when many Deno processes are writing to the deps cache at the same time. Fix is to use `atomic_write_with_retries` which is much more reliable (and the function that helped make the ecosystem tests more reliable too). After this change I no longer have this issue. Closes https://github.com/denoland/deno/issues/24073
2024-06-06refactor: remove `PermissionsContainer` in deno_runtime (#24119)David Sherret
Also removes permissions being passed in for node resolution. It was completely useless because we only checked it for reading package.json files, but Deno reading package.json files for resolution is perfectly fine. My guess is this is also a perf improvement because Deno is doing less work.
2024-06-03refactor: don't share `reqwest::HttpClient` across tokio runtimes (#24092)David Sherret
This also fixes several issues where we weren't properly creating http clients with the user's settings.
2024-05-29fix: bump cache sqlite dbs to v2 for WAL journal mode change (#24030)David Sherret
In https://github.com/denoland/deno/pull/23955 we changed the sqlite db journal mode to WAL. This causes issues when someone is running an old version of Deno using TRUNCATE and a new version because the two fight against each other.
2024-05-28feat(vendor): support modifying remote files in vendor folder without ↵David Sherret
checksum errors (#23979) Includes: * https://github.com/denoland/deno_graph/pull/486 * https://github.com/denoland/deno_graph/pull/488 * https://github.com/denoland/deno_lockfile/pull/25 * https://github.com/denoland/deno_lockfile/pull/22 * https://github.com/denoland/deno_graph/pull/483 * https://github.com/denoland/deno_graph/pull/470
2024-04-29chore: add tests for moduleGraph2 (#23603)David Sherret
Also did some renames from underscores to hyphens
2024-04-19refactor: move redirect handling into deno_graph (#23444)David Sherret
2024-04-19perf(lsp): cleanup document dependencies (#23426)Nayeem Rahman
2024-04-17perf: v8 code cache (#23081)Igor Zinkovsky
This PR enables V8 code cache for ES modules and for `require` scripts through `op_eval_context`. Code cache artifacts are transparently stored and fetched using sqlite db and are passed to V8. `--no-code-cache` can be used to disable. --------- Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
2024-03-06fix(node): improve cjs tracking (#22673)David Sherret
We were missing saying that a file is CJS when some Deno code imported from the node_modules directory at runtime.
2024-02-20perf(jsr): fast check cache and lazy fast check graph (#22485)David Sherret
2024-02-15feat(unstable): single checksum per JSR package in the lockfile (#22421)David Sherret
This changes the lockfile to not store JSR specifiers in the "remote" section. Instead a single JSR integrity is stored per package in the lockfile, which is a hash of the version's `x.x.x_meta.json` file, which contains hashes for every file in the package. The hashes in this file are then compared against when loading. Additionally, when using `{ "vendor": true }` in a deno.json, the files can be modified without causing lockfile errors—the checksum is only checked when copying into the vendor folder and not afterwards (eventually we should add this behaviour for non-jsr specifiers as well). As part of this change, the `vendor` folder creation is not always automatic in the LSP and running an explicit cache command is necessary. The code required to track checksums in the LSP would have been too complex for this PR, so that all goes through deno_graph now. The vendoring is still automatic when running from the CLI.
2024-02-14chore: rename DENO_REGISTRY_URL to JSR_URL (#22414)Nayeem Rahman
2024-02-06fix(publish): lazily parse sources (#22301)David Sherret
Closes #22290
2024-02-01refactor: load bytes in deno_graph (#22212)David Sherret
Upgrades deno_graph to 0.64 where deno_graph is now responsible for turning bytes into a string. This is in preparation for Wasm modules.
2024-01-13fix(jsr): ensure proper storage of ahead of time module infos (#21918)David Sherret
This was incorrectly being stored so the AOT cache for JSR specifiers wasn't being used. I added a wrapper type to help prevent the API being used incorrectly.
2024-01-01chore: update to Rust 1.75 (#21731)林炳权
2024-01-01chore: update copyright to 2024 (#21753)David Sherret
2023-12-12refactor: better handling for registry urls (#21545)Bartek Iwańczuk
2023-11-23feat(unstable): tar up directory with deno.json (#21228)Bartek Iwańczuk
Co-authored-by: David Sherret <dsherret@gmail.com> Co-authored-by: Luca Casonato <lucacasonato@yahoo.com> Co-authored-by: Luca Casonato <hello@lcas.dev>
2023-10-25refactor: break out ModuleInfoCache from ParsedSourceCache (#20977)David Sherret
As title. This will help use the two independently from the other, which will help in an upcoming deno doc PR where I need to parse the source files with scope analysis.
2023-10-25feat(unstable): ability to `npm install` then `deno run main.ts` (#20967)David Sherret
This PR adds a new unstable "bring your own node_modules" (BYONM) functionality currently behind a `--unstable-byonm` flag (`"unstable": ["byonm"]` in a deno.json). This enables users to run a separate install command (ex. `npm install`, `pnpm install`) then run `deno run main.ts` and Deno will respect the layout of the node_modules directory as setup by the separate install command. It also works with npm/yarn/pnpm workspaces. For this PR, the behaviour is opted into by specifying `--unstable-byonm`/`"unstable": ["byonm"]`, but in the future we may make this the default behaviour as outlined in https://github.com/denoland/deno/issues/18967#issuecomment-1761248941 This is an extremely rough initial implementation. Errors are terrible in this and the LSP requires frequent restarts. Improvements will be done in follow up PRs.
2023-09-18feat(unstable): package manager (#20517)David Sherret
Adds an experimental unstable built-in package manager to Deno, but it is currently not usable because the registry infrastructure hasn't been setup and it points to a non-existent url by default. The default registry url can be configured via the `DENO_REGISTRY_URL` environment variable.
2023-09-07feat: support import attributes (#20342)David Sherret
2023-08-08refactor: use deno_cache_dir crate (#20092)David Sherret
Uses https://github.com/denoland/deno_cache/pull/26
2023-08-02feat(unstable/lsp): support navigating to deno_modules folder (#20030)David Sherret
Closes #20015 Closes https://github.com/denoland/vscode_deno/issues/850 (only for deno_modules, but I don't think this will be possible for the global cache)
2023-08-02feat(unstable): optional `deno_modules` directory (#19977)David Sherret
Closes #15633
2023-07-08refactor: abstract away file system to be buried inside HttpCache (#19760)David Sherret
This improves the HttpCache to make it being stored on the file system an implementation detail.
2023-05-25fix(compile): handle when DENO_DIR is readonly (#19257)David Sherret
Closes #19253
2023-04-26feat(cli): don't check permissions for statically analyzable dynamic imports ↵Nayeem Rahman
(#18713) Closes #17697 Closes #17658
2023-03-27feat(core): initialize SQLite off-main-thread (#18401)Matt Mastracci
This gets SQLite off the flamegraph and reduces initialization time by somewhere between 0.2ms and 0.5ms. In addition, I took the opportunity to move all the cache management code to a single place and reduce duplication. While the PR has a net gain of lines, much of that is just being a bit more deliberate with how we're recovering from errors. The existing caches had various policies for dealing with cache corruption, so I've unified them and tried to isolate the decisions we make for recovery in a single place (see `open_connection` in `CacheDB`). The policy I chose was: 1. Retry twice to open on-disk caches 2. If that fails, try to delete the file and recreate it on-disk 3. If we fail to delete the file or re-create a new cache, use a fallback strategy that can be chosen per-cache: InMemory (temporary cache for the process run), BlackHole (ignore writes, return empty reads), or Error (fail on every operation). The caches all use the same general code now, and share the cache failure recovery policy. In addition, it cleans up a TODO in the `NodeAnalysisCache`.
2023-03-22feat(cli): --ext parameter for run, compile, and bundle (#17172)Cre3per
Adds `--ext` to `deno run`, closes #5088 Additionally - Adds `--ext` to `deno compile` and `deno bundle`
2023-03-15refactor: remove usages of `map_or` / `map_or_else` (#18212)David Sherret
These methods are confusing because the arguments are backwards. I feel like they should have never been added to `Option<T>` and that clippy should suggest rewriting to `map(...).unwrap_or(...)`/`map(...).unwrap_or_else(|| ...)` https://github.com/rust-lang/rfcs/issues/1025
2023-02-24fix(npm): lazily install package.json dependencies only when necessary (#17931)David Sherret
This lazily does an "npm install" when any package name matches what's found in the package.json or when running a script from package.json with deno task. Part of #17916 Closes #17928
2023-02-23feat(npm): support bare specifiers from package.json in more subcommands and ↵David Sherret
language server (#17891)
2023-02-22feat: support bare specifier resolution with package.json (#17864)Bartek Iwańczuk
This commit enables resolution of "bare specifiers" (eg. "import express from 'express';") if a "package.json" file is discovered. It's a step towards being able to run projects authored for Node.js without any changes. With this commit we are able to successfully run Vite projects without any changes to the user code. --------- Co-authored-by: David Sherret <dsherret@gmail.com>
2023-02-22refactor: use deno_graph for npm specifiers (#17858)David Sherret
This changes npm specifiers to be handled by deno_graph and resolved to an npm package name and version when the specifier is encountered. It also slightly changes how npm specifier resolution occurs—previously it would collect all the npm specifiers and resolve them all at once, but now it resolves them on the fly as they are encountered in the module graph. https://github.com/denoland/deno_graph/pull/232 --------- Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
2023-02-21fix(npm): improve peer dependency resolution (#17835)David Sherret
This PR fixes peer dependency resolution to only resolve peers based on the current graph traversal path. Previously, it would resolve a peers by looking at a graph node's ancestors, which is not correct because graph nodes are shared by different resolutions. It also stores more information about peer dependency resolution in the lockfile.
2023-02-15feat: wire up ext/node to the Node compatibility layer (#17785)Bartek Iwańczuk
This PR changes Node.js/npm compatibility layer to use polyfills for built-in Node.js embedded in the snapshot (that are coming from "ext/node" extension). As a result loading `std/node`, either from "https://deno.land/std@<latest>/" or from "DENO_NODE_COMPAT_URL" env variable were removed. All code that is imported via "npm:" specifiers now uses code embedded in the snapshot. Several fixes were applied to various modules in "ext/node" to make tests pass. --------- Co-authored-by: Yoshiya Hinosawa <stibium121@gmail.com> Co-authored-by: Divy Srivastava <dj.srivastava23@gmail.com>
2023-02-15refactor: use deno_graph's semver and npm structs (#17791)David Sherret