Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: deno vendor #13670

Merged
merged 38 commits into from
Feb 16, 2022
Merged
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
fadf48a
Committing for backup purposes.
dsherret Feb 3, 2022
8532d19
Add copyrights.
dsherret Feb 5, 2022
9699a5c
Merge branch 'main' into deno_vendor
dsherret Feb 5, 2022
e0db996
Starting to add text changes.
dsherret Feb 7, 2022
383a962
Somewhat working.
dsherret Feb 7, 2022
85ab7ed
Slight fix.
dsherret Feb 7, 2022
31c4996
Better import map and leave output files the same more
dsherret Feb 8, 2022
e5fd1f4
Merge remote-tracking branch 'upstream/main' into deno_vendor
dsherret Feb 8, 2022
591042d
Small fixes. Need to resolve the todos... might be a bit of work.
dsherret Feb 8, 2022
5e6022a
Lots of bugs, but slowly getting there. Will need deno_graph improvem…
dsherret Feb 10, 2022
52c7818
- Inspect the source code itself to get the specifier text
dsherret Feb 10, 2022
6a54b35
Basic testing infrastructure.
dsherret Feb 10, 2022
677f492
Fix more issues. It seems to be working decently now.
dsherret Feb 10, 2022
8317c87
Maintain casing of extensions when able
dsherret Feb 11, 2022
855dbe4
Merge branch 'main' into deno_vendor
dsherret Feb 11, 2022
cf746ae
More tests.
dsherret Feb 11, 2022
a981b53
- derived import map and functionality test
dsherret Feb 11, 2022
bd77405
Merge remote-tracking branch 'upstream/main' into deno_vendor
dsherret Feb 14, 2022
2974cd1
Support x-typescript-types headers
dsherret Feb 14, 2022
fe1413f
Fix and add tests for default export detection.
dsherret Feb 14, 2022
36bc916
Add flags test.
dsherret Feb 14, 2022
9762a40
More integration tests and fixes.
dsherret Feb 14, 2022
b77ddb5
Add sub command description
dsherret Feb 14, 2022
8c16515
Add test for an existing import map.
dsherret Feb 14, 2022
4286ffd
Tests for dynamic imports
dsherret Feb 14, 2022
8d10e4d
Add support for more flags.
dsherret Feb 14, 2022
72fb9b4
Updates based on self review.
dsherret Feb 15, 2022
c8a8aa5
Fix fs_util test
dsherret Feb 15, 2022
3d097a5
Maybe fix mac.
dsherret Feb 15, 2022
f04366c
Fix mac issue on a mac
dsherret Feb 15, 2022
4056996
Probably fix on all operating systems.......
dsherret Feb 15, 2022
d147900
Referencing a different origin should add entry to "imports"
dsherret Feb 16, 2022
8b10b6a
Add message on success.
dsherret Feb 16, 2022
b3b98cc
Merge branch 'main' into deno_vendor
dsherret Feb 16, 2022
326f399
Fix clippy issues on main.
dsherret Feb 16, 2022
dd8d561
Update for new deno_graph changes.
dsherret Feb 16, 2022
bfa2ee9
Add context to canonicalize.
dsherret Feb 16, 2022
90bce37
Use first non-remote entry point in output message.
dsherret Feb 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Starting to add text changes.
  • Loading branch information
dsherret committed Feb 7, 2022
commit e0db996cf96c1281406bdbe2c835ca2ce8a68553
108 changes: 108 additions & 0 deletions cli/tools/vendor/analyze.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
// Copyright 2018-2022 the Deno authors. All rights reserved. MIT license.

use std::path::Path;
use std::path::PathBuf;

use deno_ast::LineAndColumnIndex;
use deno_ast::ModuleSpecifier;
use deno_ast::SourceTextInfo;
use deno_ast::swc::common::BytePos;
use deno_ast::swc::common::Span;
use deno_graph::Module;
use deno_graph::ModuleGraph;
use deno_graph::Position;
use deno_graph::Range;
use deno_graph::Resolved;

use super::mappings::Mappings;
use super::text_changes::TextChange;

pub struct CollectSpecifierTextChangesParams<'a> {
pub mappings: &'a Mappings,
pub module: &'a Module,
pub graph: &'a ModuleGraph,
}

struct Context<'a> {
mappings: &'a Mappings,
module: &'a Module,
graph: &'a ModuleGraph,
text_info: &'a SourceTextInfo,
text_changes: Vec<TextChange>,
local_path: PathBuf,
}

impl<'a> Context<'a> {
pub fn byte_pos(&self, pos: &Position) -> BytePos {
// todo(https://github.com/denoland/deno_graph/issues/79): use byte indexes all the way down
self.text_info.byte_index(LineAndColumnIndex {
line_index: pos.line,
column_index: pos.character,
})
}

pub fn span(&self, range: &Range) -> Span {
let start = self.byte_pos(&range.start);
let end = self.byte_pos(&range.end);
Span::new(start, end, Default::default())
}
}

pub fn collect_specifier_text_changes(params: &CollectSpecifierTextChangesParams) -> Vec<TextChange> {
let text_info = match &params.module.maybe_parsed_source {
Some(source) => source.source(),
None => return Vec::new(),
};
let mut context = Context {
mappings: params.mappings,
module: params.module,
graph: params.graph,
text_info,
text_changes: Vec::new(),
local_path: params.mappings.local_path(&params.module.specifier).to_owned(),
};

// todo(dsherret): this is may not good enough because it only includes what deno_graph has resolved
// and may not include everything in the source file
for (specifier, dep) in &params.module.dependencies {
handle_maybe_resolved(&dep.maybe_code, &mut context);
handle_maybe_resolved(&dep.maybe_type, &mut context);
}

context.text_changes
}

fn handle_maybe_resolved(maybe_resolved: &Resolved, context: &mut Context<'_>) {
if let Resolved::Ok { specifier, range, .. } = maybe_resolved {
let span = context.span(range);
let local_path = context.mappings.local_path(specifier);
let new_specifier = get_relative_specifier(&context.local_path, &local_path);
context.text_changes.push(TextChange::from_span_and_text(
Span::new(span.lo + BytePos(1), span.hi - BytePos(1), Default::default()),
new_specifier,
));
}
}

fn get_relative_specifier(
from: &Path,
to: &Path,
) -> String {
let relative_path = get_relative_path(from, to);

if relative_path.starts_with("../") || relative_path.starts_with("./")
{
relative_path
} else {
format!("./{}", relative_path)
}
}

pub fn get_relative_path(
from: &Path,
to: &Path,
) -> String {
let from_path = ModuleSpecifier::from_file_path(from).unwrap();
let to_path = ModuleSpecifier::from_file_path(to).unwrap();
from_path.make_relative(&to_path).unwrap()
}
52 changes: 52 additions & 0 deletions cli/tools/vendor/mappings.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
// Copyright 2018-2022 the Deno authors. All rights reserved. MIT license.

use std::collections::HashMap;
use std::collections::HashSet;
use std::path::Path;
use std::path::PathBuf;

use deno_ast::ModuleSpecifier;
use deno_core::error::AnyError;
use deno_graph::Module;
use deno_graph::ModuleGraph;

use super::specifiers::dir_name_for_root;
use super::specifiers::get_unique_path;
use super::specifiers::make_url_relative;
use super::specifiers::partition_by_root_specifiers;
use super::specifiers::sanitize_filepath;

/// Constructs and holds the remote specifier to local path mappings.
pub struct Mappings(HashMap<ModuleSpecifier, PathBuf>);

impl Mappings {
pub fn from_remote_modules(
graph: &ModuleGraph,
remote_modules: &[&Module],
output_dir: &Path,
) -> Result<Self, AnyError> {
let partitioned_specifiers = partition_by_root_specifiers(remote_modules.iter().map(|m| &m.specifier));
let mut mapped_paths = HashSet::new();
let mut mappings = HashMap::new();

for (root, specifiers) in partitioned_specifiers.into_iter() {
let base_dir = get_unique_path(
output_dir.join(dir_name_for_root(&root, &specifiers)),
&mut mapped_paths,
);
for specifier in specifiers {
let media_type = graph.get(&specifier).unwrap().media_type;
let relative = base_dir
.join(sanitize_filepath(&make_url_relative(&root, &specifier)?))
.with_extension(&media_type.as_ts_extension()[1..]);
mappings.insert(specifier, get_unique_path(relative, &mut mapped_paths));
}
}

Ok(Self(mappings))
}

pub fn local_path(&self, specifier: &ModuleSpecifier) -> &PathBuf {
self.0.get(specifier).as_ref().unwrap_or_else(|| panic!("Could not find local path for {}", specifier))
}
}
57 changes: 34 additions & 23 deletions cli/tools/vendor/mod.rs
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
// Copyright 2018-2022 the Deno authors. All rights reserved. MIT license.

use std::collections::HashMap;
use std::collections::HashSet;
use std::path::PathBuf;

use deno_core::anyhow::bail;
use deno_core::error::AnyError;
use deno_core::resolve_url_or_path;
use deno_graph::ModuleKind;
use deno_runtime::permissions::Permissions;

use crate::flags::VendorFlags;
@@ -15,13 +14,15 @@ use crate::proc_state::ProcState;
use crate::resolver::ImportMapResolver;
use crate::resolver::JsxResolver;

use self::specifiers::dir_name_for_root;
use self::specifiers::get_unique_path;
use self::specifiers::make_url_relative;
use self::specifiers::partition_by_root_specifiers;
use self::specifiers::sanitize_filepath;
use self::analyze::CollectSpecifierTextChangesParams;
use self::analyze::collect_specifier_text_changes;
use self::mappings::Mappings;
use self::text_changes::apply_text_changes;

mod analyze;
mod specifiers;
mod text_changes;
mod mappings;

pub async fn vendor(ps: ProcState, flags: VendorFlags) -> Result<(), AnyError> {
let output_dir = resolve_and_validate_output_dir(&flags)?;
@@ -31,23 +32,33 @@ pub async fn vendor(ps: ProcState, flags: VendorFlags) -> Result<(), AnyError> {
.into_iter()
.filter(|m| m.specifier.scheme().starts_with("http"))
.collect::<Vec<_>>();
let partitioned_specifiers =
partition_by_root_specifiers(remote_modules.iter().map(|m| &m.specifier));
let mut mapped_paths = HashSet::new();
let mut mappings = HashMap::new();
let mappings = Mappings::from_remote_modules(&graph, &remote_modules, &output_dir)?;

for (root, specifiers) in partitioned_specifiers.into_iter() {
let base_dir = get_unique_path(
output_dir.join(dir_name_for_root(&root, &specifiers)),
&mut mapped_paths,
);
for specifier in specifiers {
let media_type = graph.get(&specifier).unwrap().media_type;
let relative = base_dir
.join(sanitize_filepath(&make_url_relative(&root, &specifier)?))
.with_extension(&media_type.as_ts_extension()[1..]);
mappings.insert(specifier, get_unique_path(relative, &mut mapped_paths));
}
// collect text changes
for module in &remote_modules {
let source = match &module.maybe_source {
Some(source) => source,
None => continue,
};
let local_path = mappings.local_path(&module.specifier);
let file_text = match module.kind {
ModuleKind::Esm => {
let text_changes = collect_specifier_text_changes(&CollectSpecifierTextChangesParams {
graph: &graph,
mappings: &mappings,
module,
});
apply_text_changes(source, text_changes)
},
ModuleKind::Asserted => {
source.to_string()
},
_ => {
log::warn!("Unsupported module kind {:?} for {}", module.kind, module.specifier);
continue;
}
};
std::fs::write(local_path, file_text)?;
}

Ok(())
71 changes: 71 additions & 0 deletions cli/tools/vendor/text_changes.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
// todo: before merge, delete this file as it's now in deno_ast

use std::ops::Range;

use deno_ast::swc::common::BytePos;
use deno_ast::swc::common::Span;

#[derive(Clone, Debug)]
pub struct TextChange {
/// Range start to end byte index.
pub range: Range<usize>,
/// New text to insert or replace at the provided range.
pub new_text: String,
}

impl TextChange {
pub fn new(start: usize, end: usize, new_text: String) -> Self {
Self {
range: start..end,
new_text,
}
}

pub fn from_span_and_text(span: Span, new_text: String) -> Self {
TextChange::new(span.lo.0 as usize, span.hi.0 as usize, new_text)
}

/// Gets an swc span for the provided text change.
pub fn as_span(&self) -> Span {
Span::new(
BytePos(self.range.start as u32),
BytePos(self.range.end as u32),
Default::default(),
)
}
}

/// Applies the text changes to the given source text.
pub fn apply_text_changes(
source: &str,
mut changes: Vec<TextChange>,
) -> String {
changes.sort_by(|a, b| a.range.start.cmp(&b.range.start));

let mut last_index = 0;
let mut final_text = String::new();

for change in changes {
if change.range.start > change.range.end {
panic!(
"Text change had start index {} greater than end index {}.",
change.range.start, change.range.end
)
}
if change.range.start < last_index {
panic!("Text changes were overlapping. Past index was {}, but new change had index {}.", last_index, change.range.start);
} else if change.range.start > last_index && last_index < source.len() {
final_text.push_str(
&source[last_index..std::cmp::min(source.len(), change.range.start)],
);
}
final_text.push_str(&change.new_text);
last_index = change.range.end;
}

if last_index < source.len() {
final_text.push_str(&source[last_index..]);
}

final_text
}