Sketch Mode Revamp

[pierremtb]

Important

This is currently in planning, see discussion below.

Milestone for issue tracking: https://github.com/KittyCAD/modeling-app/milestone/5

---

Useful links: #7435 (comment)

Making a quick summary of architecture alignment (or what my impression of it is):

1. Solver runs as part of the wasm bundle (might not be rust, but definitely part of local execution)
2. Drawing is done locally by THREE (could change in future if we move to server side execution, but out of scope for current revamp)
3. Sketch will have a special sketch block (allows us to constrain what's possible, put special syntax within, do partial execution of that block for edit/drag animations)
4. There's some alignment around syntax too, points will be first class, and we broadly agreed with what NRC showed in last meeting (15/07/2025)
5. Code-mods will be handled by rust, and we'll have an API with TS land, because drag animation is the most demanding thing the API will be something like TS informs rust of what things are selected and where the user wants to drag them (plus modifiers, snaping on/off), rust returns all of the things to be drawn(or updated) with metadata (like dimensions, what segments are constrained, what the constraints are for adding icons etc)
6. This is a completely separate system to the current sketch mode, no backwards compat. (we can consider making migration seamless, but since we can run both sketch UX at the same time, we don't need to worry about that until we're at the point where we want to kill old sketch mode).
7. There's loose agreement on region support that doesn't require closed profiles

[franknoirot]

Agenda for June 10th meeting:

1. Question: do we agree that sketch is the primary system we need to focus on improving in the near term?
2. Question: do we agree that making sketch mode more approachable to developers will improve delivery speed?
   1. If yes, what smaller team will begin work on that tech debt?
3. Question: does the constraint solver discussed previously need to be begun?
   1. If yes, question: can the KCL team begin that work on their own?
4. Question: do we agree what sketch mode should feel like?
   1. @nicboone8 will demo a simple sketch workflow in a few apps. That might be a good proxy for the "traditional CAD" user experience that some customers have raised.

[franknoirot]

Thanks to everyone for coming to the meeting! The meeting has uploaded to our shared recorded meetings site under the title "Sketch mode post-1.0". Our answers to the questions in the agenda were:

1. Answer: Yes, while balancing with our other ZDS high-level priorities, as they all fall under our true top goal, which is a familiar point-and-click-only user experience.
2. Answer: Yes. The TypeScript side of sketch mode does and knows too much, and must be rewritten, in the nearer term to make it easier to implement fixes and features to the current sketch KCL paradigm, and to prepare for a constraint-enabled
3. Answer: Yes. We have no other way to implement backward references and overall dimensions, which are the primary blockers to a familiar sketch point-and-click user experience.
   1. @nrc raised that the boundary layer between Rust and TS and the information sharing between them has encumbered development, and could threaten development speed of a constraint system, and that moving that boundary entirely to the Rust side would speed up all future language work. The KCL team needs to discuss what level of effort refactors to move that boundary, how necessary it is to implement a constraint solver, how parallelize-able that work is with a constraint solver implementation.
4. Answer: It should feel like a traditional CAD sketch user flow first and foremost. Once we achieve that, we can do a lot to improve the legibility and power of those traditional user experiences. @nicboone8 explained the demo GIF within his write-up "Advanced Sketch Features" and why that user experience of "technical play" is important.

Note

The efforts to create a more familiar sketch mode experience need to be done in parallel with improvements to the true primary goal, which is a familiar and robust point-and-click experience. A subset of the KCL team's time and a subset of the Design Studio frontend team will drive hard at these efforts, pulling in help and advice from the rest of the team as needed, while the rest of the team continues to make user's experience of the app better today. An excellent example of critical efforts that fall under this goal and support these sketch mode efforts without having anything to do with this sketch mode rewrite is #6629. Efforts like this need to be pushed towards by everyone else now, and will be tracked in the Zoo Design Studio project and as milestones for public viewing.

[franknoirot]

This thread is resolved ✅

[franknoirot]

Ahead of our June 17th meeting, please post any "fundamental questions" as top-level comments and discuss in the threads.

[franknoirot]

Question: Is KCL input or output to the solver? @nrc, @Irev-Dev, and @nickmccleery have weighed in previously on Slack.

[Irev-Dev]

I think we were all working under the assumption of it as input, it kinda breaks code as the source of truth by treating it as output.

[franknoirot]

This thread is resolved ✅: KCL is the input to the solver

[franknoirot]

Question: Where does the solver live?

[franknoirot]

@nrc @Irev-Dev please give us a final word on why it must live client-side. I think we are resolved that this will be the case, but I'd like to have responses to @max-mrgrsk's points above before we mark this as settled.

[Irev-Dev]

solver will live in the executor (may not be rust but will be local). because rust will also be doing the code-mods well probably have an API where (using editing by draging as an example) the TS tells rust/executor what's selected and where they'r being draged to (plus modifies like snapping on/off). And rust can update the code and return all of the data of what to draw (the segments with enough detail to draw them, plus metadata like which are contained, what the constraints are for icon purposes, dimensions too).

snapping to bodies will have to be things that are setup "hand-wavey-before" execution because nothing in the sketch block will move these entities, we can assume they won't move during the partial execution, so I think we're good. I say "hand-wavey-before" because the syntax could maybe still be put into the sketch block just so it feels cohesive.

We're drawing using three.js so rotating the camera while sketching won't be a problem.

From what I can tell, 3d curves do not use solvers in any of the incumbents, so in that case it's a separate case, and I think it's out of scope for this re-vamp. My 2c anyway.

[Irev-Dev]

I'm changing my mind a little on the 3d curves stuff. I'll add more but probably not tonight.

[Irev-Dev]

compute region intersection using gpu

What is region intersection? how is this different from regular regions? is this 2d CSG.
Why is the GPU important, I'm not sure anything in 2D really needs GPU (aside from animations)

[nickmccleery]

from what I can tell, 3d curves do not use solvers in any of the incumbents, so in that case it's a separate case, and I think it's out of scope for this re-vamp. My 2c anyway.

I think that's fair—the reason I mention 3D stuff is less for geometry creation (e.g. curves) and more for kinematics and mates. Having a library of functions for computing point-point, point-line, point-surface etc. etc. distances and a mechanism of stuffing those into a Newton's method solve would be useful there, but it's not like we're proposing to do any of the heavy lifting in Typescript here so re-usability isn't out of the question anyway.

[franknoirot]

Question: What in the system should have backwards compatibility, and what should break?

[Irev-Dev]

My vote is

• Ofc all old kcl should continue to work.
• We run both sketch modes in parallel for a time (can edit old sketches with old sketch mode but all new sketches use new sketch mode and syntax)
• Bonus we're able to convert old sketch to new sketch syntax (I think we can probably infer what the constraints need to be)
• We eventually kill old sketch mode UI

[franknoirot]

I think I support this. The only probably minor issue could be that we have to heuristically determine if a sketch is "old" or "new", and users could conceivably put both into their KCL. but if we get the new sketch mode to handle that we'd have no issue, either by converting or otherwise

[franknoirot]

This thread is resolved ✅: we will follow this approach recommended by @Irev-Dev

[max-mrgrsk]

Is snapping to existing bodies, sketching in 3D, and including existing bodies into region intersections with the sketch-on-face in the current scope? If not, it would be nice to make sure the current work is extendable to those things. We’ll need them eventually.

For example, here’s a simple rectangle sketched on the body face. But it interacts with the whole face, and you’re free to choose any combination for further extrusion.

[Irev-Dev]

So I've talked to some of the engine team about "projecting" edges and segments elsewhere in the scene onto the users's current sketch plane, the end point is actually been added we just never added it.

But I think that would achieve what we want here, but getting the integration as smooth as this might not be as straight forward.
I imagined a user would have to select what they want projected onto their current sketch plane, but are these kind of things normally automatically there?
Or is it because you started a sketch on that face?

[franknoirot]

We have an engine endpoint for getting what profiles enclosing the current point too. I had Serena build one for an initial effort to support regions, but never integrated it because I couldn't see the end product in KCL and point-and-click clearly. Just something else in our toolbox

[max-mrgrsk]

TL;DR: We need regions (intersection/union/difference).

Copied from Slack:

Let’s say a user prompts “split washer” — t2c is trained on conventional CAD washers and sketching, which is 2 circles and the gap done with 2 parallel lines. (CAD recognizes all regions and lets the user select what to extrude). Boom.

The only way to make a washer right now in Design Studio is to make the simple one first, then create a 3D wedge to cut the gap.

Which is not what our LLM was trained to do.

We need regions, and most importantly, we need to make them the same way current CAD does. They have to work with both closed and open profiles.

[Irev-Dev]

Will depend a little on some other decisions, but here's an idea, where I think the only assumption needed is first class point support, which I think most of us are on board with so

When the users clicks a region, we take the closes segment and closes intersection point, and that allows us to generate the following code

Because if we have a segment and an intersection, crawl along the segments to form a closed profile, but notice the direction here, is basically which way we turn at each intersection we find, I've used LEFT here but CCW and CW would also work.

Notice if the user move their cursur slightly to the right, we use the same interception point and segment, it's just the direction changes to select a different region

Here's one more example

If we want this, I think this push in favor of local solver that has all the 2d context, because getting real time feed back for "given this coordinate, is it in valid region, and what are the closest segment, point, and the direction" sounds difficult over an API for a completely different system, and over the network.

[nrc]

When we talk about region support, is this any different to 2d csg? With this example, one confounding factor is that we're using two parallel lines rather than a rectangle. I would imagine if this was 2 circles and a rectangle, it would be very simple to express the desired region as bigCirc - littleCirc - rect. Is there a strong reason for supporting the parallel lines vs a rectangle? And are there cases where we need to think about selecting a region rather than in terms of union/subtraction of profiles/shapes?

[Irev-Dev]

pretty confident that 2d csg is not going to be sufficient, purely because ME's are not going to want to have to close their profiles just to make regions work.

[pierremtb]

I agree with @Irev-Dev there

[franknoirot]

The most recent syntax proposal from @nrc has a promising syntax for this: please add a snippet when you share that document @nrc. A region KCL function that simple takes an array of path segments—which may or may not be joined by constraints yet—are passed in and fed into a region detection system. Some points of disagreement are still being worked out:

1. The proposal showed regions being returned from the sketch block. I think this should not be the case, and that the region function should simply be used at the time of modeling operation later. This will require both every segment be reference-able after the sketch block, which is another discussion we had yesterday.
2. There weren't examples in the proposal showing how inner profiles would be cut out from regions. Perhaps they would just be handled by the region detection system internally?

[nicboone8]

One thing I don't believe we've discussed at all yet is sketching in 3D. Not that it needs to be part of the constraint solver necessarily- most cad systems basically treat it as a standalone tool. Really only used with sweep to model wire routings and flexible hoses

[franknoirot]

Here is a relevant short video of sketch in 3D for anyone else who hasn't seen it in a while.

1. Correct me if I'm wrong, but the key difference here is that every segment can be associated with a different plane
2. It looks like the cursor is doing the work of showing which plane your next segment will be on

[Irev-Dev]

every segment can be associated with a different plane

I'm no expert here, but I think maybe that's just this example, all of the line fall on one of major axis, each segment is 90 degrees from each other. not sure it's the best example because you could have a line going from (0,0,0) to (3,5,4). Obviously here there is a custom plane that you associate with it, but what about true 3d curves? what if we want to add handles to a helix?

[franknoirot]

but what about true 3d curves? what if we want to add handles to a helix?

Ah good point it's not so simple, my point 1 is incorrect.

[franknoirot]

How might, in the future, this plug into @nrc's syntaxt proposal?

I'll hazard some fanfic:

1. The sketch() {} block might be able to take a different argument that makes it clear that more than one plane will be involve in this sketch
2. Then adding a plane keyword to all the associated sketch geometry functions would make them able to be used directly with a provided plane
3. Constraints and dimensions could be restricted or otherwise be made to behave differently when two pieces of geometry do not share the same plane argument

[franknoirot]

My goal is just to confirm that we satisfy this enough for ourselves to call this thread resolved:

it would be good to give it enough thought to leave the door open for to add later. Because it would be very easy to make this a giant PITA to add in later.

[Irev-Dev]

Do we keep a distinct sketch mode

[Irev-Dev]

I think yes, NRC did mention maybe not, but to me it makes sense to keep things separate, just because adding constraints is a very distinct way of thinking. but maybe I'm too stuck in how "CAD is done"

[pierremtb]

Would take a lot to get me out of the dedicated sketch mode camp I think, at least for the point-and-click UX. If there's a world where the implementation doesn't require a dedicated sketch mode and that's allowing us to move away from that pattern later even in point-and-click then I wouldn't mind at all!

[franknoirot]

Yeah I agree with @pierremtb, in the spirit of meeting users where they are I think we should stick with a separate mode for now

[franknoirot]

tldr; yes we keep a distinct sketch mode

@jtran and I met last Wednesday, and discussed this issue just a bit more. We both came to the conclusion that we will stick with a separate mode for now. Most CAD users are very comfortable with the Sketch button entering a mode where new buttons become available, and we don't want to waste time or focus while we try to make sketching feel familiar to them.

We did envision a future case where we employ 3 changes to make a less distinct "mode" for sketching. We realized that the user state that makes this a "mode" is the persistence of a "current sketch plane" argument that is held around for the user to operate on repeatedly without specifying it, and these build on that. We recommend making only the 1st of these 3 changes.

1. Make the "sketch" argument explicit in all sketch mode tools. Refactor current sketch mode to use this argument, and skip it if it's present.
   1. We recommend making this adjustment now, because it allows users to be able to select a sketch tool from outside of sketch, and then select a sketch plane to get started. Today, users have to know they need to be sketching to enable a sketch tool.
2. Refactor the toolbar layout to show both sketch and modeling tools all the time, once 1 is complete.
   1. Solving potential layout density issues associated with this was one of the reasons we recommend not pursuing this change. In the future we would like to make the toolbar fully customizable. That would help give users tools to surface only what they want to see in the toolbar, hopefully aiding in the density problem.
3. Make the Sketch button persist a UI element that shows the currently-active sketch, which is dismissible.
   1. We do not recommend pursuing this change currently, as it relies on the 2nd change and is too removed from making a familiar sketching experience for traditional CAD users.

[franknoirot]

This thread is resolved ✅: we will keep a distinct sketch mode

[Irev-Dev]

[syntax] Does sketch get it's own block?
(something along the lines of mySketch = sketch {...})

[Irev-Dev]

Kurt and NRC say yes.

[franknoirot]

I have an impulse away from this but I think it's only for vague aesthetic reasons. Your points around parsing and "locking down" sketch mode behaviors using a custom block sound necessary, and there is prior art including FeatureScript.

A question I have is how would we pave a way for sketch in 3D with this? Maybe if within the sketch block a sketch plane can be set and used until it is changed, then sketch in 3D is simply redeclaring the sketch surface as the sketch progresses?

[franknoirot]

A question I have is how would we #7435 (comment) with this? Maybe if within the sketch block a sketch plane can be set and used until it is changed, then sketch in 3D is simply redeclaring the sketch surface as the sketch progresses?

This question should be resolved in the thread about 3D sketching, not here.

[franknoirot]

This thread is resolved ✅: sketches will receive their own code block. @nrc's latest proposal shows a syntax like mySketch = sketch(myPlane) { ... }

[Irev-Dev]

Follows from #7435 (comment)
How restrictive is a sketch (or sketch block)

[Irev-Dev]

Kurt thinks locking down. There's a special sketch block, within it there's no branches, no loops. No need to complicate things, mixing constraints with branches is like us being schizophrenic. better support partial execution
NRC has a block but it's not locked down.

[franknoirot]

@Irev-Dev I believe this thread is resolved, right? We agreed in our last (14-07) meeting that especially for a first pass, we will be quite restrictive within the sketch block, as restrictive as we need to be to limit complexity, very in-line with what you've put here.

[Irev-Dev]

Do we need to support partial execution of the sketch?

[Irev-Dev]

Kurt thinks it's critical for low latency editing/animations. But animating the whole sketch is maybe optional (partial execution also implies local solver)

[Irev-Dev]

[syntax] what is the syntax for relationsips/constraints?

[Irev-Dev]

Completely up in the air really, this is probably where we diverged the most, but probably the exact type of solver we use will help inform this, i.e. inkling style means any continuous expression should work.

how well we support custom/user-defined constraints will probably fall out of this

[franknoirot]

This issue still needs discussing, tagging @jtran @nrc @adamchalmers

[Irev-Dev]

What draws the sketch?

[Irev-Dev]

I have since thought of a way of maybe doing stuff on the engine without the latency penalties, but my take away is probably that whatever draws the sketch, should also have the solver.

This is a lot to solve to figure out "animations" because that's were perf gets us is trying to update the sketch as the user drags things.

The alternative is basically if the engine is aware of all constraints, it also knows how to animate the sketch if it's got the solver there too. BUT this only works with very declarative constraints, it starts to break down with constraints as general expressions. i.e. IIL1|| = 0.5 * ||L2||, has to be the data only approach like { constraint: "equalLength", segments: [...]}
Than when the user stops dragging, that's when we fire a code mode to update some of the "temp data" in the sketch

[Irev-Dev]

I'm probably getting ahead of myself with some of this stuff.

[max-mrgrsk]

conventional cad is unhappy with stuff like sheet metal patterns. Engineers normally use textures in this case to speed things up. Since we are already doing so much on the gpu side - those things might get possible. But only if sketches are on the engine side.

[franknoirot]

@Irev-Dev this thread is a little mixed with this one above about engine- vs client-side. Could you, @nrc, or another proponent of the client-side approach give a response to @max-mrgrsk's note about patterning from this thread in your resolution up there? I have gotten the sense that we have settled on the client-side approach, but I have not heard final words addressing these concerns, either that we have a path forward or that we are not prioritizing them.

[Irev-Dev]

I guess I got a little lost, like I don't think pattern should be done in sketch mode, it's not really something where a solver is needed no?
Because it's just like repeat this shape, I mean ofc you can pattern a shape made in sketch mode, but it's still getting patterned after. which means it will be engine side.

[pierremtb]

Cross posting @adamchalmers's writeup regarding shapes at #7508

This is relevant to both this and #7461

[pierremtb]

Depending on the output of this discussion, #7508 could become its own discussion, but keeping it here for now

[franknoirot]

This thread is resolved ✅: @adamchalmers reported during our meeting last night that he and the rest of the KCL team do not see those shape helpers as necessary to the functioning of the new sketch mode.

We discussed briefly how, through the code mod API, they could be made transparent to the frontend in the future (so, for example, a rectangle() could be written in KCL without changing the data and interactivity seen by the frontend sketch mode, which would continue to operate on a set of points, lines, and constraints). However, this will be a later iteration. @adamchalmers, @jtran or @nrc please correct me if I'm wrong.

[Irev-Dev]

Since we want to support animating, (that is solving in real time as users drag handles)
What is the minimum performance we think is acceptable?

[nrc]

Constraint solution can rapidly (and surprisingly) get into very long exec times because it's inherently an exponential time problem. Algorithms restrict the search space, but then it's possible to fall off the happy path pretty easily and you go from 30fps to 30spf :-) We probably need to have fallback in the UI for handling these high latencies in the solver.

[franknoirot]

@nicboone8 @nickmccleery @max-mrgrsk @pierremtb go ahead and weigh in with the worst performance you've remembered working with in sketch mode. My hunch is that no commercial sketcher lets the user ever feel or experience the 30spf solutions @nrc, I wonder if those would just be marked as "unable to solve" cases. A concrete example you could show in an existing CAD tool would be helpful here.

[nicboone8]

Generally all these sketch modes feel pretty snappy to use in my experience. Notable exceptions might be editing a sketch with a huge number of 2d patterns, or working within an assembly with hundreds of parts loaded in. Projecting a complex curve sometimes takes a few seconds. Everything else feels almost instant.

When you hit a 30spf scenario, you ctrl+s because it's a reliable indicator that the entire program is about to crash. It's common to see this, but not limited to sketch mode

[nickmccleery]

Likewise, don't think I've ever encountered a noticeable performance issue in a 2D sketcher. Big models tip over CAD systems, but my experience of the CATIA and Onshape sketchers has been painless.

Off the top of my head, I don't think I can dream up a suitably cursed but still realistic arrangement that would be so difficult for the system to solve—although I guess the typical point and click workflow means you add constraints progressively, so generally your initial values at each solve step will be fairly good and you're probably only playing tunes with one or two elements per solve call.

I can try and come up with a nasty sketch with the intention of pushing the thing into a corner, but out of curiosity, I went and hackily profiled some old code I wrote for simulating suspension system kinematics.

Roughly speaking, the more complex setup (3D, 18DOF, SLSQP solve with simultaneous vertical wheel displacement and steering rack displacement) was generally double digit milliseconds per solution (M4 MBP). The less complex solution (3D, 18DOF, Newton's method solve with just vertical displacement) was sub-millisecond per solution—and a JAX JIT version got to lots of the same solutions an order of magnitude faster than that.

Gut says any really nasty/slow solves will be relatively rare, particularly if we're using a third party solution that's intended for this use-case—but I'll see if I can prove that

[nickmccleery]

Should also say my solutions there are quite naive; no sparse solver, no optimisations based on what we know about the system etc. Basically just asking the computer to go brr.

[pierremtb]

Keeping track of things: the canvas Sketch mode common ground (and not) that was worked on after the meeting on Tuesday

[nickmccleery]

Posting here to get all the information (hopefully) referenced in one spot, even if just for posterity.

I spent a bunch of time ironing out a very high-level requirements-esque document for a 2D sketcher with constraints, really without any consideration of current functionality. This covers user requirement with respect to element types, constraint categorisation, which elements can have which constraints applied etc. It then goes into a little bit of the maths of how the constraint solve will work, just so that everyone's got some visibility over that—even if we go with a third-party lib.

That work sits alongside having ~exhaustively swept through all of the sketcher constraint options in some other systems for each of the constraints I think we need, putting together a spreadsheet which separately identifies every constraint possible between each primitive, along with some commentary.

These documents really go together; they reference a lot of the same concepts... and hopefully the individual constraint options can be prioritised and worked into a fairly actionable set of requirements.

• Doc: https://gist.github.com/nickmccleery/5de47e465474a3420e69727d4806d0ef
• Spready: https://docs.google.com/spreadsheets/d/1YHkqXJOuhxwJFZztyvKRlPVCdAboQ9oqNGhnD6Ukn5M/edit?usp=sharing

[davreev]

Excellent write-up! One thing to highlight on the solver side - each Newton step involves solving a linear system so this will likely eat up the biggest chunk of the compute budget. Not sure where the solve is intended to happen but it'll be important to have access to an efficient sparse linear solver for this reason (most of these systems will be very sparse).

[franknoirot]

Thank you so much for this @nickmccleery. Last night we agreed to every bit of this as a requirements checklist. This will be the doc we use going forward.

The one point of feedback from the implementing folks is that there were "a lot of 1's" in the spreadsheet, to quote @nrc, so we will have to have them negotiate the priority when they start writing code. @nickmccleery my understanding is that you put a lot of 1s because the underlying geometry for a lot of these types of constraints are shared, is that right?

[nickmccleery]

Some of it is because they're shared, but generally I took the view that anything that's a 1 in there is just a requirement to hit the 'viable' part of a functional MVP.

That said, happy to review what I put in there—my brain was fairly fried by the time I got to rank things, but I do think there's a high likelihood that functional sketcher 'table stakes' is relatively large in scope.

Also worth getting @nicboone8 and @max-mrgrsk to chip in with their rankings I think.