Fix using vite packages when they're ignored from compiling

Fix using vite packages when they're ignored from compiling #2954