Reflect Store

[iDevelopThings]

Just a random idea that crossed my mind

Do you think it would be better/possible to store the types/regular getType call on the window/process object(global basically)?

From time to time, I get random issues with the package not being imported from nested getType calls. Not sure if this is just cause of jest/ts-jest also

But this would also remove the need for that import + provide a global store that would work from everywhere right? 🤔

[iDevelopThings]

Interesting... I actually hacked something together for this in another local branch, was just playing/testing so it's an absolute mess, but it works

Also figured a somewhat okay method for finding the entry ts/js file of the source

This object is added to any sources which are considered an "entry point", which is a lot, but doesn't depend on any importing... which can also be problematic

    process["tst-reflect-store"] = {
        store: {},
        __typeGetter: function (description, typeId) {
            return require("tst-reflect").getType(description, typeId);
        },
        __lazyTypeGetter: function (typeId) {
            return function lazyType() {
                return this.__typeGetter(typeId);
            };
        },
        __setDescription: function (identifier, data) {
            this.store[identifier] = data;
            this.__typeGetter(data, identifier);
        },
        __getDescription: function (identifier) {
            return this.store[identifier];
        }
    };
}

"process" holds the main store of types, rather than "typesMetaCache" in the reflect lib

Then for the rest of the code, they reference this "store" object above to set/get types, reflect also uses this store

Example:

It feels a little weird, but at the same time, seems to be working really great. If you're interested in it, I can push my branch to my fork^^

[Hookyns]

Also figured a somewhat okay method for finding the entry ts/js file of the source

It's interesting.. is that entry file detection reliable? It probably detect more files, that's quite Ok, but can it miss one?

From time to time, I get random issues with the package not being imported from nested getType calls.

Sorry, what does it mean? Ctors are not imported by that new require you wrote or runtime package is not imported or something else?

My plan was make the metadata lib file mode default, store everything in one file and let user import this one file at startup. (And provide lookup methods from #2)
Now you inspired me and we can use your detection method to make auto import. Or it should be possible to do transformation on runtime package itself and add that import into runtime, cuz I publish TS file.. it was required by older versions.. I made a lot of changes since that.. I don't know if it is still needed and I don't remember why it was needed. 😅

BTW, we can change config metadata: "infile" | "libfile" | "inprocess" if you want to merge it. This is still zero version, we are looking the best method.. so people should use it to find out pros and cons.

[iDevelopThings]

It's interesting.. is that entry file detection reliable? It probably detect more files, that's quite Ok, but can it miss one?

It doesn't seem to, its via program.getRootFileNames() if you specifically call ttsc src/index.ts for example, it will find only that file, which is what you want, but if not... it can be a range of files(maybe depending on the import resolution of which files will actually be included & used in the project)

Sorry, what does it mean? Ctors are not imported by that new require you wrote or runtime package is not imported or something else?

It seems like runtime package isn't imported correctly, either it doesn't get imported, or it imports as something like const tst_reflect_1 = require('...') for example, but then code tries to reference tst_reflect_2. No idea how this happens, i had a similar issue with my ctor code also

My plan was make the metadata lib file mode default, store everything in one file and let user import this one file at startup. (And provide lookup methods from #2) Now you inspired me and we can use your detection method to make auto import. Or it should be possible to do transformation on runtime package itself and add that import into runtime, cuz I publish TS file.. it was required by older versions.. I made a lot of changes since that.. I don't know if it is still needed and I don't remember why it was needed. 😅

Yeah I'm also experimenting a little with my solution, it has some problems 😅 I feel like I could make a nice solution for the metadata lib file also, I think that's what i'll have a look at now.
Problem with my solution now, is that although it is a "global" store and works in that way, until the files with their meta defined are imported and resolved, the meta isnt added to the store, so it's basically the same as before 🤦‍♂️

BTW, we can change config metadata: "infile" | "libfile" | "inprocess" if you want to merge it. This is still zero version, we are looking the best method.. so people should use it to find out pros and cons.

I will see what i come up with, and If it's any good, will push it, but as I say, i just went in with a hammer to play/experiment, so it's not great code at all 🤣

[Hookyns]

It must be some issue with https://github.com/Hookyns/ts-reflection/blob/b96406ad6972e1f0ac21d242030c65561f40cace/transformer/src/contexts/SourceFileContext.ts#L65-L77

[iDevelopThings]

I have no idea about that piece but, i noticed when working with my code that did a similar thing in the past, that even if you create the identifier once, it wouldn't stay the same between different source files(if i remember correctly)

I've also done a bit of refactoring on my code and changed it up to work with a meta file similar to your original implementation and it works great :D

It also adds the lib import to all files(could be optimised so that it only adds it to ones who need it), but i don't think it matters too much

Circular dependency isn't an issue(i don't think) since the new ctor implementation i did doesn't need the classes to be imported in the meta lib file

However I just stumbled across one issue with this :( with those ctors and their path being relative to the source file who originally made the getType() call, they're invalid when in the meta file

I think I can fix this though, might be over-thinking it, but pretty sure they just need to be relative to the meta file instead :D hopefully i can make a push soon so you can see!

[iDevelopThings]

😅

[Hookyns]

Circular dependency isn't an issue(i don't think)

It isn't thanks to your ctor requires and my lazy getType #465342f I've made recently.

class Foo 
{
    doSomething(): Foo 
    { 
        return this; 
    }
}

or

class A
{
	getB()
	{
		return new B();
	}
}

class B
{
	getA()
	{
		return new A();
	}
}

getType<A>();

It was endless recursion.
Now it generates:

(0, tst_reflect_1.getType)({
	k: 1,
	n: "B",
	fn: "@@this/dev/quick-tests/index.ts:B#21877",
	meths: [{n: "getA", params: [], rt: tst_reflect_1.getType.lazy(21875), tp: [], o: false, am: 2}], // <<-------- HERE IS LAZY GETTYPE
	ctors: [{params: []}]
}, 21877);
(0, tst_reflect_1.getType)({
	k: 1,
	n: "A",
	fn: "@@this/dev/quick-tests/index.ts:A#21875",
	meths: [{n: "getB", params: [], rt: (0, tst_reflect_1.getType)(21877), tp: [], o: false, am: 2}],
	ctors: [{params: []}],
	ctor: () => A
}, 21875);

---

I have no idea about that piece but, i noticed when working with my code that did a similar thing in the past, that even if you create the identifier once, it wouldn't stay the same between different source files(if i remember correctly)

Well I look for existing import of getType in each source file and store it in per-file context. Every time you create getTypeCall, it use the stored getType identifier, so even TS change names in final JS, it is still the same identifier instance and it change all occurrences.
If it does not exist in source file, it is generated and stored in per-file context. Each time the getType identifier is needed it should call the getGetTypeIdentifier() in SourceFileContext.

[Hookyns]

I've just got an idea how to detect those situations,.. I gonna try to write some code and I hope that buggy identifier thing happen.

[Hookyns]

Okay, I have detection code and code which generate that tst_reflect_2.getType which is not imported...
Now I'll try to figure out what's wrong and how to fix it.

[iDevelopThings]

Oh nice :D thankfully you could reproduce, was so random for me ^^

I also pushed my wip code, there's a lot that isn't documented/tested/covered and i imagine in breaks the current in-line setup, but if you want to pull and have a little look, it works pretty damn well ^^

https://github.com/iDevelopThings/ts-reflection/tree/testing-hot-swappable-store-and-meta

[iDevelopThings]

I will go chill for the rest of the night but just @ me if you have any questions/issues etc :D
Today has been a long one 😅

[iDevelopThings]

@Hookyns are you against me extracting descriptions for the runtime package to separate files?

My OCD is going slightly insane, it's also getting tiring scrolling for ever looking for things 😅

[Hookyns]

NP, put Description interfaces elsewhere,.. it doesn't matter. 😉 Just keep export stuff exporting from main file.

[iDevelopThings]

Okay sweet, I will probably do this tomorrow, tidies things up a lot :D

[iDevelopThings]

So... been playing some more and tidying up all my code from yesterday + documenting.

Also ran the regular inline generation through my code, it works well. Figured I can also just use it's current flow to add the reflect import into the file, this way we don't need to track the "getType import". Which will clean up the code base a lot

I'll push this code, then I'll remove the code related to that in a separate commit so I can roll it back and then you can see what you think - idk if this refactor is a little overkill or not, but it allows us to modify key parts pretty easily and from main places, rather than all over the place :D

[iDevelopThings]

Yeah that's cool :D adjust it to your style as you wish :)

I'm also not a fan of the "Impl" but naming is hard and I'm also not a fan of "IThing" 😅

Looking forward to it :)

[Hookyns]

😅 I am the "I" guy.. well .NET

But I don't use it on interfaces is TS if it is interface just to describe structure of an object.. but I keep it on interfaces which are rly interfaces for classes. But I will leave it.. for you.

Just "Impl" should be on classes, it is "Implementation" and interface just cannot be Implementation.. This Java guys use usually. Interfaces without I, but all classes with Impl. I interface is less evil, cuz u usually have less interfaces than classes.. 😅

[iDevelopThings]

Make it the way you normally would, it's your project :D I can live with either!

I noticed there's one small thing I forgot to do with the stores. Are you in the middle of working on your changes or?
Don't want the conflicts to be painful 😅

[Hookyns]

Nope,.. I'm waiting for your PR to devel 😆 I didn't know you've finished,.. avoiding conflicts too..

[iDevelopThings]

No problem 😅

iDevelopThings@f312163

Mine are all done now, I think it should be good 😄

[iDevelopThings]

Was just also thinking about a way to mark things as "indexable"

Like, it would be nice if we could use getType<T>(); as well as, a way to tell the transformer "generate types in these places", and it wouldn't need "getType" calls. It would be hella useful.

Maybe specifying an array of glob locations in tsconfig

"reflection" : {
    "metadata" : {
        "paths" : [
            "src/Services/**/*.ts",
            "src/SomeExplicitFile.ts",
        ]
    }
}

If you're open to the idea, it's something I can work on a little whilst you're busy with this stuff^^ I feel it'd be quite a simple addition

[iDevelopThings]

Ahhh you have a good point, I totally forgot that we're not exporting stuff for that 😅

I was thinking that maybe we can make another meta writer which is tree-shakable, and rather than the transformed code just calling get_type(id) it would actually use an export... but this code would still be referenced and not get removed right 🤦‍♂️

I guess the only other options are the exclude config, using inline meta or some kind of depth config(maybe a configurable depth for your own source files & one for external modules)

[Hookyns]

Hope you are doing no big changes cuz I'm doing quite big changes.. renaming and moving files and some methods/functions... 😇

[Hookyns]

I'm super tired but I think I finished that (almost; I left some TODOs in code, I'll review that later).
https://github.com/Hookyns/ts-reflection/tree/feature/refactored-stores

I found 2 issues when running dev/metadata-lib.

1. Nested type doesn't have props, seems like the same problem with ctor parameters,.. where were the problem?

2. It is required to set rootDir and outDir; okay np, but rootDir is "../.." and outDir is "./"
   metadata lib file is generated in rootDir ../.. and generated JS files are in dev/metadata-lib/dev/metadata-lib/. Last dev/metadata-lib/ is generated cuz of ../.. root. And the problem? const ___tst_reflection_meta = require("./../../reflection.meta"); should be two levels down const ___tst_reflection_meta = require("./../../../../reflection.meta");

I'll continue tomorrow(/later today).

[iDevelopThings]

All looks good to me, except for this in helpers.ts -> createCtorGetter - I can't remember why, but this was quite important for when you turned off the meta file mode

let relative = getRequireRelativePath(context.currentSourceFile.fileName, constructorDescription.srcPath);

if (context.metaWriter.is('ts-lib-file'))
{
	relative = context.metaWriter.getPathRelativeToLib(constructorDescription.srcPath);
}

And I'm not sure if I fully understand your problem from above, the way I always saw it, tsconfigs "rootDir" is your src directory. To me it made sense to generate a ts file in your src, then we can just transpile that as normal and output it into your build path.
The data is also there if you want to manually import it and just compile regularly with tsc

With dev metadata lib, your rootDir should be "./" as that's where your entry point and source files are, right?
Then your out dir is the same location, so it should be fine

Maybe to prevent any confusion etc, it would be best for us to add a config for the src/out paths? (this would help a lot any way, rather than us trying to guess where the end user will have their files and how they're structured, especially when it comes to creating the ctor require statements

I had a similar problem a compiler project I'm tackling too, it's a pain in the ass dealing with these paths and trying to create relative paths etc

[Hookyns]

I've fixed the issue with wrong relative path to metadata lib file.
To get the right out path I've used ts.getOutputFileNames().

I've extended our config object

{
	/**
	 * Base absolute path which will be used as root for type full names.
	 */
	rootDir: string;

	/**
	 * Path of tsconfig.json file.
	 */
	tsConfigPath: string;
	
	/**
	 * Output directory.
	 */
	outDir: string;

	/**
	 * Project directory
	 * @description It is directory containing tsconfig.json in most cases.
	 */
	projectDir: string;
}

All the values are based on tsconfig or inferred from cwd(). You can check createConfig in config.ts and TransformerContext.prepareConfig.

Path of the metadata file is no more important, cuz you've made auto-import, so it can be guessed.

---

except for this in helpers.ts -> createCtorGetter - I can't remember why, but this was quite important for when you turned off the meta file mode

let relative = getRequireRelativePath(context.currentSourceFile.fileName, constructorDescription.srcPath);

if (context.metaWriter.is('ts-lib-file'))
{
	relative = context.metaWriter.getPathRelativeToLib(constructorDescription.srcPath);
}

There is new method MetadataWriter.getRequireRelativePath() which is implemented in each type of MetadataWriter differently. Behavior is the same as original code you've posted.

[Hookyns]

@iDevelopThings U've ruined my life by that separation of reflect.ts into separate files. 😆
There are too many circular dependencies... 😭

[Hookyns]

Everything import Type and Type imports everything.. 😆

[Hookyns]

Fixed! 🎉

Well,.. it is awful, but it is working,.. I think there is nothing we can do with it... without big changes..

[Hookyns]

@iDevelopThings
I hope I didn't offend you, I'm sorry if I did. It was just a joke. 😞
That one-file was terrible, you did a great job. Original design was bad so that separation had consequences. I'm working on v0.10 which is separated into two standalone packages. First one is an abstract package and second one in an implementation.