Implement Dynamictable

.github/workflows/lint.yml

@@ -2,6 +2,8 @@ name: Lint
 
 on:
   push:
+    branches:
+      - main
   pull_request:
     branches: [main]
 

.github/workflows/tests.yml

@@ -2,6 +2,11 @@ name: Tests
 
 on:
   push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
 
 jobs:
   test:
@@ -34,8 +39,20 @@ jobs:
         run: pytest
         working-directory: nwb_linkml
 
-      - name: Report coverage
-        working-directory: nwb_linkml
-        run: "coveralls --service=github"
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      - name: Coveralls Parallel
+        uses: coverallsapp/[email protected]
+        if: runner.os != 'macOS'
+        with:
+          flag-name: run-${{ join(matrix.*, '-') }}
+          parallel: true
+          debug: true
+
+  finish-coverage:
+    needs: test
+    if: ${{ always() }}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Coveralls Finished
+        uses: coverallsapp/[email protected]
+        with:
+          parallel-finished: true

README.md

@@ -9,4 +9,4 @@ Translating NWB schema language to linkml
 
 Just submitting to pypi to squat the package name 
 
-[![Coverage Status](https://coveralls.io/repos/github/p2p-ld/nwb-linkml/badge.svg)](https://coveralls.io/github/p2p-ld/nwb-linkml)
+[![Coverage Status](https://coveralls.io/repos/github/p2p-ld/nwb-linkml/badge.svg)](https://coveralls.io/github/p2p-ld/nwb-linkml) 

docs/index.md

@@ -284,6 +284,7 @@ api/nwb_linkml/schema/index
 
 meta/todo
 meta/changelog
+meta/references
 genindex
 ```
 

docs/intro/translation.md

@@ -20,6 +20,11 @@
 
 ### DynamicTable
 
+```{note}
+See the [DynamicTable](https://hdmf-common-schema.readthedocs.io/en/stable/format_description.html#dynamictable)
+reference docs
+```
+
 One of the major special cases in NWB is the use of `DynamicTable` to contain tabular data that
 contains columns that are not in the base spec. 
 
@@ -284,8 +289,35 @@ When generating pydantic models we...
 
 There are several different ways to create references between objects in nwb/hdmf:
 
-- ...
-
+- [`links`](https://schema-language.readthedocs.io/en/latest/description.html#sec-link-spec) are group-level
+  properties that can reference other groups or datasets like this:
+  ```yaml
+  links:
+  - name: Link name
+    doc: Required string with the description of the link
+    target_type: Type of target
+    quantity: Optional quantity identifier for the group (default=1).
+  ```
+- [Reference `dtype`](https://schema-language.readthedocs.io/en/latest/description.html#reference-dtype)s are
+  dataset, and attribute-level properties that can reference both other objects and regions within other objects:
+  ```yaml
+  dtype:
+    target_type: ElectrodeGroup
+    reftype: object
+  ```
+- `TimeSeriesReferenceVectorData` is a compound dtype that behaves like VectorData and VectorIndex combined
+  into a single type. It is slightly different in that each row of the vector can refer to a different table,
+  and has a different way of handling selection (with `start` and `count` 
+  rather than a series of indices for the end of each cell)
+- Implicitly, hdmf creates references between objects according to some naming conventions, eg.
+  an attribute/dataset that is a `VectorIndex` named `mydata_index` will be linked to a `VectorData`
+  object `mydata`.
+- There is currently a note in the schema language docs that there will be an additional
+  [Relationships](https://schema-language.readthedocs.io/en/latest/description.html#relationships) system
+  that explicitly models relationships, but it is unclear how that would be different than references. 
+
+We represent all of these by just directly referring to the object type, preserving the source type
+in an annotation, when necessary.
 
 
 ## LinkML to Everything

docs/meta/references.md

@@ -0,0 +1,11 @@
+# References
+
+## Documentation
+
+- [hdmf](https://hdmf.readthedocs.io/en/stable/)
+- [hdmf-common-schema](https://hdmf-common-schema.readthedocs.io/en/stable/)
+- [pynwb](https://pynwb.readthedocs.io/en/latest/)
+
+```{todo}
+Add the bibtex refs to NWB papers :)
+```

docs/meta/todo.md

@@ -7,6 +7,7 @@ NWB schema translation
 - handle compound `dtype` like in ophys.PlaneSegmentation.pixel_mask
 - handle compound `dtype` like in TimeSeriesReferenceVectorData
 - Create a validator that checks if all the lists in a compound dtype dataset are same length
+- [ ] Move making `target` optional in vectorIndex from pydantic generator to linkml generators!
 
 Cleanup
 - [ ] Update pydantic generator
@@ -22,7 +23,7 @@ Cleanup
 
 - [ ] Make a minimal pydanticgen-only package to slim linkml deps?
 - [ ] Disambiguate "maps" terminology - split out simple maps from the eg. dataset mapping classes
-- [ ] Remove unnecessary imports
+- [x] Remove unnecessary imports
   - dask
   - nptyping
 - [ ] Adapt the split generation to the new split generator style