Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev #34

Open
wants to merge 100 commits into
base: master
Choose a base branch
from
Open

Dev #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
231f137
Add ACKNOWLEDGMENTS files with licenses of 3rd party software
trevorspreadbury Jun 2, 2021
3d91ad7
add functionality to search for image on internet if not found locall…
trevorspreadbury Jun 3, 2021
1929cf6
Modify results_dirs to allow additions from the UI
trevorspreadbury Jun 3, 2021
d42347e
modify function docs for ui javascript
trevorspreadbury Jun 3, 2021
eae51a3
Add ability for users to download results of ui query
trevorspreadbury Jun 3, 2021
c59590b
Add scale bar and subfigure label info to ui query download
trevorspreadbury Jun 3, 2021
2803d9a
change google drive locations for checkpoints to materialeyes account
trevorspreadbury Jun 3, 2021
ad044bf
fix bug that removes captions during repeated runs
trevorspreadbury Jun 3, 2021
c70040b
Update nature scraper and test to deal with new html format
trevorspreadbury Oct 13, 2021
948ce02
update open access checking to account for new nature html
trevorspreadbury Oct 14, 2021
9c20f76
Fix bugs in pipeline.py -- not properly saving results_dir and subfig…
trevorspreadbury Oct 17, 2021
050671e
Fix ACS encoding error - header was accepting wrong encodings
trevorspreadbury Oct 28, 2021
9d9ae75
Update entity_ruler to match new spaCy specs
trevorspreadbury Oct 28, 2021
5da9641
Fix ACS encoding by removing 'Accept-Encoding' header
trevorspreadbury Oct 28, 2021
0e2da40
Resolve #13 and refactor JournalFamily to be more comprehensible
trevorspreadbury Nov 15, 2021
3086369
remove ui code -- moved to https://github.com/MaterialEyes/exsclaim-ui
trevorspreadbury Dec 19, 2021
dc77ddc
make journal family variables more explicit
trevorspreadbury Dec 19, 2021
c2a7278
Changes to made to add the Wiley publication family to the tool
Dec 20, 2021
45eed0b
update tests to use new nature html
trevorspreadbury Feb 18, 2022
cf156a5
initial rsc scraper
trevorspreadbury Feb 18, 2022
47f1a27
Add docker to allow using selenium for rsc
trevorspreadbury Feb 19, 2022
703961c
add selenium to requirements
trevorspreadbury Feb 19, 2022
5194020
updated query names to standardize across journals
trevorspreadbury Feb 19, 2022
d6f7dd8
adds ability to turn page to rsc scraper
trevorspreadbury Feb 19, 2022
0f88154
Merge pull request #15 from MaterialEyes/rsc
trevorspreadbury Feb 19, 2022
9a48fa7
Made changes to PR questions:
Feb 21, 2022
bd278f0
fix typo
trevorspreadbury Feb 25, 2022
9b4087e
Move journal-specific logic out of base class
trevorspreadbury Feb 25, 2022
2e03c1a
Merge branch 'dev' into wiley
trevorspreadbury Feb 25, 2022
ab8f7b0
Merge pull request #14 from MaterialEyes/wiley
trevorspreadbury Feb 25, 2022
346ab4c
Converted journal class to use properties with docstrings
trevorspreadbury Feb 26, 2022
7e566f6
reformat methods as abstract, added typing
trevorspreadbury Feb 26, 2022
2cc4f2b
reorganize and clean journalfamily methods
trevorspreadbury Feb 26, 2022
29facaa
update readme with docker
trevorspreadbury Mar 12, 2022
83c6615
fix minor bugs in journal scraper
trevorspreadbury Mar 26, 2022
f26491d
correct line_label_distance typo
trevorspreadbury Mar 26, 2022
ffb47f1
create continuous integration checks
trevorspreadbury Mar 26, 2022
ce4b97e
Merge pull request #18 from MaterialEyes/ci
trevorspreadbury Mar 26, 2022
9ba4042
update gitignore
trevorspreadbury Mar 29, 2022
6cc5732
move checkpoint loading code to utilities
trevorspreadbury Mar 29, 2022
d7ac607
create dockerignore
trevorspreadbury Mar 29, 2022
79395ec
Apply first round of style checkers -- flake8, black, isort
trevorspreadbury Mar 29, 2022
6034d13
Merge remote-tracking branch 'origin' into style
trevorspreadbury Mar 29, 2022
4b10422
lint bug report
trevorspreadbury Mar 29, 2022
c3b34ea
fix black error https://github.com/psf/black/issues/2964
trevorspreadbury Mar 29, 2022
598f3a5
Merge pull request #19 from MaterialEyes/style
trevorspreadbury Mar 29, 2022
e67c00c
Merge branch 'dev' into journal_cleanup
trevorspreadbury Mar 29, 2022
751f4ab
Add type hints to journal functions
trevorspreadbury Mar 29, 2022
2ca74f0
Reformat journal.py to use sphinx documentation format
trevorspreadbury Mar 29, 2022
9cd915f
Merge pull request #16 from MaterialEyes/journal_cleanup
trevorspreadbury Mar 29, 2022
d9613d2
Revert "Reformat journal.py to use sphinx documentation format"
trevorspreadbury Mar 31, 2022
ef06413
added docstring to journalfamily class
trevorspreadbury Mar 31, 2022
270e6d0
Add module docstrings to utilities package
trevorspreadbury Mar 31, 2022
e9b190f
Add additional documentation that is sphinx compatable
trevorspreadbury Mar 31, 2022
50bfb20
Ignore sphinx builds
trevorspreadbury Mar 31, 2022
41f7878
Initial docs using sphinx and napoleon to auto generate docs
trevorspreadbury Mar 31, 2022
6d85daa
Merge pull request #22 from MaterialEyes/docs
trevorspreadbury Mar 31, 2022
012eac7
Fix bug in model utility from copy/paste error
trevorspreadbury Mar 31, 2022
f975445
Simplify results location to either specified or default location
trevorspreadbury Mar 31, 2022
1d85336
New test queries for every supported journal family
trevorspreadbury Mar 31, 2022
0dfd378
Modify dockerfile to be smaller, run with input query argument
trevorspreadbury Mar 31, 2022
4540a65
Allow easy development in vscode from within a container
trevorspreadbury Mar 31, 2022
695a4f8
Update documentation for docker usage
trevorspreadbury Mar 31, 2022
26cbb07
Merge pull request #23 from MaterialEyes/docker
trevorspreadbury Mar 31, 2022
89ece6e
image save fixes
katerinavr Jan 22, 2023
8aafb47
update journal.py
katerinavr Jan 23, 2023
3db58ed
update journal
katerinavr Jan 23, 2023
64bf54c
update
katerinavr Jan 23, 2023
ea5a917
update
katerinavr Jan 23, 2023
ebc374c
update
katerinavr Jan 23, 2023
0ccd3d4
update
katerinavr Jan 23, 2023
3031fa1
update
katerinavr Jan 23, 2023
36e0539
update
katerinavr Jan 23, 2023
81ba6de
update
katerinavr Jan 23, 2023
ff172a5
update
katerinavr Jan 23, 2023
386259b
update
katerinavr Jan 23, 2023
a37f95a
update
katerinavr Jan 23, 2023
6c53bf6
update
katerinavr Jan 23, 2023
3b53cf8
update
katerinavr Jan 24, 2023
8f3b3f6
update
katerinavr Jan 24, 2023
1779712
update
katerinavr Jan 24, 2023
eb15e46
update
katerinavr Jan 24, 2023
7633605
update
katerinavr Jan 24, 2023
a44cae7
update
katerinavr Jan 24, 2023
b60384d
update
katerinavr Jan 24, 2023
59091b0
update
katerinavr Jan 24, 2023
c5abd69
update
katerinavr Jan 24, 2023
386d1da
update
katerinavr Jan 24, 2023
dd5a8f3
update
katerinavr Jan 25, 2023
d1a4dae
update
katerinavr Jan 25, 2023
3b68e84
update
katerinavr Jan 25, 2023
6a5134a
update
katerinavr Jan 28, 2023
45cf34f
update journal.py
katerinavr May 1, 2023
cf681fc
upadate get_page_info function
katerinavr May 1, 2023
5fd7dcf
Update main.workflow.yml
katerinavr May 2, 2023
3b5279a
update GitHub Actions workflow to use Node.js 16
katerinavr May 2, 2023
f77498f
Merge branch 'dev' into katerinavr-patch-1
trevorspreadbury May 4, 2023
355e367
Merge pull request #38 from MaterialEyes/katerinavr-patch-1
trevorspreadbury May 4, 2023
4cbf0fc
Merge branch 'dev' into javascript_fixes
katerinavr Aug 10, 2023
d5f468f
Merge pull request #39 from MaterialEyes/javascript_fixes
katerinavr Aug 10, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
{
"name": "exsclaim-dev",
"build": {
"dockerfile": "../Dockerfile",
"context": "..",
"args": {},
"target": "dev"
},
// Set *default* container specific settings.json values on container create.
"settings": {
"terminal.integrated.profiles.linux": {
"bash": {
"path": "/bin/bash"
}
},
"python.defaultInterpreterPath": "/usr/local/bin/python3",
"python.languageServer": "Default",
"python.linting.enabled": true,
"python.linting.pylintEnabled": true,
"python.formatting.autopep8Path": "/usr/local/py-utils/bin/autopep8",
"python.formatting.blackPath": "/usr/local/py-utils/bin/black",
"python.formatting.yapfPath": "/usr/local/py-utils/bin/yapf",
"python.linting.banditPath": "/usr/local/py-utils/bin/bandit",
"python.linting.flake8Path": "/usr/local/py-utils/bin/flake8",
"python.linting.mypyPath": "/usr/local/py-utils/bin/mypy",
"python.linting.pycodestylePath": "/usr/local/py-utils/bin/pycodestyle",
"python.linting.pydocstylePath": "/usr/local/py-utils/bin/pydocstyle",
"python.linting.pylintPath": "/usr/local/py-utils/bin/pylint"
},
// Add the IDs of extensions you want installed when the container is created.
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance",
"ms-vscode-remote.remote-containers",
"ms-toolsai.jupyter",
],
"features": {
"github-cli": "latest"
},
"runArgs": [
"--shm-size=16gb"
],
// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [9000],
// Use 'portsAttributes' to set default properties for specific forwarded ports. More info: https://code.visualstudio.com/docs/remote/devcontainerjson-reference.
// "portsAttributes": {
// "9000": {
// "label": "Hello Remote World",
// "onAutoForward": "notify"
// }
// },
// Use 'otherPortsAttributes' to configure any ports that aren't configured using 'portsAttributes'.
// "otherPortsAttributes": {
// "onAutoForward": "silent"
// },
// Use 'postCreateCommand' to run commands after the container is created.
// "postCreateCommand": "pip3 install -r requirements.txt",
// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
"workspaceMount": "source=${localWorkspaceFolder},target=/src,type=bind",
"workspaceFolder": "/src"
}
37 changes: 37 additions & 0 deletions .devcontainer/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
version: '3.9'
services:
# Update this to the name of the service you want to work with in your docker-compose.yml file
exsclaim:
# If you want add a non-root user to your Dockerfile, you can use the "remoteUser"
# property in devcontainer.json to cause VS Code its sub-processes (terminals, tasks,
# debugging) to execute as the user. Uncomment the next line if you want the entire
# container to run as this user instead. Note that, on Linux, you may need to
# ensure the UID and GID of the container user you create matches your local user.
# See https://aka.ms/vscode-remote/containers/non-root for details.
#
# user: vscode

# Uncomment if you want to override the service's Dockerfile to one in the .devcontainer
# folder. Note that the path of the Dockerfile and context is relative to the *primary*
# docker-compose.yml file (the first in the devcontainer.json "dockerComposeFile"
# array). The sample below assumes your primary file is in the root of your project.
#
# build:
# context: .
# dockerfile: .devcontainer/Dockerfile

volumes:
# Update this to wherever you want VS Code to mount the folder of your project
- .:/workspace:cached

# Uncomment the next line to use Docker from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker-compose for details.
# - /var/run/docker.sock:/var/run/docker.sock

# Uncomment the next four lines if you will use a ptrace-based debugger like C++, Go, and Rust.
# cap_add:
# - SYS_PTRACE
# security_opt:
# - seccomp:unconfined

# Overrides default command so things don't shut down after the process ends.
command: /bin/sh -c "while sleep 1000; do :; done"
6 changes: 6 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
docs/
*/tests
build/
dist/
.ipynb_checkpoints/
exsclaim.egg-info/
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Steps to reproduce the behavior:
A clear and concise description of what you expected to happen.

**Outputs**
If applicable, add screenshots to help explain your problem. For error messages, please copy and paste them here.
If applicable, add screenshots to help explain your problem. For error messages, please copy and paste them here.

**Environment (please complete the following information):**
- OS: [e.g. Ubuntu]
Expand Down
28 changes: 28 additions & 0 deletions .github/workflows/main.workflow.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Development Checks

on:
# Trigger the workflow on push or pull request,
# but only for the main branch
pull_request:
branches:
- main
- master
- dev

jobs:
linter:
runs-on: ubuntu-latest
steps:

- name: Checkout Code Repository
uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v3
with:
python-version: "3.9"
cache: pip
cache-dependency-path: |
requirements.txt
- name: Run pre-commit
uses: pre-commit/[email protected]
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,7 @@ cookie
dist/
build/
*.sh
*/results_dirs
*/results_dirs
.ipynb_checkpoints/
**/local.ini
docs/build/
34 changes: 34 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
exclude: "^docs/"
default_stages: [commit]

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.1.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml

- repo: https://github.com/psf/black
rev: 22.1.0
hooks:
- id: black
additional_dependencies: ['click==8.0.4']

- repo: https://github.com/PyCQA/isort
rev: 5.10.1
hooks:
- id: isort

- repo: https://github.com/PyCQA/flake8
rev: 4.0.1
hooks:
- id: flake8
args: ["--config=setup.cfg"]
additional_dependencies: [flake8-isort]

# sets up .pre-commit-ci.yaml to ensure pre-commit dependencies stay up to date
ci:
autoupdate_schedule: weekly
skip: []
submodules: false
Loading