Skip to content

added some stuff to adapters.md and minor formatting edits #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

added some stuff to adapters.md and minor formatting edits #17

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a6f2a0c
added some stuff to adapters.md
A1Liu Oct 15, 2018
cebb7bd
added more content
A1Liu Oct 15, 2018
a13f8e0
added some more stuff to adapters.md
A1Liu Oct 16, 2018
9bea2ce
reformatted links in service_map so that issue 15 is easier to solve …
A1Liu Oct 16, 2018
b6baefe
reformatted links
A1Liu Oct 16, 2018
af41214
Merge branch 'master' into master
A1Liu Oct 18, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions api/usage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ name | type | example | desc
id | integer | 42 | the id of the resource
uuid | string | "b35e6500-2086-4893-affb-8c57ae2bb6b9" | the universally unique id
shortname | string | 1200 | the course number or code
tags | array\<string\> | ["uhhhhh"] | the course's tags
tags | array\<string\> | ["uhhhhh"] | the course's tags


### Terms
Expand Down Expand Up @@ -154,4 +154,3 @@ seats | integer | 40
seats_taken | integer | 20 | the number of students currently registered for this section
conflict_ids | array\<integer\> | [4, 8, 15, 16, 23] | the ids of all sections that overlap with this section
periods | array<hash> | [{ "day": 1, "start": 800, "end": 950, type: "LEC"}] | the section's periods. each period has a day of the week, start time, end time, and type

82 changes: 70 additions & 12 deletions architecture/adapters.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<!--

- ->
# Service Map

## Welcome!
Expand All @@ -12,16 +12,35 @@ They can be written in any language, and must simply respond to a set of HTTP en
## Basic Concept
Here's a diagram illustrating the basic concept of Yacs and adapters:

<!-- ![alt-text](../_media/adaptersfig1.png) -->
<!-- ![alt-text](../_media/adaptersfig1.png) - ->
========
DIAGRAM GOES HERE
========
<!-- TODO: Make diagram -->
<!--
When Yacs needs information that isn't stored in its database, it sends an HTTP request to the adapter that is responsible for that information. Then that adapter works its magic, and sends the information back in JSON format. This document is going to focus on the [`yaml-rpi`][yaml-rpi-adapter] adapter and use its code in examples.
<!-- TODO: Make diagram --><!--

- ->
When Yacs needs information that isn't stored in its database, it sends an HTTP request to the adapter that is responsible for that information. Then that adapter works its magic, and sends the information back in JSON format. This document is going to focus on the [`yaml-rpi`][yaml-rpi-adapter] adapter and use its code in examples. The document `app.rb` will also be edited for clarity to this:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't technically correct. Every adapter is polled on a fixed interval, and Yacs saves the combined result/


``` ruby
require 'sinatra'
require 'sinatra/json'
require 'oj'
require 'yaml'

set :bind, '0.0.0.0'
set :port, 4600

ENV['YAML_SOURCE'] ||= 'schools-and-subjects.yml'

get "/:term_shortname" do
file = open(ENV['YAML_SOURCE'])
yaml_content = YAML.load(file)
json yaml_content
end
```

## Data Source
The first thing your adapter needs is a place to get data from. In the `yaml-rpi` adapter, that place is a file called `schools-and-subjects.yml`.
The first thing your adapter needs is a place to get data from. In the `yaml-rpi` adapter, that place is a file called `schools-and-subjects.yml`:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if this is the best example to start with. It is the simplest, but it is also the least useful. It is not typical that data will be coming from a config file, and I don't necessarily want to encourage people to use adapters that way. Although they can be used like this, they are intended to ingest dynamic data from some other source on the web.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm, do you think this markdown should use a script in another language? There's quite a few ruby/sinatra specific parts for the adapters as far as I can tell. Maybe it'd be better to use python or pseudo-code in the toy example code?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly, I don't think ruby is a bad choice here, we just need to find the right format for this guide.


```yml

Expand All @@ -41,15 +60,22 @@ schools:

```

Now that we have a place to get information from, we need to load it into the adapter. We do so with this line:
Now that we have a place to get information from, we need to load it into the adapter. We do so with this line in `app.rb`:

```ruby
``` ruby

# ...
require 'yaml'

ENV['YAML_SOURCE'] ||= 'schools-and-subjects.yml'

# ...
file = open(ENV['YAML_SOURCE'])
yaml_content = YAML.load(file)
# ...
```

Here, `ENV` is just a container for the environment variables in Ruby, and `'YAML_SOURCE'` is the name of the environment variable. We're setting its value to the name of the file, `'schools-and-subjects.yml'`, and essentially that 'loads' it into the adapter.
Here, `ENV` is just a container for the environment variables in Ruby, and `'YAML_SOURCE'` is the name of the environment variable. We're setting its value to the name of the file, `'schools-and-subjects.yml'`. Then, when the adapter is asked for information, it first opens a file at the location, then uses a function from the `'yaml'` package to convert the file's contents to a ruby object.

## JSON API and REST
Your adapter then needs to convert the data that it loaded into a format that Yacs can read - that's what the JSON API is for. This document will give a brief explanation of the API - a more in-depth explanation can be found [here][json-api].
Expand All @@ -68,17 +94,49 @@ The JSON API is based off of JavaScript Object Notation:

```

Notice that objects are surrounded by `{curly brackets}`, and both attributes and values are surrounded by `"double quotes"`. Each attribute-value pair is specified by a single line, with a colon separating attributes and values, and attribute-value pairs are separated by a comma. Note that spacing is irrelevant to the formatting of the document except in strings -- anything between double quotes must be on a single line.
Objects are surrounded by `{curly brackets}`, and both attributes and values are surrounded by `"double quotes"`. Each attribute-value pair is specified by a single line, with a colon separating attributes and values, and attribute-value pairs are separated by a comma. Spacing is irrelevant to the formatting of the document except in strings -- anything between double quotes must be on a single line. Finally, there is no syntax for making comments.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's within the scope of this documentation to explain how JSON works. I'd rather just provide a link as a reference.


In the `yaml-rpi` adapter, this is done through these lines in `app.rb`:

```ruby

In the `yaml-rpi` adapter, this is done through
require 'sinatra/json'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While helpful for understanding this particular adapter's code, I think including such specific information about ruby distracts the reader from what is important. If I'm new to this project and want to write an adapter, I probably won't use ruby if I don't already know it.

require 'oj'

# ...
json yaml_content

```

`require 'sinatra/json'` and `require 'oj'` tell ruby to load the Sinatra package's JSON helper function and the Optimized JSON package respectively. Then when the adapter needs to convert the data in `yaml_content` to JSON, the `json` function from `'sinatra/json'` is called.


## HTTP Protocol

<!-- IDK what to do here, I don't really understand this part fully yet
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just setting up the web server, and is very specific to Sinatra, the web framework we use in most of the adapters.


- ->

```ruby

# ...
require 'sinatra'

set :bind, '0.0.0.0'
set :port, 4600

# ...

get "/:term_shortname" do
# ...
end

```

## Implementation Agnostic
Notice that the only parts of the adapter that actually interact with YACS itself are format of the data (JSON) and the data transfer protocol (HTTP). Since both of these systems are implementation agnostic, you can write your adapter in whatever language you want! As long as it responds to HTTP requests with a JSON object, it's good to go!

[yacs-adapters]: https://github.com/YACS-RCOS/yacs/tree/master/adapters
[yaml-rpi-adapter]: https://github.com/YACS-RCOS/yacs/tree/master/adapters/yaml-rpi
[json-api]: http://jsonapi.org/format/
-->
<!-- -->
4 changes: 3 additions & 1 deletion architecture/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ While automatically pulling and combining data from various sources is great, we
This allows us to correct for out of date sources, add data that isn't present anywhere else, or even use Yacs as a primary catalog and information system.

The way core handles overrides is by storing two sets of attributes for each record: `auto_attributes`, and `override_attributes`.
These are both [`jsonb` columns](https://blog.codeship.com/unleash-the-power-of-storing-json-in-postgres/), and are therefore unstructured.
These are both [`jsonb` columns][json-cols], and are therefore unstructured.
`auto_attributes` stores the state of the record exactly as it came from malg.
`override_attributes` stores, attributes that were set manually, and, you guessed it, overrides `auto_attributes`.

Expand All @@ -107,3 +107,5 @@ That's how data gets in and out of Yacs.
This pipeline was developed through a good amount of trial and error.
It is not 100% perfect, but it is simple to develop against, easy to contribute to, and it gets the job done.
Pretty neat!

[json-cols]: https://blog.codeship.com/unleash-the-power-of-storing-json-in-postgres/
31 changes: 21 additions & 10 deletions architecture/service_map.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ This is a good place to start if you don't know where something lives, or want t
The above diagram shows how each of the services connect.
For more in-depth information on how Yacs operates as a whole, see [overview](architecture/overview)

### [core](https://github.com/yacs-rcos/yacs/blob/master/core)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think all of these changes are in your other PR, can we remove them from here and vice versa?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll open another PR with just adapter.md changes

### [core][core-repo]

Yacs is currently the largest service.
To avoid confusion, we'll call this "Yacs Core", or just "Core".
Expand All @@ -23,7 +23,7 @@ It provides a REST API for accessing and manually updating this data.
Yacs Core is also responsible for generating schedules, and computing section conflicts.
It is a Rails 5 app, and uses PostgreSQL to store data.

### [web](https://github.com/yacs-rcos/yacs/blob/master/web)
### [web][web-repo]

Yacs Web is the primary student-facing frontend for Yacs.
It is written in Typescript with Angular 6.
Expand All @@ -32,50 +32,61 @@ It provides the interface for browsing, searching, and selecting courses, as wel
It will also provide an interface for receiving notifications (planned).
If you are a new contributor, this is probably the best place to look for work to do as well.

### [admin](https://github.com/yacs-rcos/yacs-admin)
### [admin][admin-repo]

Yacs Admin is the administrative frontend for Yacs.
It is an interface for administrators, instructors, and sysadmins to make manual updates to the Yacs database.
In practice, it is typically used for correcting errors, updating course details, and managing non-catalog (special topics) courses.

### [malg](https://github.com/yacs-rcos/yacs/blob/master/malg)
### [malg][malg-repo]

Yacs Malg is the 'secret sauce' that connects Yacs to your university.
It pulls data from JSON data sources, and intelligently and configurably combines (amalgamates) it into a master graph.
The graph serialized and stored in Redis for internal persistance, and every entity is written to a Kafka topic as well for further processing and storage.
Malg polls each data source at regular intervals, and continually writes changes to Redis and Kafka.

### [adapters](https://github.com/yacs-rcos/yacs/blob/master/adapters)
### [adapters][adapters-repo]

Yacs uses services called adapters to pull data from a university's systems, and transform that data into a form Yacs can ingest.
Each data source has a corresponding adapter, which, when combined, provide a complete picture of a university's academic data.
It is very easy to create new adapters - more information explaining them can be found on the [adapters documentation](https://yacs.io/#/architecture/adapters).
They can be written in any language, and must simply respond to a set of http endpoints.

### [users](https://github.com/yacs-rcos/yacs/blob/master/users)
### [users][users-repo]

Yacs Users is the user authentication server for Yacs.
It is a full stack Rails 5 app, and uses devise for authentication.
It also handles storing user data, and provides a REST API for accessing and modifying that data (planned).

### [notifications](https://github.com/yacs-rcos/yacs/blob/master/notifications)
### [notifications][notifications-repo]

Yacs Notifications serves the streaming API for changes to Yacs objects.
It allows clients to receive notifications regarding courses and sections they are interested in.

### [docs](https://github.com/yacs-rcos/docs)
### [docs][docs-repo]

Docs is the repository for this documentation!
It uses Docsify, and awesome static site generator that generates beautiful documentation from everyday Markdown files.
Because we have a bunch of repos, documentation goes in here so it is easier to find, with the exception of each service's README.

### [nginx](https://github.com/yacs-rcos/yacs-nginx)
### [nginx][nginx-repo]

The nginx configuration we use to deploy Yacs.
It has reverse proxies for the external APIs and services the frontend builds.

### [auth](https://github.com/yacs-rcos/yacs-auth)
### [auth][auth-repo]

Yacs Auth is a microservice authorization library.
It contains our authorization logic and session management, which uses JSON Web Tokens (JWT) with Redis for session handling and validation.
This library allows any service to authenticate a user and validate or invalidate their session, as long as that service is connected to the shared Redis instance.

[core-repo]: https://github.com/yacs-rcos/yacs/blob/master/core
[web-repo]: https://github.com/yacs-rcos/yacs/blob/master/web
[admin-repo]: https://github.com/yacs-rcos/yacs-admin
[malg-repo]: https://github.com/yacs-rcos/yacs/blob/master/malg
[adapters-repo]: https://github.com/yacs-rcos/yacs/blob/master/adapters
[users-repo]: https://github.com/yacs-rcos/yacs/blob/master/users
[notifications-repo]: https://github.com/yacs-rcos/yacs/blob/master/notifications
[docs-repo]: https://github.com/yacs-rcos/docs
[nginx-repo]: https://github.com/yacs-rcos/yacs-nginx
[auth-repo]: https://github.com/yacs-rcos/yacs-auth
18 changes: 12 additions & 6 deletions contributors/getting_started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ There is always plenty of work to be done, and if you ever have any questions or

## Where Do I Start?

First, read the [Code of Conduct](overview/code_of_conduct.md) if you haven't already.
First, read the [Code of Conduct](overview/code_of_conduct.md) if you haven't already.

The next step is this document here.
You may be eager to get started working, and that's super!
Expand All @@ -26,13 +26,13 @@ This document will explain how we do things, our development and project managem
### Issues

First thing's first: issues!
We use [Github Issues](https://guides.github.com/features/issues/) to keep track of what we need to do.
We use [Github Issues][github-issues] to keep track of what we need to do.
Every issue describes a single task, problem, or request.
Issues should be small, specific, and well defined.

### Projects

We use [Github Projects](https://help.github.com/articles/about-project-boards/) to keep track of our issues across our multiple services and initiatives (more on this [later](contributors/service_map.md)). Our main project board is the [Core Project](https://github.com/orgs/YACS-RCOS/projects/3).
We use [Github Projects][github-projects] to keep track of our issues across our multiple services and initiatives (more on this [later](contributors/service_map.md)). Our main project board is the [Core Project][core-project].
Issues on this project are organized into five categories:

1) **Needs Review**. Newly created issues go here.
Expand Down Expand Up @@ -63,8 +63,8 @@ Once the fix for an issue has been released, a maintainer will move the issue to
## Finding an Issue

Now that you see how things work, you are ready to find your first issue!
The [Ready for Development](https://github.com/orgs/YACS-RCOS/projects/3#column-2034428) column on the [Core Project](https://github.com/orgs/YACS-RCOS/projects/3) is a great place to start.
There are a number of issues here labeled [good-first-issue](https://github.com/orgs/YACS-RCOS/projects/3?card_filter_query=label%3A%22good+first+issue%22).
The [Ready for Development][ready-for-development] column on the [Core Project][core-project] is a great place to start.
There are a number of issues here labeled [good-first-issue][good-first-issue-label].
Issues with this label have been selected by the maintainers as good starting points for new contributors.

You are not, however, limited to issues with this label by any means!
Expand All @@ -89,4 +89,10 @@ You can also always skip ahead to the [Setup Guide](contributors/setup_guide) to
So now you have something to work on.
Congratulations on making it this far!
The next step is to get Yacs running on your computer.
Check out the [Setup Guide](contributors/setup_guide) for the next steps!
Check out the [Setup Guide](contributors/setup_guide) for the next steps!

[github-issues]: https://guides.github.com/features/issues/
[github-projects]: https://help.github.com/articles/about-project-boards/
[core-project]: https://github.com/orgs/YACS-RCOS/projects/3
[ready-for-development]: https://github.com/orgs/YACS-RCOS/projects/3#column-2034428
[good-first-issue-label]: https://github.com/orgs/YACS-RCOS/projects/3?card_filter_query=label%3A%22good+first+issue%22