Foil Communication / File Data

[amcguiga]

Relates to #70

FOIA Statuses

I did some research on FOIA statuses. Based on this and on the way they're rendered on the muckrock site, I'm renaming and reorganzing some statuses.

Ideal Workflow

• ack : Awaiting Acknowledgement
  • Request has been sent to department and has not yet been acknowledged
• processed : Awaiting Processing
  • Request is acknowledged by the department and is being processed
• done : Completed
  • Request has been satisfactorily completed by the department and documents have been returned
  • In practice, usually just consists of some records since 2000-2010 (often not including "unsubstantiated" allegations)
  • No records released by any department pre 1990
  • Sometimes marked as done though the request was actually rejected by the department

Temporary Blockers

• fix : Additional Action Required
  • Request requires additional forms submitted or clarification needed for the department to proceed
  • This status seems to override subsequent processing statuses, so requests will stay in fix after they've already been addressed
• payment : Awaiting Payment
  • Request requires submission of payment to department for them to proceed
• partial : Partially Completed
  • Request has been partially completed but there is an indication that additional documents are being withheld
  • This isn't used consistently, as no departments are actually completing the entire request
• Typically requests in this status have a similar level of completion as done requests
• appealing : Appealing
  • Request has not received a response and has been escalated to additional agencies
  • Many request that should fit this criteria are labeled as ack or fix instead, unfortunately

Rejections

• rejected : Rejected
  • Request has been rejected by the department and won't be fulfilled
  • Usually the department cites FOIL requirements in New York that the request must be "specific and reasonable"
• no_docs : No Documents Found
  • Request has been "completed" by department and no records were returned
  • Usually they will cite the decision that records aren't available for "unsubstantiated" allegations
• lawsuit : Pending Litigation
  • Request will not be completed until department considers litigation on 50a finalized
  • Departments cited New York Circuit / Supreme Court stay orders, which have since expired

New York State Supreme Court Justice Carol R. Edmead, issued a Stay Order (Index No. 154982/2020) prohibiting the public disclosure of any records concerning unsubstantiated and non-final allegations or settlement agreements concerning disciplinary matters against individual police officers, firefighters and correction officers.

It is my understanding that the Second Circuit Court of Appeals in NYC reinstated a stay order regarding NYC police/fire/correction officers involving the same subject matter as your FOIL request. There is also a separate subject matter-related New York State Supreme Court stay order from upstate New York I believe is still in effect.

Abandoned

• abandoned : Duplicate Request
  • Request has been abandoned because it was delivered to an incorrect department
  • In every case, the request has been filed again in the appropriate department
  • We should probably filter these out of the data altogether

API Changes

We would like to be able to display the lifecycle of each FOIL request. This would show how foia requests are being processed by police departments in aggregate. Their API exposes a limited journal of these activities through the communications object on a Foia Request.

Rather than exposing the whole object in their format, I'm choosing to expose a summary of the data that's included.

{
      "foiaReq": {
        "id": "96380",
        "status": "done",
        "datetime_submitted": "2020-06-19T02:11:48",
        "datetime_done": "2021-02-11T16:00:54.197509",
        "price": "0.00",
        "absolute_url": "https://www.muckrock.com/foi/weedsport-22361/police-department-misconduct-allegations-and-disciplinary-proceedings-96380/",
        "agency": "21969",
        "user": "34697",
        "communications": [
          {
            "from_user": 9535,
            "to_user": 35651,
            "datetime": "2020-06-19T02:11:48.775853",
            "status": null,
            "file_stats": {
              "count": 1,
              "pages": 7,
              "formats": [
                "pdf"
              ]
            }
          },
          {
            "from_user": 4420,
            "to_user": 35651,
            "datetime": "2020-06-26T05:01:47.019957",
            "status": null,
            "file_stats": null
          },
          {
            "from_user": null,
            "to_user": null,
            "datetime": "2020-06-26T15:37:27.593235",
            "status": "processed",
            "file_stats": {
              "count": 1,
              "pages": 1,
              "formats": [
                "pdf"
              ]
            }
          },
          {
            "from_user": 4420,
            "to_user": 35651,
            "datetime": "2021-02-05T05:01:15.235900",
            "status": null,
            "file_stats": null
          },
          {
            "from_user": 4420,
            "to_user": 35651,
            "datetime": "2021-02-08T05:00:54.185420",
            "status": null,
            "file_stats": null
          },
          {
            "from_user": 35651,
            "to_user": 34697,
            "datetime": "2021-02-08T11:31:59.637205",
            "status": "fix",
            "file_stats": null
          },
          {
            "from_user": 35651,
            "to_user": 34697,
            "datetime": "2021-02-11T16:00:54.197509",
            "status": "done",
            "file_stats": {
              "count": 2,
              "pages": 55,
              "formats": [
                "png",
                "pdf"
              ]
            }
          }
        ]
      },
      "agency": {
        "id": "21969",
        "agencyName": "Weedsport Village Police Department"
      },
      "jurisdiction": {
        "id": "22361",
        "jurisdictionName": "Weedsport"
      }
    }

Key elements that we could use are the communication.status, communication.datetime as those could show a funnel of how the FOIL request changes over time.

We could also attempt to model how many records have been released by looking at the file statistics for done and partial requests. communication.file_stats will show total number of pages for files returned in document formats (pdf, doc). It's possible for police records to be released as zip, but those aren't estimated at pages. As far as I can tell, the image files tend to be from email signatures (png, jpg, gif).