Convert available funds / contributions/expenditures from area to step charts #216

hancush · 2024-10-01T17:02:18Z

Overview

This PR updates the charts on candidate and committee pages. Namely it:

Changes them from area charts to step charts, see Convert available funds / contributions/expenditures from area to step charts #216 (comment)
Updates chart titles, adds axis labels
Forces a consistent date range along the x-axis for easier comparison, see Convert available funds / contributions/expenditures from area to step charts #216 (comment)
Fixes tooltip formatting
Updates the available funds chart to use the end of the filing period, rather than the date the filing was made

It also updates the aggregate queries to force a continuous interval of aggregate data. Previously, it was sparse, which posed a challenge for charting. See #216 (comment)

Demo

Testing instructions

Update your local Docker Compose file to connect to a read-only version of the prod database (see Money Trail NM - Docker Env - Readonly Prod DB note in Bitwarden), then view some donation / expenditure charts and confirm they look good.

camp_fin/static/js/chart_helper.js

fgregg · 2024-10-01T17:53:13Z

chart style

i think it's hacked into columns, because we were trying to evoke a bar chart, and not an area chart. If it was a true area chart, then the area under the curve should equal the total money. If you divided the monthly money by the number of days in the month, then you would have an approximation of that.

In distinction, with a barchart, you just pay attention to how high the whole number reaches. Why we didn't use a barchart, directly, I don't quite remember.

Even if we did want to use an area chart, i think it would still make sense to have something stair-step like this since we don't know when in a month the donation happened.

IMO, the best alternatives would be

A barchart
A stairstep area chart, with monthly amounts normalized by days in the month
A starstep line chart.

Of these, the line chart is probably the simplest to do.

Why data isn't loading?

I don't know. The aggregate_data.py file is responsible for making the view that power those. We could pair on that if you'd like.

hancush · 2024-10-01T19:06:20Z

@fgregg Thanks, got it. Updated to a stepped line chart.

For the data issue, I resolved it by:

Recreating the materialized views (seems like we didn't do this after bringing in the last change to the aggregate queries in 27a86ba, which updated the contribution transaction types to capture)
Updating "Monetary contribution" to "Monetary Contribution" in the TransactionType table (the aggregate query looks for the specific capitalization we control, but we haven't reimported pre-2023 data since we made that change, so those transactions had the old description and weren't being captured)

This is ready for another look.

fgregg

this code looks quite clean, nice job! some minor, optional suggestions.

where was this change made?

Updating "Monetary contribution" to "Monetary Contribution" in the TransactionType table (the aggregate query looks for the specific capitalization we control, but we haven't reimported pre-2023 data since we made that change, so those transactions had the old description and weren't being captured)

Also, since we are switching to line charts here, it might be good to switch to line charts for the cash on hand and debt at present charts above this chart?

fgregg · 2024-10-01T19:24:38Z

camp_fin/models.py

+            ) months
+            LEFT JOIN (
+              SELECT
+                {table}.amount AS amount,


how about a COALESCE(table.amount, 0) as amount here, and then you can simplify some of the following python code.

optional suggestion, tbc.

added up above!

fgregg · 2024-10-02T15:11:09Z

camp_fin/models.py

+              months.month,
+              {table}.amount
+            FROM (
+              SELECT


optional suggestion: this looks like a good place to use a CTE.

CTEs aren't supported in the version of Postgres we're using for local development (which might not be a constraint in deployment, actually, but don't feel like upgrading Postgres here).

yeah, we should have the development env match up, so I can open an issue for that. fine not to change that here.

hancush · 2024-10-02T16:48:16Z

@fgregg Good idea to update the funds available, since these are point in time numbers, not an accumulating value. I revised that chart, and added axis labels and forced a consistent date range / x-axis for both, since the first chart should be a function of the second, i.e., they're connected and should be easy to compare?

hancush · 2024-10-02T16:50:41Z

camp_fin/models.py

-            if summed_filings[0].opening_balance:
-                first_opening_balance = summed_filings[0].opening_balance
-            else:
-                first_opening_balance = 0
-
-            if summed_filings[0].debt_carried_forward:
-                first_debt = summed_filings[0].debt_carried_forward
-            else:
-                first_debt = 0
-
-            init_date = summed_filings[0].initial_date
-
-            first_initial_date = [int(since), 1, 1]
-
-            # If the first available filing date is on or before the start date
-            # passed into this method, use that as the first date in the trendline
-            if init_date:
-                init_date_parts = [init_date.year, init_date.month, init_date.day]
-                if datetime(*init_date_parts) <= datetime(*first_initial_date):
-                    first_initial_date = init_date_parts
-
-            debt_trend.insert(0, [first_debt, *first_initial_date])
-            balance_trend.insert(0, [first_opening_balance, *first_initial_date])


i'm not sure about doing this inference, since the other values are as-of the filing date, and we don't have the date for the filing before the first one in our database. i omitted it, but open to your thoughts.

i think it's fine to omit.

hancush · 2024-10-02T16:53:47Z

camp_fin/templates/camp_fin/candidate-detail.html

@@ -83,10 +83,10 @@ <h4>
 {% endblock %}

 {% block charts %}
-  <h3><i class="fa fa-fw fa-area-chart"></i> Net funds over time</h3>
+  <h3><i class="fa fa-fw fa-area-chart"></i> Funds available and debts by filing date</h3>


Net is a difference in values. We're showing both values.

fgregg

Great job lining up the data ranges!

i think it's okay for the available fund chart to be a normal line chart and not a step chart. If you have $100 dollars on july 1, and $200 dollars on august 1, it's reasonable to think that you had $150 at some point in between. But, it's up to you.

fgregg · 2024-10-02T17:17:27Z

camp_fin/models.py

            FROM (
              SELECT
                DISTINCT DATE_PART('year', month) AS year,
                GENERATE_SERIES(1, 12) AS month
              FROM {table}_by_month
              ORDER BY year, month
            ) months
-            LEFT JOIN (


wait if we don't need to do LEFT JOIN here, then that suggests that every month we need is already in the generated table, in which case why do we need this subquery at all?

if we don't need that left join here, then we should probably move the coalese logic up to the aggregate functions.

Yeah, that would be awesome, because right now, the aggregate tables are sparse, so we can't distinguish between when there were $0 in transactions or we don't have data. That's why I did away with this left join - for the months on either end of the range, it reported $0 in transactions, but that might not actually be accurate.

hancush · 2024-10-02T20:52:51Z

Ok, @fgregg, last revision is in. Updated the funds chart to use end of filing period, adding the filing description/s to the tooltip, and fixed tooltip formatting on both charts.

hancush · 2024-10-02T20:56:24Z

camp_fin/templates/base-detail.html

-    for (i = 0; i < donation_trend.length; i++) {
-      donation_trend_f.push([Date.UTC(donation_trend[i][1],donation_trend[i][2]-1,donation_trend[i][3]), donation_trend[i][0]]);
-    }
+    const chartValues = Array.prototype.concat(balance_trend, debt_trend, expend_trend, donation_trend)

-    // format expenditures
-    for (i = 0; i < expend_trend.length; i++) {
-      expend_trend_f.push([Date.UTC(expend_trend[i][1],expend_trend[i][2]-1,expend_trend[i][3]), expend_trend[i][0]]);
-    }
+    const startYear = chartValues.reduce(function(prev, current) {
+      return (prev && prev.year > current.year) ? current : prev
+    }).year
+
+    const endYear = chartValues.reduce(function(prev, current) {
+      return (prev && prev.year > current.year) ? prev : current
+    }).year


Calculate start and end year across all chart data, to ensure consistency.

hancush · 2024-10-02T20:58:22Z

camp_fin/models.py

+            period_end = {
+                "description": description,
+                "year": end_date.year,
+                "month": end_date.month,
+                "day": end_date.day,
+            }
+            balance_trend.append(
+                {
+                    "amount": closing_balance,
+                    **period_end,
+                }
+            )
+            debt_trend.append(
+                {
+                    "amount": total_unpaid_debts * -1,
+                    **period_end,
+                }
+            )


Now returning lists of dictionaries, in order to pass filing description through to the view. I don't see that trends is used anywhere else, besides the deprecated races view.

fgregg

looking good, but why did you turn them back into area charts?

fgregg · 2024-10-03T13:06:40Z

camp_fin/static/js/chart_helper.js

@@ -114,7 +114,7 @@ ChartHelper.netfunds = function(el, title, sourceTxt, yaxisLabel, data, startYea
  return new Highcharts.Chart({
      chart: {
          renderTo: el,
-          type: "line",
+          type: "area",


why did you turn this back to area?

…vailable for a continuous interval

hancush · 2024-10-03T17:08:39Z

camp_fin/management/commands/aggregate_data.py

+            continuous_interval_query = """
+                CREATE MATERIALIZED VIEW {transaction_type}_by_{interval} AS (
+                    WITH {interval}_summaries AS (
+                        {summary_query}
+                    )
+                    SELECT
+                        continuous_interval.entity_id,
+                        continuous_interval.{interval},
+                        COALESCE({interval}_summaries.amount, 0) AS amount
+                    FROM (
+                        SELECT DISTINCT
+                            entity_id,
+                            generate_series(
+                                MIN({interval}) OVER (PARTITION BY entity_id),
+                                MAX({interval}) OVER (PARTITION BY entity_id),
+                                '1 {interval}'
+                            ) AS {interval}
+                        FROM {interval}_summaries
+                    ) continuous_interval
+                    LEFT JOIN {interval}_summaries
+                    USING (entity_id, {interval})
+                    ORDER BY entity_id, {interval}
+                )
+            """


If we have transaction info for a candidate in Jan, March, and April, then we can infer from the absence of transactions that they clocked a total of 0 in Feb. This query creates a continuous sequence of aggregates for each entity, beginning with the first date we have a transaction for and ending with the last date we have a transaction for. Left joining the summaries then fills in intervals with no transactions with 0.

This is useful in particular for the chart. Given sparse data, a step chart will draw a line from point A to C, suggesting a potentially misleading value for B (which gets worse the more values are missing). This registers months without transactions appropriately as 0.

Revise contribution/expenditure chart as a true area chart

3da50d3

hancush commented Oct 1, 2024

View reviewed changes

camp_fin/static/js/chart_helper.js Outdated Show resolved Hide resolved

hancush temporarily deployed to openness-pro-hcg-revise-xr6h82 October 1, 2024 17:04 Inactive

Revert JavaScript formatting

e5796ec

hancush temporarily deployed to openness-pro-hcg-revise-xr6h82 October 1, 2024 17:08 Inactive

hancush requested a review from fgregg October 1, 2024 17:45

Convert area chart to step chart

8c7b59a

Remove unneeded option

26d670d

fgregg approved these changes Oct 2, 2024

View reviewed changes

hancush changed the title ~~Revise contribution/expenditure chart as a true area chart~~ Revise contribution/expenditure chart as step chart Oct 2, 2024

Convert funds by filing into line chart

8a95238

hancush commented Oct 2, 2024

View reviewed changes

hancush requested a review from fgregg October 2, 2024 16:54

hancush changed the title ~~Revise contribution/expenditure chart as step chart~~ Convert available funds / contributions/expenditures from area to step charts Oct 2, 2024

fgregg approved these changes Oct 2, 2024

View reviewed changes

fgregg requested changes Oct 2, 2024

View reviewed changes

hancush added 3 commits October 2, 2024 15:37

Use filing period end for available funds

4338d93

Update chart title

99fca26

Group filings ending on same date

f21a76e

hancush commented Oct 2, 2024

View reviewed changes

hancush requested a review from fgregg October 2, 2024 20:58

fgregg requested changes Oct 3, 2024

View reviewed changes

Update aggregate queries so that contributions and expenditures are a…

16dfa3d

…vailable for a continuous interval

hancush commented Oct 3, 2024

View reviewed changes

hancush marked this pull request as ready for review October 3, 2024 17:21

hancush requested a review from fgregg October 3, 2024 17:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert available funds / contributions/expenditures from area to step charts #216

Convert available funds / contributions/expenditures from area to step charts #216

hancush commented Oct 1, 2024 •

edited

Loading

fgregg commented Oct 1, 2024 •

edited

Loading

hancush commented Oct 1, 2024 •

edited

Loading

fgregg left a comment

fgregg Oct 1, 2024

fgregg Oct 2, 2024

hancush Oct 2, 2024

fgregg Oct 2, 2024

hancush Oct 2, 2024

fgregg Oct 2, 2024

hancush commented Oct 2, 2024

hancush Oct 2, 2024

fgregg Oct 2, 2024

hancush Oct 2, 2024

fgregg left a comment

fgregg Oct 2, 2024

fgregg Oct 2, 2024

hancush Oct 2, 2024

hancush commented Oct 2, 2024

hancush Oct 2, 2024

hancush Oct 2, 2024

fgregg left a comment

fgregg Oct 3, 2024

hancush Oct 3, 2024

hancush Oct 3, 2024 •

edited

Loading

hancush Oct 3, 2024

Convert available funds / contributions/expenditures from area to step charts #216

Are you sure you want to change the base?

Convert available funds / contributions/expenditures from area to step charts #216

Conversation

hancush commented Oct 1, 2024 • edited Loading

Overview

Demo

Testing instructions

fgregg commented Oct 1, 2024 • edited Loading

chart style

Why data isn't loading?

hancush commented Oct 1, 2024 • edited Loading

fgregg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hancush commented Oct 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fgregg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hancush commented Oct 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fgregg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hancush Oct 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hancush commented Oct 1, 2024 •

edited

Loading

fgregg commented Oct 1, 2024 •

edited

Loading

hancush commented Oct 1, 2024 •

edited

Loading

hancush Oct 3, 2024 •

edited

Loading