1.42.9

Merged PRs

dolt

• 8219: Bug fix: events should remain enabled after sql-server restarts
  Fixing an event scheduler bug that caused events to be disabled and not run when the sql-server is restarted.
  Fixes #8167
• 8209: add constraint violations to dolt status
  fixes: #8114

go-mysql-server

• 2623: Fix anti-join correctess bug
  We had some strange logic for accepting a join anti-match, ripped it out and everything seems to be working correctly now.
• 2620: implement name_const function
  MySQL docs: https://dev.mysql.com/doc/refman/8.4/en/miscellaneous-functions.html#function_name-const
• 2618: More aggressively elide IN filters used for indexed lookups
  re: #8215
• 2617: More QueryProps, missed max1rowiter usage
• 2613: Query properties rule filtering
  Edit most of the analyzer interfaces to pass a new context object that accumulates query specific properties. Currently the object is called QueryFlags, and accumulates information about the query to inform better rule filtering and more efficient spooling strategies.
  The change that has the biggest effect on oltp_point_select perf is the sql.QFlagMax1Row setting, which lets us skip the default results iter boilerplate when we're only returning one row. Added a couple other skips for rules that are easy to whitelist correctly and show prominently on CPU profiles, like aggregations and subqueries.

vitess

• 361: adding instant as non-reserved keyword
  The INSTANT keyword isn't in the MySQL docs, but it is a non reserved keyword.
  MySQL Docs: https://dev.mysql.com/doc/refman/8.4/en/keywords.html
  fixes: #8220
• 360: Bug fix: Preserve sign for integers in prepared statements
  Bound integer values for prepared statements are parsed from the wire and packaged into int64 values that are then passed to the SQL engine to execute with the prepared statement. For int8, int16, int24, and int32 types those bytes from the wire weren't getting cast to the correct type first, before they were cast to int64, which meant if the signed bit was set, the value was interpreted incorrectly.
  Customer issue: #8085

Closed Issues

• 8220: Correctness Issue, queries run differently on MySQL and Dolt
• 8206: Unknown JSON struct tag in schema_marshaling.go
• 8167: Scheduled events fail to execute after a server restart
• 8114: dolt_status table unhelpful during merge when there are constraint violations.

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	2.07	3.02	1.5
groupby_scan	13.7	17.63	1.3
index_join	1.37	2.66	1.9
index_join_scan	1.3	2.18	1.7
index_scan	34.95	54.83	1.6
oltp_point_select	0.18	0.3	1.7
oltp_read_only	3.49	5.99	1.7
select_random_points	0.34	0.65	1.9
select_random_ranges	0.39	0.83	2.1
table_scan	34.95	55.82	1.6
types_table_scan	75.82	144.97	1.9
reads_mean_multiplier			1.7

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.13	5.99	0.7
oltp_insert	3.82	3.02	0.8
oltp_read_write	8.58	12.08	1.4
oltp_update_index	3.89	3.02	0.8
oltp_update_non_index	3.89	2.97	0.8
oltp_write_only	5.47	6.21	1.1
types_delete_insert	7.7	6.55	0.9
writes_mean_multiplier			0.9

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	98.12	38.64	2.5
tpcc_tps_multiplier			2.5

Overall Mean Multiple	1.70

1.42.8

Merged PRs

dolt

• 8211: Bug fix: binlog replication: logging levels, use MySQL's older date serialization format
  Small bug fixes for binlog replication while testing with the python-mysql-replication library:
  • Adjusting DEBUG logging to be less verbose
  • Using MySQL's older date serialization format for compatibility
  • Adding a nil check for gtidPosition
• 8207: removing unconditional recursive calls
  fixes:
  • #8206
  • #8205
  • #8204
  • #8203
  • #8202
• 8201: chore: update lib:eventsapi 2022-> 2024
  Ran into issues when I ran:

  go install github.com/dolthub/dolt/go/cmd/dolt@main
  # github.com/dolthub/dolt/go/cmd/dolt/commands
  /go/pkg/mod/github.com/dolthub/dolt/[email protected]/cmd/dolt/commands/cherry-pick.go:79:19: undefined: eventsapi.ClientEventType_CHERRY_PICK
  /go/pkg/mod/github.com/dolthub/dolt/[email protected]/cmd/dolt/commands/dump.go:112:19: undefined: eventsapi.ClientEventType_DUMP
  /go/pkg/mod/github.com/dolthub/dolt/[email protected]/cmd/dolt/commands/filter-branch.go:95:19: undefined: eventsapi.ClientEventType_FILTER_BRANCH
  /go/pkg/mod/github.com/dolthub/dolt/[email protected]/cmd/dolt/commands/gc.go:76:19: undefined: eventsapi.ClientEventType_GARBAGE_COLLECTION
  /go/pkg/mod/github.com/dolthub/dolt/[email protected]/cmd/dolt/commands/migrate.go:74:19: undefined: eventsapi.ClientEventType_MIGRATE
  /go/pkg/mod/github.com/dolthub/dolt/[email protected]/cmd/dolt/commands/profile.go:89:19: undefined: eventsapi.ClientEventType_PROFILE
  /go/pkg/mod/github.com/dolthub/dolt/[email protected]/cmd/dolt/commands/rebase.go:74:19: undefined: eventsapi.ClientEventType_REBASE
  /go/pkg/mod/github.com/dolthub/dolt/[email protected]/cmd/dolt/commands/reflog.go:63:19: undefined: eventsapi.ClientEventType_REFLOG
  /go/pkg/mod/github.com/dolthub/dolt/[email protected]/cmd/dolt/commands/show.go:72:19: undefined: eventsapi.ClientEventType_SHOW
  /go/pkg/mod/github.com/dolthub/dolt/[email protected]/cmd/dolt/commands/tag.go:81:19: undefined: eventsapi.ClientEventType_TAG
  /go/pkg/mod/github.com/dolthub/dolt/[email protected]/cmd/dolt/commands/tag.go:81:19: too many errors
  

  Realized the eventsapi library version in go.mod is outdated
• 8187: Feature: Auto-start binlog replication on server restart
  Once replication has been started with START REPLICA;, replication now automatically restarts when the server is restarted. If replication is stopped with STOP REPLICA;, it is not restarted automatically on server restart. This matches MySQL's behavior.
  This PR also includes a few other small improvements to binlog replication:
  • allows 'show replica status' before replication is started
  • moves logging of binlog messages from DEBUG to TRACE level
  • removes several references to GetRunningServer() global function
    Resolves #8168
• 8184: [dtables] fix from_commit index behavior
• 8182: go/libraries/doltcore/env/actions: Improve error message when a clone fails mid-download.
  Fixes a dropped error return in the clone implementation.
• 8181: Bug fix: resolving case-sensitive table name for TableEditor
  A MySQL primary server may send table names in TableMap binlog events that do not match the case of the table name. We use a case-insensitive table name lookup when getting the table's schema, but not when creating the TableWriter that applies row updates/inserts/deletes. This change fixes that so that mixed case table names can replicate correctly.
• 8177: Bump fast-xml-parser and @aws-sdk/client-ses in /.github/actions/ses-email-action
  Bumps fast-xml-parser to 4.4.1 and updates ancestor dependency @aws-sdk/client-ses. These dependencies need to be updated together.
  Updates fast-xml-parser from 4.2.5 to 4.4.1
  Changelog

  Sourced from fast-xml-parser's changelog.

  Note: If you find missing information about particular minor version, that version must have been changed without any functional change in this library.

  4.4.1 / 2024-07-28

  • v5 fix: maximum length limit to currency value
  • fix #634: build attributes with oneListGroup and attributesGroupName (#653)(By Andreas Naziris)
  • fix: get oneListGroup to work as expected for array of strings (#662)(By Andreas Naziris)

  4.4.0 / 2024-05-18

  • fix #654: parse attribute list correctly for self closing stop node.
  • fix: validator bug when closing tag is not opened. (#647) (By Ryosuke Fukatani)
  • fix #581: typings; return type of tagValueProcessor & attributeValueProcessor (#582) (By monholm)

  4.3.6 / 2024-03-16

  • Add support for parsing HTML numeric entities (#645) (By Jonas Schade )

  4.3.5 / 2024-02-24

  • code for v5 is added for experimental use

  4.3.4 / 2024-01-10

  • fix: Don't escape entities in CDATA sections (#633) (By wackbyte)

  4.3.3 / 2024-01-10

  • Remove unnecessary regex

  4.3.2 / 2023-10-02

  • fix jObj.hasOwnProperty when give input is null (By Arda TANRIKULU)

  4.3.1 / 2023-09-24

  • revert back "Fix typings for builder and parser to make return type generic" to avoid failure of existing projects. Need to decide a common approach.

  4.3.0 / 2023-09-20

  • Fix stopNodes to work with removeNSPrefix (#607) (#608) (By [Craig Andrews]https://github.com/candrews))
  • Fix #610 ignore properties set to Object.prototype
  • Fix typings for builder and parser to make return type generic (By Sarah Dayan)

  4.2.7 / 2023-07-30

  • Fix: builder should set text node correctly when only textnode is present (#589) (By qianqing)
  • Fix: Fix for null and undefined attributes when building xml (#585) (#598). A null or undefined value should be ignored. (By Eugenio Ceschia)

  4.2.6 / 2023-07-17

  • Fix: Remove trailing slash from jPath for self-closing tags (#595) (By Maciej Radzikowski)

  4.2.5 / 2023-06-22

  • change code implementation

  4.2.4 / 2023-06-06

  • fix security bug

  4.2.3 / 2023-06-05

  • fix security bug

  ... (truncated)

  Commits

  • d40e29c update package d...

1.42.7

Merged PRs

dolt

• 8174: check for pg_catalog table when resolving a table
  Fixes:
  dolthub/doltgresql#442
  dolthub/doltgresql#513
• 8172: Reveal dolt archive command
  Move the dolt admin archive command to dolt archive
• 8164: archive guard rails
  Add features to ensure that archive files are not used when running a remotesrv or when pushing backups.
  Also provide the --revert flag which can return a database back to the classic format.
• 8137: Dolt index searchable, lookup caching
  Two main changes
  • cache schema hashes on root values, so that we don't have to a table lookup to get a schema hash
  • Implement sql.IndexSearchable in a way that we cache strict key lookups for a given table schema. The lifecycle for these objects is the span between ALTER TABLE statements

go-mysql-server

• 2610: use datetime precision in information_schema.columns.datetime_precision
  When determining if a schema change occurred, one of the tables Prisma looks at is the information_schema.columns.
  Here, we incorrectly mark all datetime and timestamp columns as with a precision of 6.
  If a table has type DATETIME(3), Prisma would think there was a schema change, and perform a migration when one isn't needed.
  This PR addresses this issue by having the information_schema.columns table accurately reflect the datetime preicision of the columns,
  fixes: #8173
• 2604: Index searchable edits
  We previously added support for integrators choosing their own indexes with an sql.IndexSearchable interface. This was for a customer use case. This PR expands the interface to let Dolt cache information about strict key lookups.
  The motivation is that (1) strict key lookups will always be the best-case scenario result of index costing, (2) caching this information in-between ALTER statements is usually a long enough lifecycle for the overhead to be worth it.
  I added a streamlined range builder as part of this optimization that only accepts literal values in the order expected by the target lookup. The user of this range builder takes responsibility for feeding the values in the correct order. As a result, we sidestep expensive string formatting, map creating, and map lookups during range building.
  Follow-on fixes to functional dependencies permuted plans a bit more. Inner joins are chosen more frequently in some of our test plans now that we're reflecting strict key max-1-row cardinalities.

vitess

• 360: Bug fix: Preserve sign for integers in prepared statements
  Bound integer values for prepared statements are parsed from the wire and packaged into int64 values that are then passed to the SQL engine to execute with the prepared statement. For int8, int16, int24, and int32 types those bytes from the wire weren't getting cast to the correct type first, before they were cast to int64, which meant if the signed bit was set, the value was interpreted incorrectly.
  Customer issue: #8085
• 359: fix detection of multi-statements in ComPrepare
  Currently, preparing multi-statements is not supported; so we can't prepare a query like select ?; select ?;.
  However, the check for this condition just looked for any characters after the first ;, which meant that queries like select ?; \n would incorrectly throw an error.
  This was made apparent using the Prisma ORM, which runs the query:

  SELECT TABLE_NAME AS view_name, VIEW_DEFINITION AS view_sql
  FROM INFORMATION_SCHEMA.VIEWS
  WHERE TABLE_SCHEMA = ?;

  The above query ends in a newline character.
  The fix is to use SplitStatementToPieces(), which trims these white space characters, and check if there's exactly one piece; this was taken from the vitessio repo: https://github.com/vitessio/vitess/blob/main/go/mysql/conn.go#L1204
  fixes #8157

Closed Issues

• 4127: Support for date and time literals
• 8173: prisma migrate dev, generating changes to columns of the date type in loop
• 7200: Weird behavior for datetime literal syntax.
• 8157: Error when using Prisma ORM: "Error: unknown error: can not prepare multiple statements"

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	2.11	3.02	1.4
groupby_scan	13.22	17.01	1.3
index_join	1.34	2.81	2.1
index_join_scan	1.27	2.22	1.7
index_scan	38.25	54.83	1.4
oltp_point_select	0.18	0.44	2.4
oltp_read_only	3.49	7.56	2.2
select_random_points	0.34	0.78	2.3
select_random_ranges	0.39	0.92	2.4
table_scan	38.94	55.82	1.4
types_table_scan	80.03	144.97	1.8
reads_mean_multiplier			1.9

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.13	6.09	0.7
oltp_insert	3.82	3.02	0.8
oltp_read_write	8.58	13.7	1.6
oltp_update_index	3.89	3.07	0.8
oltp_update_non_index	3.89	3.02	0.8
oltp_write_only	5.47	6.32	1.2
types_delete_insert	7.7	6.67	0.9
writes_mean_multiplier			1.0

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	99.65	36.43	2.7
tpcc_tps_multiplier			2.7

Overall Mean Multiple	1.87

1.42.6

Merged PRs

dolt

• 8166: [kvexec] Fix panic in non-covering strict lookup
  I missed a test case in nonCovLaxSecondaryLookupGen. I tried to add more tests surrounding the specific panic query here: dolthub/go-mysql-server#2607.
  Also rename the rowexec to kvexec.
• 8139: Feature: Log binlog events to disk
  Major changes:
  • Moved binlogging initialization code from sqlengine to server, so that bin logging only happens when in sql-server mode.
  • binlogProducer now sends events to the new logManager type that writes binlog events to files on disk. binlogStreamer now reads events from those logs and streams them to replicas (instead of receiving events directly from binlogProducer).
  • DoltBinlogPrimaryController now validates that the missing GTIDs from a replica are available in the binlog files and sends an error if the primary doesn't have enough binlog data on disk to get a replica in sync.

go-mysql-server

• 2606: [memo] assume self-join stats cardinality continuity
  Self-join stats estimation is particularly expensive because all of the buckets exactly overlap. If the index is unique, the cardinality distribution will not change. If the index is non-unique, the cardinality will expand proportional to rowCount/distinctCount.

  before
  BenchmarkOltpJoinScan-12    	    1766	    694524 ns/op	  462834 B/op	    8240 allocs/op
  after
  BenchmarkOltpJoinScan-12    	    2460	    481166 ns/op	  193569 B/op	    7129 allocs/op
  

  sysbench perf here: #8159

vitess

• 359: fix detection of multi-statements in ComPrepare
  Currently, preparing multi-statements is not supported; so we can't prepare a query like select ?; select ?;.
  However, the check for this condition just looked for any characters after the first ;, which meant that queries like select ?; \n would incorrectly throw an error.
  This was made apparent using the Prisma ORM, which runs the query:

  SELECT TABLE_NAME AS view_name, VIEW_DEFINITION AS view_sql
  FROM INFORMATION_SCHEMA.VIEWS
  WHERE TABLE_SCHEMA = ?;

  The above query ends in a newline character.
  The fix is to use SplitStatementToPieces(), which trims these white space characters, and check if there's exactly one piece; this was taken from the vitessio repo: https://github.com/vitessio/vitess/blob/main/go/mysql/conn.go#L1204
  fixes #8157
• 358: Feature: parser support for PURGE BINARY LOGS syntax
  https://dev.mysql.com/doc/refman/8.4/en/purge-binary-logs.html
• 357: Bug fix: Send an error response when the server fails to handle COM_BINLOG_DUMP_GTID
  A MySQL primary needs to be able to send back an error response when handling the COM_BINLOG_DUMP_GTID command. Previously, when the integrator returned an error, it was logged in the primary server logs, but it was not being sent back to the replica who sent the command. This change causes an error packet to be sent to the replica, containing the details of the error the integrator returned.
  This change is difficult to test in isolation, but I have tests in dolt that will exercise this codepath.

Closed Issues

• 8157: Error when using Prisma ORM: "Error: unknown error: can not prepare multiple statements"
• 6816: subquery insert into not null column throws error

1.42.5

Merged PRs

dolt

• 8162: Better error message when attempting to push from a shallow clone
  Push from a shallow clone is possible, but we should message better when it's not possible. In order to do this we need to error better from the Generational Chunk Store with a custom error.
  Fixes: #8156
• 8158: fix no-op merge msg in cli
  Fixes: #8148
• 8154: When removing from a secondary index during merging, use the pre-merge ordinal mapping, not the merged mapping, to identify the secondary key to be removed.
  Basically, we have a bug where dropping a column on the remote side of a merge can interfere with updating secondary indexes.
  Example:
  Base Schema: (pk INT PRIMARY KEY, a TINYINT, b INT, UNIQUE KEY b_idx (b))
  "theirs" drops column a
  Merged Schema: (pk INT PRIMARY KEY, b INT, UNIQUE KEY b_idx (b))
  In effect, the merger would see that in the final table, b is the second column. Then, when updating b_idx for each resolved row, it would use the second column of "ours" to find the index entry to remove. But this is incorrect, because the second column of "ours" is a.
  If the user is lucky, these two columns will be different sizes and the merger will panic. But if the two columns are the same size, merge proceeds with an incorrect value. This will cause it to either fail to remove the old row from the secondary index, or remove a different row. Either way, the secondary index is now incorrect.
• 8150: fix warnings in dolt sql shell
  This PR fixes a bug where warnings were incorrectly being suppressed in the dolt sql shell.
  The bug is caused by the shell making queries to check if the working set is dirty, what database we're on, and what branch we're on. The fix is to set a special flag in the session to not clear the warnings for those specific queries.
  companion pr: dolthub/go-mysql-server#2605
  fixes: #8016
• 8124: [stats] limit stats boostrap to server start
  fixes: #8123

go-mysql-server

• 2606: [memo] assume self-join stats cardinality continuity
  Self-join stats estimation is particularly expensive because all of the buckets exactly overlap. If the index is unique, the cardinality distribution will not change. If the index is non-unique, the cardinality will expand proportional to rowCount/distinctCount.

  before
  BenchmarkOltpJoinScan-12    	    1766	    694524 ns/op	  462834 B/op	    8240 allocs/op
  after
  BenchmarkOltpJoinScan-12    	    2460	    481166 ns/op	  193569 B/op	    7129 allocs/op
  

  sysbench perf here: #8159
• 2605: add lock to prevent warnings from being cleared
  This PR adds two functions to BaseSession that toggle a boolean, so integrators can prevent warnings from being cleared.
  This is mostly useful for dolt sql shell.
  addresses #8016

Closed Issues

• 8123: stats refresh warning a little too zealous
• 8156: Unknown push Error When Pushing Large DB to Remote
• 8016: dolt sql suppresses warnings.
• 8148: Ghost Commit Error On Empty Merge
• 7638: Syntax Error Occurs When Using AS Clause with ON DUPLICATE KEY UPDATE

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	2.11	2.97	1.4
groupby_scan	13.46	17.32	1.3
index_join	1.37	2.81	2.1
index_join_scan	1.3	2.22	1.7
index_scan	34.33	53.85	1.6
oltp_point_select	0.18	0.46	2.6
oltp_read_only	3.49	7.7	2.2
select_random_points	0.34	0.77	2.3
select_random_ranges	0.39	0.89	2.3
table_scan	34.33	54.83	1.6
types_table_scan	74.46	142.39	1.9
reads_mean_multiplier			1.9

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.13	6.09	0.7
oltp_insert	3.82	3.02	0.8
oltp_read_write	8.58	13.95	1.6
oltp_update_index	3.82	3.07	0.8
oltp_update_non_index	3.89	3.02	0.8
oltp_write_only	5.37	6.43	1.2
types_delete_insert	7.7	6.67	0.9
writes_mean_multiplier			1.0

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	99.43	34.46	2.9
tpcc_tps_multiplier			2.9

Overall Mean Multiple	1.93

1.42.4

Merged PRs

dolt

• 8155: Bug fix: loading persisted @@server_id from config.json
  This fixes binlog replicas loading @@server_id values persisted in config.json so that they are properly converted from strings to the right type.
  Also added extra connection error logging
• 8153: make tags case insensitive
  This PR makes it so tags are case-insensitive.
  Additionally, fixes a display error in the sql shell.
  fixes #8147
• 8152: Revive ability to read really old dolt_schema tables
  We have several old databases which we don't touch but are public on DoltHub. These databases present an error to users with the removal of the migration ability in 1.42.0. The change will bring them back.
• 8135: Build Archives more robustly
  This change is long in coming. Several findings from building a large user database are incorporated here. Specifically:
  • No longer storing all chunks in memory
  • Cancellable process with ^C
  • Progress reporting for major stages of the build
  • No chunk grouping by default, --group-chunks to enable. Pathological cgo dictionary building problem needs to be fixed before we enable by default.
  • Writing a useful metadata json block with dolt version and origin table file ID.

go-mysql-server

• 2605: add lock to prevent warnings from being cleared
  This PR adds two functions to BaseSession that toggle a boolean, so integrators can prevent warnings from being cleared.
  This is mostly useful for dolt sql shell.
  addresses #8016
• 2603: [memo] reorder should add new plans to intermediate expr join child
  There was a bug where we'd add reordered join plans to project or distinct nodes, rather than their join children. Code comment explains more clearly how this works.
• 2602: disallow forward slash in database name
  fixes #8126
• 2600: Support information_schema.columns hook for doltgres
• 2593: custom row exec
  Additions for custom row operators on Dolt side: #8072

Closed Issues

• 8147: Tag releases: database not found

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	2.07	2.97	1.4
groupby_scan	13.46	17.01	1.3
index_join	1.34	2.81	2.1
index_join_scan	1.27	2.57	2.0
index_scan	34.33	54.83	1.6
oltp_point_select	0.18	0.45	2.5
oltp_read_only	3.49	7.56	2.2
select_random_points	0.34	0.75	2.2
select_random_ranges	0.39	0.89	2.3
table_scan	34.95	55.82	1.6
types_table_scan	75.82	142.39	1.9
reads_mean_multiplier			1.9

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.13	6.09	0.7
oltp_insert	3.82	3.02	0.8
oltp_read_write	8.58	13.95	1.6
oltp_update_index	3.89	3.07	0.8
oltp_update_non_index	3.89	3.02	0.8
oltp_write_only	5.47	6.32	1.2
types_delete_insert	7.7	6.67	0.9
writes_mean_multiplier			1.0

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	99.27	34.31	2.9
tpcc_tps_multiplier			2.9

Overall Mean Multiple	1.93

1.42.3

Merged PRs

dolt

• 8144: Don't panic when performing a GC on a shallow clone
• 8143: Add support for visualizing prolly tree and blob messages in dolt show
  This PR does a couple things:
  1. dolt show #address can now display the internals of a ProllyTreeMap message (typically used for storing indexes). Previously, only splunk/noms show could do this.
  2. Both dolt show and splunk can now display the contents of a Blob message.
  3. If a ProllyTreeMap leaf node contains a value that is itself an address (example: the value of text and json columns), that value is shown as a human readable address, which can be fed back into dolt show or splunk to explore the whole tree.
• 8141: support STAGED as commit hash
  This PR adds support for STAGED as a commit_hash when filtering dolt diff system tables.
  fixes for #7978
• 8140: Continue to support writes when archives are in play
  Error in hasMany effectively prevented writes after moving database to archives.
• 8084: enable key range iter
  There was an issue merging #8025 ontop of a revert. Enable the key iteration optimization.
• 8072: [rowexec] dolt-side lookup execution operator
  This PR adds custom Dolt execution operators for lookup joins. When building an execution plan, we try to replace joinIter with a Dolt equivalent that inlines the key building and map get. This is a lot faster than repeatedly building the secondary iterator and materializing sql.Rows in-between lookups.
  The main downside is that this PR hoists filters in join children to after materializing lookup join rows.
  This brings index_join from 5.18 ms/query to 2.64 ms/q, which will be about 2.0x MySQL's latency.
  This PR falls short of some aspiration goals:
  • We hoist table filters until after the final join row is built because we don't have a way to call scalar expressions on val.Tuple yet. There are edge case queries that might be dramatically slower because of this. To fix this, we could need to convert sql.Expression filters into a format that we could execute on val.Tuple KV pairs.
  • We do not yet try to to optimize consecutive lookup joins. I'm not sure if a materialization block would be better represented iteratively or recursively beyond a simple string of lookups. A lot of interfaces and indexing considerations to think about there.
    Safety comments:
  • we fallback to GMS when lookup source/dest keys are not prolly.Encoding compatible
  • the source iterators are the same as what we used before, but without projection mapping to sql.Rows. The keyless iterator required a change to return duplicate rows at the KV layer (vs the sql layer).
  • the secondary iterators are a generalization of what we currently use, but return KV pairs instead of rows
  • projection mapping is the same but generalized to merge an arbitrary list of KV pairs after the join
    There are extra tests here: dolthub/go-mysql-server#2593

go-mysql-server

• 2598: Adding the binlog_expire_logs_seconds system variable
  MySQL @@binlog_expire_logs_seconds reference docs
• 2596: Fix information_schema.columns for databases with schemas
  Missed this table in my original PR
• 2593: custom row exec
  Additions for custom row operators on Dolt side: #8072

Closed Issues

• 8126: CREATE DATABASE statements should not create databases hierarchicaly
• 7698: Incorrect return typefor UNIX_TIMESTAMP
• 7978: Should be able to use "STAGED" as commits in dolt_commit_diff_[tablename]
• 8131: Docker Mysql Server Resets When Connecting Immediately After Launch

1.42.2

Merged PRs

dolt

• 8132: Detect reading from non-indexed JSON documents and store it in a LazyJSONDocument instead of an IndexedJSONDocument.
  This adds tests that we properly read JSON documents written before the migration to the new indexed JSON storage format. When loading such documents from storage, we should represent them as a LazyJSONDocument instead of an IndexedJSONDocument. This ensures we don't accidentally use the optimized versions of the JSON functions when there aren't any keys in the prolly tree to guide the optimization.

go-mysql-server

• 2596: Fix information_schema.columns for databases with schemas
  Missed this table in my original PR
• 2595: Adding user name and host length validation to CREATE USER
  This change matches MySQL's behavior of limiting user names to 32 chars and host names to 255 chars. Attempting to create a user with a name or host longer than that limit now returns a validation error.
  Customer issue: #8120

Performance

Read Tests	MySQL	Dolt	Multiple
covering_index_scan	2.07	3.02	1.5
groupby_scan	13.22	17.32	1.3
index_join	1.37	5.37	3.9
index_join_scan	1.27	2.57	2.0
index_scan	34.33	54.83	1.6
oltp_point_select	0.18	0.46	2.6
oltp_read_only	3.49	7.7	2.2
select_random_points	0.34	0.75	2.2
select_random_ranges	0.39	0.9	2.3
table_scan	34.33	56.84	1.7
types_table_scan	74.46	144.97	1.9
reads_mean_multiplier			2.1

Write Tests	MySQL	Dolt	Multiple
oltp_delete_insert	8.13	6.09	0.7
oltp_insert	3.82	3.02	0.8
oltp_read_write	8.58	13.95	1.6
oltp_update_index	3.89	3.07	0.8
oltp_update_non_index	3.89	3.02	0.8
oltp_write_only	5.37	6.32	1.2
types_delete_insert	7.7	6.67	0.9
writes_mean_multiplier			1.0

TPC-C TPS Tests	MySQL	Dolt	Multiple
tpcc-scale-factor-1	99.02	35.06	2.8
tpcc_tps_multiplier			2.8

Overall Mean Multiple	1.97

1.42.1

Merged PRs

dolt

• 8128: Bug fix: dolt_reset() should commit the active transaction
  Fixes: #6272
• 8127: cache table list
  RootValue caches a hash of its table list, which is invalidated when we update the table address list. The hash is used to cache a map of table names in the root value and expedites table retrieval during planbuilder. The cache could be used elsewhere for performance in the future.
  dolthub/doltgresql#503

Closed Issues

• 6272: CALL DOLT_RESET('--hard') does not implicitly commit the transaction
• 8120: CREATE USER allows username over 32 chars, then SELECT throws error
• 8117: REGEXP isn't collation aware

1.42.0

Backwards incompatible changes in this release:

• The dolt_schemas and dolt_procedures tables now return empty results and disallow edits. In previous releases, select * from either table would result in an error if there was no data. These tables are internal tables to Dolt which are intended to be managed by Dolt itself, and not modified by users. If your application is currently writing to these tables, it will no longer be able to do so and changes will be required in order to upgrade your database.
• Username and host names are now restricted in length (32 and 255 respectively) to match the behavior of MySQL. Attempting to create a user with a name or host longer than that limit now returns a validation error. Existing users and hostnames which are longer than those limits are not affected.

Per Dolt’s versioning policy, this is a minor version bump because these changes may impact existing applications. Please reach out to us on GitHub or Discord if you have questions or need help with any of these changes.

Merged PRs

dolt

• 8116: Return empty dolt_procedures and dolt_schemas tables
  Previously dolt_procedures and dolt_schemas tables could not be selected from if there were not procedures or views. That resulted in errors in the when a dolt CLI client attempted to run dolt diff. This was previously swallowed by the client, but it made the server logs dirty for no reason.
  Now, select * from dolt_schemas will result in an empty result set.
  I did not add new tests for this as several existing tests were broken by this behavior change. I fixed those instead. Due to change in existinging behavior which people may depend on, this requires a minor version bump.

go-mysql-server

• 2595: Adding user name and host length validation to CREATE USER
  This change matches MySQL's behavior of limiting user names to 32 chars and host names to 255 chars. Attempting to create a user with a name or host longer than that limit now returns a validation error.
  Customer issue: #8120
• 2594: Fixed REGEXP
  This fixes the case-sensitivity issue found in: #8117
  Although we had moved REGEXP_LIKE to the ICU engine, we forgot to also move REGEXP, which is a synonym for REGEXP_LIKE according to the docs. This makes that change, and also completely removes all remnants of the old regex code.

Closed Issues

• 8110: ARM64 Docker Image does not contain arm binaries