Skip to content

Commit ba28830

Browse files
jeremyevansjeremyevans
committed
Add pg_auto_parameterize_in_array extension, for converting IN/NOT IN to = ANY or != ALL for more types
When I originally developed the pg_auto_parameterize, I only handled integer arrays in order to avoid having that extension depend on the pg_array extension. This extension depends on both and adds support for the additional types.
1 parent c33ec61 commit ba28830

File tree

6 files changed

+351
-2
lines changed

6 files changed

+351
-2
lines changed

‎CHANGELOG

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
=== master
22

3+
* Add pg_auto_parameterize_in_array extension, for converting IN/NOT IN to = ANY or != ALL for more types (jeremyevans)
4+
35
* Fix literalization of infinite and NaN float values in PostgreSQL array bound variables (jeremyevans)
46

57
=== 5.71.0 (2023-08-01)

‎doc/testing.rdoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -176,7 +176,7 @@ SEQUEL_MODEL_PREPARED_STATEMENTS :: Use the prepared_statements plugin when runn
176176
SEQUEL_MODEL_THROW_FAILURES :: Use the throw_failures plugin when running the specs
177177
SEQUEL_NO_CACHE_ASSOCIATIONS :: Don't cache association metadata when running the specs
178178
SEQUEL_NO_PENDING :: Don't skip any specs, try running all specs (note, can cause lockups for some adapters)
179-
SEQUEL_PG_AUTO_PARAMETERIZE :: Use the pg_auto_parameterize extension when running the postgres specs
179+
SEQUEL_PG_AUTO_PARAMETERIZE :: Use the pg_auto_parameterize extension when running the postgres specs. Value can be +in_array+ to test the pg_auto_parameterize_in_array extension, and +in_array_string+ to test the pg_auto_parameterize_in_array extension with the +:treat_in_string_list_as_text_array+ Database option set.
180180
SEQUEL_PG_TIMESTAMPTZ :: Use the pg_timestamptz extension when running the postgres specs
181181
SEQUEL_PRIMARY_KEY_LOOKUP_CHECK_VALUES :: Use the primary_key_lookup_check_values extension when running the adapter or integration specs
182182
SEQUEL_QUERY_PER_ASSOCIATION_DB_0_URL :: Run query-per-association integration tests with multiple databases (all 4 must be set to run)

‎lib/sequel/extensions/pg_auto_parameterize_in_array.rb

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# frozen-string-literal: true
2+
#
3+
# The pg_auto_parameterize_in_array extension builds on the pg_auto_parameterize
4+
# extension, adding support for handling additional types when converting from
5+
# IN to = ANY and NOT IN to != ALL:
6+
#
7+
# DB[:table].where(column: [1.0, 2.0, ...])
8+
# # Without extension: column IN ($1::numeric, $2:numeric, ...) # bound variables: 1.0, 2.0, ...
9+
# # With extension: column = ANY($1::numeric[]) # bound variables: [1.0, 2.0, ...]
10+
#
11+
# This prevents the use of an unbounded number of bound variables based on the
12+
# size of the array, as well as using different SQL for different array sizes.
13+
#
14+
# The following types are supported when doing the conversions, with the database
15+
# type used:
16+
#
17+
# Float :: if any are infinite or NaN, double precision, otherwise numeric
18+
# BigDecimal :: numeric
19+
# Date :: date
20+
# Time :: timestamp (or timestamptz if pg_timestamptz extension is used)
21+
# DateTime :: timestamp (or timestamptz if pg_timestamptz extension is used)
22+
# Sequel::SQLTime :: time
23+
# Sequel::SQL::Blob :: bytea
24+
#
25+
# String values are also supported using the +text+ type, but only if the
26+
# +:treat_string_list_as_text_array+ Database option is used. This is because
27+
# treating strings as text can break programs, since the type for
28+
# literal strings in PostgreSQL is +unknown+, not +text+.
29+
#
30+
# The conversion is only done for single dimensional arrays that have more
31+
# than two elements, where all elements are of the same class (other than
32+
# nil values).
33+
#
34+
# Related module: Sequel::Postgres::AutoParameterizeInArray
35+
36+
module Sequel
37+
module Postgres
38+
# Enable automatically parameterizing queries.
39+
module AutoParameterizeInArray
40+
# Transform column IN (...) expressions into column = ANY($)
41+
# and column NOT IN (...) expressions into column != ALL($)
42+
# using an array bound variable for the ANY/ALL argument,
43+
# if all values inside the predicate are of the same type and
44+
# the type is handled by the extension.
45+
# This is the same optimization PostgreSQL performs internally,
46+
# but this reduces the number of bound variables.
47+
def complex_expression_sql_append(sql, op, args)
48+
case op
49+
when :IN, :"NOT IN"
50+
l, r = args
51+
if auto_param?(sql) && (type = _bound_variable_type_for_array(r))
52+
if op == :IN
53+
op = :"="
54+
func = :ANY
55+
else
56+
op = :!=
57+
func = :ALL
58+
end
59+
args = [l, Sequel.function(func, Sequel.pg_array(r, type))]
60+
end
61+
end
62+
63+
super
64+
end
65+
66+
private
67+
68+
# The bound variable type string to use for the bound variable array.
69+
# Returns nil if a bound variable should not be used for the array.
70+
def _bound_variable_type_for_array(r)
71+
return unless Array === r && r.size > 1
72+
classes = r.map(&:class)
73+
classes.uniq!
74+
classes.delete(NilClass)
75+
return unless classes.size == 1
76+
77+
klass = classes[0]
78+
if klass == Integer
79+
# This branch is not taken on Ruby <2.4, because of the Fixnum/Bignum split.
80+
# However, that causes no problems as pg_auto_parameterize handles integer
81+
# arrays natively (though the SQL used is different)
82+
"int8"
83+
elsif klass == String
84+
"text" if db.typecast_value(:boolean, db.opts[:treat_string_list_as_text_array])
85+
elsif klass == BigDecimal
86+
"numeric"
87+
elsif klass == Date
88+
"date"
89+
elsif klass == Time
90+
@db.cast_type_literal(Time)
91+
elsif klass == Float
92+
# PostgreSQL treats literal floats as numeric, not double precision
93+
# But older versions of PostgreSQL don't handle Infinity/NaN in numeric
94+
r.all?{|v| v.nil? || v.finite?} ? "numeric" : "double precision"
95+
elsif klass == Sequel::SQLTime
96+
"time"
97+
elsif klass == DateTime
98+
@db.cast_type_literal(DateTime)
99+
elsif klass == Sequel::SQL::Blob
100+
"bytea"
101+
end
102+
end
103+
end
104+
end
105+
106+
Database.register_extension(:pg_auto_parameterize_in_array) do |db|
107+
db.extension(:pg_array, :pg_auto_parameterize)
108+
db.extend_datasets(Postgres::AutoParameterizeInArray)
109+
end
110+
end

‎spec/adapters/postgres_spec.rb

Lines changed: 79 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,17 @@
1313
end
1414
DB.extension :pg_hstore if DB.type_supported?('hstore')
1515
DB.extension :pg_multirange if DB.server_version >= 140000
16-
DB.extension :pg_auto_parameterize if uses_pg && ENV['SEQUEL_PG_AUTO_PARAMETERIZE']
16+
17+
if uses_pg && ENV['SEQUEL_PG_AUTO_PARAMETERIZE']
18+
if ENV['SEQUEL_PG_AUTO_PARAMETERIZE'] = 'in_array_string'
19+
DB.extension :pg_auto_parameterize_in_array
20+
DB.opts[:treat_string_list_as_text_array] = 't'
21+
elsif ENV['SEQUEL_PG_AUTO_PARAMETERIZE'] = 'in_array'
22+
DB.extension :pg_auto_parameterize_in_array
23+
else
24+
DB.extension :pg_auto_parameterize
25+
end
26+
end
1727

1828
describe 'PostgreSQL adapter' do
1929
before do
@@ -64,6 +74,74 @@ def c.exec_prepared(*); super; nil end
6474
Sequel.datetime_class = Time
6575
end
6676

77+
it "should be able to handle various types of IN/NOT IN queries" do
78+
ds = @db.select(1)
79+
ds.where(2=>[2, 3]).wont_be_empty
80+
ds.where(4=>[2, 3]).must_be_empty
81+
ds.exclude(4=>[2, 3]).wont_be_empty
82+
ds.exclude(2=>[2, 4]).must_be_empty
83+
84+
ds.where('2'=>%w[2 3]).wont_be_empty
85+
ds.where('4'=>%w[2 3]).must_be_empty
86+
ds.exclude('4'=>%w[2 3]).wont_be_empty
87+
ds.exclude('2'=>%w[2 3]).must_be_empty
88+
89+
ds.where(2=>[2, 3].map{|i| BigDecimal(i)}).wont_be_empty
90+
ds.where(4=>[2, 3].map{|i| BigDecimal(i)}).must_be_empty
91+
ds.exclude(4=>[2, 3].map{|i| BigDecimal(i)}).wont_be_empty
92+
ds.exclude(2=>[2, 3].map{|i| BigDecimal(i)}).must_be_empty
93+
94+
ds.where(Date.new(2021, 2)=>[2, 3].map{|i| Date.new(2021, i)}).wont_be_empty
95+
ds.where(Date.new(2021, 4)=>[2, 3].map{|i| Date.new(2021, i)}).must_be_empty
96+
ds.exclude(Date.new(2021, 4)=>[2, 3].map{|i| Date.new(2021, i)}).wont_be_empty
97+
ds.exclude(Date.new(2021, 2)=>[2, 3].map{|i| Date.new(2021, i)}).must_be_empty
98+
99+
ds.where(DateTime.new(2021, 2)=>[2, 3].map{|i| DateTime.new(2021, i)}).wont_be_empty
100+
ds.where(DateTime.new(2021, 4)=>[2, 3].map{|i| DateTime.new(2021, i)}).must_be_empty
101+
ds.exclude(DateTime.new(2021, 4)=>[2, 3].map{|i| DateTime.new(2021, i)}).wont_be_empty
102+
ds.exclude(DateTime.new(2021, 2)=>[2, 3].map{|i| DateTime.new(2021, i)}).must_be_empty
103+
104+
ds.where(Time.local(2021, 2)=>[2, 3].map{|i| Time.local(2021, i)}).wont_be_empty
105+
ds.where(Time.local(2021, 4)=>[2, 3].map{|i| Time.local(2021, i)}).must_be_empty
106+
ds.exclude(Time.local(2021, 4)=>[2, 3].map{|i| Time.local(2021, i)}).wont_be_empty
107+
ds.exclude(Time.local(2021, 2)=>[2, 3].map{|i| Time.local(2021, i)}).must_be_empty
108+
109+
ds.where(Sequel::SQLTime.create(2, 0, 0)=>[2, 3].map{|i| Sequel::SQLTime.create(i, 0, 0)}).wont_be_empty
110+
ds.where(Sequel::SQLTime.create(4, 0, 0)=>[2, 3].map{|i| Sequel::SQLTime.create(i, 0, 0)}).must_be_empty
111+
ds.exclude(Sequel::SQLTime.create(4, 0, 0)=>[2, 3].map{|i| Sequel::SQLTime.create(i, 0, 0)}).wont_be_empty
112+
ds.exclude(Sequel::SQLTime.create(2, 0, 0)=>[2, 3].map{|i| Sequel::SQLTime.create(i, 0, 0)}).must_be_empty
113+
114+
ds.where(2=>[2, 3].map{|i| Float(i)}).wont_be_empty
115+
ds.where(4=>[2, 3].map{|i| Float(i)}).must_be_empty
116+
ds.exclude(4=>[2, 3].map{|i| Float(i)}).wont_be_empty
117+
ds.exclude(2=>[2, 3].map{|i| Float(i)}).must_be_empty
118+
119+
ds.where(2=>[2.0, 3.0, 1.0/0.0, -1.0/0.0, 0.0/0.0]).wont_be_empty
120+
ds.where(4=>[2.0, 3.0, 1.0/0.0, -1.0/0.0, 0.0/0.0]).must_be_empty
121+
ds.exclude(4=>[2.0, 3.0, 1.0/0.0, -1.0/0.0, 0.0/0.0]).wont_be_empty
122+
ds.exclude(2=>[2.0, 3.0, 1.0/0.0, -1.0/0.0, 0.0/0.0]).must_be_empty
123+
124+
ds.where(Sequel.blob('2')=>%w[2 3].map{|i| Sequel.blob(i)}).wont_be_empty
125+
ds.where(Sequel.blob('4')=>%w[2 3].map{|i| Sequel.blob(i)}).must_be_empty
126+
ds.exclude(Sequel.blob('4')=>%w[2 3].map{|i| Sequel.blob(i)}).wont_be_empty
127+
ds.exclude(Sequel.blob('2')=>%w[2 3].map{|i| Sequel.blob(i)}).must_be_empty
128+
129+
ds.where(2=>[2, 3.0]).wont_be_empty
130+
ds.where(4=>[2, 3.0]).must_be_empty
131+
ds.exclude(4=>[2, 3.0]).wont_be_empty
132+
ds.exclude(2=>[2, 4.0]).must_be_empty
133+
134+
ds.where(2=>[2]).wont_be_empty
135+
ds.where(4=>[2]).must_be_empty
136+
ds.exclude(4=>[2]).wont_be_empty
137+
ds.exclude(2=>[2]).must_be_empty
138+
139+
ds.where(2=>[2, 3, nil]).wont_be_empty
140+
ds.where(4=>[2, 3, nil]).must_be_empty
141+
ds.exclude(4=>[2, 3, nil]).must_be_empty # NOT IN (..., NULL) predicate always false
142+
ds.exclude(2=>[2, 4, nil]).must_be_empty
143+
end
144+
67145
it "should provide a list of existing ordinary tables" do
68146
@db.create_table(:test){Integer :id}
69147
@db.tables.must_include :test

‎spec/extensions/pg_auto_parameterize_in_array_spec.rb

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
require File.join(File.dirname(File.expand_path(__FILE__)), "spec_helper")
2+
3+
describe "pg_auto_parameterize_in_array extension" do
4+
before do
5+
@db = Sequel.connect('mock://postgres')
6+
@db.synchronize{|c| def c.escape_bytea(v) v*2 end}
7+
@db.opts[:treat_string_list_as_text_array] = 't'
8+
@db.extension :pg_auto_parameterize_in_array
9+
end
10+
11+
types = [
12+
["strings if treat_string_list_as_text_array Database option is true", proc{|x| x.to_s}, "text"],
13+
["BigDecimals", proc{|x| BigDecimal(x)}, "numeric"],
14+
["dates", proc{|x| Date.new(2021, x)}, "date"],
15+
["times", proc{|x| Time.local(2021, x)}, "timestamp"],
16+
["SQLTimes", proc{|x| Sequel::SQLTime.create(x, 0, 0)}, "time"],
17+
["datetimes", proc{|x| DateTime.new(2021, x)}, "timestamp"],
18+
["floats", proc{|x| Float(x)}, "numeric"],
19+
["blobs", proc{|x| Sequel.blob(x.to_s)}, "bytea"],
20+
]
21+
22+
if RUBY_VERSION >= '2.4'
23+
types << ["integers", proc{|x| x}, "int8"]
24+
else
25+
it "should fallback to pg_auto_parameterize extension behavior when switching column IN/NOT IN to = ANY/!= ALL for integers" do
26+
v = [1, 2, 3]
27+
nv = [1, nil, 3]
28+
type = "int8"
29+
30+
sql = @db[:table].where(:a=>v).sql
31+
sql.must_equal %'SELECT * FROM \"table\" WHERE ("a" = ANY(CAST($1 AS #{type}[])))'
32+
sql.args.must_equal ['{1,2,3}']
33+
34+
sql = @db[:table].where(:a=>nv).sql
35+
sql.must_equal %'SELECT * FROM \"table\" WHERE ("a" = ANY(CAST($1 AS #{type}[])))'
36+
sql.args.must_equal ['{1,NULL,3}']
37+
38+
sql = @db[:table].exclude(:a=>v).sql
39+
sql.must_equal %'SELECT * FROM \"table\" WHERE ("a" != ALL(CAST($1 AS #{type}[])))'
40+
sql.args.must_equal ['{1,2,3}']
41+
42+
sql = @db[:table].exclude(:a=>nv).sql
43+
sql.must_equal %'SELECT * FROM \"table\" WHERE ("a" != ALL(CAST($1 AS #{type}[])))'
44+
sql.args.must_equal ['{1,NULL,3}']
45+
end
46+
end
47+
48+
types.each do |desc, conv, type|
49+
it "should automatically switch column IN/NOT IN to = ANY/!= ALL for #{desc}" do
50+
v = [1,2,3].map(&conv)
51+
nv = (v + [nil]).freeze
52+
53+
sql = @db[:table].where(:a=>v).sql
54+
sql.must_equal %'SELECT * FROM \"table\" WHERE ("a" = ANY($1::#{type}[]))'
55+
sql.args.must_equal [v]
56+
57+
sql = @db[:table].where(:a=>nv).sql
58+
sql.must_equal %'SELECT * FROM "table" WHERE ("a" = ANY($1::#{type}[]))'
59+
sql.args.must_equal [nv]
60+
61+
sql = @db[:table].exclude(:a=>v).sql
62+
sql.must_equal %'SELECT * FROM "table" WHERE ("a" != ALL($1::#{type}[]))'
63+
sql.args.must_equal [v]
64+
65+
sql = @db[:table].exclude(:a=>nv).sql
66+
sql.must_equal %'SELECT * FROM "table" WHERE ("a" != ALL($1::#{type}[]))'
67+
sql.args.must_equal [nv]
68+
end
69+
end
70+
71+
it "should automatically switch column IN/NOT IN to = ANY/!= ALL for infinite/NaN floats" do
72+
v = [1.0, 1.0/0.0, -1.0/0.0, 0.0/0.0]
73+
nv = (v + [nil]).freeze
74+
type = "double precision"
75+
76+
sql = @db[:table].where(:a=>v).sql
77+
sql.must_equal %'SELECT * FROM \"table\" WHERE ("a" = ANY($1::#{type}[]))'
78+
sql.args.must_equal [v]
79+
80+
sql = @db[:table].where(:a=>nv).sql
81+
sql.must_equal %'SELECT * FROM "table" WHERE ("a" = ANY($1::#{type}[]))'
82+
sql.args.must_equal [nv]
83+
84+
sql = @db[:table].exclude(:a=>v).sql
85+
sql.must_equal %'SELECT * FROM "table" WHERE ("a" != ALL($1::#{type}[]))'
86+
sql.args.must_equal [v]
87+
88+
sql = @db[:table].exclude(:a=>nv).sql
89+
sql.must_equal %'SELECT * FROM "table" WHERE ("a" != ALL($1::#{type}[]))'
90+
sql.args.must_equal [nv]
91+
end
92+
93+
it "should not automatically switch column IN/NOT IN to = ANY/!= ALL for strings by default" do
94+
@db.opts.delete(:treat_string_list_as_text_array)
95+
v = %w'1 2'
96+
sql = @db[:table].where([:a, :b]=>v).sql
97+
sql.must_equal 'SELECT * FROM "table" WHERE (("a", "b") IN ($1, $2))'
98+
sql.args.must_equal v
99+
100+
sql = @db[:table].exclude([:a, :b]=>v).sql
101+
sql.must_equal 'SELECT * FROM "table" WHERE (("a", "b") NOT IN ($1, $2))'
102+
sql.args.must_equal v
103+
end
104+
105+
it "should not convert IN/NOT IN expressions that use unsupported types" do
106+
v = [Sequel.lit('1'), Sequel.lit('2')].freeze
107+
sql = @db[:table].where([:a, :b]=>v).sql
108+
sql.must_equal 'SELECT * FROM "table" WHERE (("a", "b") IN (1, 2))'
109+
sql.args.must_be_nil
110+
111+
sql = @db[:table].exclude([:a, :b]=>v).sql
112+
sql.must_equal 'SELECT * FROM "table" WHERE (("a", "b") NOT IN (1, 2))'
113+
sql.args.must_be_nil
114+
end
115+
116+
it "should not convert multiple column IN expressions" do
117+
sql = @db[:table].where([:a, :b]=>[[1.0, 2.0]]).sql
118+
sql.must_equal 'SELECT * FROM "table" WHERE (("a", "b") IN (($1::numeric, $2::numeric)))'
119+
sql.args.must_equal [1, 2]
120+
121+
sql = @db[:table].exclude([:a, :b]=>[[1.0, 2.0]]).sql
122+
sql.must_equal 'SELECT * FROM "table" WHERE (("a", "b") NOT IN (($1::numeric, $2::numeric)))'
123+
sql.args.must_equal [1, 2]
124+
end
125+
126+
it "should not convert single value expressions" do
127+
sql = @db[:table].where(:a=>[1.0]).sql
128+
sql.must_equal 'SELECT * FROM "table" WHERE ("a" IN ($1::numeric))'
129+
sql.args.must_equal [1]
130+
131+
sql = @db[:table].where(:a=>[1.0]).sql
132+
sql.must_equal 'SELECT * FROM "table" WHERE ("a" IN ($1::numeric))'
133+
sql.args.must_equal [1]
134+
end
135+
136+
it "should not convert expressions with mixed types" do
137+
sql = @db[:table].where(:a=>[1, 2.0]).sql
138+
sql.must_equal 'SELECT * FROM "table" WHERE ("a" IN ($1::int4, $2::numeric))'
139+
sql.args.must_equal [1, 2.0]
140+
141+
sql = @db[:table].where(:a=>[1, 2.0]).sql
142+
sql.must_equal 'SELECT * FROM "table" WHERE ("a" IN ($1::int4, $2::numeric))'
143+
sql.args.must_equal [1, 2.0]
144+
end
145+
146+
it "should not convert other expressions" do
147+
sql = @db[:table].where(:a=>1).sql
148+
sql.must_equal 'SELECT * FROM "table" WHERE ("a" = $1::int4)'
149+
sql.args.must_equal [1]
150+
151+
sql = @db[:table].where(:a=>@db[:table]).sql
152+
sql.must_equal 'SELECT * FROM "table" WHERE ("a" IN (SELECT * FROM "table"))'
153+
sql.args.must_be_nil
154+
end
155+
end

‎www/pages/plugins.html.erb

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -683,6 +683,10 @@
683683
<span class="ul__span">Automatically parameterizes queries when using the postgres adapter with the pg driver.</span>
684684
</li>
685685
<li class="ul__li ul__li--grid">
686+
<a class="a" href="rdoc-plugins/files/lib/sequel/extensions/pg_auto_parameterize_in_array_rb.html">pg_auto_parameterize_in_array </a>
687+
<span class="ul__span">Builds on pg_auto_parameterize, but handles additional types when converting IN/NOT IN to = ANY/!= ALL.</span>
688+
</li>
689+
<li class="ul__li ul__li--grid">
686690
<a class="a" href="rdoc-plugins/files/lib/sequel/extensions/pg_enum_rb.html">pg_enum </a>
687691
<span class="ul__span">Adds support for PostgreSQL enums.</span>
688692
</li>

0 commit comments

Comments
 (0)