Skip to content

Fix non base 10 number parsing #529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
TomWright wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Fix non base 10 number parsing #529

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
87384f1
Fix non base 10 number parsing issues in YAML
TomWright Mar 10, 2026
5cf1594
Fix integer parsing in various parsers and functions
TomWright Mar 10, 2026
01434f5
Improve YAML number handling
TomWright Mar 30, 2026
16c95d4
Fix a bad rebase
TomWright Mar 30, 2026
4af26b6
Merge branch 'master' into fix-yaml-int-parsing
TomWright Mar 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- Fixed a bug in YAML, TOML and dasel query parser that caused it to fail when parsing non-base10 numbers (e.g. hex, binary, octal).
- Fixed a bug in the `toInt` function that caused it to fail when parsing non-base10 numbers (e.g. hex, binary, octal).
- XML child element ordering now has more comprehensive round-trip handling. Thanks @takeokunn.

## [v3.4.1] - 2026-03-30
Expand Down
2 changes: 1 addition & 1 deletion execution/func_to_int.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ var FuncToInt = NewFunc(
return nil, err
}

i, err := strconv.ParseInt(stringValue, 10, 64)
i, err := strconv.ParseInt(stringValue, 0, 64)
Copy link
Copy Markdown

@ccoVeille ccoVeille Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been there. I would discourage doing this.

YAML spec is not linked to what Go supports with base 0

Add tests for 0x42 0X42 0o42 0b11 0B11

I feel like at least 0B11 0X42 are invalid in YAML, while valid in Go

Also 1_000 is valid for YAML spec 1.2, but not 1.1

0775 is the decimal number 775 is YAML 1.2, and the octal number 775 in YAML spec 1.1

Go parse 0775 as the octal number 775

1e3 is valid in YAML, but invalid for strconv.ParseInt but valid for strconv.ParseFloat

Please read strconv.ParseInt documentation and check what YAML spec would expect

Copy link
Copy Markdown
Owner Author

@TomWright TomWright Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

func_to_int is fine to use go conventions, but all very good points for the yaml parser.
I'll revisit the implementation.

Thank you for the in-depth details! While I'm familiar with all these markup languages, I definitely haven't gotten into the finer details like octal parsing for example

Copy link
Copy Markdown
Owner Author

@TomWright TomWright Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's been a while, but I eventually got back to this. I've added some special hanging which I think covers some edge cases.

Do you have any thoughts or test cases you think I've missed? I'm no YAML expert

if err != nil {
return nil, err
}
Expand Down
2 changes: 1 addition & 1 deletion parsing/toml/toml_reader.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ func (j *tomlReader) readNode(p *unstable.Parser, n *unstable.Node) (string, *mo
}
return "", model.NewFloatValue(f), nil
case unstable.Integer:
i64, err := strconv.ParseInt(string(n.Data), 10, 64)
i64, err := strconv.ParseInt(string(n.Data), 0, 64)
Copy link
Copy Markdown

@ccoVeille ccoVeille Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not an expert in TOML, but I took a quick look at the spec

  • Please add tests for these:

Valid

42, -42, 0x12, 0o7, 0b111, 12_000, -12_000

Valid but as string: "042"

Invalid

-0x12, -0o7, -0b111, 042

All these values are valid with base 0 with ParseInt but not in TOML

Here are also feedbacks provided by an LLM

# ==========================================================
# HEXADECIMAL, OCTAL, AND BINARY BASES
# ==========================================================

# --- Valid (Case Insensitivity for Prefixes and Values) ---
hex_lowercase = 0xff12       # Standard hex
hex_uppercase = 0XFF12       # Uppercase prefix and values are VALID

bin_lowercase = 0b1101
bin_uppercase = 0B1101       # Uppercase 'B' is VALID
oct_lowercase = 0o755
oct_uppercase = 0O755        # Uppercase 'O' is VALID

# --- Invalid ---
# Negative signs are NOT supported for non-decimal bases
invalid_hex = -0xFF12        
invalid_oct = -0o755         
invalid_bin = -0b1101        

# ==========================================================
# SCIENTIFIC NOTATION (FLOATS)
# ==========================================================

# --- Valid ---
standard_sci = 1e6           # 1,000,000
positive_exp = 5e+10         # 50,000,000,000
negative_exp = -1.5E-4       # -0.00015 (Uppercase 'E' is VALID)
large_float  = 6_022.140e23  # Avogadro's number

# --- Invalid ---
# Decimal points must always have digits on both sides
invalid_point_1 = .1e2       
invalid_point_2 = 1.e2       

# Underscores are NOT allowed in the exponent part
invalid_underscore = 1e1_0

if err != nil {
return "", nil, err
}
Expand Down
24 changes: 22 additions & 2 deletions parsing/yaml/yaml_reader.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"fmt"
"io"
"strconv"
"strings"

"github.com/tomwright/dasel/v3/model"
"github.com/tomwright/dasel/v3/parsing"
Expand Down Expand Up @@ -96,11 +97,11 @@ func (yv *yamlValue) UnmarshalYAML(value *yaml.Node) error {
case "!!bool":
yv.value = model.NewBoolValue(value.Value == "true")
case "!!int":
i, err := strconv.Atoi(value.Value)
i, err := parseYAMLInt(value.Value)
if err != nil {
return err
}
yv.value = model.NewIntValue(int64(i))
yv.value = model.NewIntValue(i)
case "!!float":
f, err := strconv.ParseFloat(value.Value, 64)
if err != nil {
Expand Down Expand Up @@ -186,3 +187,22 @@ func (yv *yamlValue) UnmarshalYAML(value *yaml.Node) error {
}
return nil
}

func parseYAMLInt(s string) (int64, error) {
// Strip leading sign for prefix detection.
clean := s
if len(clean) > 0 && (clean[0] == '+' || clean[0] == '-') {
clean = clean[1:]
}

switch {
case strings.HasPrefix(clean, "0x") || strings.HasPrefix(clean, "0X"):
return strconv.ParseInt(s, 0, 64)
case strings.HasPrefix(clean, "0o") || strings.HasPrefix(clean, "0O"):
return strconv.ParseInt(s, 0, 64)
case strings.HasPrefix(clean, "0b") || strings.HasPrefix(clean, "0B"):
return strconv.ParseInt(s, 0, 64)
default:
return strconv.ParseInt(s, 10, 64)
}
}
85 changes: 85 additions & 0 deletions parsing/yaml/yaml_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,91 @@ name2: Tom
`,
}.run)

t.Run("base numbers", func(t *testing.T) {
Copy link
Copy Markdown

@ccoVeille ccoVeille Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are some edge cases you could consider

  • Explicit plus sign (+) with integers / prefixed bases
    The implementation strips a leading sign before prefix detection, but no tests cover the + form.

  • +42 -> 42

  • +0x10 -> 16

  • +0o10 -> 8

  • +0b10 -> 2

  • -42 is tested, but not negative hex/octal/binary.

  • -0x10 -> -16

  • -0o10 -> -8

  • -0b10 -> -2

  • Numeric separators _

  • 1_000 -> 1000

  • 0xFF_FF -> 65535

  • 0b1010_1010 -> 170

Go’s strconv.ParseInt / ParseFloat won’t accept _ directly, so this needs either normalization or a documented rejection + tests.

t.Run("standard", rwTestCase{
in: `10
`,
out: `10
`,
}.run)

t.Run("zero", rwTestCase{
in: `0
`,
out: `0
`,
}.run)

t.Run("negative", rwTestCase{
in: `-42
`,
out: `-42
`,
}.run)

t.Run("hex lowercase", rwTestCase{
in: `0x10
`,
out: `16
`,
}.run)

t.Run("hex uppercase letters", rwTestCase{
in: `0xff
`,
out: `255
`,
}.run)

t.Run("octal", rwTestCase{
in: `0o10
`,
out: `8
`,
}.run)

t.Run("binary", rwTestCase{
in: `0b10
`,
out: `2
`,
}.run)

t.Run("leading zero is decimal", rwTestCase{
in: `010
`,
out: `10
`,
}.run)

t.Run("hex in map", rwTestCase{
in: `val: 0x10
`,
out: `val: 16
`,
}.run)

t.Run("octal in map", rwTestCase{
in: `val: 0o77
`,
out: `val: 63
`,
}.run)

t.Run("mixed types in map", rwTestCase{
in: `dec: 42
hex: 0xff
oct: 0o77
bin: 0b1010
`,
out: `dec: 42
hex: 255
oct: 63
bin: 10
`,
}.run)
})

t.Run("bounded yaml expansion", func(t *testing.T) {
in := `a: &a ["lol","lol","lol","lol","lol","lol","lol","lol","lol"]
b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a]
Expand Down
2 changes: 1 addition & 1 deletion selector/parser/parse_literal.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ func parseNumberLiteral(p *Parser) (ast.Expr, error) {
}, nil

default:
value, err := strconv.ParseInt(token.Value, 10, 64)
value, err := strconv.ParseInt(token.Value, 0, 64)
if err != nil {
return nil, err
}
Expand Down
Loading