Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace sqlite3 with sqlean #220

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Replace sqlite3 with sqlean #220

wants to merge 5 commits into from

Conversation

kracekumar
Copy link
Contributor

@kracekumar kracekumar commented Mar 20, 2025

Description

  1. Sqlean is drop in replacement for sqlite3 with extra extensions.
  2. It's built 12 extra extensions. From the website
Think of them as the extended standard library for SQLite:

    [crypto](https://antonz.org/sqlean-crypto/): Hashing, encoding and decoding data.
    [define](https://antonz.org/sqlean-define/): User-defined functions and dynamic SQL.
    [fileio](https://antonz.org/sqlean-fileio/): Reading and writing files and catalogs.
    [fuzzy](https://github.com/nalgeon/sqlean/blob/main/docs/fuzzy.md): Fuzzy string matching and phonetics.
    [ipaddr](https://github.com/nalgeon/sqlean/blob/main/docs/ipaddr.md): IP address manipulation.
    [math](https://github.com/nalgeon/sqlean/blob/main/docs/math.md): Math functions.
    [regexp](https://antonz.org/sqlean-regexp/): Pattern matching using regular expressions.
    [stats](https://github.com/nalgeon/sqlean/blob/main/docs/stats.md): Math statistics — median, percentiles, etc.
    [text](https://antonz.org/sqlean-text/): Powerful Unicode-aware string functions.
    [time](https://antonz.org/sqlean-time/): High-precision date/time.
    [uuid](https://github.com/nalgeon/sqlean/blob/main/docs/uuid.md): Universally Unique IDentifiers.
    [vsv](https://github.com/nalgeon/sqlean/blob/main/docs/vsv.md): CSV files as virtual tables.
  1. This provides some extra extensions inside sqlite which can be handy as reported in writefile function with fileio #119
  2. It's worth experimenting to see how it goes especially in terms of regression.
  3. Note: I ran ruff formatter and it fixed a bunch of style issues.

Notes:

  • I haven't used sqlean in production or in any other project not sure of any issues
  • This may sound risky to replace sqlite3, though it's a drop-in replacement.

Checklist

  • I've added this contribution to the CHANGELOG.md file.

@@ -361,7 +361,7 @@ def run_cli(self):
else:
history = None
self.echo(
'Error: Unable to open the history file "{}". ' "Your query history will not be saved.".format(history_file),
'Error: Unable to open the history file "{}". Your query history will not be saved.'.format(history_file),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the formatting done by ruff.

assert set(executor.table_columns()) == set([("a", "x"), ("a", "y"), ("b", "z"), ("t", "t")])
assert set(executor.tables()) == set([("sqlean_define",), ("a",), ("b",), ("t",)])
assert set(executor.table_columns()) == set(
[("sqlean_define", "type"), ("sqlean_define", "body"), ("sqlean_define", "name"), ("a", "x"), ("a", "y"), ("b", "z"), ("t", "t")]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding extra table and columns from sqlean define

@kracekumar
Copy link
Contributor Author

Here is a simple benchmark

# -*- coding: utf-8 -*-

from litecli.main import LiteCli
import timeit


def perf():
    dbname=":memory:"
    cli = LiteCli()
    cli.connect(dbname)
    # Create a table
    cli.run_query("create table test (a text)")
    # Insert 10000 rows
    for i in range(10000):
        cli.run_query(f'insert into test values("{i}")')
    # Delete the table
    cli.run_query("drop table test")


def main():
    # timeit is not needed when using hyperfine
    print(timeit.timeit("perf()", globals=globals(), number=10))


if __name__ == "__main__":
    main()

Default Sqlite3

hyperfine --shell /opt/homebrew/bin/fish  --runs 5 --show-output  'uv run python litecli/performance.py'
Benchmark 1: uv run python litecli/performance.py
3.7350782089924905
3.7475891669892007
3.77843445900362
3.740818833000958
3.749061916998471
  Time (mean ± σ):      3.966 s ±  0.027 s    [User: 3.899 s, System: 0.050 s]
  Range (min … max):    3.944 s …  4.008 s    5 runs

Sqlean

 hyperfine --shell /opt/homebrew/bin/fish  --runs 5 --show-output  'uv run python litecli/performance.py'
Benchmark 1: uv run python litecli/performance.py
3.648521625000285
3.724126499990234
3.765500624998822
3.648744167003315
3.666765707996092
  Time (mean ± σ):      3.921 s ±  0.083 s    [User: 3.824 s, System: 0.066 s]
  Range (min … max):    3.854 s …  4.027 s    5 runs

I don't see much deviation in performance and I think it should be okay.

Let me know what you think.

@kracekumar kracekumar requested a review from amjith March 21, 2025 01:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant