memory was not properly released after insertion

[cooronx]

I am currently using sea-orm with PostgreSQL.

It works fine, however, when i insert a huge entry using ActiveModel, I discovered that memory wasn't being released and was accumulating, causing my program to have high memory usage.

I wrote a small demo to demonstrate.

async fn test_process_signal_service() -> anyhow::Result<()> {
        let mut opt: ConnectOptions = ConnectOptions::new("postgres://ly:*****@127.0.0.1:5432/postgres");
        opt.max_connections(20)
            .min_connections(5)
            .sqlx_logging(false);
        let db = Database::connect(opt).await?;
        
        let content = fs::read_to_string("test.txt").unwrap();
        let array : Vec<ProcessData> = serde_json::from_str(&content).unwrap();
        let active_model = align_data_segment::ActiveModel {
            align_data_id: Set(10),
            data: Set(array),  // data is a huge json, approximately 10 MB.
            start_at: Set(2132133245231),
            end_at: Set(53242113321),
            sequence_id: Set(0),
            ..Default::default()
        };
        println!("the memory before insert: {}",get_memory_usage()); // print 10518KB
        active_model.insert(&db).await?;
        println!("the memory after insert:: {}",get_memory_usage()); // print 121443KB
        Ok(())
}

pub fn get_memory_usage() -> i64 {
    let pid = process::id();
    
    let process = Process::new(pid as i32).unwrap();

    let status = process.stat().unwrap();
    status.rss  // RSS (Resident Set Size)，KB
}

Because this table will be inserted multiple times in my project, this leads to increasingly higher memory usage.

I don't understand why, but when I use raw SQL with sea-orm, the memory usage returns to normal.

async fn test_process_signal_service() -> anyhow::Result<()> {
        let mut opt: ConnectOptions = ConnectOptions::new("postgres://ly:*****@127.0.0.1:5432/postgres");
        opt.max_connections(20)
            .min_connections(5)
            .sqlx_logging(false);
        let db = Database::connect(opt).await?;
        
        let content = fs::read_to_string("test.txt").unwrap();
        let array : Vec<ProcessData> = serde_json::from_str(&content).unwrap();
        let sql = r#"
            insert into align_data_segment (align_data_id, data, start_at, end_at, sequence_id)
            VALUES ($1, $2::jsonb, $3, $4, $5)
        "#;

        let stmt = Statement::from_sql_and_values(
        DbBackend::Postgres,
        sql,
        vec![
            Value::from(10),
            Value::from(content), // content is a huge json, approximately 10 MB.
            Value::from(124213),
            Value::from(2345443),
            Value::from(0),
        ],
    );
        println!("the memory before insert: {}",get_memory_usage()); // print 10518KB
        let _ = db.query_one(stmt).await.unwrap();
        println!("the memory after insert:: {}",get_memory_usage()); // print 13164KB
        Ok(())
    }

Is this normal ORM behavior, or am I using it incorrectly?

[tyt2y3]

I don't think RSS is a good fine grained measure for the real memory usage of a single statement.
If you've found that raw sql is better, then just stick to it.
Also, not sure which RC version you were using. In anycase please use the latest 2.0 rc.

I don't believe there is memory leak or anything specifc about SeaORM.
Heap fragmentation is a real problem in Rust in general, especially with deeply nested JSON kind of data structures.
I've found mimalloc to be an effective solution in some cases. YMMV

Or don't even parse the JSON as serde_json::Value. process it as a string and cast_as jsonb in the insert statement.

[cooronx]

thanks for answering👍
i'm currently using sea-orm = { version = "1.1.15", features = [ "runtime-tokio-rustls", "sqlx-postgres", ] }
RSS is the most common measure of how much physical RAM a process is actually using as far as i know.
i just trying to figure out why. And it makes pretty odd to mixed use orm and raw sql in the project. i was guessing if there is a caching mechanism in SeaORM's insertion.
The reason I'm asking this question is that when my program runs for a long time, the memory usage keeps increasing, and I suspect this module might be the cause.

[Huliiiiii]

We don't have caches. The default allocator does not return memory to the OS immediately. Memory fragmentation can also prevent memory from being reclaimed by the system.

Compared to the raw SQL, we performed more heap allocations.

We did not implement any special Drop. If memory still "leaks" after switching allocators, it may be due to one of our dependencies.

[cooronx]

thanks for answering👍
maybe i will try other allocators like the mimalloc mentioned before