Skip to content

Commit 190c81b

Browse files
rgbkrkrgbkrk
authored
Merge pull request #10 from rgbkrk/roadmap
Update README and ROADMAP
2 parents 36b4930 + 31b709b commit 190c81b

File tree

5 files changed

+265
-397
lines changed

5 files changed

+265
-397
lines changed

‎README.md

Lines changed: 99 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -1,103 +1,141 @@
11
# Anode
22

3-
A real-time collaborative notebook system built with LiveStore event sourcing.
3+
A real-time collaborative notebook system built on LiveStore, focusing on seamless AI ↔ Python ↔ User interactions.
44

5-
**Current Status: ✅ FULLY OPERATIONAL** - Python code execution working end-to-end with reactive architecture.
5+
**Current Status: ✅ FULLY OPERATIONAL** - Zero-latency Python execution with reactive architecture working end-to-end.
66

7-
## Architecture
8-
9-
- **Schema Package** (`@anode/schema`): LiveStore schema definitions
10-
- **Web Client** (`@anode/web-client`): React-based web interface
11-
- **Document Worker** (`@anode/docworker`): Cloudflare Worker for sync backend
12-
- **Kernel Client** (`@anode/dev-server-kernel-ls-client`): Python execution server
7+
## What Makes Anode Different
138

14-
### Key Design
15-
- Each notebook = one LiveStore store (`NOTEBOOK_ID = STORE_ID`)
16-
- Execution queue system: `pending` → `assigned` → `executing` → `completed`
17-
- **Reactive kernel architecture** using LiveStore's `queryDb` subscriptions
18-
- Manual kernel management (start one per notebook)
9+
- **Real-time collaboration** built on event sourcing (LiveStore)
10+
- **Zero-latency execution** using reactive subscriptions (no polling)
11+
- **AI-first design** for intelligent code assistance and context-aware suggestions
12+
- **Local-first architecture** with offline capability
1913

2014
## Quick Start
2115

22-
### 1. Start Core Services
16+
### 1. Install and Start Core Services
2317
```bash
2418
pnpm install
2519
pnpm dev # Starts web client + sync backend
2620
```
2721

28-
### 2. Create Notebook
22+
### 2. Create Your First Notebook
2923
1. Open http://localhost:5173
30-
2. URL gets notebook ID: `?notebook=notebook-123-abc`
31-
3. Create cells and edit
24+
2. URL automatically gets notebook ID: `?notebook=notebook-123-abc`
25+
3. Start creating cells and editing
3226

3327
### 3. Enable Python Execution
3428
```bash
35-
# In new terminal - use your actual notebook ID
29+
# In new terminal - use your actual notebook ID from the URL
3630
NOTEBOOK_ID=notebook-123-abc pnpm dev:kernel
3731
```
3832

33+
**Pro tip**: Click the **Kernel** button in the notebook header to copy the exact command for your notebook!
34+
3935
### 4. Execute Code
40-
- Add code cell in web interface
41-
- Write Python: `import random; random.random()`
42-
- Press Ctrl+Enter or click Run
43-
- See results appear in real-time
36+
- Add a code cell in the web interface
37+
- Write Python: `import numpy as np; np.random.random(5)`
38+
- Press **Ctrl+Enter** or click **Run**
39+
- See results appear instantly
4440

45-
## What's Working
41+
## Architecture
4642

47-
- ✅ Real-time collaborative editing
48-
- ✅ Python code execution via Pyodide
49-
- ✅ Event sourcing and sync
50-
- ✅ **Reactive work queue management** (instant response)
51-
- ✅ Multiple isolated notebooks
52-
- ✅ Output generation and display
53-
- ✅ Zero-latency execution (no polling delays)
43+
Anode uses a breakthrough reactive architecture for instant execution:
5444

55-
## Development Commands
45+
```
46+
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
47+
│ Web Client │◄──►│ Document Worker │◄──►│ Python Kernel │
48+
│ (React UI) │ │ (CF Workers) │ │ (Pyodide) │
49+
└─────────────────┘ └──────────────────┘ └─────────────────┘
50+
│ │ │
51+
▼ ▼ ▼
52+
LiveStore DB Event Sync Hub Execution Queue
53+
```
5654

57-
```bash
58-
# Core development
59-
pnpm dev # Start web + sync
60-
NOTEBOOK_ID=your-id pnpm dev:kernel # Start kernel
55+
### Key Design Principles
56+
- **One notebook = One LiveStore store** (`NOTEBOOK_ID = STORE_ID`)
57+
- **Reactive work queue**: `executionRequested` → `executionAssigned` → `executionStarted` → `executionCompleted`
58+
- **Event sourcing**: All changes are events, enabling perfect collaboration and audit trails
59+
- **Session-based kernels**: Each kernel gets a unique session for isolation
6160

62-
# Utilities
63-
pnpm reset-storage # Clear all data
64-
pnpm build:schema # Build schema after changes
61+
## What's Working Right Now
6562

66-
# Individual services
67-
pnpm dev:web-only # Web client only
68-
pnpm dev:sync-only # Sync backend only
69-
```
63+
- ✅ **Instant Python execution** with zero polling delays
64+
- ✅ **Real-time collaborative editing** across multiple users
65+
- ✅ **Reactive architecture** using LiveStore's `queryDb` subscriptions
66+
- ✅ **Multiple isolated notebooks** with separate kernels
67+
- ✅ **Rich output display** for Python results
68+
- ✅ **Offline-first operation** with sync when connected
69+
- ✅ **Event sourcing** for complete history and debugging
7070

7171
## Project Structure
7272

7373
```
74-
packages/
75-
├── schema/ # LiveStore events and tables
76-
├── web-client/ # React notebook interface
77-
├── docworker/ # Cloudflare Workers sync
78-
└── dev-server-kernel-ls-client/ # Python kernel process
74+
anode/
75+
├── packages/
76+
│ ├── schema/ # LiveStore events & state definitions
77+
│ ├── web-client/ # React notebook interface
78+
│ ├── docworker/ # Cloudflare Workers sync backend
79+
│ └── dev-server-kernel-ls-client/ # Python execution server
80+
├── ROADMAP.md # Development priorities
81+
└── AGENTS.md # AI agent context
82+
```
83+
84+
## Development Commands
85+
86+
```bash
87+
# Core development workflow
88+
pnpm dev # Start web + sync
89+
NOTEBOOK_ID=your-notebook-id pnpm dev:kernel # Start kernel for specific notebook
90+
91+
# Utilities
92+
pnpm reset-storage # Clear all local data
93+
pnpm build:schema # Rebuild schema after changes
94+
95+
# Individual services (for debugging)
96+
pnpm dev:web-only # Web client only
97+
pnpm dev:sync-only # Sync backend only
7998
```
8099

81-
## Architecture Notes
100+
## Next Phase: AI-First Notebooks
82101

83-
**Simplified Schema**: Removed all timestamp fields to eliminate complexity and database errors. Simple schemas are more reliable for prototypes.
102+
Anode is designed around **AI ↔ Python ↔ User interactions**. The next major milestone is completing the AI cell architecture:
84103

85-
**Reactive Over Polling**: Kernels use LiveStore's `queryDb` reactive subscriptions for instant work detection. No more polling delays - executions start immediately when cells are run.
104+
### Coming Soon
105+
- **AI cells** that understand notebook context
106+
- **Code completions** with LSP + kernel integration
107+
- **Intelligent suggestions** based on current data and code
108+
- **Context-aware assistance** for data analysis workflows
86109

87-
**One Store Per Notebook**: Each notebook gets its own LiveStore database for clean isolation.
110+
See [ROADMAP.md](./ROADMAP.md) for detailed development priorities.
88111

89112
## Troubleshooting
90113

91-
**Build failures**: Run `pnpm build:schema` first
92-
**Execution not working**: Start kernel with correct `NOTEBOOK_ID`
93-
**Stale state**: Run `pnpm reset-storage`
94-
**Slow execution**: Should be instant with reactive architecture - check kernel logs for errors
114+
| Problem | Solution |
115+
|---------|----------|
116+
| Build failures | Run `pnpm build:schema` first |
117+
| Execution not working | Start kernel with correct `NOTEBOOK_ID` (use copy button in UI) |
118+
| Stale state | Run `pnpm reset-storage` |
119+
| Slow execution | Should be instant - check kernel logs |
120+
121+
## Architecture Highlights
122+
123+
**Simplified Event Schema**: Removed timestamp complexity for reliability. Simple schemas work better for rapid prototyping.
124+
125+
**Reactive Over Polling**: Kernels use LiveStore's reactive subscriptions for instant work detection. This breakthrough eliminates polling delays entirely.
126+
127+
**Local-First Design**: Everything works offline first, syncs when connected. Your work is never lost.
128+
129+
## Contributing
130+
131+
Anode is an open source project focused on developer experience. The system provides a solid foundation for collaborative notebook execution and can be extended incrementally.
95132

96-
## Next Steps
133+
Key areas for contribution:
134+
- AI cell implementation
135+
- Code completion systems
136+
- Rich output formats
137+
- Performance optimizations
97138

98-
1. Clean up tests (remove timestamp field references)
99-
2. Add error recovery for kernel restarts
100-
3. Improve kernel lifecycle management
101-
4. Add proper authentication
139+
## License
102140

103-
The system provides a solid foundation for collaborative notebook execution with **instant reactive execution** and can be extended incrementally.
141+
[License TBD - Open Source]

‎ROADMAP.md

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# Anode Development Roadmap
2+
3+
**Current Status**: Fully operational reactive architecture with zero-latency Python execution ✅
4+
5+
This roadmap outlines the next phases of development for Anode, building on the working reactive architecture to achieve Jupyter parity while maintaining the real-time collaborative advantages of LiveStore.
6+
7+
## Phase 1: Production Foundation
8+
9+
### 1.1 Authentication & Authorization 🔐
10+
**Priority: HIGH** - Critical for multi-user deployment and kernel-document isolation
11+
12+
**Current State**: Hardcoded `'insecure-token-change-me'` tokens
13+
**Target**: JWT-based auth with kernel session isolation
14+
15+
- [ ] **Google OAuth Integration**
16+
- Replace hardcoded auth with Google OAuth flow
17+
- Token exchange service: Google token → Anode JWT
18+
- Basic user management (profile, preferences)
19+
20+
- [ ] **Kernel Security Model**
21+
- Kernels get session-specific JWTs
22+
- Document worker validates kernel permissions
23+
- Kernel isolation (can only access assigned notebook)
24+
25+
### 1.2 Kernel Lifecycle Management 🐍
26+
**Priority: HIGH** - Manual kernel management creates friction
27+
28+
**Current State**: Manual `NOTEBOOK_ID=x pnpm dev:kernel` startup
29+
**Target**: Automatic kernel lifecycle with session management
30+
31+
- [ ] **Kernel Session Service**
32+
- `/api/kernels/ensure-session` endpoint
33+
- Session state tracking
34+
- Kernel status: `claiming` → `provisioning` → `starting` → `ready` → `busy` → `shutdown`
35+
36+
- [ ] **Enhanced Document Worker**
37+
- Kernel session validation in sync hooks
38+
- Integration with kernel service APIs
39+
40+
### 1.3 Demo Deployment 🚀
41+
**Priority: MEDIUM** - Showcases capabilities
42+
43+
- [ ] **CloudFlare Pages Deployment**
44+
- Web client static hosting
45+
- Demo environment configuration
46+
- Basic rate limiting
47+
48+
- [ ] **Demo Kernel Strategy**
49+
- Session-isolated kernel instances
50+
- Consider browser-based Pyodide as fallback
51+
52+
## Phase 2: Jupyter Parity
53+
54+
### 2.1 Enhanced Python Kernel 🐍
55+
**Priority: HIGH** - Core execution improvements
56+
57+
- [ ] **Package Management**
58+
- Pre-installed scientific stack (numpy, pandas, matplotlib)
59+
- Dynamic package installation
60+
- Environment isolation between notebooks
61+
62+
- [ ] **Rich Output Support**
63+
- Plot rendering (matplotlib, plotly)
64+
- DataFrame HTML display
65+
- Image and media handling
66+
67+
- [ ] **Code Completions & IntelliSense**
68+
- LSP integration for Python (Pylsp/Pyright)
69+
- Kernel-based completions (runtime introspection)
70+
- Context-aware suggestions from notebook variables
71+
- Auto-imports and documentation on hover
72+
- Error highlighting and diagnostics
73+
74+
### 2.2 AI Cell Architecture 🤖
75+
**Priority: HIGH** - Enable AI <> Python <> User interactions
76+
77+
**Design Vision**: AI cells function like kernel adapters
78+
- Input: User prompt + notebook context
79+
- Output: AI response (markdown, code, suggestions)
80+
- Execution flow: `aiExecutionRequested` → `aiExecutionStarted` → `aiExecutionCompleted`
81+
82+
- [ ] **AI Kernel Adapter**
83+
- Similar to Python kernel but calls LLM APIs
84+
- Notebook context extraction (previous cells, outputs)
85+
- Response streaming
86+
87+
- [ ] **Context Management**
88+
- Intelligent context window management
89+
- Cell dependency tracking
90+
91+
### 2.3 SQL Cell Integration 🗄️
92+
**Priority: MEDIUM** - SQL analysis on Python data
93+
94+
**Design Vision**: SQL cells work in tandem with Python kernel
95+
- SQL source gets translated to Python pandas/DuckDB execution
96+
- Results flow back through Python kernel execution queue
97+
- Shared data context between SQL and Python cells
98+
99+
- [ ] **SQL → Python Translation**
100+
- DuckDB integration
101+
- Database connection management through Python
102+
- Result set handling and display
103+
104+
## Phase 3: Collaboration & Polish
105+
106+
### 3.1 Performance & Scale 📈
107+
**Priority: MEDIUM** - Handle larger notebooks
108+
109+
- [ ] **Large Output Handling**
110+
- Offload images/media (avoid base64 in notebook)
111+
- Efficient binary data storage
112+
- Output compression
113+
114+
- [ ] **Memory Management**
115+
- Kernel resource limits
116+
- Garbage collection strategies
117+
118+
### 3.2 Real-Time Collaboration 👥
119+
**Priority: LOW** - Build on LiveStore's existing sync
120+
121+
**Current State**: Basic LiveStore sync working
122+
**Target**: Enhanced collaborative features
123+
124+
- [ ] **Basic Presence**
125+
- Active user list per notebook
126+
- Simple "User X is editing" indicators
127+
128+
- [ ] **Conflict Resolution**
129+
- Leverage LiveStore's event sourcing
130+
- Execution queue ordering (already working)
131+
132+
## Phase 4: Developer Experience
133+
134+
### 4.1 Import/Export 📁
135+
- [ ] **Jupyter Compatibility**
136+
- Import/export .ipynb files
137+
- Maintain notebook format compatibility
138+
139+
### 4.2 API & Extensions 🛠️
140+
- [ ] **REST API**
141+
- Notebook management
142+
- Execution control
143+
- Output retrieval
144+
145+
- [ ] **Custom Cell Types**
146+
- Framework for new cell types (GraphQL, etc.)
147+
- Cell type registry
148+
149+
## Current Strengths to Preserve
150+
151+
- ✅ **Reactive Architecture**: Zero-latency execution via LiveStore `queryDb`
152+
- ✅ **Event Sourcing**: Clean audit trail and state management
153+
- ✅ **Local-First**: Offline capability and fast interactions
154+
- ✅ **Type Safety**: End-to-end TypeScript with Effect
155+
156+
## Technical Debt to Address
157+
158+
- [ ] Manual kernel lifecycle management
159+
- [ ] Hardcoded authentication
160+
- [ ] Limited error handling and recovery
161+
- [ ] Missing production monitoring
162+
163+
---
164+
165+
*This roadmap focuses on achieving Jupyter parity while leveraging Anode's unique real-time collaborative architecture. Priorities will evolve based on user feedback and adoption.*

0 commit comments

Comments
 (0)