-
-
Notifications
You must be signed in to change notification settings - Fork 271
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
3.0.0 sync with GCloud suddenly stopped working - "Failed to synchronize with server" #3395
Comments
This is a bit of a stretch, but can you try with 3.0.2? We fixed some handling of error messages, and hopefully that shows some more information. That error comes from here, which is adding context to an underlying error. So it's a bummer that only the context, and not the underlying error, appears. |
Authenticated with gcloud again and then attempted to
And here's diagnostics again:
|
Are you able to recompile from source? If so, I can make a nice patch to hopefully get better debugging info. I filed #3411 to track the error message problem. |
I am able to build from source on all my devices, yes. Also had an important development in this issue. I did a normal update of system packages for all my devices and it fixed my ability to sync on one of them; specifically it's a normal Linux install of Arch Linux. Unsure exactly what caused that to start working, but my best guess is that I was using a pretty old version of Of the other two devices that still don't work, one is a Windows WSL instance of Arch Linux and the other is a completely separate device running a normal Linux install on EndeavourOS. I'm able to build taskwarrior on both of them fine, although when running Assuming we can resolve that somehow, then we can do the patch for more debugging information and I'll follow your instructions the same way for both devices if that would be helpful. |
The version of the Here's a patch which should hopefully include the error info: diff --git taskchampion/taskchampion/src/replica.rs taskchampion/taskchampion/src/replica.rs
index bad6cceba..66d72c3c2 100644
--- taskchampion/taskchampion/src/replica.rs
+++ taskchampion/taskchampion/src/replica.rs
@@ -225,16 +225,17 @@ impl Replica {
/// indicate it is urgent (snapshot urgency "high"). This allows time for other replicas to
/// create a snapshot before this one does.
///
/// Set this to true on systems more constrained in CPU, memory, or bandwidth than a typical desktop
/// system
pub fn sync(&mut self, server: &mut Box<dyn Server>, avoid_snapshots: bool) -> Result<()> {
self.taskdb
.sync(server, avoid_snapshots)
+ .map_err(|e| dbg!(e))
.context("Failed to synchronize with server")?;
self.rebuild_working_set(false)
.context("Failed to rebuild working set after sync")?;
Ok(())
}
/// Return undo local operations until the most recent UndoPoint, returning an empty Vec if there are no
/// local operations to undo. |
I rebuilt with the patch and still got that Anyways here is the results after
|
Huh, that error only occurs when communicating with a taskchampion-sync-server instance, not when using gcloud. Did your config change from, or to, gcloud? |
I set it up using gcloud immediately after upgrading to 3.0.0 and haven't changed any part of my config since then; my taskrc is the same on all my devices and they all have the json file containing the gcloud credentials as mentioned in My taskrc has the |
Oh, I'm sorry - I misread the source. This can happen for cloud syncs, too. This is one of those things that "shouldn't" happen, where the linear history of changes isn't linear anymore. Is the screenshot above showing all of the objects in your bucket? If not, can you send that whole list? The values are just random UUIDs so don't reveal anything about your data. Also, please show the output of |
Yes that's right; the image in the original issue message is every object in the bucket. It's still accurate as of today. Also I use the XDG base directory system to organize my configs, which I believe began support in taskwarrior 2.6.0, so I modified your command a bit. Hope that's ok.
|
Thanks! So what I see in the bucket is two versions (
Your local replica keeps track of the version on which its operations are based, in this case ec3c99a1... Sync begins by looking for any new versions on the server, by looking for a version with parent ec3c99a1... -- finding none. So it assumes that ec3c99a1... is the latest version. This is plausible from the screenshot -- the content of the Your local replica then packages up its local operations and tries to upload them as a new version at the end of the chain, with parent version ec3c99a1.. -- and the server rejects it for not having the latest version as its parent. Which means that
Here xxxxxx and yyyyyy are different, and The first question is, how did ec3c99 get uploaded and then disappear? Taskwarrior only writes to the As for why ec3c99 wasn't "on the list": that means that The object names in the bucket actually contain the version info, so we can at least learn Also, if you click the "latest" object and download it, it should be a text file containing the latest version. What is that value? |
Wow, what a cool explanation! It's interesting to learn the inner workings of how taskwarrior handles its syncing. Anyways, here's a full view of the names of every object stored in the bucket (+ some extra columns just in case the timestamps end up mattering) Here is the entire contents of the
|
Huh, so the table is actually
which means that the replica uploaded ec3c99, but somehow Here's the code that handles updating taskwarrior/taskchampion/taskchampion/src/server/cloud/server.rs Lines 371 to 386 in ef9613e
So, if that compare_and_swap operation fails, it deletes the object from the bucket and returns AddVersionResult::ExpectedParentVersion with the value from latest . Here's the code that calls add_version :taskwarrior/taskchampion/taskchampion/src/taskdb/sync.rs Lines 82 to 113 in ef9613e
That updates the base_version in sync_meta if it gets AddVersionResult::Ok , or tries again if it gets AddVersionResult::ExpectedParentVersion .
So presumably there was an attempt to upload new version ec3c99 which
I think what that means is that taskwarrior/taskchampion/taskchampion/src/server/cloud/gcp.rs Lines 105 to 167 in ef9613e
and looking at all the places Ok(..) occurs there (the return values), those are all false before the call to upload_object , and then false if that failed with a 412 error. The upload_res? statement there should handle any other error as an actual error, propagated back and causing task sync to fail. Looking at the implementation of is_http_error suggests that even the 412 is represented as Result::Err so I think that's a valid assumption.
So, I'm stumped -- what have I missed? Thanks for your patience, @jcoffa. The good news is, the fix here is pretty easy. Just put |
I swapped out the value in That sync failed ("Failed to synchronize with server"), but it did change some files in the bucket! Here's a screenshot of what that looked like: I then tried to use the local build that had the patch you sent, and then that one worked!
After this successful sync the gcloud bucket appears completely unchanged. Another Very weird. I'll try with my other devices and report back. |
It looks like it successfully uploaded a new version, and subsequent runs had nothing new to upload so didn't fail. Maybe try modifying a task and running |
Damn, got the same error again:
gcloud bucket appears unchanged since the last screenshot (made sure to refresh too). I DID NOT end up trying with my other devices like I said I was going to at the end of my last comment, by the way. |
Has the value in I'll be busy most of the day tomorrow, but will check back after that. It might be worth putting some logging ( |
Value of
So looks like it is reproducible at least! I'll see if I can do some print debugging to figure out what's going on. I've never used rust before but I'll try my best! No worries on any delays by the way, everyone's got a life to live and this is all practically volunteer work anyways! Syncing isn't a huge part of my daily workflow so I personally am not affected by this issue that much. |
I believe I may have stumbled into something. TL;DR I got syncing working on my main device by manually updating I'll do my best to explain what I did in the exact order I did them in:
TL;DR is the message at the end: I double-checked the instructions from And it worked!
Same issue as before; I'd like to highlight that I didn't manually swap out the value in
And here's my full print output just in case there's something in here that's even remotely helpful (although it's kind of a mess):
(that last line "Syncing with GCP bucket" is simply the success message when taskwarrior itself performs a sync without error; that's not one of my print statements)
Here is a screenshot of what my bucket looks like now: So... things appear to be working now? At least for my main device I use taskwarrior on. I haven't attempted syncs on other devices as I figured I'd let you digest this absolute wall of text (sorry!) before trying to do anything else without guidance; for fear of messing something up and making it harder to debug. |
Something I just thought of while sleeping on this issue and thinking about why |
Nice work! I set up a service account to replicate this, without the However, I don't reproduce exactly the error you've seen -- a newer version in
which before #3411 would have just said "Failed to synchronize with server". My bucket has
so two versions: faeb7f followed by fb1e11. The
It works?! And even more bizarre, Ah, I see what's happening here! On the last (successful) run of So, there are two bugs here now.
Any chance you want to make a PR for the second one? Thanks so much for your patience tracking this down. That's three bugs found and soon to be fixed in one issue! |
Yeah I can make a PR for the 2nd one, it'd be my pleasure :) And thank YOU for your patience as well! No chance could I have fixed my syncing issues by myself. |
Set up syncing with google cloud by following the steps outlined in
man task-sync
after upgrading from 2.6.2 -> 3.0.0. Verified that it was working and have been using this solution for syncing for a couple weeks now across 3 devices. Have performed successful syncs on all 3 of these devices over the past couple weeks. Everything was working as expected until today for some reason.Attempted to run
task sync
as normal. Got an error message saying "Failed to synchronize with server". Fair enough, I usually forget to rungcloud auth login
before attempting a sync anyways. Successfully authenticate in the browser window that opens using the same google account I've always used. Runtask sync
again, same generic error message "Failed to synchronize with server".man task-sync
task
directory as expected, and the full absolute path + file name matches the value of mysync.gcp.credential_path
taskrc propertygcloud config get-value project
Output of
task diag
:At a loss at how to debug this; Taskwarrior only gives me a generic failure message when sync fails.
The text was updated successfully, but these errors were encountered: