Write operations seem to be hanging

[adamrogas]

I am seeing odd behavior and was wondering if anyone else was as well,

When testing my CloudFS plugin, that was based heavily on the FILE plugin write operations are hanging and never completing.

anytime the content.CopyTo(stream) is being called the program hangs.

It appears that for some reason the Dokan stream is never actually getting copied to the GatherStream because assigned blocks is always empty and so GetAvailableBytes is coming back as 0.

As a sanity check I also attached the original untouched FILE plugin and had similar results.

has anything changed that could be affecting this ?

[viciousviper]

You're right - the latest changes that were introduced to support filesystems with "immutable" files (read: MEGA) seem to have broken something.
Please revert to v 1.0.4-alpha for the time being - I'll get to the root of this.

[adamrogas]

The crux of the problem seems to be that in Dokan Operations set allocation Size

            var context = (StreamContext)info.Context;
            context.Stream = scatterStream;

and the content is never copied from the Dokan context stream to the scatterStream so everything is blank :)

[viciousviper]

Could you please check if my latest checkin to the develop branch does any good for your problem?
I haven't quite gotten around to reproducing your observation with the call to SetAllocationSize(), however, there definitely was a bug in determining file sizes for encrypted files - which may account for your hangs in Stream.CopyTo().

With the fix applied I was able to write a ~12Mb binary file to a FILE based volume, verify its CRC while on the mounted volume and copy it back to a normal HD intact.

[adamrogas]

I will check however I should have clarified before I am using the library with the encryption key set to "" which, I understood, would bypass the encryption all together, I am wondering if this could also account for the problems I am seeing

[viciousviper]

You are correct in assuming that encryptionKey = string.Empty bypasses encryption.
I will doublecheck tonight that everything works as it should with encryption turned off.

BTW, do you access your mounted cloud volume through Windows Explorer or directly from .NET code?

My primary design goal for DokanCloudFS has been to enable cloud access for Windows Explorer, so there is a chance that not all code paths are covered for "raw" .NET - although this will definitely work for the CloudFS gateways.

[adamrogas]

I access the mounted cloud volume via windows Explorer,

I am still seeing issues with writes after the changes.

I wonder if the proxy file creation could be interfering as that seems to be a fairly recent change.

the reason I mentioned the SetAllocation is that I am seeing scenarios where Write is called before set allocation and as far as I can tell that written content would not end up in the scatterstream.

[viciousviper]

the reason I mentioned the SetAllocation is that I am seeing scenarios where Write is called before set allocation and as far as I can tell that written content would not end up in the scatterstream.

That's interesting. I've not seen this behavior so far. IIRC the only case where SetAllocationSize() is not called first is for small files (< 4kb) where the entire content is written in a single Write().

Do you have a specific scenario that I could try to reproduce, like

• do you write a single file/multiple files/a single folder with its files/nested folders?
• what size are the affected files?
• what filesystem type is the originating volume?
• what OS platform are you testing on?

[adamrogas]

I was testing with small files.. but in those cases the file save hangs.

[viciousviper]

Ok, bad habit. I just saw that your reply came between my original comment and my edits.
If you'd be so kind as to give me a couple more specifics of your scenario I'd be happy to look into this tonight.

[adamrogas]

Sorry about that.

My setup is windows 8.1 and windows 10
I am testing with large txt files and small txt files from notepad as that uses simple file io and and office documents such as doc and docx files which has a tendency to use overlapped io
file system I am writing to is NTFS
all files are affected.

I am using BTSYNC that is selectively syncing files from a remote share but as far as access goes all my reads and writes are local and I am not seeing file access issues.

[viciousviper]

Please verify if #9c11912 improves anything.

[adamrogas]

These are tests with small files.

1: So new file saving no longer hangs but the files are written to the disk and then immediately are removed

2: If you attempt to open an existing txt file and then attempt to save it an error is thrown as the driver attempts to use the original stream that only has read permissions

when using the file sample cloud disk saving a file now returns a permissions error

Attached are the two log files from my notepad test with my virtual drive "CloudShare" and the "File"
virtual drive

for this test I literally started the drive,
opened up notepad
typed "this is a test"
and then hit the close button at which point it asked if I wanted to save and I selected each of the drives named the file something like test.txt and attempted to save.

CloudShare Simple Text Save.txt
File Simple Text Save.txt

[viciousviper]

Thanks very much for this detailed report - I now understand the reason behind your observed errors much better.
To be frank, I haven't viewed the online editing use case as very important so far and rather focused on a backup/sync scenario.

[viciousviper]

Ok, most of your simple test scenario should work now:

• I was able to create a new text file by saving a short text from Notepad exe.
• After restarting Notepad.exe and opening the text file I could save a modified text to the opened file.
  All this was tested on a FILE gateway, but it should work just as well on any of the other gateways - with the exception of hubiC (currently not fully operational).

Thanks again for your cooperation, @adamrogas. Please feel free to report any additional shortcomings or to close this issue if there are none.

[adamrogas]

So I just tested this, the problem of initial writing seems to have been resolved.

but if I open a file in notepad, and type some changes and hit save or close it immediately comes back with "Invalid Parameter".....

at first glance it looks like what is really happening is the write cant access the file because the read already has the file open and has not released the file yet. this error then cascades down and ultimately the driver attempts to remove the file for some reason :) which luckily it can't for the same reason, "The File is locked"

I have included a screen shot of my system during this error, which gives a surprising amount of detail.

I have the call stack visible as well as the locals where you can see the error message,

[viciousviper]

Nice shot. But why do you think the amount of detail to be surprising?

[viciousviper]

The reason for the driver trying to remove a file when errors occur (except UnauthorizedAccessException) is again related to my bias towards a backup/sync scenario.

In the presence of unrecoverable write errors in the cloud (use of a fully transactional file system is ruled out for the time being) I'd rather have a file missing entirely in my backup than keeping a corrupt copy. Especially when considering that there is no such thing as a partially corrupt file with AES encryption.

But I'd be interested in your view on this matter. Maybe there are other cases than authorization failure that can be compensated without purging the target file.

[adamrogas]

On the file deleting
I understand the concept on a new file creation, the problem, at least from my perspective is that the error that is occurring is nothing more than a conflict when attempting to gain access to write back to the file, so in that case nothing about the file has changed but yet the original file is potentially getting removed from the system.

Now from the perspective of a remote file system view for backups this makes tons of sense. In an interactive view of a file system this is incredibly dangerous.

On the open then save issue
The file is opened and kept open with read only locking so it then is failing when the attempt is made to open the file for writing by what is essentially another process,the assumption by the file system and the application is that the same "process" will be the ones actually making the writes to the file system.

In the case of this driver being a proxy in the middle that is no longer the case and that is where a majority of the file issues are coming from in the dokan projects that are actually proxing file system access.

[viciousviper]

On the file deleting
I understand the concept on a new file creation, the problem, at least from my perspective is that the error that is occurring is nothing more than a conflict when attempting to gain access to write back to the file, so in that case nothing about the file has changed but yet the original file is potentially getting removed from the system.

I totally agree with your view. What I'd like to do is identify better those situations when "nothing about the file has changed" so there is no danger of the file having gotten corrupted by a partial write. I started checking for UnauthorizedAccessException because in this case I can be fairly certain that the target file remains unchanged. The problem arises with the unspecific IOException that is thrown in the file locking case. The same Exception could also signal an aborted network file transfer. At least this is what I think will happen - please correct my if I'm wrong.

On the open then save issue
...

You sound like you have run into some painful issues with Dokan in the past :-(.

I'm basically aware of the implications behind read/write file access contention. However, I have so far failed in tracking down the actual reason for this contention to occur in DokanCloudFS. Whenever I break into the debugger in CloudOperations.WriteFile() the read-Stream has already been disposed and the write succeeds. It is only when letting the driver run free that the Exception is thrown.

Annoying case of Heisenbug - maybe related to a threading issue.

[viciousviper]

Also, I have so far only observed the contention for the local FILE gateway in the unusual case of turned-off encryption.
This is hardly a primary use case for DokanCloudFS.

[adamrogas]

I thought about writing a file manager that was responsible for all of the actual io that gave out handles to to the processes as they were accessing the files, the theory is that the same process doesn't have to abide by the locking even if it's in conflict so it never really comes up for these apps when they are working with the disk directly.

So no file close until there are zero handles and everything else would be virtual.

-Adam

Sent from my iPhone

On May 13, 2016, at 4:04 PM, viciousviper [email protected] wrote:

On the file deleting
I understand the concept on a new file creation, the problem, at least from my perspective is that the error that is occurring is nothing more than a conflict when attempting to gain access to write back to the file, so in that case nothing about the file has changed but yet the original file is potentially getting removed from the system.

I totally agree with your view. What I'd like to do is identify better those situations when "nothing about the file has changed" so there is no danger of the file having gotten corrupted by a partial write. I started checking for UnauthorizedAccessException because in this case I can be fairly certain that the target file remains unchanged. The problem arises with the unspecific IOException that is thrown in the file locking case. The same Exception could also signal an aborted network file transfer. At least this is what I think will happen - please correct my if I'm wrong.

On the open then save issue
...

You sound like you have run into some painful issues with Dokan in the past :-(.

I'm basically aware of the implications behind read/write file access contention. However, I have so far failed in tracking down the actual reason for this contention to occur in DokanCloudFS. Whenever I break into the debugger in CloudOperations.WriteFile() the read-Stream has already been disposed and the write succeeds. It is only when letting the driver run free that the Exception is thrown.

Annoying case of Heisenbug - maybe related to a threading issue.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[adamrogas]

Understood, it's a fairly unique use case.

Sent from my iPhone

On May 13, 2016, at 4:12 PM, viciousviper [email protected] wrote:

Also, I have so far only observed the contention for the local FILE gateway in the unusual case of turned-off encryption.
This is hardly a primary use case for DokanCloudFS.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[viciousviper]

Nevertheless I may have found an ugly workaround - at least it worked for me in about 10 consecutive manual tests.
Please try the above two and below one commits (FileGateway must be updated, too).

[viciousviper]

Hi @adamrogas,

have you had a chance to check if the latest changes to CloudFS - or version 1.0.6-alpha for that matter - fixed your situation?

[drunkenbeaver]

It seems that the problem is still present while saving the MS Office files (.docx file opened - trying to save changes, two temporary files created by MS Word) - tested with Box.

There is no problem with classic Text files.

Exception occurs in Read method of ScatterStream class (during the SetAllocationSize dokan operation).

[viciousviper]

Hi @drunkenbeaver,

made some headway with the MS Office file handling issue.
I now have an internal version working that is able to open, edit and save (under a different name) a .docx file in MS Word 2016.
What I couldn't work out is in-place saving of a modified .docx file from within MS Word. The DokanCloudFS console trace looks to be without errors but still Word complains about some missing file system rights or other on the original file. I don't see any unsupported calls (like setting a readonly file attribute) nor are any Exceptions raised or any non-success DokanResults returned.
I'm inclined to leave it at that for the time being and rather bundle up the current state of affairs into a new beta release.

Maybe someone with deeper knowledge of the inner workings of MS Office file operations can help out: What is it that MS Word expects the file system to do after it saves (and apparently verifies by re-reading) a temporary ~WRD0000.tmp file?

[drunkenbeaver]

Hi there! First of all, I'm really happy that you still actively maintain this code base and I look forward to switch to develop branch to check out the latest progress on this and relevant issues.

This is what I've found MS is doing while Opening, Saving or Closing it's documents (.docx eg.) where Word.docx is a sample file I'm working with:

Open Word.docx

Create ~$Word.docx

Save Word.docx

Create ~WRD0000.tmp
Rename Word.docx > ~WRL00001.tmp
Rename ~WRD00000.tmp > Word.docx
Delete ~WRL00001.tmp

Close Word.docx

Delete ~$Word.docx

You're probably familiar with this or similar scenario.

It's quite difficult to track what's happening here, especially if at the endpoint you're trying to version these documents (ofc, original Word.docx is actually the only file that's really of interest to the users).

[viciousviper]

Hi @drunkenbeaver,

your helicopter view of MS Word's file operations is certainly correct. At this level of detail I have been able to perform the steps until

Open Word.docx

Create ~$Word.docx

Save Word.docx

Create ~WRD0000.tmp

At this point, MS Word initiates a series of Read operations on both the original and the temporary file (mostly just for the leading 512 bytes) and looks to be in some obscure way dissatisfied with the results. Next, MS Word pops up an error window citing "Word cannot complete the save due to a file permission error." and terminates the write operation.
This might be a general problem of Word not being able to update .docx files in place on a network shared volume - in which case there isn't much I can do about it.

Apart from that I have tracked down the error you reported back in December, so you'll at least be able to save a .docx file under a different name with the next DokanCloudFS release.