Pmv 3.17 Blocks On Foc If Last Drive Used Is Unavailable, And sometimes even when it is! Options

[asavage]

I have caught up on my work for this batch. Another couple of hours of work should occur next week.

[asavage]

I had a batch of 128 jpeg files to convert in-place to png yesterday.
The directory contained those files, plus seven .php and two .html files.

Open file name = *_b.*
Type = <Custom>

Thumbnail Type = Mixed
Icon Thumbnails = Manual

During the convert process, the box went to 100%, the swap file remained at 20Mb, and the system ground to a halt. I left it run overnight, but no change by morning. After reboot, I found that it had actually converted 23 of the 128 files.

[Peter]

Hmm. Now you threw in TWO new factors at the same time, and we don't have a clue which one that caused the problem to appear.

It could be the switch to Mixed, or it could be the usage of the * mask.

Please switch back to On-The-Fly and see if that remains stable.

I'm trying to establish a setting where a problem does NOT occur. Possible problems are:
-Strange file that causes PMView's format auto detection to use up all memory
-Problems with EAs in one or more of the file system drivers on your system

However, a problem with the format detection should normally be repeatable, so it's more likely that this is a problem with EAs.

[Peter]

Another thing you could try is to disable "Use idle time priority" on the Special page in PMView's options notebook.

[asavage]

I worked with it several hours yesterday, and with on-the-fly and *.* it did not have the 100% CPU problem (although I got several out-of-memory dialogues, and when it does that it closes).

=============================================
I normally run two PMV windows. I scan two pages at a time on my large-format scanner. I open the scan in one PMV, then copy half of the scan to the clipboard, then in the other PMV I Edit->Paste->As New (<Alt+E,P,N).

Often, the copy to clipboard fails and PMV closes, so I have to open PMV again and load the scan again, this time cropping to 1/2 size and saving a single page to open in the other PMV.

[later edit: actually, if I <Ctrl+X> a small section of the large scan, prepatory to a Paste, that's when I would see this behavior. I think I also saw it on a copy to clipboard, but I'm not so certain about that any more.

In the Win version, on XP SP3, copy to clipboard often silently fails with large copy operations. PMV doesn't say the copy to clipboard fails, but there's nothing to paste elsewhere.]

I have Undo enabled, which doubles the mem requirement. I can't do without Undo.

It would be nice if PMV did not die when it gives the "out of memory" message.
=============================================

FWIW, with 200Mb scan files and on-the-fly for thumbnails, it's a very slow process to work continuously with PMV. The outline of the thumbnails is not shown, nor the file names, until it has read the file (that's apparently what's going on; the actual mechanism isn't apparent), so if I open a dir with, say 1.6Gb of scans, it's minutes until I can even click on one, since there's not even empty thumbnails in the FOC.

I never knew this before, because I never use on-the-fly.

[Peter]

I worked with it several hours yesterday, and with on-the-fly and *.* it did not have the 100% CPU problem (although I got several out-of-memory dialogues, and when it does that it closes).

That's what I'm afraid of: The reason for the hangs you are seeing are most likely because it hangs during processing of EAs (Extended Attributes) in the file system driver. This could explain why I can't repeat the problem here (different system configuration).

Now that you mention it, it's possible that the file system driver works flawlessly under normal conditions but hangs in low memory conditions. Repeating the situation may be difficult.

Actually, when thinking more about this, I guess there's a chance that the problem is in PMView too and is caused by an out-of-memory situation where memory is allocated directly (not via C++ new/delete). I will have a quick look at this and see if some improvements are needed.

It would be nice if PMV did not die when it gives the "out of memory" message.

PMView was conceived back in 1992 when 32-bit OS/2 was brand new, most machines had 16MB memory or less, and nobody could imagine hitting the 2GB (later 3GB) 32-bit barrier of the operating system. Actually most developers did not realize that this addressing barrier existed, or envisioned it would somehow be solved the day that much memory is needed. (Microsoft fixed it on Windows by moving to 64 bits).

Consequently I designed PMView to assume memory is always available. The "out of memory" message is actually a fairly new addition that was added in PMView v3.24. The message is displayed once PMView catches a C++ runtime out-of-memory exception. At this point it's already too late to do anything fancy.

Today the problem can be solved by switching to 64-bits, so it just does not make sense to spend a lot of time on reworking all existing code to handle the situation. That time is much better spent on adding new features and fixing bugs.

FWIW, with 200Mb scan files and on-the-fly for thumbnails, it's a very slow process to work continuously with PMV. The outline of the thumbnails is not shown, nor the file names, until it has read the file.

The file names (and thumbnail placeholder outlines, if applicable) should be shown almost immediately UNLESS you have configured PMView to sort files based on a criteria that requires examining the files in detail (file format, image size, image depth, etc.). As long as you sort by a feature that is available by a common "dir" statement, file names and outlines should appear almost immediately. (When EAs are used, PMView store the additional info in the EAs, which is why you can sort on any criteria and still keep it quick).

Thumbnails are added by a background thread, so they should appear one by one (slowly if you have really big files).

[Peter]

FWIW, this great article may be of some value:

http://www.os2voice.org/VNL/past_issues/VN.../feature_3.html

[asavage]

Now that you mention it, it's possible that the file system driver works flawlessly under normal conditions but hangs in low memory conditions. Repeating the situation may be difficult.

You're telling me! It's been quite the chore to try to replicate it here, and when I'm not trying, it always seems to rear its ugly head, esp. when I don't have work saved to disk [grrrrr].

Actually, when thinking more about this, I guess there's a chance that the problem is in PMView too and is caused by an out-of-memory situation where memory is allocated directly (not via C++ new/delete). I will have a quick look at this and see if some improvements are needed.

Consequently I designed PMView to assume memory is always available. The "out of memory" message is actually a fairly new addition that was added in PMView v3.24. The message is displayed once PMView catches a C++ runtime out-of-memory exception. At this point it's already too late to do anything fancy.

Noted. I can live with it, as on the OS/2 platform it's fairly predictable. That is, it will work for some number of images and operations, and then it will consistently fail, even if I close & reopen every PMV instance, until a reboot. It's still faster than working on the 'doze side, even so.

Even swapping to disk is preferable to losing work, though. If it were possible to catch the situation before the exception handler, a dialogue to "save now"?

Today the problem can be solved by switching to 64-bits, so it just does not make sense to spend a lot of time on reworking all existing code to handle the situation. That time is much better spent on adding new features and fixing bugs.

Agreed, for this issue.

The file names (and thumbnail placeholder outlines, if applicable) should be shown almost immediately UNLESS you have configured PMView to sort files based on a criteria that requires examining the files in detail (file format, image size, image depth, etc.). As long as you sort by a feature that is available by a common "dir" statement, file names and outlines should appear almost immediately. (When EAs are used, PMView store the additional info in the EAs, which is why you can sort on any criteria and still keep it quick).

And I've always used the Mixed setting, so until now I only saw the initial delay upon the first read of the image. But we needed to know if the 100% CPU problem would appear using on-the-fly, so I gritted my teeth and suffered it.

I do not have representative 200Mb images here right now to test, so I can't replicate, and 3.72 is working fine with the stuff I have available, but the lack-of-placeholders was a real problem yesterday. I was pointing the FOC to a local drive and it would take, oh, maybe a full minute to even show placeholders.

To be fair, I do not know if I can blame PMV, as I am running two PMV instances plus SANE scanning in the background, and I/O is very high. CPU is only high when I'm rotating (100% for about 30 seconds to two minutes, depending on the image). I don't expect anything to work snappy when CPU is 100%, but even with high disk I/O I do expect the placeholders to draw.

The problem is that I will be saving an image in one PMV (and that will take about two minutes, because it's saving non-local, and they're large files and one segment between here & there is still 10Mb/s) and while that's saving, I want to load the next image into another PMV instance, but I can't, because the image I want to load was just scanned in, so PMV's never indexed it before. I am frequently 5 to 10 scans behind with PMV -- that is, I can scan quite a bit faster than I can process them in PMV, so let's assume there are five "new" images in the local scan dir and I want to load one in PMV whilst the other PMV is busy saving to the remote drive.

But the FOC will not immediately give me anything to click on. I don't need the thumbnails, the scans are in filename sort order (that's the way TAME & SANE name them), I just need to load the next image. So I wait.

Thumbnails are added by a background thread, so they should appear one by one (slowly if you have really big files).

I don't mind the slow thumbnail creation, but I wasn't getting the placeholders fast, either.

I have no new work to scan on the horizon, but next time I do, I'll test this part further.

Meanwhile, do give the 100% CPU problem (or possibly low memory situation) another thought. And, Thanks.

[Peter]

Thanks to your valuable feedback, and after thinking some more on it, I think I can do a relatively easy fix that will be helpful and serve as an "early warning" before work is lost:

You mentioned the Undo as a possible memory hog, and I think that's a very good hunch! I believe 50% of the time, the undo-memory may be what triggers an out-of-memory condition. I will see if I can make PMView recover from this with only a warning message that lets you continue work. Possibly this will be a good help for preventing lost work...

[asavage]

FWIW, this great article may be of some value:

http://www.os2voice.org/VNL/past_issues/VN.../feature_3.html

That's a great article. I read it entirely, and while I do not claim to understand it all, it's still enlightening. Thanks for the link.

[Peter]

Al,

Do you think there's a chance that this problem occurred with the release of v3.68?

v3.68 fixed a 10-year old bug with thread synchronization. This is the only change I can imagine would have an impact on the the FOC behavior. It is possible that the added synchronization brings out new problems if underlying drivers are not well behaved when accessed concurrently by the same application. The added synchronization in v3.68 combined with unexpected resource locking by drivers may possibly result in a deadlock situation.

If/when you have time to try this out, please e-mail me and I will send you links to download v3.67 and v3.68. Comparing these two should give you a clear indication whether the problem is caused by the synchronization added in v3.68 or not.

Another user reported seeing the same problem as you when he upgraded from v3.66 to v3.72. Apparently the problem appeared sometime between these two versions. To me, v3.68 looks like the most likely one, but I need to get this verified before I can work on a solution.

Now, in your original post you said you were going to backlevel to v3.70 which made it sound like this bug is new in v3.71. If this is the case, can you confirm that v3.70 indeed works fine and v3.71 does not? My thinking is that v3.70 has the same problem and that you need to go back all the way to v3.67 before the problem goes away. If I can get this confirmed, then we're half the way to solving the issue...

Thanks,
Peter

[asavage]

Do you think there's a chance that this problem occurred with the release of v3.68?

I had an opportunity to test this theory today.

I've been re-creating thumbnail icons as EAs. Several thousand files. I turned on Automatic thumbnail creation and Icon thumbnailing.

I used the FOC to navigate through directories, letting it finish creating thumbnails in each dir before moving on.

With 3.74, it seems reasonably OK unless I open two PMV windows and use two FOCs to do the thumbnail creation. In that situation, one of the FOCs locks up, CPU goes to 100% and the system comes to an effective stop: the clock does not advance seconds.

I have CADH installed, so I can C-A-D, run TOP and find that one instance of PMV is using 100% of the CPU. I can kill it from TOP, and the system resumes normally. The other PMV's FOC continues to create thumbnails fine.

I can start another instance of PMV and can bring the system down again pretty regularly this way, given enough files that need thumbnails created for them.

I installed PMV 3.66 instead of 3.78, and for a while it seemed to not exhibit this behavior. I though you were on to something, but about a half-hour later it did the same thing. Since I was leaving the house for a while, I left it running at 100% for four hours. When I returned, the clock was advancing jerkily (updating every ten seconds or so) but the CPU (Pulse) was still at 100% and no new thumbnails had been created as far as I could tell.

I used CADH & TOP to kill it, and things resumed as normal again.

I can't give you better clues that that.

[Peter]

Al,

It seems like something in your disk system is deadlocking. Sorry I can't provide more help than that.

Thanks,
Peter