Very slow indexing

[Ed Hawco]

Hello. When I publish a project (HTML Help) the indexing phase of the publication compile used to be very quick. Since I upgraded to H+M 7 it's now very slow.

For example, I just did a direct comparison of a small project. Indexing under H+M 6 took seven seconds. Indexing the same project under H+M 7 took over a minute.

That's a significant lag, and I worry that I'll be sitting on my hands for five minutes of indexing for larger projects. Is this due to some new indexing algorithm? Or is there some configuration change with H+M 7 that maybe I can undo?

[Tim Green]

Hi Ed,

When I publish a project (HTML Help) the indexing phase

If there is an indexing phase then it's not HTML Help (local CHM files displayed in the Microsoft CHM Viewer for local software documentation), it's WebHelp (web-based HTML displayed in a browser).

If the indexing is slow, the first thing you need to do is make sure you're using the latest version of HM7. Since this is a new release, updates are going to be released quite frequently in the first period. You can check your installed version with Help > About. The current version is 7.0.8 Build 3780. If you need to update you can download the latest installer from

http://www.helpandmanual.com/downloads.html?upd

Just install on top of your current version to update.

WebHelp search indexing in Help & Manual 7 should not be appreciably slower than in HM6. Slow performance is generally related to network issues, in our experience. We would need to know more about where HM and your projects are stored to be able to say more on that.

[Ed Hawco]

Thanks Tim. I meant Web help not HTML help.

The projects are stored locally, so there should be no network issues. And as I said, I did an immediate side-by-side test. I opened a project in H+M 6 and published it (also to a local folder) and the indexing took seven seconds. Then I closed it, and opened the same project in H+M 7 -- which had been freshly installed only three days earlier (I'm not on the same machine now, so I can't check the build number), ran the same Publish task, and indexing took over a minute.

I ran the test because I had noticed that the indexing in publishing tasks was running long on the two or three different projects I had published since upgrading.

Update: I just installed H+M 7 on the machine I'm using now (I regularly use two different machines for my projects, and the files are shared on Dropbox, so they are "local" yet shared over the cloud). Ran the same project. Indexing took over a minute.

[Tim Green]

Hi Ed,

I think we had one case like this a while ago and there was a cause that could be dealt with. I'm going to confer with our developers on it and get back to you as soon as I have more information. Generally speaking, however, there should only be very minor speed differences between HM6 and HM7.

[Alexander Halser]

Hi Ed,

Which version/build of H&M 7 do you use? And: do you use the internal version of the Webhelp indexer or the "full" version of the Zoom HTML indexer? If the latter, which version of Zoom? Please also include Windows version details.

The significant lag that Tim mentioned was caused by an indexing task that had actually finished but did not properly return. When creating the Webhelp index, H&M starts an external indexer (a custom build of the Zoom search engine for H&M), which indexes the HTML pages just created and waits . If the indexer does not return to H&M, it gets terminated after a timeout. This is about the period that you are experiencing.

[Ed Hawco]

Hi Alexander. I'm using H+M 7.0.8, build 3780. Windows 7 Home Premium version 6.1 (build 7601, SP 1), 64 bit on a virtual machine (Parallels) running on a Macbook Air with OS X 10.11.2. I have both H+M 6 and H+M 7 installed, and the problem only occurs on H+M 7.

My other setup (home vs. office) is exactly the same except it's on a Mac Mini. Same version of Parallels and Windows, same versions of H+M, same problem on both machines.

I have no idea what indexer I'm using, so I guess that means I'm using the internal one. (I.e., I have not explicitly installed or pointed to another indexer.)

[Alexander Halser]

Hi Ed,

My colleage Michael just tested your configuration, Windows 7 under Parallels on OS/X and HM 7.0.8 worked without problems. Indexing time for a small project was about 10-20 seconds (which is under the timeout mark). Can you try this with a small new project that is saved locally inside the VM and report the compile times of the full-text indexer?

[Ed Hawco]

OK, I created a new small project. It lives 100% on the local virtual machine (not Dropbox). It uses a default H+M skin and does not reference an external typography repository. Indexing took about 27 seconds.

Then I made a copy of the project (with a few small changes). It also lives in the virtual machine, but uses the same skin file and typography repository (both on Dropbox) that my regular projects use. Indexing took about 27 seconds.

I don't know if helps at all, but here are the compiler messages. "ALL-IN" is the 100% self-contained project. "Tentacles" is the one that references Dropbox.

ALL-IN Compiler Messages:
Generating Full-text Search Index...
Wrensoft Zoom Search Engine [Core Engine] Version 7.0.1022b (Build: Custom for EC Software)
Core Engine: Version 7.0.1022b (Build: Custom for EC Software) on Windows 7 [Professional Edition]
Core engine IPC ready.
Start indexing (offline mode) at Thu Jan 21 11:40:02 2016
INDEX SUMMARY
Files indexed: 8
Files skipped: 47
Files filtered: 0
Unique words found: 396
Variant words found: 37
Total words found: 2957
Avg. unique words per page: 49.50
Avg. words per page: 369
Errors: 0
Total bytes scanned/downloaded: 136139
File extensions:
Successfully created all required files


TENTACLES Compiler Messages:
Generating Full-text Search Index...
Wrensoft Zoom Search Engine [Core Engine] Version 7.0.1022b (Build: Custom for EC Software)
Core Engine: Version 7.0.1022b (Build: Custom for EC Software) on Windows 7 [Professional Edition]
Core engine IPC ready.
Start indexing (offline mode) at Thu Jan 21 11:37:43 2016
INDEX SUMMARY
Files indexed: 6
Files skipped: 49
Files filtered: 0
Unique words found: 341
Variant words found: 27
Total words found: 2570
Avg. unique words per page: 56.83
Avg. words per page: 428
Errors: 0
Total bytes scanned/downloaded: 54351
File extensions:
Successfully created all required files

[Tim Green]

Hi Ed,

I just also tested the demo project you sent on both Windows directly and Windows 7 running in Parallels on a Mac. On a Windows machine indexing completes in about 2 seconds with Help & Manual 7 and about 5 seconds on HM6. On the Mac (an old Mac Pro) under Parallels it takes about 5-7 seconds, with no appreciable difference between the two HM versions.

So there is something on your configuration that is causing the problem. What versions of Parallels and OS X are you running? I tested on OS X El Capitan with all current updates and Parallels 11.

[Alexander Halser]

Indexing took about 27 seconds.

Sounds reasonable. This is certainly not a timeout that you are experiencing when it takes less than a minute. It really takes your VM that long to load the indexer, scan the pages, index them and write the Javascript files (the indexing application has increased in functionality and size and memory usage, compared to the older one in HM6).

[Ed Hawco]

Tim, I'm using El Capitan with lastest updates and Parallels 11. And remember; I use two different machines, each with the same setup, and the problem occurs on both machines. :-/

Alexander, the times reported are for a very small test project. For a small "real" project that indexes 14 files it took 52 seconds to index. For my largest project (67 files) it took 1 minute 26 seconds to index.

I suspect this will remain one of those odd mysteries. Fortunately it's not a show-stopper, just a small annoyance, so I will just have to live with it.

[Alexander Halser]

I suspect this will remain one of those odd mysteries.

I guess so, yes. There is unfortunately not much we can do about it. If the lag of the indexer is caused by a timeout because it does not return to H&M, then we can test and improve this. But under a minute, it is not a matter of timeouts. This is the real time that the indexer needs to create the full-text index and I suppose it is working properly.

The time difference between HM6 and HM7 is caused by an updated version of the Zoom indexer. This new version has more functionality and might be somewhat slower than the older version. In our tests, the time difference was barely noticable. But I guess that your special configuration of Windows in Parallels on OS/X is the reason, why the indexer is a bit slower than it could be. 15-30 seconds for creating the full-text index is perfectly normal. If it is over a minute, it is typically a large index that is being created.

We are going to report this to Wrensoft, the vendor of the Zoom indexer. But this configuration is so special - we cannot even duplicate the times with the same setup on our test machines, that it will be hard to figure out what exactly causes the 30 extra seconds.

I remember in our tests of the updated indexer, that it temporarily requires more RAM than the version in HM6. Could this be a bottle neck on your VM? Does the compile time get shorter when you increase the available RAM for the VM?

[Ed Hawco]

AHA! I didn't think it was a RAM bottleneck because I have 4 GB of RAM assigned to the VM, and that should be sufficient. But I looked again at the configuration, and I saw that I had assigned only one processor to the VM. So I switched it to four processors.

The project that was taking 52 seconds to index now indexed in about six seconds.

DING!

As a test I set the configuration back to TWO processors. No perceptible change; it took six or seven seconds.

So the take-away is that the indexer really prefers to use at least two processors on a VM. RAM seems to be less of a factor.

PROBLEM SOLVED!

[Alexander Halser]

Perfect
Thank you for reporting back!