Hi, I have a decent computer with a metric crap-ton of RAM and trying to determine the best way to get my outputs to regenerate super quickly...
e.g. during my scripts to rebuild all my help and PDF files I never see the CPUs rise about 20% for HELPMAN.EXE and never see the RAM utilisation or IO for that process or related processes show any bottleneck
I also utilise a RAM drive for system and user \temp\ files (as my computer has so much RAM its silly not to)
Is there anything else I can do to optimise the processes to go faster? my scripts for about a dozen PDF publishing destinations and maybe 1000 web pages takes about 15 minutes... and well, you know what developers are like, they want their updated content committed like five minutes ago (unfortunately they do not use a build server or publish help to the web for such things. Sigh.)
Any suggestions for optimising speed of content compiling?
Moderators: Alexander Halser, Tim Green
Any suggestions for optimising speed of content compiling?
David Scott
Documentation Infrastructure Consultant
https://www.sourceone.co.nz
SourceOne. Documentation, engineered.
Documentation Infrastructure Consultant
https://www.sourceone.co.nz
SourceOne. Documentation, engineered.
- Tim Green
- Site Admin
- Posts: 23189
- Joined: Mon Jun 24, 2002 9:11 am
- Location: Bruehl, Germany
- Contact:
Re: Any suggestions for optimising speed of content compilin
Hi David,
Fifteen minutes for that load sounds pretty much par for the course. What takes time is not the processing -- as you see yourself the processor load is not all that high -- but all the physical IO of reading and writing all the files. In PDF in particular very large temporary files need to be written while rendering the final PDF, and then translated to PDF, and that takes a while. PDF rendering time will double if you use Page X of Y in your page numbering scheme, because then the entire document needs to be rendered twice to get the page references right.
What will slow your process down most is doing anything at all on a network drive. That is always a bottleneck and it will increase rendering time by orders of magnitude. The next factor is drive speed: Ideally everything should be on the fastest local SSD drives possible, and they should all be installed inside your local computer. External USB drives will also be much slower, including USB3, even if they are SSDs. External USB-C-Thunderbolt SSD drives can sometimes be as fast as internal SSD, but only if everything is working correctly, and on Windows PCs it often isn't. External USB-C/Thunderbolt connections still very wildly in their performance, so it's not possible to say definitively that they are faster or even fast enough, at least not yet.
Of course, the computer should also be fast enough to make full use of the I/O capabilities of really fast SSDs, but if you're investing in drives like that, it's unlikely that the computer will be far behind.
We've done testing with RAM drives and they only provide very marginal improvements over current SSDs. On balance, they're not really worth the trouble.
Fifteen minutes for that load sounds pretty much par for the course. What takes time is not the processing -- as you see yourself the processor load is not all that high -- but all the physical IO of reading and writing all the files. In PDF in particular very large temporary files need to be written while rendering the final PDF, and then translated to PDF, and that takes a while. PDF rendering time will double if you use Page X of Y in your page numbering scheme, because then the entire document needs to be rendered twice to get the page references right.
What will slow your process down most is doing anything at all on a network drive. That is always a bottleneck and it will increase rendering time by orders of magnitude. The next factor is drive speed: Ideally everything should be on the fastest local SSD drives possible, and they should all be installed inside your local computer. External USB drives will also be much slower, including USB3, even if they are SSDs. External USB-C-Thunderbolt SSD drives can sometimes be as fast as internal SSD, but only if everything is working correctly, and on Windows PCs it often isn't. External USB-C/Thunderbolt connections still very wildly in their performance, so it's not possible to say definitively that they are faster or even fast enough, at least not yet.
Of course, the computer should also be fast enough to make full use of the I/O capabilities of really fast SSDs, but if you're investing in drives like that, it's unlikely that the computer will be far behind.
We've done testing with RAM drives and they only provide very marginal improvements over current SSDs. On balance, they're not really worth the trouble.
Regards,
Tim (EC Software Documentation & User Support)
Private support:
Please do not email or PM me with private support requests -- post to the forum directly.
Tim (EC Software Documentation & User Support)
Private support:
Please do not email or PM me with private support requests -- post to the forum directly.
Re: Any suggestions for optimising speed of content compilin
Thanks, good to know that at least I'm not doing something wrong or missed an obvious optimisation opportunity
David Scott
Documentation Infrastructure Consultant
https://www.sourceone.co.nz
SourceOne. Documentation, engineered.
Documentation Infrastructure Consultant
https://www.sourceone.co.nz
SourceOne. Documentation, engineered.