Non-searchable in Google; custom server-side code

cindynotbrady · Tue Jan 19, 2016 8:22 pm

I am working with our web team to get our online manuals behind security, so the information cannot be searched through a google search. So my web team gave me this code to place in the html/css:

Code: Select all

   <%
		Dim CurrentDomain : CurrentDomain = Request.ServerVariables("HTTP_HOST")
        Dim currentURL
		if not session("viewtrain") And currentURL = "https://secure.goldpointsystems.com/" Then
			response.redirect("https://secure.goldpointsystems.com/default.asp")
		ElseIf not session("viewtrain") And currentURL = "https://gpssecure.securexfr.com/" Then
			response.redirect("https://gpssecure.securexfr.com/default.asp")
		ElseIf not session("viewtrain") And currentURL = "http://markettest/" Then
			currentURL = "http://markettest/"
		ElseIf not session("viewtrain") And currentURL = "http://d5588/" Then
			currentURL = "http://d5588/"
		ElseIf not session("viewtrain") And currentURL = "http://localhost/" Then
			currentURL = "http://localhost/"
		End if
		%>

I tried pasting it in the head of our skin's layout, but when I publish it, it wrecks havoc on the layout (the search and home index are pushed down). Do you know where I can safely paste this code so it doesn't mess up the layout? I can be more specific in further posts, or I can email you our code.

Thanks so much!

Cindy Fisher

Unread post by **Tim Green** » Wed Jan 20, 2016 10:21 am

Hi Cindy,

That code is not JavaScript -- I think it's Visual Basic -- so it would not be directly compatible at all unless the entire page was built for it. You also don't need to add scripting to your page to prevent Google searches and it is really a not such a good idea anyway. It is also highly unlikely that you could add something like this to a HM skin without a complete rewrite.

In fact, you don't need to do anything at all to the skin or pages themselves to prevent Google searches. Your web administrator just needs to edit the robots.txt file on the server to exclude the folder and Google and other leading search engines will all respect that. Your web people just need to Google configuring robots.txt to find instructions if they are not familiar with this.

If you want a belt and suspenders solution, the only thing you need to change in your project to achieve the same aim is to add this line to the <head> section of all the templates in your skin:

Code: Select all

<meta name="robots" content="noindex,nofollow" />

You need to check the HTML Page Template > Default, Layout (if your skin uses one), Table of Contents, Keyword Index and Search templates in your skin and add the line together with the other <meta> tags there. If the robots tag already exists then replace it with the above version. Otherwise just add it. (All the templates except the page template are in the Publishing Options > WebHelp section of the skin.)

Simon Dismore · Wed Jan 20, 2016 1:32 pm

Tim's answer will prevent indexing, but the code you posted seems to be for denying people access to your documentation until they have logged in. You might have business reasons for that, e.g. to prevent potential buyers from evaluating the functionality without being guided by a sales person, or for security (denying hackers information about the system), or simply to track who is reading which help topics.

As it stands at the moment, once someone has a link to any page of the documentation, they can browse any of your HTML user guides (e.g. I reached CIM GOLD, GOLDWriter, GOLDMiner). However, the guides that have been published as ASP pages (e.g. GOLDPrint) are blocked, because the server does a 302 redirect to the default login screen. I suppose you are looking for similar security for all your guides?

If so, it might be as simple as republishing your web guide with different filenames so that they are processed by ASP.NET on your server. I don't have a ton of experience with this but these steps worked on my local (IIS 8) server:

In Help & Manual's HTML Export Options set the Extension for HTML files to ".aspx". You'll have to type this manually in the field, it's not one of the listed values.
In H&M's WebHelp publishing task, set the Index page to "default.aspx".
Publish and copy all the files to an ASP.NET folder on your IIS web server. I won't attempt to explain how to set this up: I'm not an expert, you are running an older version of IIS, and anyway there must be someone in your web team who already knows this stuff.
If you don't already have a 'web.config' file in the folder, a minimal one for test purposes might be:
Code: Select all
```
<configuration>
    <system.web>
        <customErrors mode="Off"/>
    </system.web>
</configuration>
```

Now point your browser at the folder and see what happens. It may be that the entire server is already configured to force a login, in which case your code fragment won't be needed. On that other hand if it is needed, try putting it at the beginning of your HTML template, above the <%DOCTYPE%>, and publish to aspx as above. Note that your secure.goldpointsystems.com domain is being hosted on IIS 6 whereas http://www.goldpointsystems.com is hosted on IIS 8.5, so that might account for some differences of behavior.

Good luck...

Simon Dismore · Wed Jan 20, 2016 1:43 pm

By the way, if you are doing any tests using IIS, you might find this snippet useful. It lists a lot of useful data about the current session. Just paste the code into an HTML Code Object and republish to aspx.

Code: Select all

<h2>ServerVariables listing</h2>
This ASP.NET page (demo created by <a href="http://www.4guysfromrolla.com/demos/servervariables.asp">4guysfromrolla</a>) lists the server and environment variables for the current user's IIS Session.
<p><hr><p>
<table>
    <% Dim Name as String %>
    <tr>
        <td><strong>Variable</strong></td>
        <td><strong>Value</strong></td>
    </tr>
    <% For Each Name In Request.ServerVariables %>
    <tr>
        <td><%= Name %></td>
        <td><%= Request.ServerVariables(name) %></td>
    </tr>
    <% Next %>
</table>

cindynotbrady · Wed Jan 20, 2016 5:57 pm

Holy smokes, this is good feedback! We don't want our site searchable, and yes, we do want people to first log on before accessing our documentation. So thanks for all this great information. I will share it with our web gurus, and hopefully these tips will help them. I will let you know how it turns out.

I will add Tim Green's information to start, because I know how to do that. The other stuff I will have to confer with them.

Thanks again!

cindynotbrady · Wed Jan 20, 2016 11:16 pm

So I did exactly as you suggested and it worked! If we use the aspx format the code is correct.

One quick question, however. My web gurus were wondering why when we're testing, the browser brings up this address:

http://127.0.0.1:81/....

Isn't there a way to tell it to bring up the testing server? We couldn't find it. We wanted it to build to here:

http://localhost/test/....

Thanks!

Simon Dismore · Thu Jan 21, 2016 12:02 am

It's late so this is just a quick note.

On most Windows systems localhost and 127.0.0.1 are the same thing: a web server running on your local machine. In my experience this is *not* a good way to check things, because you aren't acting as a real user. Everything about authentication and authorization will behave differently, and issues with fonts etc won't be visible because you already have them installed locally. To do a proper test you must put the documentation onto a web server and browse it from vanilla systems. That's why we have an IIS test server on our intranet. It doesn't have to be expensive: ours is a retired workstation running Windows Server 2012 Standard on a 3GHz Xeon with 8GB RAM and a couple of 500GB drives. If your company is serious about technical documentation, that box costs c. $700 today new at Dell Outlet, excluding the operating system. After a few hours configuration you would be testing on something almost identical to your production web servers, and it would last for years. That's a no-brainer in my opinion!

cindynotbrady · Thu Jan 21, 2016 7:59 pm

Thanks for the tip. We do test it on a testing web server, then push it to a live web server. I was just testing it on my local machine, though, because I personally don't have security clearance to our testing server. (Me writer...me no touch anything....but actually through this process, they did give me security clearance to test files out there.) For some reason, when I would overwrite "localhost" for the 127.0.0.1 location in the web browser, my code would work correctly. Strange.

(And I'm pretty sure I'm leaving out details that would make it clear as to why that was happening.)

Unread post by **Tim Green** » Fri Jan 22, 2016 7:50 am

Hi Cindy,

For some reason, when I would overwrite "localhost" for the 127.0.0.1 location in the web browser, my code would work correctly. Strange.

That's normal. For a browser, 127.0.0.1 and the word "localhost" are synonymous -- they both point to the same address, which is the local "home" web address on every computer on every platform (Windows, Mac, Linux, Unix..).

As a trivia point, this is also the reference behind those geek T-shirts that read "There's no place like 127.0.0.1".

cindynotbrady · Mon Jan 25, 2016 10:12 pm

Ha, ha! So funny that I just saw a bumper sticker with that, and I was telling my husband that I've seen that when I test my help files. Now I know. This site helps me so much in so many ways!

cindynotbrady · Tue Mar 15, 2016 5:50 pm

So I've been trying to publish with aspx, and it looks like it works until it gets out to our live site. And then we are thrown for a loop. What happens is that someone puts in their user name and log in, logs onto our documentation site. Then clicks a link in the navigation that takes them to one of the manuals that is built with Help&Manual, but then the log in screen appears yet again.

I wonder if it has to do with the fact that our documentation is on a newer server than the main site.

I keep trying different things (like publishing the manual to a .asp instead of .aspx), but nothing seems to be working. I noticed that even though I have set my export files to be .asp, the main index file still only publishes to .html. When I overwrite the .html index extension to .asp, it doesn't build at all and gives me an error.

Any other hints perhaps? I know this is a difficult issue and not easy to explain.

Thanks,

Cindy Fisher

cindynotbrady · Tue Mar 15, 2016 5:54 pm

Note: It may help to understand our situation. We made the horrible decision to put all our documentation in Dreamweaver. We've been slowly trying to move all our content over to Help&Manual a manual at a time (loans, deposits, user's guides, etc.), but we have thousands of pages of content and at least 35 different manuals. So some stuff is still in Dreamweaver and other stuff has been transferred over to Help&Manual. So as not to confuse our users, we have tried our best to make the Help&Manual skin match our other CSS template we created in Dreamweaver. It's not perfect, but it's a good solution until we find the time to make sure all manuals are in Help&Manual.

cindynotbrady · Tue Mar 15, 2016 7:14 pm

Nevermind. We figured it out. We had a weird piece of code in there. We placed this code at the top:

<%@ Page Language="VB" %>

Once I took it out, it all worked fine.

Help & Manual Forums

Non-searchable in Google; custom server-side code

Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code

Re: Non-searchable in Google; custom server-side code