Protect Sensitive Information on Hugo Sites From Scrapers

Web scrapers use bot scripts — often running on networks of rotating proxies — to crawl websites, parse the HTML markup, and pattern-match text that resembles an email address, phone number, or any other data they can use to sell your information.

For this reason, I recommend taking the necessary precautions to keep sensitive details displayed in plain text on your website(s) out of the hands of scrapers.

While spam filters are more advanced than ever at preventing bad actors from poaching your inbox, I also know from my experience working in the industry that they’re never perfect.

The Solution

Luckily, Hugo provides built-in template functions that allow you to base64 encode or decode content with simple pipe filters: base64Encode and base64Decode.

We can use the encoding functionality above alongside some basic JavaScript to “cloak” the plain text emails and make it unlikely for your email to be compromised (most scrapers can’t or won’t execute JavaScript).

Note: This method will likely impact search engine crawlers as well. It’s best to avoid it when working with schema markup, structured data, or other SEO considerations.

Encoding Your Content in Hugo Templates

The encoding process is relatively simple. For illustration purposes, let’s say you have a few site parameters in your Hugo config file for displaying an email and phone number.

Here’s what that might look like in a vulnerable state:

1{{ with .Site.Params.myEmailAddress }}
2  <a href="mailto:{{.}}">{{.}}</a>
3{{ end }}
4
5{{ with .Site.Params.myPhoneNumber }}
6  <a href="tel:{{.}}">{{.}}</a>
7{{ end }}

The above would output something like this during the build process:

1<a href="mailto:user@example.com">user@example.com</a>
2<a href="tel:+1-111-111-1111">+1-111-111-1111</a>

Not ideal. Easy pickings for even your most rudimentary parser.

Let’s start by making it a little bit harder.

To encode the values, pipe a base64Encode function to the context (or variable if outside of a block’s context). I’ve also added a .needs-decoding CSS class and empty mailto:/tel: href attributes to the elements in the template. You’ll see why in a moment.

1{{ with .Site.Params.myEmailAddress }}
2  <a href="mailto:" class="needs-decoding">{{ . | base64Encode }}</a>
3{{ end }}
4
5{{ with .Site.Params.myPhoneNumber }}
6  <a href="tel:" class="needs-decoding">{{ . | base64Encode }}</a>
7{{ end }}

The result of the encoded value will be displayed on your page. It’ll look like a long string of random characters with an equals sign at the end. But we’re not done yet. We only want robots to see the encoded value. Regular users should get the decoded email address and phone number.

Decoding The base64 String With JavaScript

The next step is to write some basic JavaScript to decode the string dynamically. First, we need to select all the elements with the .needs-decoding class.

Let’s start writing our function. You can use an arrow function instead if you prefer.

1function decodeStringBase64() {
2  const elsToDecode = document.querySelectorAll('.needs-decoding');
3}

Next, we’ll iterate through all the elements with encoded content using forEach(). If the NodeList is empty, it should silently fail since we called the method on a valid type.

 1function decodeStringBase64() {
 2  // ...
 3
 4  elsToDecode.forEach(function(el) {
 5    const decodedText = atob(el.innerText);
 6    el.innerText = decodedText;
 7    if (el.nodeName && el.nodeName.toLowerCase() === 'a') {
 8      el.href += decodedText;
 9    }
10  });
11}

In our loop, we first replace the innerText of the element with the decoded string, regardless of what type of element it is. For example, a <span> or <p> with sensitive text content should work the same way despite not having an href attribute.

atob() is the built-in that does all the fancy decoding for us behind the scenes.

Lastly, for <a> elements only, we concatenate the placeholder href attribute and the decoded text to create a valid hyperlink.

The Full JavaScript Code

That’s all there is to it! It’s not the most groundbreaking code ever written, but it’ll serve our purpose just fine.

 1function decodeStringBase64() {
 2  const elsToDecode = document.querySelectorAll('.needs-decoding');
 3
 4  elsToDecode.forEach(function(el) {
 5    const decodedText = atob(el.innerText);
 6    el.innerText = decodedText;
 7    if (el.nodeName && el.nodeName.toLowerCase() === 'a') {
 8      el.href += decodedText;
 9    }
10  });
11}

To test this out, call the function somewhere with decodeStringBase64();, rebuild your site if necessary, and fire up the page. You should now see the decoded version of the email and phone number. Both the element texts and hyperlinks should contain your parameter values.

I purposefully kept this tutorial simple so those without much JavaScript experience can get it working without much hassle.

Hopefully, you can be at peace knowing the contact information on your websites is safe moving forward.