A Complete Breakdown of Technical SEO

EDITOR’S NOTE: Being able to run a sound tehcnical SEO audit has become a stable in any good SEO strategy. What is technical SEO you ask? Technical SEO is any sufficiently technical action undertaken with the intent to help search engine bots crawl, index, and render your site. Understanding technical SEO will help your website rank higher in search results overall. There are a lot of great free resources out there on SEO (like our blog!) that teach you how search engines work and how you can apply the information to your own site. Remember to be patient with your results! SEO is not a short term strategy, but it will result in having higher traffic for less work in the long run. If you have any questions about technical SEO or want us to take care of it for  you, reach out to us at Prebuilt Sites or The BBS Agency. We’d love to help you out!

(This article was written with the help and input from Alexis Sanders, Paul Shapiro, and Bartosz Goralewicz.)

I very well remember the cold sweat on my forehead when I opened our client’s analytics reports in May 2012. I was working for an SEO agency in Germany and traffic of a few of our clients had dropped sharply. It was about a week after Google had rolled out “Penguin”.

Back then, it was perfectly normal to buy some backlinks as part of your SEO strategy. Nowadays, not so much.

Instead, when starting to learn SEO, we start with keyword research, trying to understand user intent, screen the existing content, and… do a technical SEO audit. The latter has become a stable part of every SEO strategy with dozens of tools able to help you.

What is Technical SEO?

At TechSEO Boost, a conference organized by Paul Shapiro, Russ Jones gave a good definition to explain what is Technical SEO. He said, “any sufficiently technical action undertaken with the intent to improve search results”.

Alexis Sanders, Technical SEO manager @ Merkle, gives a more specific definition: “Technical SEO covers the crawl, index, and render portion of the “crawl, index, rank” model. At a high level, you need to learn how to answer these questions:

• Can search engine bots crawl/find your page? (includes topics such as status codes, sitemaps, information architecture, robots.txt, facets)
• Can search engine bots index your page? (includes: meta robots)
• Can search engine bots render your page? (includes: JavaScript, the DOM, page speed)
• Can bots understand content on your page? (includes: structured data (Schema.org, HTML), accessibility)
• Are you sending search engines the proper signals for dealing with ranking content? (e.g., canonical tags, dealing with pagination)
• Is this page worthy of ranking? (includes: content relevance, authority, HTTP, UX, mobile-friendly, site latency)”

 

(Spider graph used by Sanders to illustrate her skill confidence) Photo: SEM Rush

The big technical SEO topics are:

• Crawling
• Rendering
• Indexation
• Mobile
• SSL
• Structured data
• Migrations
• Page speed
• Content optimization
• Status codes
• Site structure

Quite a lot, isn’t it? When I learned SEO, Google was much simpler. If you enter Search Engine Optimization nowadays, you surely feel overwhelmed.

That is why I wrote this article: to help you navigate the deep sea that is technical SEO.

How Technical SEO Got Put On The Map

There has always been a certain degree of technicality in SEO. However, in the last 5–7 years technical SEO became its own discipline next to content (marketing) and link building.

 

Photo: SEM Rush

That is also how Bartosz Goralewicz, co-founder, and head of SEO @ Elephate, got started:

I started in SEO around 2011–12, most SEOs know that it wasn’t the best moment to start. I figured that this was a huge opportunity for me. I remember people making fun of “white hat SEO” and saying that only links matter. I started looking into what was called “on-page SEO” and as things evolved, I went deeper and deeper.

The rise of Technical SEO is the result of three trends:

• First, the ongoing war of Google against backlink spam and “overly active” link building.
• Second, the rapid progression of Google’s algorithm.
• Third, the sophistication of websites.

The Penguin algorithm and sudden spike in manual link penalties pushed SEOs to look for other levers to grow organic traffic. Besides content and content marketing, technical SEO turned out to be much more potent than assumed pre-penguin.

Google’s algorithm(s) advanced a lot leading up to Penguin, as you can tell from the progression of its updates:

• 2003 — Florida (anti-keyword stuffing spam)
• 2005 — Jagger (anti-link spam)
• 2009 — Caffeine (near real-time indexation)
• 2010 — MayDay (anti-thin content)
• 2011 — Panda (“quality”)
• 2011 — Google starts using SSL in search
• 2011 — Freshness (prioritizing fresh content)
• 2012 — Penguin (anti-link “spam”)
• 2012 — Knowledge Graph
• 2012 — EMD (downgrade exact match domain)
• 2013 — Phantom (quality update)
• 2013 — Hummingbird (core algorithm overhaul)
• 2015 — Rankbrain (contextual search)

The two short years between 2011 and 2013 shaped much of what Google is today. Google also introduced standards that made it easier for Webmasters to control and monitor indexation:

• 2005 — Google introduces XML sitemaps
• 2009 — rel-canonical tag
• 2011 — Schema.org
• 2011 — rel=next/prev

Of course, the technical SEO toolset is much greater. It covers so many topics that it is a role by itself, and that is how organizations should treat it.

Websites themselves have become more complicated and interactive, which challenges page speed, indexation, and rendering.

Consumer behavior changes towards using other devices to browse the web. For many years, the revolution was focused on “mobile devices”, and in the upcoming years, it will be driven by voice search.

That begs the question: “what technical SEO actually is and what fields it covers.”

How do you learn technical SEO?

Learning anything always takes the same ingredients: mindset, knowledge, application. Too many people focus on the knowledge part, and that is why they fail, get overwhelmed or feel like they are spinning their wheels.

Let’s start with mindset.

The Right Mindset for Learning Technical Search Engine Optimization

Every brain surgeon starts with basic biochemistry in med school. In technical SEO, you also start with the basics and then specialize. It takes a certain humbleness to do that but it is important for success.

Part of that is admitting that you don’t can’t know everything. Bartosz Goralewicz is a strong proponent of that mindset:

If you can’t explain why something happened (e.g., a website drop), the healthy thing to do is say ‘I don’t know what caused that drop’.

Good technical SEOs don’t know the answer to everything, but they can find it out.

The only constant in this industry is change, so you’re always going to learn SEO trends to stay ahead of the game. Google never stops evolving, and we have to reverse engineer ranking factors.

There is a word for keeping up with subjects that always change: continuously learning. It means that you can never stop educating yourself — ever. SEOs should spend at least 10–15% of their time keeping up and learning.

 

Photo: SEM Rush

Alexis Sanders agrees: “Learning Technical SEO is definitely a moving target. It is constantly evolving and advancing, so learning it is an ongoing process, with many rabbit holes (so many crawl traps to explore! ?).”

That leads us to another requirement for becoming a good technical SEO: being ready to change your mind when the data proves it.

When you look closer, some are not able to drop their theories and beliefs even when the data and experiments say otherwise. SEO is very dynamic. If your SEO beliefs haven’t evolved over the last 5–7 years, then your chances of success are close to 0.

— Bartosz Goralewicz

This mindset is fundamental: humbleness, continuous learning, and adaptiveness.

Next, let’s talk about what you should actually learn: knowledge.

What should you learn to become proficient at technical SEO?

The good thing is that all of this information is much readily available online, to be learned for free. There are tons of websites where you can take university classes on these subjects, read free textbooks, and tutorials on them.

— Paul Shapiro

The amount of material shouldn’t be a problem nowadays. It is all free.

Alexis Sanders recommends “keeping up with industry news and following relevant publications and blogs, such as:

• Google Webmaster Central Blog
• Google Research Blog 
• Google’s Blog
• Bing Search Blog
• Moz’s Blog
• Search Engine Round Table
• Search Engine Journal
• The SEM Post
• SEMrush Blog
• Search Engine Land
• SEO Skeptic by Aaron Bradley
• SEO by the Sea by Bill Slawski
• Deep Crawl’s Webmaster Hangout Notes
• Merkle’s Digital Marketing Reports

The way you want to build your knowledge is to start broad and then go narrow. If you want to build a high house, you need a strong base.

 

Photo: SEM Rush

Here is a rough outline of what a syllabus could look like:

First, start with basic web development (HTML, CSS, a bit of Javascript) and then dive deeper into information retrieval and eventually, computer programming. There are numerable free online courses to learn basic web development out there, for example from W3schools or Code Academy. Mozilla offers free introductions to HTML, CSS, Javascript.

A list of YouTube channels about web development:

• LearnCode.academy
• DevTips
• Wes Bos
• Codecourse
• Quentin Watt Tutorials
• LevelUpTuts
• The Net Ninja
• howCode
• Traversy Media
• LearnWebCode
• Google Developers
• Brad Hussey
• freeCodeCamp

Learning programming takes time and lots of practice. One tool I find continuously helpful is “Anki”, a spaced repetition app. It allows you to learn with digital flashcards and a system that helps you to store information in your long-term memory. Spaced repetition works by repeating things you find hard to memorize more often and things you keep easily in mind less often.

 

Photo: SEM Rush

Second, check out some of the most utilized SEO tools out there and begin to put your learnings into practice. Chrome developer tools and the Mozilla developer tools documentation. Both are very underrated technical tools, and the documentations provide interesting and helpful information about many web technology topics.

They will also aid you in assessing the “health” of a page or site later on. In 2018, many technical performance problems are related to:

• How Javascript is rendered.
• Progressive web apps and AMP.
• Entity-based ranking.

Third, learn SEO aspects such as how a search engine works and its technical components. Understand the difference between “Indexer” and “Crawler”, what a “Document-Term Matrix” is and the concept of “Semantic Search”. The patent on “ Scheduler for search engine crawler” should give you an idea of the complexity of the topic.

 

Photo: SEM Rush

Read the original Google paper, the PageRank patent and the other 9 patents of Bill Slawski’s top 10. On top of that, Ian Roger’s explanation of PageRank is great and don’t forget the “How search works” section by Google.

The purpose of digging deeper into search engine architecture, information retrieval, and Google patents is not to find out how exactly things are going — it is to understand the processes and challenges of search engines. That will also help you to judge whether certain claims are reasonable or just hot air.

Then, dive deeper into the big topics:

• Crawling: log file analysis, XML / HTML sitemaps, mobile bot crawl behavior
• Mobile: AMP, Progressive Web Apps, Responsive design
• SSL: HTTPS, HTTP/2
• Structured data: Schema markup, Microdata & JSON-LD, Rich Snippets
• Migrations: domain migrations, relaunches, CMS changes, HTTP to HTTPS
• Page speed: rich media and script compression, CSS sprites, CDNs, server speed optimization, parallel downloads and minify, caching
• Rendering: critical rendering path/lazy loading, DOM rendering, Javascript  frameworks rendering
• Content optimization: entity optimization, duplicate content, thin content
• Status codes: 3xx, 4xx, 5xx
• Indexation: canonicalization, robots.txt, meta-tags
• Site structure: internal linking, URL structure, taxonomy

On top of that, get to know Excel, Screaming Frog (or another crawler), MySQL and how to use the (MAC) terminal. They are not core technical SEO skills but will be needed and are good to know for many applications.

Photo: SEM Rush

Also, everyone says this, but building a website is helpful to understanding basic web infrastructure. It definitely helped me empathize with my clients’ dev teams. Using those skills can also help you to test, which I have found is useful for gaining a deeper understanding of SEO.

— Alexis Sanders

Remember, the best source of information is a combination of people and material. It is much better and faster to learn from someone who is already in the game and can even show you how to do things. When I started out, I went through a 9 months traineeship, in which I learned the craft from the ground up.

Alexis is “participating in mentorships (both as a mentee and mentor)”. Be proactive about mentorship and don’t shy away from learning from several people. Some may be really good at mobile but not good at international, etc. Look at what people share and read it. Ask questions on Twitter. The SEO industry is a very supportive community!

A couple more tips from Alexis Sanders:

• “Work with application program interfaces (APIs).
• Author comprehensive, useful publications.
• Participate in online courses, code tutorials, and educational videos on Lynda.com, Udacity, Coursera, Codecademy, CodeSchool, and Google.
• Creepin’ on John Mueller’s “Tweets and Replies”.

Now onto the last thing, application.

How to Apply What You Learned About Technical SEO

There are many mediums to learn SEO from: books, videos, guides, ebooks, blog articles. But you never truly learn something without applying it. It doesn’t matter how many books about “driving a car” you have read, without driving one you won’t be a good driver. The same idea applies to technical SEO.

The most important advice I give to people starting out in SEO is to have your own project. Start a blog, a little web-shop, a wiki — anything! Apply what you learn to it and draw your conclusions. You also need to develop an understanding of how long certain recommendations might take to implement and how to measure their success. One of the best ways to do this is by creating an on page SEO checklist, working your way through it and analysing how it improves your site.

Just like Paul Shapiro, I became very interested in the web in high school and taught myself web design and development (“I had done freelance web design and development in high school, and a self-taught programmer at a very young age”).

My first blog was about Muscle Cars. It started as a pure learning project and ended up yielding a couple hundred bucks before I burned it (another lesson learned). A good SEO always has a couple of “projects” running.

The Biggest Mistakes People Make When Learning Technical SEO

The first big mistake you can make is to think you can learn technical SEO within a short time.

I think the biggest mistake is potentially thinking that technical SEO is something you can just check the box off as something you know after taking a single Udacity course. Since it requires knowledge of web development, computer science and more to be adept, it’s a long process, and something you’ll likely develop knowledge in over time.

— Paul Shapiro

People often underestimate the nuances of complex web developments: “There are so many variables, especially as complexity is added. Patrick Stox had a stellar talk at TechSEO Boost entitled ‘ Everything That Can Go Wrong Will Go Wrong’, which covers some SEO snafus.” (Alexis Sanders)

One of the reasons for things taking a bit longer than people wish is the necessary practical experience I mentioned before.

When you learn technical SEO, remember to rely only on hard data and experiments. If you do that, I am not worried about your future career.

— Bartosz Goralewicz

 

Photo: SEM Rush

Bias and the inability to change your mind based on evidence are two of the biggest mistakes SEOs can make. Not continuing to learn is a safe way to get left behind in any digital marketing discipline. Knowing better than what the data suggests is another sure shot at failing.

Technical SEO doesn’t exist in isolation. According to Alexis Sanders, it is important to be aware of the bigger picture (“For example — Failing to balance recommendations with client needs and capabilities”) and the relationships with UX and dev teams:

If you’re not bringing in donuts for your UX and dev teams, today’s the day! In all seriousness though, technical SEO, UX, and development work are all deeply intertwined, so play nice.

Lastly, don’t only look at SEO. One mistake SEOs make is to only consume content around SEO, visit SEO conferences, and dismiss everything else. There is nothing to be learned in doing this after you have reached a certain point. Instead, look at disciplines like conversion optimization, social media marketing, e-mail marketing, and, of course, paid search. That will help you to put SEO into greater context and expand SEO with the creative use of other channels.

The Most Important Lessons from Alexis Sanders, Paul Shapiro, and Bartosz Goralewicz

Bartosz: “Most of my work is research. What I learned and what continues to amaze me is that I often assumed that a lot of concepts were too complex for me and that others understood it. We are all characters from “The Emperor’s New Clothes,” and we don’t want to look stupid so we don’t ask questions about what we don’t understand.

My dead-simple JavaScript experiment article gathered 600+ links from 200+ referring domains on top of ~40k visits just because I couldn’t find a clear answer regarding how Google is really indexing and ranking JavaScript pages. What is even more interesting is that this isn’t a “one-hit wonder”. If you look around our community there are hundreds of things we are very excited about, but nobody has gotten to the bottom of how these things work. You find out, and I guarantee that you’ll be one of the most successful technical SEOs.”

Paul: “You don’t have to learn alone, and you can learn from others. There are a lot of smart people who can help as catalysts to your learning. The SEO community is special in that I think there’s a lot of people willing to help you. I’m always learning from other smart minds in the industry, people like Patrick Stox, JR Oakes, Russ Jones, Emily Grossman, Dawn Anderson, Hamlet Batista, Max Prin and so many others who I appreciate, but can’t name due to brevity. I’m thankful to all of them though.”

Alexis: “In terms of my process, I recommend:

• Be disciplined, thorough, and consistent with learning.
• Research on your own first, then ask an expert.
• Have fun with it. Interact and engage with information. There are a ton of resources available to learn in a playful environment.
• Some fun Technical SEO things:
  • Practice Schema.org (by Max Prin and myself)
  • Schema.org Item Types D3 Interactive Visualization
  • http://oap.ninja (by Dean Cruddace)
  • https://technicalseo.expert
  • SEO Tests“

How to Learn Technical SEO and Where It Is Going

Learning technical SEO takes time, patience, continuous learning, and application. The best way to start is to gain knowledge in a broad sense and then go deeper into topics that are important in the moment and interest you. You can learn it completely for free, just from material on the internet. But the best teachers are people who know their craft. It is good to have a methodical approach to learning new topics because they will keep coming up in technical SEO.

Alexis Sanders’ process for Learning SEO Issues:

1. “Start with Google SERPs
2. Consult Google documentation
3. Read every article/piece of content on page one and page two (yes that me, that <1% of CTR on page two…)
4. Make note of questions that arise during initial exploration
5. Copy/paste highlights into a document
6. Organize information in a slide deck
7. Process information and try to visualize
8. Consider how various data and trends connect
9. Get answers to the recorded questions
   – Try to find answers via research (minimal half hour)
   – Ask someone for help and advice
   Places to ask:
   – Reddit (e.g., https://www.reddit.com/r/bigseo/)
   – Google Webmaster Hangouts
   – Twitter (e.g., Google Webmasters, John Mueller, Gary Illyes, etc.)
10. Try to speak on the topic (talking through usually helps pinpoint gaps in understanding)
11. Rinse, repeat (until learning plateau is hit)”

You have to look at a highly dynamic topic, such as technical SEO, also in terms of where it is going. Otherwise, what you are learning right now might be outdated by the time you have really understood it.

The strong advances in AI and especially machine learning might make you think that at some point Google can perfectly understand websites. That is only partially true. Only because something is understood, it doesn’t mean it can’t be optimized. It would be foolish to assume that technical SEO isn’t necessary anymore at some point.

So far, Google seems to double in complexity every year but that only made technical SEO more important. Instead of vanishing, I can see technical SEO splitting up into several disciplines: mobile technical SEO, crawl optimization (crawling, rendering, page speed, log file analysis), and classic technical SEO (status codes, SSL, structured data, site structure, migrations, indexation). All of these fields are progressing on their own and becoming increasingly complex.

So, what happened to the clients that got hit by Penguin at the beginning of this article? We demoted many links and built them a strong technical base. Once the next iteration of Penguin released them from the algorithmic penalty, the technical optimization helped them performing stronger than ever!

Originally posted on SEM Rush.