Google analytics engagement time not working - how I found out the problem

This is very niche, and is mainly of interest if you're using the Webtoffee Cookie consent plugin (there is a free version available too). However, if you're interested in debugging analytics problems, then this gives you some techniques.

The problem showed itself that the engagement time on our GA4 analytics was zero. As you can see from the graph above, not exactly zero, but certainly very small. I turned to webmasters.stackexchange.com and got a response from jen. It didn't give me the answer, but pointed me to an analytics debugger extension for Chrome. I could see that using that the engagment time (_et parameter) wasn't zero, but was a small number of milliseconds, less than 50.

I tried a number of things on the cookie plugin settings, such as putting the Google Tag Manger code into cookie categories, as recommend by this article (only for the free plugin though). That made the analytics script run when you accepted the cookies, but never after that. I tried disabling it, but it still didn't make engagement time get registered.

In the end though, the problem was that there was code in our site theme designed to work with GTM, but we hadn't been asked to do the corresponding changes in our GTM config. Once I removed that code the engagement time was a healthy number of seconds. I don't quite know why that caused this problem, but my theory is that the events it was triggering ran pretty quickly after the page loaded, and at that point the user engagement was recorded on those events, rather than when the user subsequently did something (scroll, navigate).

Other things to help with debugging

• Chrome application tab, cookie section, has a handy clear all cookies button to the right of the filter box.
• Make sure you enable the analytics debugger extension when trying to debug!
• Clear the site cache after making changes to the cookie plugin settings

FormAssembly and Salesforce connectors - advanced techniques

Following on from my initial post about FormAssembly and Salesforce, here are some advanced techniques.

We use a particular technique with our FormAssembly forms and Salesforce when it comes to looking for existing contacts, creating new ones, and doing further updates on Salesforce.

The Form Submitted connector runs after the user clicks the submit button. Once it's finished the user gets the acknowledgement message, or gets redirect to the page saying it's worked. The reason we put some of our processing here, is because if it fails the user gets told with an error message. If we used the After Form Submitted connector then they wouldn't see the error message they would think it worked. Although we get emails if the connector fails, it may be a few days before someone gets round to reading that email and contacting the user to say it didn't work.

The error message won't mean much to them, as it'll be Salseforce jargon, but hopefully they'll pass it onto us and we can help them. Or if it's something that is fixable (e.g. an invalid email address) then they may retry. (FormAssembly email validation lets email addresses through that Salesforce says are invalid.)

So in the Form Submitted connector we look up the user by email, and create a contact if they don't exist.  If they do exist we also take the opportunity to update fields, like their name. Then in the After Form Submitted we look them up again. We know this will work, because the previous connector created them if necesssary. Then we do whatever the form requires, e.g.

• if they're signing up for a subscription then add them to a campaign
• if it's something we need to do, create a case
• if it's giving consent for something, add a consent (we use GiveClarity's own consent system)

Duplicates

One problem with creating contacts like this, is that you may get a duplicate message. Even though Salesforce considers this a warning, FormAssembly considers it a fatal error. (I suggested to them that they don't do that, but I couldn't get them to understand what I mean.) So we had to set our duplicate settings on Salesforce so it does nothing about them, apart from add them to the possible duplicates report. It does mean though, that we need to be vigilant about running that report.

Do you have any tips about using FormAssembly and Salesforce together?

FormAssembly Salesforce connector review

One of the consultants we used for our switch to Salesforce and NPSP recommended we use FormAssembly for getting data from webforms into Salesforce. That was a very good recommendation, and so I'd like to say a bit about my experience with the FormAssembly to Salesforce integration.

FormAssembly has various integrations, Salesforce is only one of them. They can run at one of three points - when a form is loaded, when it's submitted and after it's submitted. The difference between the latter two is subtle. When you click the submit button then "when" run is one and the user gets the success page when it's complete, or an error if it fails for any reason. At that point the "after" one runs and the user is unware.

I'm not going to do a full tutorial as the FormAssembly documentation is very good. I'm just writing here to say that it's very easy to use, even if you're not a Salesforce admin. When you start using the connector you can create, update or lookup an object. Depending on how that went you can then create, update or lookup a subsidiary object and so on.

So a typical case might be that you lookup the email address on the form to find a contact with that email address. If you find them carry on to the next step, otherwise you add them. 

You can do all this by clicking around the user interface, so it's very #clicksnotcode. FormAssembly has a formula function, not specific to the Salesforce connector, and this can be used, for example, to turn two address line fields into a single Mailing Street field with a carriage return between the two lines:

@CONCATENATE(%%tfa_10%%,@IF(%%tfa_11%%,@CONCATENATE(@CHAR(13),%%tfa_11%%),""))

tfa_nn are the internal names for the fields.

This combination of FormAssembly and Salesforce has met our needs so I'm very pleased with it.

Update: I wrote a followup post on FormAssembly and Salesforce advanced techniques

Internaut day - 25 years ago I got onto the web

25 years ago I heard about this program from a USENET newsgroup called Mosaic. I downloaded and installed it. As they say: Mind. Blown. All the pages had a grey background. The images would download one by one, and you could see each line of pixels loading. A later version would download the pictures simultaneously.

I remember downloading a quicktime movie of a jet powered sledge. It took ages, but it was groundbreaking.

Someone once asked me about the difference between the web and the internet. I didn't give a very good answer, but since then I've though of a good analogy: we had roads (internet) before we had cars (webpages). Other traffic travels on the roads like horses (FTP) but it's mostly cars (webpages).

How hard is it to change your email address?

We're in the middle of moving to Google Apps (I'll write on the experience when we're nearer the end). As part of the process we've taken the opportunity to change our email addresses to use the wycliffe.org.uk domain. So I've gone round various websites changing my email address. The process varies from simply changing a field once you've logged in, to a more complex flow involving verifying the new address. Some sites I couldn't change it, so I had to contact someone to do it by hand.

Some sites have the email address as the username, but you can't change it. I think this is just badly designed. It's not unusual for people to change emails, or for anything that might be work related, for a different person to take over a role. When the old email that hangs around is for someone who has died it just looks bad.

So when designing your site login system, consider how easy you want to make it for someone to change their email address.

I saw a post on a blog, I can't remember which, where someone had wanted to change their email, or retrieve a password on their own site. They were surprised how poor the experience was given the amount of effort they'd put into other parts of their site. Let me know if you've come across that post too and I'll link it here.

Kingdom Code

Awesome to have 60 people at Impact Hub last night for the "Kingdom Code" Christians in Tech event #kingdomcodeuk pic.twitter.com/BTZKenssIp
— Andy Geers (@andygeers) June 17, 2014

You can see me on that picture if you look very closely.

Kingdom Code was an event organised by Andy Geers and others:

Are you a Christian in tech? This event is being held to draw together a foundational group of Christians who work as developers, UX / UI designers, data geeks, product manager or entrepreneurs and who are excited to see mobile and tech used to serve Kingdom purposes. One output of this group may be to organise a Code for the Kingdom Christian weekend hackathon in 2015. 

The event was really well organised. After a bit of mingling we moved into a room with chairs. There was a brief introduction from Andy and Rupert Edwards and a big of plugging from the sponsors. We were then told to chat to the person next to us, and a bit later, to the person on the other side. There was a pre-event questionnaire asking about what people hoped to get out of the event and what sort of work they were in. There was a wide variety - not just developers and designers, but digital marketing people, business developments and entrepreneurs. I think I put myself down as jack of all trades. Some people were doing not specifically Christian type jobs (I hate the term "secular job"), but had Christian related side projects, in the way that Andy G does Prayermate on the side. Some people were visionary types who could see potential, e.g. in the internet playing a similar role to Roman roads which enabled the growth of the church after Jesus' death and resurrection.

We then split into 8 groups according to those types - I went with database/IT. After a bit we were then asked to mix about talk about this possible Code for the Kingdom event next year.

There was a plugging spot, where about 20 people got a minute to plug the thing they were involved in and ask for help, mostly volunteers or funding. There was a variety of visions for, say, easy software for administering churches, or helping those turning 18 who are forcibly returned to the country they sought asylum from, or getting the Bible out via mobiles. I plugged the paid job I blogged about recently. There was someone from Scripture Union who won the "oldest organisation" award, in contrast to the new ventures that people were working on. I think we came second.

There's another event planned in October, but I left before the end so I didn't hear much about that. Update: It's Monday 13 October.

Update2: Andy's written a post about it, though I blush to link to it.

I met a couple of people with links to Wycliffe:

• someone who did a short-term placement with us a couple of years who is part-time working on Tyndale House's online scripture tools. Update: they've launched a new version of the site now.
• Jason Ramasami who did this great picture for our magazine:
Post by Wycliffe Bible Translators UK.

It will be interesting to see what this event has started.

Is it OKto.bike?

I wrote in a previous post about a web thingy so you could find out if it was OK to bike or not? Back then it was basically working, but since then I've added the ability to specify your own preferences, rather than being stuck with mine, or using a really long URL. Now it's got its own domain name and a logo.

(I am not a designer and have no ability in that area.)

So now you can bookmark http://okto.bike and check every day if it's OK to make your regular bike trip.

Next step with this - rewrite it in Ruby on Rails because that's what the cool kids are using.

New office - furniture and rolling display

There's a good post with lots of pictures of our new office on the Wycliffe Centre move blog. I've been involved in the office layout planning since near the start of the process and one of the things we needed to consider was which office furniture to get. Because our previous furniture was from all sort of different sources it was very mismatched, so we wanted to spend money on getting new desks. (We managed to reuse a number of other bits of furniture, such as non-desk chairs, bookcases and some cabinets.) In the end we went with furniture company called Ofquest. Even with a generous discount it wasn't cheap. What we got for our money though was solid construction, a degree of reconfigurability, and really useful features, like a cable tray under the desk and clip on cable guides for the legs. It should last us a number of years, and survive another office move should we move before it wears out.

One of things of lesser importance that I've worked on with Ruth, our content person, and Stuart and Andy, my IT colleagues, is the TV display you can see above. Stuart and Andy got a Raspberry Pi going booting into the Chrome browser and displaying a page from our intranet. The page changes every few seconds and as well as showing prayer items shows things to praise God for, internal notices, the picture of the week from wycliffe.net. For that page, and another one which screenscraped our own wycliffe.org.uk site I used a bit of  XPath, obtained via Chrome's web inspectory thingy, which makes it very easy to get at HTML content. Here's a fragment of the code I used:

$pageHTML = file_get_contents($url);

if ($pageHTML) {

$dom = new DomDocument();

$dom->loadHTML($pageHTML);

$xpath = new DOMXPath($dom);

$entries = $xpath->query('//*[@id="image"]/img');

if ($entries) {

$tag1HTML = nodeContent($entries->item(0), true);

}

}

The user-editable pages were done with Perch, a very neat content management system. It's not suitable for large sites, probably defined by complexity of navigation rather than number of pages. It's very quick to get going on though. I used the API to write a custom app which removed items from a page when an associated expiry date had passed. As you've got access to the source code as well as the API documentation I was able to write something which delved into the Perch innards without too much effort, once I understood factory classes. When I needed support I got a quick response (_/- confusion - doh!).

Stonemasons' forum

At lunchtime someone who's doing an MA in Digital Education gave us a talk about what he'd learnt. They are learning about Digital Education using digital methods (online forums, Skype, blogs). It made me wonder if other professions learnt in a similarly recursive way then what would it be like. Imagine what a stonemasons' discussion forum would look like...

tap, tap, tap, tap
tap, tap, tap, tap, tap
tap, tap
tap, tap
tap, tap, tap, tap
tap
tap
tap, tap, tap, tap, tap
tap
tap, tap, tap
tap, tap
tap, tap, tap
tap, tap, tap, tap
tap, tap
tap, tap, tap, tap
tap
...
...
tap, tap, tap
tap, tap, tap, tap
tap, tap
tap, tap, tap
tap
tap, tap, tap
tap, tap, tap

tap, tap, tap
tap
tap
tap, tap, tap
tap
tap

"IMHO"

And if you're familiar with Terry Pratchett's Discworld novels then you will know what I mean about trolls having a whole new meaning.

Daddy, daddy, look what I found on YouTube

It used to be that my children would come to me and say "daddy, daddy, look what I made out of Lego". Now they say, "dad, look what I found on YouTube". (They don't really say "daddy" as I put in the title. That's a carefully crafted sentence designed to get you to click through from my social media feeds.)

What they are showing me is episodes from their favourite "series". I say series, because apart from the length, they are like TV series, with new episodes coming out regularly, weekly for example. The quality is professional. I wonder who's paying for this stuff? The money for those cameras has to come from somewhere, the people who do the work need to eat.

In the case of Rhett and Link (500 web videos produced) they "support themselves by creating and distributing their branded entertainment", though they do get paid to do stuff for other people. I'm not clear though whether the video or the money came first.

Video Games High School is another interesting one. 5,661 pledged $273,725 on Kickstarter to make this series. All those people paid all that money and we get to watch it for free. (Internet connection charges apply.) How does that work?

I don't have time to dig into all the other people my kids are telling me about. (Jenny Bee signed our ukelele, does that give me street cred?) I'm just bemused that all this good quality stuff is up on YouTube and people are managing to eat too.

What was Tim Berners Lee thinking of?

With Tim Berners Lee winning the Queen Elizabeth Prize for Engineering along with some other people, some I'd heard of, and some I'd not, it's worth rereading Tim's original proposal for what became the World Wide Web. He was trying to solve the problems that CERN was facing, but envisioned it being useful to the rest of the world as well. He was right.

What does it mean to be logged in?

The last time I got fed up of my lack of knowledge on something I drew a poster about public key encryption. This time it's about what being logged in means. What is prompting this is the fact that more recent updates to Chrome have seemed to make it so that in some cases when Chrome restarts you remain logged in to a website where I'm sure you weren't before. So I'm finding out the different ways that websites keep you logged in. I usually write a complete blog post before I publish it, but this is going to be a public work in progress.

Wireframe:
HTTP is stateless. What about headers?
What different technologies keep the fact that you're logged in?

• cookies - session, time-limited
• technologies - PHP, ASP, others
• ASP session state
• anything else?

When you tick "remember me" what's going on? Is it clear whether it's remembering username and password?
How does the browser remember passwords? Is it secure?
What's going on with Single Sign On? (Ref this video: https://www.polderconsortium.org/fedid-in-global-missions)

Disclaimer

If you know about this stuff, and I get something wrong, please tell me, so we can make the world a better place, or at least this blog post. Bear in mind though, that I am simplifying. So when, for example, I say as below "you get a web page back" the proper way of saying that probably is something like "you get an HTTP response" which is in all likelihood a web page, but may not be in every circumstance. However I'm not going to say that, because I'm simplifying.

HTTP is stateless

You ask for a web page, you get a web page. You ask for another, you get that. The server isn't required to keep track of the fact that you asked for one just now. There are ways to get round that, one of which is cookies. I'm logged into a local MediaWiki (the software that Wikipedia runs on) site. Here's the cookie that gets sent to my computer every page I request.

I've obscured some of the strings in case they are a security risk, you never know. So there you can see a couple of relevant variables. PHPSESSID is the PHP way of keeping track of sessions, and bsm_bsm_session (where bsm is the name of my mediawiki instance) is probably a mediawiki specific cookie. __utma and __utmc are Google Analytics tracking. I don't know why they are involved.

This is the site that I first noticed I stayed logged into every time Chrome started up. That cookie is a session cookie, which are supposed to be deleted by the browser every time you shut it down. Chrome chooses not to do this though, if you enable "continue where I left off" in settings. I do that so that my tabs from my previous session are open, and apparently session cookies aren't deleted too, which is a feature.

ASP.NET

Which roughly means "how Windows servers do it" (ignoring older technologies). An authentication token is stored in a cookie or a page URL (source).

So what's in that cookie?

What determines whether a cookie or a URL is used?

Tim Berners-Lee and an anecdote involving Bible Translation

In 2008 Tim Berners-Lee founded the World Wide Web Foundation. In the speech where he announced it he told an interesting anecdote:

This person, this man had taught himself English by using various books that he’d come across. Among them are the bible I understand, something that he could get in both languages. Having taught himself English and having got some sort of rudimentary Internet connection, then he could sell his services as a translator on the open market out there in the big wide world. He could not only – so he could bring money into the village but also he could be a channel to bring information into the village. He could translate the stuff that was out there into the village. And he could operate – so he could enable commerce.

The reason he told it was because he was talking about access to the internet could enable development goals. Naturally, given my job, the thing that interested me was the fact that having the Bible in English and presumably his own language, was what enabled him to sell his translation services.

(I first heard about this anecdote on the Guardian Tech Weekly podcast.)

Federated Identity and Identity Assurance - why you should care

Yesterday I read a blog entry from the UK Government Digital Service about Identity Assurance.

We’re helping develop a secure service that lets people log in to online government services more easily.

They link to this article by the Telegraph which describes it well (apart from the headline).

I've blogged before about the Polder Consortium where we're thinking about such things. Identity Assurance is part of Federated Identity. If you're in IT then I expect you're going to get friends and family asking you if it's OK to use your Facebook login to access government services. (I suspect a Facebook login may not reach the required standard of assurance.) I'm personally encouraged by the fact that they are consulting with people who worry about privacy.

I think, though, that if you're in IT it's worth understanding about federated identity, authentication, authorisation and assurance levels so at least you can have an informed opinion. (I'll point to a document that's soon going to become available when it does.)

In other news, as they say, the Polder Consortium has released some new standards, recommendations and notes, either because they've gone to Proposed state or because we've decided that some drafts are worth making public.

Evolving layouts for web pages

Seven years ago (surely a century in internet time) I posted a blog post which I won't summarise, but repeat here as it's short:

It's a big deal when a magazine or a newspaper has a redesign. Similarly with websites. It's something to do with the fact that it's easy to churn out issue after issue/page after page using the same layouts, fonts etc.

I would have thought that in the dynamic web world you could spend more time with your templates, spending more effort making them work harder for you, so each new page looks like an evolution, or a minor redesign, whereas you haven't put any effort into that page, you just put it in at the beginning. Does that make any sense?

Today, I've just read an article in Contents magazine, Made to Measure, which expands that idea, and is so much better written than I could do. It doesn't say that all the work can be done at the beginning, but with thought I agree with them that some work needs to be done for each article.

So if static templates are too limiting, but per-piece art direction is too costly, how else can we make scalable, sustainable digital publications that are beautiful and accessible? To find a middle path, we can take a cue from the art of the tailor.

What is a website exactly?

I've come across a blog recently called The Pastry Box project - "30 People Shaping The Web. One Thought Every Day. All Year Round. Sugar For The Mind.".

This recent post talks about interviewing candidates, and a standard question for them:

What's your favourite website?

If I were asked that question it might take me a while to answer. My thinking went like this:

Well, I use Gmail a lot, but that's my email, not really a website. Google Reader too. I use Librarything to keep track of the books I've read, but I think of that as my library catalogue, not a website. Flickr's where I store my photos, Facebook is... well Facebook.

I can't think of my favourite website, because the websites I use most are places for doing things. You could ask the same question about books, and I'd be able to give you an answer (apart from the Bible and Lord of the Rings it's probably Wintersmith actually). Books are more all alike than websites.

Is there something else that could give rise to head-scratching if you asked about a favourite, because it is such a diverse thing?

"What's your favourite activity?" maybe.

Visual display of tracking cookies - collusion

Thanks to the Guardian I've come across a Firefox plugin which gives you a visual display of tracking cookies. It's called Collusion. (Update 20 March 2021 It's now Lightbeam. Thanks to someone at Comparitech.com for telling me about this broken link.)

I've been vaguely aware that website were passing information to each other, as I noticed that if I visited crucial.com to look at memory then I'd see more Crucial adverts on other sites. Similarly with Dell. Then one day I found a link on a site next to an advert (unfortunately I didn't note it down) which gave more information on why I was getting these Crucial adverts. It was easy to understand and very upfront about how it used information from other sites to display relevant adverts.

Here's what you get if you visit the following sites:

• crucial.com
• dabs.com
• ebay.co.uk
• bucksfreepress.co.uk
• guardian.co.uk

Each circle represents a site that places cookies in your browser and a line means that the site at one end placed a cookies for the site at the other end of the line.

The most circles appeared when I visited bucksfreepress.co.uk (one of the red squares with a yellow i in the middle). I guess it's a sign of a struggling industry that it has to try so hard to be clever with its advertising.

Getting rid of cookies on our website - Wordpress, YouTube, AddThis, Google maps

The deadline for complying with the EU Cookie law is 26 May, which is just over a month away.

Rather than get permission to use cookies on our website which would either be intrusive or ignored we've decided to not use cookies. Here's what I've found as I work through the various things that use cookies.

Wordpress Comments

The built in wordpress comments feature sets cookies so that it can remember the commenters details for next time. I commented out (no pun intended) these three lines in wp-comments-post.php. (If you upgrade and this file gets changed you'll have to repeat this.)

// setcookie('comment_author_' . COOKIEHASH, $comment->comment_author, time() + $comment_cookie_lifetime, COOKIEPATH, COOKIE_DOMAIN);

// setcookie('comment_author_email_' . COOKIEHASH, $comment->comment_author_email, time() + $comment_cookie_lifetime, COOKIEPATH, COOKIE_DOMAIN);

// setcookie('comment_author_url_' . COOKIEHASH, esc_url($comment->comment_author_url), time() + $comment_cookie_lifetime, COOKIEPATH, COOKIE_DOMAIN);

Wordpress Jetpack Stats

The Wordpress jetpack plugin uses quantcast cookies (HT A. Cemal Eki). The wp-donnottrack plugin will stop this.

YouTube

When you get the YouTube embed code you can now tick the "low privacy" option. All this does is use the youtube-nocookie.com domain, so you could use that on existing iframe embed code. The SmartYoutube Wordpress plugin allows you to set an option so that this happens on all your embedded videos.

Addthis.com

For the addthis wordpress plugin go to the Advanced tab and put
{ data_use_cookies: false };
in the addthis_config values field. For an embedded button use this code:

<script type="text/javascript">
var addthis_config = { data_use_cookies: false };
</script>
This won't stop individual services, like twitter, using their own cookies, but it will stop the addthis.com cookies.

Google maps

For google maps use maps.googleapis.com rather than maps.google.com. HT barryhunter.

Getting a list of postcodes in a parliamentary constituency

Whilst there are a few places where you can look up a single postcode and get which constituency it's in, I didn't find a single place to get a list of postcodes. Here's how I did it:

• I got a list of postcodes and which ward they were in from here. (Also available from the Ordnance Survey.)
• I got a list of wards and their codes from the electoral commission website.
• Fortunately the whole constituency I wanted was in one region, i.e. the letter or two at the beginning. The reason it was fortunate is because the data is in a file per region. So I used Excel to do a vlookup from the postcode list to the ward list and filtered by those which were in the ward list.

"Content" is a horrible word

I try and keep work/technical things on this blog, and other stuff on my other blog. However the boundaries are blurred, so you if you're only reading this one you might be interested in my post on my other blog - "Content" is a horrible word.