Saltar al contenido principal

Choose learning analytics tools by the decisions they support

Jonas Lindqvist·6 min de lectura·

Choose a learning analytics tool by asking which decisions your team needs to make, what evidence those decisions require, and who can act on the result. A dashboard that counts course activity can be useful. It becomes misleading when that activity is presented as proof of capability, changed behaviour or financial return without additional evidence.

Start with a small requirements sheet before requesting demonstrations. Name a decision owner, define each metric and write down what would make the result unreliable. That preparation gives vendors a concrete problem to address and makes it easier to recognise when a simple report is sufficient.

Start with a decision, not a dashboard

Consider three requests that sound similar: “show completion”, “identify people who need support”, and “show whether training improved service”. They require different evidence. Completion may come from an assignment record. Support decisions need context and a conversation. Service improvement needs a defined operational measure and an evaluation approach that considers other explanations.

CIPD’s learning evaluation factsheet connects evaluation to identified performance gaps and business objectives. Translate that into a procurement question: what action would change if this report moved? If nobody can answer, postpone the chart and clarify the underlying need.

Give each proposed report an owner and a review rhythm. A store manager checking outstanding safety instruction has a different job from an L&D lead comparing the usefulness of two programmes. They should not receive the same unrestricted individual-level dashboard simply because the software can generate it.

Separate four kinds of evidence

Evidence Useful question What it does not establish alone
Assignment and activity Did the intended audience receive and use the material? Understanding or skill
Assessment response How did someone answer these particular tasks? Reliable performance in every work situation
Observed application What happened during a defined work task? That training caused any wider change
Operational outcome Did an agreed service or business measure change? Attribution to the programme without further evaluation

Keep these categories visible in requirements and reporting. An assessment score can be informative without becoming a universal readiness label. A manager observation can be valuable while remaining a judgement with a date, scope and possible bias.

Docebo’s Learning Intelligence page describes reporting and dashboard capabilities. That is evidence of the vendor’s stated product scope, not independent proof that a particular installation establishes business impact. Ask any supplier to show how its records support your specific interpretation.

Write a metric specification a colleague can challenge

For each metric, record its purpose, numerator, denominator, time window, source and limitations. “Completion rate” is incomplete until you define the eligible population, assignment rules and cut-off date. Decide how to treat new starters, people on leave, reassigned courses and historical versions before comparing teams.

Also distinguish unique people from events. Ten attempts by one learner are not ten learners. A course launch is not necessarily a completed session. When a vendor joins several systems, ask which identifier connects them, how duplicates are handled, and what happens when someone changes role or leaves.

The draft requirements worksheet turns these questions into columns. It includes a clearly fictional example and blank rows for your own requirements. Keep unsupported data joins marked unknown rather than allowing a tidy visual to imply a dependable connection.

Worked example: a fictional service team

Imagine a fictional team of 24 advisers introducing a new escalation procedure. The programme owner wants to know whether advisers can recognise situations requiring a supervisor. The question is narrower than “did training work?” and can support a practical review.

The initial specification contains three observations. First, the owner checks which eligible advisers received the current procedure. Second, advisers work through two illustrative scenarios, with the response rationale recorded. Third, a supervisor reviews a sample of suitable work examples using agreed criteria. These observations are kept separate instead of being blended into a single score.

Suppose 20 advisers complete the material, while four have not. That tells the owner to investigate access or scheduling; it does not justify calling those four incapable. If a scenario is frequently misunderstood, the next step is to inspect its wording and the instruction. If a work observation is missing, the record should say missing, not zero skill.

The example does not predict a completion rate or service improvement. Its value is a decision path: identify an issue, understand the context, then decide whether to revise instruction, provide support or gather better evidence.

Test the reporting workflow during procurement

Bring a small fictional dataset to the demonstration. Include a duplicate person, a late assignment, a changed team and one missing assessment. Ask the supplier to explain how the report treats each case. A useful demonstration makes definitions and exceptions visible rather than avoiding difficult records.

Request an export and reconcile a few rows against the screen. Check whether the definition survives outside the product and whether the owner can explain it without a consultant. Ask who can edit a metric, whether changes are recorded, and how historical reports behave after a definition changes.

Discuss operational effort too. Someone must maintain the roster, resolve exceptions, review access and explain results. Record those responsibilities alongside licensing and implementation questions. A sophisticated report is not low-maintenance merely because its first screen appears automatically.

Decide what should stay out

Jisc’s learning analytics code addresses educational institutions and stresses transparency about purposes and data use. It is not a workplace compliance checklist. It nevertheless provides useful questions to adapt with your own privacy and employment-policy owners.

Explain what information will be used, who will see it and how people can question an interpretation. Avoid collecting personal information simply because it might be interesting later. For consequential decisions, require a suitable human review and evidence appropriate to that purpose.

Turn learning evidence into a next step

A clearer report is valuable when someone can use it to choose support. Our view is that the evaluation should continue beyond the dashboard: choose a work task, agree useful practice and make time to review it. Put Omie on the shortlist when that ongoing professional-learning routine is the job you want to support.

Omie’s daily format brings the evaluation down to a manageable unit: a short personalised lesson, about ten minutes, around the learner’s role and goals. Bring your metric specification to the Business pilot discussion and ask which evidence the current product can actually provide. Keep any additional analytics integration as an explicit requirement to demonstrate. See Omie’s current product overview for the format and product context.

Choose the smallest workable next step

A spreadsheet and a clear review meeting may answer an early question. A reporting module may be enough when course administration is the main need. A broader analytics tool becomes relevant when the decisions genuinely require reliable data from several systems and the organisation can govern that work.

For the assessment workflow behind the report, see the skills gap analysis framework. If your team wants to explore a learning programme with an agreed baseline and review questions, Start your free 90-day pilot. That is a discussion request, not a promise of a particular dashboard, integration or business outcome.

El día 0 lleva
90 segundos.

Luego, el día 7 y el día 30 muestran qué se ha quedado.

  • ISO 27001 previsto
  • ISO 9001 previsto
  • RGPD
  • DPA listo para firmar
Contáctanos